Journalists are suckers for anything that looks like science. And selection bias makes it even worse. But I was unfair to NPR.

Journalists are suckers. Marks. Vics. Boobs. Rubes.

You get the picture.

Where are the classically street-trained reporters, the descendants of Ring Lardner and Joe Liebling, the hard-bitten journos who would laugh in the face of a press release?

Today, nowhere in evidence.

I’m speaking, of course, about the reaction in the press to the latest bit of “p less than .05” clickbait to appear in PPNAS. Here’s what I wrote yesterday regarding the article, “Physical and situational inequality on airplanes predicts air rage”:

NPR will love this paper. It directly targets their demographic of people who are rich enough to fly a lot but not rich enough to fly first class, and who think that inequality is the cause of the world’s ills.

This morning I was curious so I googled the name of the article’s first author and NPR. No hits on this study. But a lot from other news organizations:

Screen Shot 2016-05-03 at 11.39.30 AM

Let’s go through and take a look:

Deborah Netburn in the L.A. Times presents the story completely uncritically. Zero concerns. From the authors’ lips to God’s ears.

Carina Storrs at CNN: 12 paragraphs of unskeptical parroting of the authors’ claims, followed by three paragraphs of very mild criticism (quoting psychologist Michael McCullough as saying that the study “is provocative, but it does not strike me as an open and shut case”), followed by two more paragraphs by the study’s author.

Gillian Mohney at ABC News: no skepticism at all, she buys into the whole study, hook, line, and sinker.

Bob Weber, CTV News: Again takes it at face value. A regression with p less than .05 in PPNAS is good enough for Bob Weber.

Unsigned, ABC Radio: A short five-paragraph story, the last paragraph of which is, “Although this study points to a link between air rage and first class cabins, it does not prove causation.”

Vanessa Lu, Toronto Star: Straight P.R., no chaser.

Peter Dockrill, Science Alert: A nearly entirely credulous story, marred only by a single paragraph buried in the middle of the story, quoting Michael McCullough from that CNN article.

And Sophie Ryan at the New Zealand Herald buys into the whole story. Again, if it’s published in PPNAS and it tells us something we want to hear, run with it.

LA Times, ABC News, Toronto Star, sure, fine, what can you expect? But the New Zealand Herald? I’m disappointed. You can do better. If NPR dodged this bullet, you can too.

Where were the savvy reporters?

Where were Felix Salmon, Ed Yong, Sharon Begley, Julie Rehmeyer, Susan Perry, etc., in all this? The quantitative and science reporters who know what they’re doing? They didn’t waste their time with this paper. They see the equivalent in Psychological Science each week, and they just tune it out.

You don’t see the most respected pop music critics reviewing the latest Nickelback show, right? OK, maybe at the Toronto Star. But nowhere else.

Where were Nate Silver’s 538 and the New York Times’s Upshot team? They didn’t waste their time with this. They like to analyze their own data. They know that data analysis is hard, and they don’t trust any numbers they haven’t crunched themselves.

We have a classic case of selection bias. The knowledgeable reporters don’t waste their time on this, leaving the suckers to write it up.

Comparison to himmicanes

Here’s a data point, though. This air rage study, like the power pose study, got nearly uniformly positive coverage, whereas the ovulation-and-clothing study and the himmicanes study were accompanied in their news reports with a bit of skepticism (not as much as was deserved, but some). Why?

I suspect a key factor is that the conclusions of this new paper told people what they want to hear: flying sucks, first-class passengers are assholes, social inequality is a bad thing, and it’s been proved by science!

Also, the ovulation-and-clothing and himmicanes studies had particularly obvious errors in their conceptualization and measurement, whereas the statistical flaws in the air rage study are more subtle and have to do with scaling of ratios and the interpretation of multiple regression coefficients.

A template for future news stories

OK, fine, you might say. But what’s a reporter to do? They can’t always call Andrew Gelman at Columbia University for a quote, and they typically won’t have the technical background to evaluate these papers by themselves.

But I do have a suggestion, a template for how reporters can handle PPNAS studies in the future, a template that respects the possibility that these papers can have value.

I’ll share that template in my next post.

P.S. BoingBoing fell for it too. Too bad. You can do better, BoingBoing!

P.P.S. Felix Salmon pointed out that the study was also promoted completely uncritically in Science magazine. Tabloids gonna tabloid.

P.P.P.S. And . . . NPR fell for it too. I guess it was inevitable.

20 thoughts on “Journalists are suckers for anything that looks like science. And selection bias makes it even worse. But I was unfair to NPR.

  1. “You don’t see the most respected pop music critics reviewing the latest Nickelback show, right? OK, maybe at the Toronto Star. But nowhere else.”

  2. There is a redeeming feature to the naive credulity on the part the media cited by Andrew. The damage is relatively unimportant compared to the harm done in the medical/health/nutrition field where large amounts of money are needlessly expended and lives are lost. Most lay journalists are on deadline, a press release is a handy tool in a busy day and besides, statistics is (are?) confusing. An excellent source for reviewing the media reviews in the medical/health/nutrition field is

  3. Holy Cow this is depressing stuff. Love the enhancement to “largely determined by….” There is nothing largely determined in these analyses at all. I knew it wouldn’t make a headline to say that the effects are less than X, or confounded with Z. But the credulity is troubling.

    On the other hand, last week I heard the morning radio DJ quoting from the report that the majority of cancer studies had failed to replicate. He was taking the approach “Duh, everyone knows that if the scientist has an incentive to find something, they will find it.” Kind of interesting to hear the “crisis in science” ideas circulating in such a popular forum. Times might get tough for social science down the road….

    I remember the Economist headline about Enron (and other financial scandals). It said “Will capitalists bring down capitalism?” or something like that. Will scientists bring down science??

    • Anon:

      Yeah, good point. One can write a skeptical article almost as fast as a press release, but with rare exceptions I expect the skeptical article will get fewer hits. Unless you go full Buzzfeed and give the clickbait headline, What Causes Air Rage?, with the deflating subhead, These Canadian Researchers Don’t Have a Clue!

      • > Canadian Researchers

        Did not know Boston is now part of Canada ;-)

        (Organizational Behaviour and Human Resource Management Area, Rotman School of Management, University of Toronto, Toronto, ON, Canada
        Marketing Unit, Harvard Business School, Boston, MA 02163)

        As an aside, I am an alumni of the Rotman School of Management, learned more in their program than in any other studies and helped set up their Executive MBA in ~ 1985.

        When I left Toronto (1998) I offered to give a parting talk there. Their response was that they did not want some statistician to show up and give a talk that no one understood and made their faculty look bad. So I sent them an outline on a very introductory meta-analysis talk as the business literature was starting to do these. After repeated followups, there was no response.

        But I would say on the scale of American like Canadian faculties, they did seem to try to be very American like and sometimes referred to themselves as the Harvard of the North.

        • “I am an alumni of the Rotman School of Management”

          You are vast, you contain multitudes! ;-)

          “…sometimes referred to themselves as the Harvard of the North.”

          At my convocation at McGill, one of the high muck-a-mucks joked that Harvard was known as the McGill of the US.

        • Corey: I am both a student alumnus, an employee alumnus and an associate (consultant) alumnus – thus alumni.
          (Admittedly I benefited from the link you provided.)

  4. Andrew:

    Instead of reforming reporting, why not focus on stanching the crap at the Journals?

    Like you say, getting these things right is hard, so arguably the Journals have more subject expertise to get this right than a generalist reporter?

    Besides, the Journalists are outsiders. The people running the Journals are from your academic community. Why is self-policing so hard?

    And if you cannot get internal controls to work why the expectation that the Journalists can / should do any better?

  5. Hello.

    I find this interesting, since I am a teacher. This is relevant because I certainly cannot teach what I do not know or understand. A journalist must either know enough of a subject and whatever science and scientific method there may be behind it, or simply parrot the information in the style of an analphabet copying a manuscript. Otherwise the journalist will not give a correct account.

    Comradely greetings,
    rikard, teacher

  6. …we will battle this problem in all directions simultaneously

    “He who defends everything defends nothing”
    –Frederick The Great

    • Anon:

      Hey, nobody’s ever called me Jesus before! First time for everything, I guess. But if reading the blog is causing too much suffering to you, just send me a self-addressed stamped envelope and I’ll fully refund your subscription fee.

Leave a Reply

Your email address will not be published. Required fields are marked *