Skip to content
 

Can the science community help journalists avoid science hype? It won’t be easy.

tl;dr: Selection bias.

The public letter

Michael Eisen and Rob Tibshirani write:

Researchers have responded to the challenge of the coronavirus with a commitment to speed and cooperation, featuring the rapid sharing of preliminary findings through “preprints,” scientific manuscripts that have not yet undergone formal peer review. . . .

But the open dissemination of early versions of papers has created a challenge: how to ensure that policymakers and the public do not act too hastily on early studies that are soon shown to have serious errors. . . .

That is why we and a group of over 100 scientists are calling for American scientists and journalists to join forces to create a rapid-review service for preprints of broad public interest. It would corral a diverse contingent of scientists ready to comment on new preprints and to be responsive to reporters on deadline. . . .

My concerns

I think this proposed service could be a good idea. I have only three concerns:

1. The focus on peer review. Given all the problems we’ve seen with peer-reviewed papers, I don’t think preprints create a new challenge. Indeed, had peer review been some sort of requirement for attention, I’m pretty sure that the authors of that Santa Clara paper, with their connections, could’ve rushed it through an expedited peer review at PNAS or JAMA or Lancet or some other tabloid-style journal.

To put it another way, peer review is not generally done by “experts”; it’s done by “peers,” who often have the exact same blind spots as the authors of the papers being reviewed.

Remember Surgisphere? Remember Pizzagate? Remember himmicanes, air rage, ESP, ages ending in 9, beauty and sex ratio, etc etc etc?

2. This new service has to somehow stay independent of the power structure of academic science. For example, you better steer clear of the National Academy of Sciences, no joke, as they seem pretty invested in upholding the status of their members

3. My biggest concern has to do with the stories that journalists like to tell. Or, maybe I should say, stories that audiences like to hear.

One story people like is the scientist as hero. Another is the science scandal, preferably with some fake data.

But what about the story of scientists who are trying their best but are slightly over their head, no fake data but they’re going too far with their claims? This is a story that can be hard to tell.

For example, consider those Stanford medical researchers. They did a reasonable study but then they botched the statistics and hyped their claims. But their claims might be correct! As I and others have written a few thousand times by now, the Stanford team’s data are consistent with their story of the world—along with many other stories. The punchline is not that their claims about coronavirus are wrong; it’s that their study does not provide the evidence that they have claimed (and continue to claim). It’s the distinction between evidence and truth—and that’s a subtle distinction!

Another example came up a few years ago, when two economists published a paper claiming that death rates among middle-aged non-Hispanic whites were increasing. It turned out they were wrong: death rates had been increasing, then flat, during the time of their study. And, more relevantly, death rates had been steadily increasing among women in that demographic category but not men. The economists in their analysis had forgotten to do age adjustment, and it just happened that the baby boom passed through their age window during the period under study, causing the average age of their category to increase by just enough to show an artifactual rise in death rate.

Anyway, I had a hard time talking with reporters about this study when it came out. I made it clear on the blog that the economists had messed up by not age adjusting—but, at the same time, their key point, which was a comparison of the U.S. to other countries, still seemed to hold up.

I recall talking with a respected journalist from a major news outlet who just didn’t know what to do with this. He had three story templates in mind:

1. Brilliant Nobel-prize-winning economist makes major discovery, or

2. Bigshot Nobel-prize-winning economist gets it all wrong, or

3. Food fight in academia.

I wouldn’t give him any of the three stories, for the following reasons:

1. The published paper really was flawed, especially given that it was often taken to imply that there was an increasing mortality rate among middle-aged white men, which really wasn’t the case. This myth continues to be believed by major economists (see here, for example), I guess because it’s such a great story.

2. The paper had this big mistake but the main conclusion, comparing to other countries, seemed to hold up. So I refused to tell the reporter that the paper was wrong.

3. I didn’t want a food fight. I wanted to say that the authors of the paper made some good points, but there was this claim about increasing death rates that wasn’t quite right.

I wouldn’t play ball and create a fight, so the journalist went with storyline 1, of the scientist-as-hero.

It can be hard to report on a study that has flaws but is not an absolute train wreck of a scandal. Surgisphere—that’s easy to write about. The latest bit of messed-up modeling—not so much.

So I support Eisen and Tibshirani’s efforts. But I don’t think it’ll be easy, especially given that there are news outlets that will print misinformation put out by reporters who have an interest in creating science heroes. Yeah, I’m looking at you, “MIT’s science magazine.”

Selection bias

We’ve talked about this before; see here and here. Here’s the logic:

Suppose you’re a journalist and you hear about some wild claim made by some scientist somewhere. If you talk with some outside experts who convince you that the article is flawed, you’ll decide not to write about it. But somewhere else there is a reporter who swallows the press release, hook, line, and sinker. This other reporter would of course run a big story. Hence the selection bias that the stories that do get published are likely to repeat the hype. Which in turn gives a motivation for researchers and public relations people to do the hype in the first place.

P.S. Steve shares the above photo of Hopscotch, who seems skeptical about some of the claims being presented to him.

25 Comments

  1. I don’t see any real change unless the scientists and journalists assure us that they have no conflict of interest. I would include non-experts too. Who is to say that consumers of science should be excluded as subsets are seized by marketing ploys of types. They have concrete experiences that can be useful for peer review purposes.

    • Peter Ellis says:

      I’m not sure what you’re saying here, but if it is that conflict of interest is the problem, I think this is overstating the importance of “conflict of interest” as usually understood. While yes, it’s important to know if a study has been funded by big pharma / big tobacco / Black Lives Matter / the Committee to Re-elect the President, it seems to me that the more common and problematic cases do not have such a material conflict.

      No assurance can control for the desire to have the “big discovery” (for scientists) or “big story” (for journalists) or just the “confirms what I’ve been saying at dinner parties for years” bias or even “I have to publish something soon or my boss will get mad”, none of which could realistically expect to feature in a conflict of interest declaration.

  2. jim says:

    As always, when you’re dealing with journalism, the first priority of the Voice of Freedom and Democracy is to be the Driver of Revenue and Profit. I don’t have anything against revenue or profit, but we have to keep in mind the purpose of the “free press” if we’re going to understand what we read.

    This morning I read an article from WAPO that said at the outset, paraphrasing:

    “people are turning to conspiracy theories about how COVID is transmitted because there is no authoritative voice and Trump is a conspiracy theorist”

    1. Fauci is writ large in the news every single day, so I’m not sure where the “no authoritative voice” assertion comes from.
    2. Conspiracy theories have been widespread for all of human history and very prominent in the last few decades, so Trump has nothing to do with that. More the other way around: Trump is picking up conspiracy theories from the environment and just rebroadcasting them.

    The “story” is just plain made up by the journalists. The narrative is false, it’s fake, its garbage published to sell newspapers. It may have facts in it, it may be chalk full of facts, but the story framework is pure fiction.

    So that’s what you’re always going to get from a journalist. They might be trying to do a good job reporting the truth, insofar as it aligns with their other interests. But when truth and need diverge, the journalist will choose need.

    • Appeals to Authority are unavoidable given that we rely on our academic training & recommendations by professors/employers largely for our own career trajectories. Which profession is exempt from this?

      Jim, you are a more critical reader than most of us. Polarization along political lines plays into this obviously, to the extent that blind spots are not recognized by the side so indulging in it. Maybe that is intentional. Otherwise one has to speculate that those who don’t discern their rhetorical strategy are tools of their leaders.

      The irony is that so much of what was initially claimed by the expert community has been debated since early March. And debunked fervently on Twitter & Facebook.

      I don’t fault any one individual or group. We simply did not have sufficient knowledge about this virus. So I try to access as many perspectives as feasible.

    • Martha (Smith) says:

      Jim wrote,
      ““people are turning to conspiracy theories about how COVID is transmitted because there is no authoritative voice and Trump is a conspiracy theorist”

      1. Fauci is writ large in the news every single day, so I’m not sure where the “no authoritative voice” assertion comes from.”

      Yes, Fauci has been writ large in the news virtually every dat. But the question is: How many people see him as an authoritative voice? And more broadly, what makes someone an “authoritative. voice” in the eyes of the public? I am guessing that many people have some stereotype of what makes someone “authoritative” in their eyes — and that Fauci doesn’t fit that stereotype. [This question of what makes someone “authoritative” to people (or which people, or …) actually would be a good topic for social science research ( but the question then is, would it be good quality research?)]

      • Martha (Smith) says:

        oops — “day”, not dat”

        • I believe a very compelling account of the role ‘expertise’ is in David Kennedy’s World of Struggle: How Power, Law, and Expertise Shape the Global Political Economy. Richard Posner focuses on the role of intellectuals in the public sphere; their proclivity to saying thing for which there are few if any accountability mechanisms. Posner is alleged to say also that few pay attention to experts.

          https://prospect.org/features/posner-proves-case/

          Not sure I agree with that second claim.

          Posner suggests that by setting up accountability mechanisms, we can better assess the merits and demerits of their assertions.

          The Prospect goes on to make other interesting observations.

      • jim says:

        “the question is: How many people see him as an authoritative voice? “

        Fair point. But it’s all relative, right? Most presidential elections are within ±5%, so most of the time half the population doesn’t trust the leadership. If Fauci is trusted by 65% of the population, he’s doing extremely well.

    • somebody says:

      Is this the article you’re talking about?

      https://www.washingtonpost.com/outlook/2020/05/05/coronavirus-conspiracy-theories-pandemic/

      If so, I would dispute your account of what the article says by a wide margin.

      In any case, I’m almost certain you’re talking about an opinion piece. An opinion piece is not written by a journalist. It is supposed to be a narrative made up by the authors, that’s the point. This is an increasingly common mistake of media illiteracy, wherein people of all political persuasions lambast “journalists” for a bad opinion being published in their paper.

      1. Journalists at reputable publications actually adhere to a pretty strict research and writing process, and most articles designated as “news” read as dry recitation.
      2. Headlines are not written by journalists. Bad headlines are indeed profit motivated, but are not written by journalists.
      3. The opinion section is not written by journalists. It’s typically written by columnists with no journalistic training or even random unaffiliated citizens.

      Don’t get me wrong, I think there are issues with the canonical news organizations. In particular, I think the fact that headlines are chosen by editors for profit maximization is atrocious, and I would do away with the opinion columns altogether in the day and age when everyone can write a blog post. But I think “the media is profit motivated” has become a pretty tired cliche. It might be true, and I think it is, but we have to go deeper and more specific for that criticism to be anything more than rhetorical ammunition in political arguments. Just pointing out opinion pieces that are bad gets us nowhere.

      • somebody says:

        > rhetorical ammunition in political arguments

        I realized what I mean by this might not be clear. To take an example, since March I’ve routinely heard the New York Times accused of both downplaying the coronavirus to take advantage of people’s desire to seek reassuring news and dramatizing the coronavirus for fearful clicks. Not from one person, of course, but from various people, all convinced that the profit motive is why corrupt news organizations have sidelined their point of view, citing the publication of one opinion piece or the other. Everyone gets to feel like the renegade punching up to fight the power and a potential discussion of editorial responsibility and discretion during a national emergency gets lost.

        • Michael Nelson says:

          NPR has plenty of actual news articles that are entirely uncritical. Same for CNN, NBC, ABC, CBS, etc. etc., particularly their televised reporting. The Times is pay-walled, so I don’t read them past the headlines so much. If they and other pay-walled sites like the Post are doing a better job, I hope they go a step further and print a huge expose’ on bad science reporting by their peers.

      • jim says:

        It’s a href=”https://www.seattletimes.com/nation-world/lacking-federal-virus-guidance-americans-turn-elsewhere/”>here, , in the Seattle Times, the by line is WAPO.
        From the piece:

        “In the absence of consistent, authoritative advice from federal officials, and with President Donald Trump and some of his top aides casting doubt on their own administration’s scientific guidance, many people have decided they are on their own, left to figure out for themselves how to live safely”

        The paragraph preceding the above refers to two women discussing a “secret Facebook group” by mothers who support school re-openings, which they want to do “despite risks posed by the corona virus”. Another woman (women are used to symbolize victimhood), out fishing with her grandson, used her mask “only in stores” because “she’s decided that masks aren’t very helpful”.

        These poor women!!! Victims of conspiracy theories!!! Tragically helpless women are falling for this stuff!!! So sad.

        There aren’t any confusing messages coming from health officials. They’ve changed their tune on masks, but otherwise the word has been consistent, *especially* from Fauci, who has said time and time again to maintain social distancing, that this isn’t going away, etc etc.

        It’s obviously an “opinon” piece in that it’s making unverified claims, but it’s not labeled that way. Assertions like this are now are common in general news. The story is more important than the facts.

        • jim says:

          Since we’re talking about misinformation: you don’t need a mask when fishing with your grandson unless you’re on a crowded peer, or when around your own home or when alone in your own car.

          So here’s the very people quacking about misinformation spreading it themselves.

          • somebody says:

            Here’s the quote from the article

            > Seven hundred miles away, Mary Gail Lowery was fishing with her grandson in Tuscaloosa, Ala. A retired teacher, she carried a mask, but used it only in stores and only “as a courtesy,” because she’s decided that masks aren’t very helpful as a defense against covid-19, the disease caused by the coronavirus.

            Nowhere does the article even imply that she should be wearing a mask while fishing. The fishing is just a description that parallels the opening line (Katie Gooch and her 3-year-old son were throwing rocks in a river not far from home in Richmond, Virginia), showing that the phenomenon under description is nationwide.

            So we have someone complaining about an article complaining about misinformation spreading misinformation, spreading misinformation. What a time to be alive!

        • somebody says:

          Here’s what you said

          > This morning I read an article from WAPO that said at the outset, paraphrasing:

          > “people are turning to conspiracy theories about how COVID is transmitted because there is no authoritative voice and Trump is a conspiracy theorist”

          There’s some tension between the article and your characterization.

          1. Trump’s name doesn’t even appear for several paragraphs. I would hardly call that the outset. In the overall article, he’s not even a close to a central focus.

          2. They didn’t say there’s no authoritative voice, they say there hasn’t been consistent authoritative advice. To be fair, you did shift your argument in your second response, but this

          > There aren’t any confusing messages coming from health officials

          seems ridiculous to me. WHO, CDC, and Fauci have all flip flopped on what we know about the virus, perhaps rightfully so based as the evidence situation evolves, but if you don’t find their information confusing then you must be some kind of information processing savant. Masks are unimportant, masks are important, stay out of the parks, outdoors are mostly safe, the virus transmits from surfaces, it spreads mostly from person to person, it’s not airborne, aerosols are the primary transmission vector, check for fever, temperature checks are mostly useless, the virus is spread when you cough, viral load peaks in the presymptomatic phase, asymptomatic people don’t spread, no wait they do.

          3. They don’t say it’s because Trump is a conspiracy theorist, they say it’s because Trump undermines the very authoritative voice that you’re saying is the authority.

          4. It’s not even about conspiracy theories. It doesn’t even say the phrase conspiracy theories except to quote someone saying “I do not traffic in conspiracy theories; I provide facts and figures from local, state and national governments”. The opening anecdote certainly conspiratorial, but that’s less than a quarter of the article. There’s also this

          > When New Mexico allowed gyms to reopen, Mikaela Kosich, an epidemiologist with a passion for roller derby, was eager to get back on the track. “I searched everywhere for information that would make it OK to go,” she said. “I really wanted to find it, but I knew I shouldn’t. Luckily, I couldn’t find it.”

          > Kosich, who works for the University of New Mexico, knows from her training that “in public health, there are often no certain answers,” but she said the contradictory signals Americans are getting from different arms of government send even medical professionals hunting for guidance on social media.

          and this

          > To fill the gap created by a government that has seemed to be at odds with itself, medical organizations and individual scientists have taken to publishing their own guidance. The Texas Medical Association, for example, put out a chart ranking the risks of 37 everyday activities, from low-risk opening the mail and getting takeout food to the highest risk choices of going to a bar or attending large religious services.

          > “With no clear direction from the federal government . . . there’s no right answer,” wrote Ezekiel Emanuel, a health policy adviser to President Obama who developed a covid-19 Risk Index. “You just need to know the risks and how much risk you’re willing to take.”

          and a long bit about Andy Slavitt and Alex Berenson on twitter, neither of whom are conspiracy theorists.

          Frankly, actually reading the article, it seems like you invented a strawman saying something wrong and opinionated, then attacked it. I don’t come away from it with a thesis statement that even vaguely resembles yours. Conspiracy theories aren’t even the main subject. It says that the environment of uncertainty and the conflict between different arms of the federal government means people get different information from different sources.

          What seems to be more important than the facts here is that you get to complain about something, and the details of what you’re complaining about are almost incidental.

  3. yyw says:

    This issue won’t be resolved unless scientific literacy becomes a job requirement for science journalists, but then these days you don’t even need to be scientifically literate to be a scientist.

    • gec says:

      I don’t think “scientific” literacy per se is the problem, I think it’s what Andrew pointed out, that journalists for various reasons feel constrained to only communicate a small number of story-types. They either try to massage what they have into one of those forms or, if they can’t do that on deadline, just don’t report it at all.

      I suspect this compulsion to fit things into a small number of stories is a combination of underestimating the audience (“will it play in Peoria?”) and tradition (“this is just how the news works”) multiplied by the need to produce content on a strict deadline.

      So maybe it is not scientific literacy, so much as “literacy” in general? The more comfortable people are with reading and writing different types of stories, the more accurate our science journalism will be.

      • Michael Nelson says:

        It’s incentives. Journalistic institutions are currently incentivized to report things that are harmful to public understanding. The coalition of scientists and journalists referred to in the open letter are trying to create an incentive in the other direction: it would provide an accessible, authoritative source for journalists to cite about a preprint. It won’t work: As Andrew wrote, articles that cite a primary source as saying the article isn’t worth reporting on won’t get published, thus creating an incentive for journalists to minimize or ignore the authority. The articles from outlets that print the flawed stories will go viral, and we’ll get secondary reporting from respected outlets with fomo.

        I mean, come on: Fox News reports opinions as facts without citing any objectively qualified authority, and they’re the most popular news outlet in the nation!

  4. Michael Nelson says:

    My immediate response upon reading the quoted portion of the letter: Ugh.

    Science is not the problem with bad journalism. Bad journalism is the problem with bad journalism.

    What if, instead of pouring all our energy into faster peer review, we just have one guy whose job it is to google science news articles and then issue a warning about premature science reporting? We’ll call him the Scientist General, give him a uniform and some medals, really make a big deal out of him. And every preprint has to carry the Scientist General’s Warning: “Preprints can be harmful to public understanding if reported with excessive credulity.”

    I’ll write the open letter. Where are my 100 cosigners? Who’s got the number for the Times?

    • Jonathan (another one) says:

      But then why should you trust *that* guy? You could get an FDA-like “I’m not buying anything until I’ve seen a pre-registered RCT,” in which case every science article post-Galileo would be “we shall see” or you can get a reasonably credulous “maybe this could use a little more corroboration but makes sense to me,” or the worst of both worlds: cautious on the unusually true and impressed by the typically crappy.

      • Michael Nelson says:

        What he says is in the job description–he’s not using actual judgment. The SG warning isn’t about the accuracy of the study’s findings or the quality of the study’s methods. Each news article is evaluated using a simple checklist: Is the study a preprint? If yes, did the reporter include a statement to the effect that preprints are, on average, far less reliable than peer-reviewed articles? Was that statement in the first full paragraph where study findings are given? If no to all three, out goes the warning.

        The point is that journalists don’t need a fancy science organization to fix the problem Eisen and Tibshirani say they’re trying to fix. Journalists just need a reason to follow a three-item checklist, and the hope is that having that checklist quoted to them “on deadline” by a fancy science organization is reason enough. My idea achieves the same with a single guy (or gal) we all play up as a big deal so he (she) won’t be ignored in the press. Which is also why the position is a life-long appointment, which ends after a four-year term when the current SG is summarily fired for his part in a lurid sex scandal.

        • Jonathan (another one) says:

          I wasn’t going to volunteer for the job until I heard I had the chance to be involved in a lurid sex scandal. But really, if all the SG is doing is reporting the results of a three item checklist, then I suspect his authority as “man of science” will be quickly undermined to “clerk who can read.”

  5. Mikhail Shubin says:

    But isnt this “selection bias” concern holds true for all kinds of reporting, not just for scientific journalism?

    How was the similar problem handled in others spheres of life?

Leave a Reply to jim