Science communication is not a one-shot game

[cat picture]

In our recent discussion of Ted doubling down on power pose, commenter Michael raised an interesting question:

I think the general attitude of most people who work on communicating science to the public is that their responsibility is only to make sure that any information they present has a source with the proper credentials (published in a peer-reviewed journal, endorsed by PhD experts in the relevant disciplines at universities). Since they are not themselves PhD experts, the feeling is that “Who am I to challenge this expert? I am just telling you what my expert says, it’s not my job to get involved in these obscure internal arguments”. . . . If Slate can let Andrew Gelman write an article, or Retraction Watch can publish an interview with him expressing his position without publishing comments from experts with objectively equal qualifications who disagree, why can’t TED let Amy Cuddy put out her ideas? How should someone outside of the relevant disciplines be expected to know when what an expert is saying needs to be challenged? I can’t think of a good solution.

I replied as follows:

One difference between Cuddy’s Ted talk and my Slate articles is that I take the other side of the argument seriously, even if I express disagreement.

For example, today in Slate I looked into Jon Krosnick’s claim that the outcome of the 2016 election was determined by Trump being listed first on the ballot in many swing states. I concluded that it was possible but that I was skeptical that the effects would’ve been large. True, Slate did not invite Krosnick to respond. But in my article I linked to Krosnick’s statement, I clearly stated my sources of evidence, I linked and took seriously a research article by Krosnick and others on the topic . . . I did my due diligence.

In contrast, the Ted team avoids linking to criticisms of Cuddy’s work, and I do not consider her statements to be in the full spirit of scientific inquiry. It seems like a damage control operation more than anything else. As to the original Carney, Cuddy, and Yap article: as I noted above, it makes a claim in the abstract that is not supported by anything in the paper. And more recently Carney gave a long list of problems with the paper, which again Cuddy is not seriously addressing.

This response is fine as far as it goes, but I realized something else is going on, which is that Slate and Ted and other media outlets get multiple chances. If Jon Krosnik can make a strong case that my skepticism in his theory is misplaced, he can write about it—and even if Slate doesn’t run Krosnick’s (hypothetical) right away, they’d certainly be aware of it if they were to do more reporting on ballot-order effects. What about Ted? It’s hard to fault them for greenlighting Cuddy’s talk: at the time, the Carney/Cuddy/Yap paper had not been widely criticized, the failed replications were still in the future, and Cuddy had that Harvard credential. So, fine. My problem with Ted comes later, when they continue to endorse the work—and, more recently, to go all-in on it—in a way that dodges all criticism. I can’t expect typical journalistic outlets to discover flaws in research claims that already managed to get past peer review. But I can fault them for not updating their priors in light of new information.

Journalism, like science and bookmaking, is a multiple-round game, and the big fails can come from not knowing when to cut your losses.

6 thoughts on “Science communication is not a one-shot game

  1. How should someone outside of the relevant disciplines be expected to know when what an expert is saying needs to be challenged?

    The answer is really simple and has been known for a long time, but people don’t want to accept it for some reason. You need to always be skeptical and require accuracy of precise predictions and/or engineering feats:

    Archimedes, however, in writing to King Hiero, whose friend and near relation he was, had stated that given the force, any given weight might be moved, and even boasted, we are told, relying on the strength of demonstration, that if there were another earth, by going into it he could remove this. Hiero being struck with amazement at this, and entreating him to make good this problem by actual experiment, and show some great weight moved by a small engine, he fixed accordingly upon a ship of burden out of the king’s arsenal, which could not be drawn out of the dock without great labour and many men; and, loading her with many passengers and a full freight, sitting himself the while far off, with no great endeavour, but only holding the head of the pulley in his hand and drawing the cords by degrees, he drew the ship in a straight line, as smoothly and evenly as if she had been in the sea. The king, astonished at this, and convinced of the power of the art, prevailed upon Archimedes to make him engines accommodated to all the purposes, offensive and defensive, of a siege.

    Plutarch, The Parallel Lives. The Life of Marcellus. (~100 AD). http://classics.mit.edu/Plutarch/marcellu.html

    Dr Halley foresaw this, and therefore, with greate sgacity, fixed its return to the very time in which it has happenedl and he will, I suppose be forever remembered and admired as the first person who has foretold an event of this kind
    […]
    By its appearance at this time, the truth of the Newtonian Theory of the Solar System is demonstrated to the conviction of the whole world, and the credit of the astronomers is fully established and raised far above all the wit and sneers of ignorant men

    The Gentleman’s Magazine 29, 205 (1759). https://babel.hathitrust.org/cgi/pt?id=mdp.39015006991130;view=1up;seq=227

  2. For real scientific intrigue, turn to today’s lengthy NYT article:

    https://www.nytimes.com/2017/03/08/science/cancer-carlo-croce.html?hp&action=click&pgtype=Homepage&clickSource=story-heading&module=photo-spot-region&region=top-news&WT.nav=top-news&_r=0

    And, who is this mysterious Clare Francis of unknown gender?:

    “Whoever Clare Francis actually is, he or she has an uncanny knack for seeing improperly altered images, as well as smaller flaws that some editors are inclined to ignore.”

    He or she says of the Ohio State researcher under scrutiny, “My reply was that he has over a thousand papers [!]. That is not the same thing. You simply misunderstand science. Please stop talking about reputations and look at the reality!”

    The subject of the NYT article, Dr. Carlo Croce, said of him or her, “Clearly we are dealing with a mentally deranged person who should be recovered in a mental institution,”
    Perhaps the major differences in this dispute vs. the one Andrew is dealing with, is the amount of grant money involved, $86 million and, of course, cancer remedies vs. power posing.

    • “It is possible, but not certain, that an error was made,” he said, adding that if there was an error, “it affects nothing,” because the results were confirmed in separate experiments.

      I do not think he is being disingenuous here, we see the same thing over and over. It just doesn’t matter what these people do (misinterpret p-values, plug in the wrong data, fail to control for other explanations), nothing affects the conclusions once they have gotten the magic stamp. They really do believe in this.

  3. I agree. Every platform will have some downsides, risks, ways to game the system so that junk gets through the first filters. The question is how the platform can build counterweights in. What’s galling about TED is that they seem not even a little interested in this question.

    If your platform is vulnerable to self-interested colleagues anonymously tanking threatening but solid work, [journal editors should be coming up with ways to deal with that.](http://statmodeling.stat.columbia.edu/2017/03/06/hey-can-guess-3-goofy-tricks-new-journal-trying-improve-peer-review)

    If your platform is, you know, just for a hypothetical… vulnerable to thought leaders/hucksters provocatively announcing sound-bite ideas to millions of people based on false premises and bad evidence, think of some way, any way, to deal with it. There’s obvious value in the TED model. I just wish they cared at all, even just a little, about self-correction.

Leave a Reply to Anoneuoid Cancel reply

Your email address will not be published. Required fields are marked *