Skip to content
 

Michael Crichton on science and storytelling

Javier Benitez points us to this 1999 interview with techno-thriller writer Michael Crichton, who says:

I come before you today as someone who started life with degrees in physical anthropology and medicine; who then published research on endocrinology, and papers in the New England Journal of Medicine, and even in the Proceedings of the Peabody Museum. As someone who, after this promising beginning . . . spent the rest of his life in what is euphemistically called the entertainment business.

Scientists often complain to me that the media misunderstands their work. But I would suggest that in fact, the reality is just the opposite, and that it is science which misunderstands media. I will talk about why popular fiction about science must necessarily be sensationalistic, inaccurate, and negative.

Interesting, given that Crichton near the end of his life became notorious as a sensationalist climate change denier. But that doesn’t really come up in this particular interview, so let’s let him continue:

I’ll explain why it is impossible for the scientific method to be accurately portrayed in film. . . .

Movies are a special kind of storytelling, with their own requirements and rules. Here are four important ones:

– Movie characters must be compelled to act
– Movies need villains
– Movie searches are dull
– Movies must move

Unfortunately, the scientific method runs up against all four rules. In real life, scientists may compete, they may be driven – but they aren’t forced to work. Yet movies work best when characters have no choice. That’s why there is the long narrative tradition of contrived compulsion for scientists. . . .

Second, the villain. Real scientists may be challenged by nature, but they aren’t opposed by a human villain. Yet movies need a human personification of evil. You can’t make one without distorting the truth of science.

Third, searches. Scientific work is often an extended search. But movies can’t sustain a search, which is why they either run a parallel plotline, or more often, just cut the search short. . . .

Fourth, the matter of physical action: movies must move. Movies are visual and external. But much of the action of science is internal and intellectual, with little to show in the way of physical activity. . . .

For all these reasons, the scientific method presents genuine problems in film storytelling. I believe the problems are insoluble. . . .

This all makes sense.

Later on, Crichton says:

As for the media, I’d start using them, instead of feeling victimized by them. They may be in disrepute, but you’re not. The information society will be dominated by the groups and people who are most skilled at manipulating the media for their own ends.

Yup. And now he offers some ideas:

For example, under the auspices of a distinguished organization . . . I’d set up a service bureau for reporters. . . . Reporters are harried, and often don’t know science. A phone call away, establish a source of information to help them, to verify facts, to assist them through thorny issues. Don’t farm it out, make it your service, with your name on it. Over time, build this bureau into a kind of good housekeeping seal, so that your denial has power, and you can start knocking down phony stories, fake statistics and pointless scares immediately, before they build. . . .

Unfortunately, and through no fault of Crichton, we seem to have gotten the first of these suggestions but not the second. Scientists, universities, and journals promote the hell out of just about everything, but they aren’t so interested in knocking down phony stories. Instead we get crap like the Harvard University press office saying “The replication rate in psychology is quite high—indeed, it is statistically indistinguishable from 100%,” or the Cornell University press office saying . . . well, if you’re a regular reader of this blog you’ll know where I’m going on this one. Distinguished organizations are promoting the phony stories, not knocking them down.

Crichton concluded:

Under the circumstances, for scientists to fret over their image seems slightly absurd. This is a great field with great talents and great power. It’s time to assume your power, and shoulder your responsibility to get your message to the waiting world. It’s nobody’s job but yours. And nobody can do it as well as you can.

Didn’t work out so well. There have been some high points, such as Freakonomics, which, for all its flaws, presented a picture of social scientists as active problem solvers. But, in many other cases, it seems that science spent much of its credibility on a bunch of short-term quests for money and fame. Too bad, seeing what happened since 1999.

As scientists, I think we should spend less time thinking about how to craft our brilliant ideas as stories for the masses, and think harder about how we ourselves learn from stories. Let’s treat our audience, our fellow citizens of the world, with some respect.

47 Comments

  1. >> – Movies need villains

    Oh I hate this assumption so much.

    • Mikhail Shubin says:

      Here is one text about the issue:
      https://aeon.co/essays/why-is-pop-culture-obsessed-with-battles-between-good-and-evil

      quote: Virtually all our mass-culture narratives based on folklore have the same structure: good guys battle bad guys for the moral future of society. These tropes are all over our movies and comic books, in Narnia and at Hogwarts, and yet they don’t exist in any folktales, myths or ancient epics. In Marvel comics, Thor has to be worthy of his hammer, and he proves his worth with moral qualities. But in ancient myth, Thor is a god with powers and motives beyond any such idea as ‘worthiness’.

      • Alex says:

        An essay I read recently (a book chapter, to be more accurate) made the point that old heroes were essentially all war heroes because the concern then was literally existential: is the group next door going to invade my land and kill me. More recent heroes are less concerned with that but are more concerned with winning, as you say, the moral future of society.

        On the ‘movies need villains’ front, in high school I learned that there are only three kinds of stories: man vs man, man vs nature, and man vs himself. I don’t think we covered if one of those tended to be more popular or better than the others.

        • Those are pretty general, so they’d apply pretty widely, but even then you’ve clearly left out:

          Boy Meets Girl (or Girl Meets Boy) (When Harry Met Sally)
          Guys Doing Dumb Stuff (Bill and Ted’s Excellent Adventure)
          In The Future X (2001 A Space Odyssey)
          Man vs Supernatural (Poltergeist)
          Making Fun of X (This Is Spinal Tap / Waynes World)

          Of course you can mix and match, the plot of Waynes World isn’t that important, but it has elements of Boy Meets Girl, and Man vs Man, I still think of it as dominated by “Making Fun of Mike Meyers’ Teenage Friends”

          • Jeff says:

            Most of this seems reasonable, but how much more Men vs Themselves could “This Is Spinal Tap” be? The answer is none. None more Men vs Themselves.

          • Alex says:

            Well, the claim is that you can put literally every story into at least one of those three categories, so they have to apply broadly! And I think they apply to your examples:

            1) Boy meets girl (or someone meets someone; romance): all romance stories I’ve seen are one or both of man vs himself (the romance is threatened by some personal flaw to be overcome) and man vs man (the romance is threatened by, e.g., an overbearing family member).

            2) Guys doing dumb stuff: the same. There’s usually some kind of life lesson to be learned (vs himself) and/or some outside force to be overcome (e.g., Bill and Ted have to keep Ted’s dad from sending Ted to military school; the guys in Animal House have to keep the dean from shutting down the fraternity). Although I’m sympathetic to the idea that the plot in movies like this (and a lot of action movies) is largely there just to move between fun sequences.

            3) In the future: future stories are just stories. Depending on what you take away from 2001, it’s either man vs nature (evolution) or man vs himself (depending on what you think of AI, maybe man vs man). The Martian is man vs nature.

            4) Man vs supernatural: I guess by definition supernatural isn’t nature, but I would lump this in with man vs nature.

            5) Making fun of: basically overlaps with whatever genre they’re making fun of. Meta rom-coms are the same as #1.

        • Mikhail Shubin says:

          > man vs man, man vs nature, and man vs himself

          I prefer Borges’ classification of literature into four types:
          1) stories about siege
          2) stories about returning home
          3) stories about search
          4) stories about suicide of god.

      • Albert Hsiung says:

        > This particular point we would be the last to deny: the man who learnt to know those “good” ones only as enemies, learnt at the same time not to know them only as “evil enemies” and the same men who inter pares were kept so rigorously in bounds through convention, respect, custom, and gratitude, though much more through mutual vigilance and jealousy inter pares, these men who in their relations with each other find so many new ways of manifesting consideration, self-control, delicacy, loyalty, pride, and friendship, these men are in reference to what is outside their circle (where the foreign element, a foreign country, begins), not much better than[Pg 40] beasts of prey, which have been let loose. They enjoy there freedom from all social control, they feel that in the wilderness they can give vent with impunity to that tension which is produced by enclosure and imprisonment in the peace of society, they revert to the innocence of the beast-of-prey conscience, like jubilant monsters, who perhaps come from a ghastly bout of murder, arson, rape, and torture, with bravado and a moral equanimity, as though merely some wild student’s prank had been played, perfectly convinced that the poets have now an ample theme to sing and celebrate. It is impossible not to recognise at the core of all these aristocratic races the beast of prey; the magnificent blonde brute, avidly rampant for spoil and victory; this hidden core needed an outlet from time to time, the beast must get loose again, must return into the wilderness—the Roman, Arabic, German, and Japanese nobility, the Homeric heroes, the Scandinavian Vikings, are all alike in this need. It is the aristocratic races who have left the idea “Barbarian” on all the tracks in which they have marched; nay, a consciousness of this very barbarianism, and even a pride in it, manifests itself even in their highest civilisation (for example, when Pericles says to his Athenians in that celebrated funeral oration, “Our audacity has forced a way over every land and sea, rearing everywhere imperishable memorials of itself for good and for evil”).

        -Nietzsche

    • Phil says:

      Some science- and/or technology-oriented movies without villains have been very successful, so I think Crichton is wrong about this. Examples include Apollo 13; The Martian; and Gravity. There are others. Interstellar had a bad guy and a not-so-great guy to add some variety to the story but they aren’t major parts of the film. Huh, indeed it suddenly occurs to me that Crichton himself wrote a book-turned-movie that didn’t have a villain: The Andromeda Strain.

      Going outside science and technology, there are movies like The Poseidon Adventure, The Guardian, Cast Away, Into the Wild, and Alive.

      So you really don’t need a villain, or at least not in the normal sense of the word, which implies a foe with agency.

  2. Dale Lehman says:

    Why is “media” equated with “film?” Perhaps the excerpts skipped an explanation, but I find that these two should be quite different – the fact that they are not, is indeed a problem. It doesn’t bother me that film misrepresents science – most good movies misrepresent a great many things. In some cases, this is precisely their appear. In other cases, movies are an accurate portrayal (of mental illness, disability, abusive acts, war, etc.) and that is sometimes their appeal. But movies are a form of creative art and I don’t think it is desirable to insist that all movies mimic reality – or the opposite.

    Media – thinking of journalism here – is quite different. I think that providing realism is important there. The fact that journalism and entertainment have become so indistinguishable is a problem. Did Crichton confuse the two, did Andrew, did I misread it, or are the excerpts out of context?

    • David L. says:

      Uh, because MC made his killing in filmable schlock and films made therefrom, maybe.

      I’m reminded of “Shogun”. I read it before getting more deeply interested in Japan, but even as someone with only a vague awareness of the country, it was clear that Shogun had nothing to do with anything about Japan whatsoever. (It really doesn’t. It’s a lovely story, and the depiction of the hero quite believable/nicely done. But the details are pure Western stereotype on steroids. The Japanese really ought to get more bent out of shape about garbage like Shogun, but, whatever.)

      The folks who went ballistic over Shogun were the US academics doing Japanese Studies (literature, history, and the like). Someone even wrote a book “Learning from Shogun” to try to undo some of the damage. But the author made an interesting (albeit inanely stupid) comment “History that’s so boring no one can read it isn’t history.”

      But he was wrong: what these things are doing is retelling existing stereotypes and tropes in entertaining ways, so that people can have fun without having to bother thinking.

  3. oncodoc says:

    Just to address one point in this article, movie makers do find it necessary to create dramatic tension via unsympathetic characters. In The Dallas Buyers’ Club the protagonist is portrayed (by the extremely handsome Matthew McConaughey) as needing to circumvent the obstructions of a scientist wearing a white coat who insists on randomized trials. The reality is that those uptight science guys actually produced real progress against the disease while the real life person portrayed in this film wound up with a bunch of toxic compounds that didn’t work. The filmmakers chose to forego the real story which is twofold; a desperate man looks for help, and the formal process of scientific inquiry actually works. Instead they turned it into a cool guy beats the nerdy guy story. Successful movies show the protagonist with a pistol shooting 73 bad guys armed with automatic weapons, and the good guy never reloads.

  4. Carlos Ungil says:

    Did you intend to include a link to the piece you quote?

    I found this one (address recorded at the American Association for the Advancement of Science on January 25 1999, and broadcast on the Science Show on April 3, 1999): http://www.abc.net.au/science/slab/crichton/story.htm

    Redacted as article in Science (5 Mar 1999): http://science.sciencemag.org/content/283/5407/1461

  5. Matt Skaggs says:

    IMO scientists should never attempt to describe the significance of their work to the public. And multiply that by ten if the work is paywalled!

    Crichton wrote:

    “…under the auspices of a distinguished organization . . . I’d set up a service bureau for reporters”

    Yes! The reporter calls the “service bureau” and says “I’d like to write an article about himmicanes.” A psychologist responds – perhaps an emeritus or someone rotating through the bureau while on sabbatical – and builds an insiders picture of how himmicanes fit into a broader picture of subliminally stimulated response or whatever. These papers are generally supportive, these are conflicting, etc. The author of the paper that caused the original request may get involved to explain the methodology. The reporter then submits the article to the bureau for factual review prior to publication.

    Crichton also said (to scientists):

    “It’s time to assume your power, and shoulder your responsibility to get your message to the waiting world. It’s nobody’s job but yours. And nobody can do it as well as you can.”

    Nope. Scientist-as-activist just does not work. The entire edifice of academic science falls apart. Why isn’t getting your paper published getting your message to the waiting world? Because it is full of jargon and dense formulae and behind a paywall and cannot be understood outside the field? Then you wrote a poor abstract. Nothing is more ridiculous to me than seeing two scientists have a very public argument about the merits of their two competing papers, both of which are behind a paywall.

    And one other thing. The IPCC is built to resemble that service bureau being discussed here. But its true function is exactly the opposite of what I am advocating.

  6. > under the auspices of a distinguished organization . . . I’d set up a service bureau for reporters. . . . Reporters are harried, and often don’t know science. A phone call away, establish a source of information to help them, to verify facts, to assist them through thorny issues. Don’t farm it out, make it your service, with your name on it. Over time, build this bureau into a kind of good housekeeping seal, so that your denial has power, and you can start knocking down phony stories, fake statistics and pointless scares immediately

    Does seem to me what David Spiegelhalter’s group does – https://wintoncentre.maths.cam.ac.uk/what-we-do

  7. John Williams says:

    ” Scientists, universities, and journals promote the hell out of just about everything, but they aren’t so interested in knocking down phony stories.” This is not entirely true. Realclimate.org, for example, does a great job on phony climate stories. The Union of Concerned Scientists does pretty well at this sort of thing, too. There are good resources for reporters out there, if they want to use. them.

  8. Z says:

    I didn’t realize Michael Crichton was a scientist. Makes this innumeracy (https://twitter.com/EmpiricalDave/status/1067941351478710272) from Jurassic Park more disappointing

  9. Mark Samuel Tuttle says:

    Permit me a story …

    Took my then MBA-student girlfriend now wife to see the Movie “Quest for Fire” (1981). For those of you haven’t seen this very imaginative tale, a pre-historic tribe can use fire but can’t start it. And, they have only the most primitive mastery of language, designed for the film by Anthony Burgess. They are attacked and lose their fire. Three young members of the tribe are chosen to “find fire”, so they journey across Europe until, to their surprise, they find a tribe that can start fire.

    At the end of the movie is a particularly poignant scene. One of the travelers is pantomiming, and describing, the story of their journey to the tribe assembled around a fire. My girlfriend turned to me and said something I’ll never forget, “That’s how it all started, didn’t it; telling stories around the fire.”

    In my career, I’ve discussed the power of stories with several cognitive psychologists, but none of them summed it up so well.

  10. sensationalist climate change denier says:

    As I recall, with Crichton’s “climate change denial” he was really making much broader points and applied them to global warming as a perhaps poorly chosen example. He strongly disliked appeals to consensus, noting that fashionable positions often end up being entirely wrong and that this is often invoked when the direct evidence is not very strong.

    In many cases, only specialists can competently judge the strength of the evidence. An intelligent layman might follow the basics of the matter with some effort, but won’t have enough experience to spot specious reasoning (it’s really easy to construct convincing but erroneous mathematical proofs with small subtle flaws, for example). Consensus is valid to the extent the experts can be trusted to assess the strength of the evidence on our behalf. But if they show a willingness to go beyond the evidence and push ideas they like despite weak evidence (or to conceal evidence they don’t like), then the trust breaks down and the public will not have confidence in expert assessment. Politics and financial interests are obvious conflicts, but there are other seemingly neutral examples that seem to based on group behavior, pride, inertia, etc.

    The best thing scientists can do is earn the public’s trust.

  11. Carl says:

    “Interesting, given that Crichton… became notorious as a sensationalist climate change denier.”

    …ahh, a now standard leftish litmus test of personal credibility. AGW is gospel.

    • Peter Dorman says:

      As I mentioned in a comment a few months back, any scientific view can be disputed (that’s how science advances), but the bar for such a dispute is set by the sophistication, diversity and weight of the evidence. The problem with AGW denialism is that it doesn’t begin to approach this bar, leaving the impression that its motivation lies well outside science. I have been embroiled recently in a debate *within* the climate science community regarding the extent of marine methane deposits, their role in the carbon cycle and their stability past and present. There are a lot of strong feelings expressed by some of the disputants, and in time one side will have been shown to be quite wrong (their views are too divergent to avoid this), but I respect all participants for at least recognizing and attempting to meet the standards for debate set by the wider research community. It’s a good contest.

      As for Crichton, his denialist “thriller”, State of Fear, was embarrassingly ignorant. He had a group of scientist-conspirators trying to engineer a fake earthquake & tsunami to scare the public into accepting the risk of climate change, when, of course, earthquakes have nothing whatsoever to do with it. At least if he had them secretly planting space heaters in the arctic or something like that….

  12. Ecoute Sauvage says:

    Andrew – here’s some more statistical advice from fiction. Euripides, The Trojan Women, the queen of Troy after the fall of her city:

    “Hecuba
    Though fortune change, endure your lot; sail with the stream, and follow fortune’s tack, do not steer your ship of life against the tide, since chance must guide your course.”

    Of course it sounds a whole lot better in the original – if you have no Greek, get someone to read it aloud to you!

    Ἑκάβη
    μεταβαλλομένου δαίμονος ἀνέχου.
    πλεῖ κατὰ πορθμόν, πλεῖ κατὰ δαίμονα,
    μηδὲ προσίστω πρῷραν βιότου
    πρὸς κῦμα πλέουσα τύχαισιν.

  13. yyw says:

    I don’t think educated people have trust issues with basic sciences, where theories give precise predictions that can be tested precisely. Outside of these fields, how does one have an informed assessment of how much to trust? We all know that journalists are not equipped to make that assessment on behalf of us. Most of us don’t have the time to thoroughly educate ourselves and go through literature of a field to evaluate the quality of claims. We have seen scientific consensus flip-flopped.

    Maybe the key for scientists in these fields to earn trust is to be keenly aware of limitations of their models and theories and uncertainties in their predictions and be very transparent when communicating with the public. A bureau that journalists can turn to could be a good idea, but this bureau should not speak with a single voice but rather should try to convey the range of plausible theories/models/predictions on any single subject.

  14. jim says:

    “…under the auspices of a distinguished organization . . . I’d set up a service bureau for reporters”

    We already have bazillions of organizations that presume to be the arbiters of Real Scientific Truth for the public, from countless blogs; major science orgs like Union of Concerned Scientists; American Geophysical Union; orgs for almost every conceivable subdiscipline in science; teachers organizations, industry professional organizations, data literacy orgs and countless other groups.

    None of these groups are reliable as the sole arbiters of scientific truth, even in their specific disciplines. They’re all at least somewhat influenced by the political views of their membership, and in some cases vehemently so.

    The problem is that, while scientists seek and claim to find truth, their claims are frequently wrong.

    In my experience, the public doesn’t have much problem understanding and accepting the realities of science. Ask anyone: According to science is coffee good for you? They’ll tell you that scientists’ answer changes from one minute to the next. But ask the scientist who published the most recent paper. They’ll tell you “well our paper showed….” The scientist sees only h/her research, but the public sees the broader patterns of science.

  15. Maximilian says:

    Dear Andrew,

    I follow your blog for a few weeks now, and let me first say that I really enjoy it!

    I have a question/remark regarding your post about the cornell university press office, where you said that “Wansink (…) openly employed a hypotheses-after-results-are-known methodology which leaves those statistics meaningless, even after correcting all the naive errors.”

    I recently thought much about this question, and I came to the conclusion that I never heard a compelling reason why hypotheses have to be stated before the data is known. (And I read popper, carnap, meehl etc., and actually meehl said in one of his lectures which you find online that this is an issue where no compelling jusfificafion that the order of hypothesizing and data analysis matters, but that there is a strong instinct of the researchers.)

    Let me give an example: you hypothesize and aquire data afterwards, and the data supports your hypothesis. Then another researcher finds an alternative explanation for your data. I think you would agree, that you have to compare those two theories on reasonable grounds before you assume one is better than the other. Fortunately, occams razor tells us we should prefer the more parsimonious theory. So in this “equation” of comparing theories (that equally well explain the data), there seems to be no room for a “bonus” one theorie gets because it was stated beforehand. It seems to me that all what matters here is parsimony.

    Furthermore, it seems to me that many researchers in social sciences implicitely do research the way “my theory is complex as shit and could explain almost every possible observation, but it is okay because I stated it beforehand”.

    After all, it seems like a human thing to assume that theories stated beforehand receive stronger corroboration. It is somehow cool and exciting to see that someone predicted new facts before they where known, and boring that someone is able to find another theory that explains already known facts. But I am not quite shure if this is actually a fallacy.

    I would find it really interesting to hear your opinion about this!

    Maximilian

    • jim says:

      Hi Maximillian,

      I’m neither a statistician or a Gelman, but I’ll try out an answer to your question. The short form of the answer is that the reason it matters is the technique used to answer the question.

      Suppose you’re a major league manager and you’re searching desperately for new clues as to how to spot that next cy young award pitcher based on high school pitching stats. You really don’t have any kind of hypothesis at all. But you pile up some 20 metrics and start testing them. You’re using the standard statistical significance where if p < 0.05 the correlation between your two variables is real.

      Now, you run the data and you find badabing! One of your tests – "do left-handed pitchers named Jed with ball/strike ratios less than 0.8333 in home games on rainy days played south of the Kansas-Nebraska border on the 4th of July all become cy young award winners?" – shows p < 0.0378588, so that comparison is a true correlation!

      So now you think up a reason as to why this might be so: these pitchers are all raised on amazing Kansas cattle and therefore have extraordinarily strong arms.

      But don't forget your test has a failure rate. The failure rate is that it will be wrong one time out of every twenty – because you specified p < 0.05. And since you just ran twenty analyses your chances of having one spurious correlation are very high.

      Now, suppose you did it the other way around. You start out with the hypothesis that "left-handed pitchers named Jed (bla bla)..BECAUSE these pitchers are all raised on amazing Kansas cattle and therefore have extraordinarily strong arms." It shouldn't matter, right? Same data, same answer. And really it doesn't. But you propose the hypothesis first, you're still using p < 0.05, so you still have a 1 in 20 chance of a spurious correlation. So in one sense, same hypothesis, same data – it doesn't matter which way you do it. The chance of a spurious result is still the same.

      The difference is this: if you do your probability first, then you are actually intentionally picking the spurious result.

      With regard to: “my theory is complex as shit and could explain almost every possible observation, but it is okay because I stated it beforehand" , this is clearly an abuse of the concept. You can state your theory before hand and: 1) design a bad experiment; 2) record data poorly; 3) utilize assumptions that aren't valid; or any number of other mistakes to generate a spurious outcome.

      • Maximilian says:

        Hey Jim,

        thanks for the answer, altough I maybe not completely unterstand your point, especially

        “The difference is this: if you do your probability first, then you are actually intentionally picking the spurious result.”

        Why do you think this is the case? To stay in your example: the model the manager is using is just overly compley, I would try for example a penalized regression model instead of testing every correlation by its own.

        And then I would make the case for those results being probably more meaningful than results optained by 20 different researchers all testing a different variable to find good players.

    • Martha (Smith) says:

      Maximilian,

      See if this helps you understand:

      1. Go to https://web.ma.utexas.edu/users/mks/CommonMistakes2016/commonmistakeshome2016.html
      2. Scroll down to the link titled “Jerry Dallal’s Simulation of Multiple Testing”.
      3. Before clicking on the link to Jerry Dallal’s simulation page, read the blurb describing what the simulation does. Then click on the link and follow the instructions in the blurb you read before clinking.
      4. Then maybe go back to my page in step 1 and try the links in the next items (Jelly Beans and Three More Jerry Dallas simulations.)

      • Maximilian says:

        Dear Martha,

        thanks, I followed the link. I understand the issue of multiple testing and the problems that come along with it. Maybe to clarify my point:

        What difference does it make, if

        1. Someone measures 100 Variables to predict cancer and some correlations occur which are in reality not there (or rather since there are no true 0-correlation, the size of some correlations is estimated much to high or the sign of the correlation is wrong)

        2. each of 100 indepentend researchers state a hypothesis beforehand that something is related to cancer and measure and test it afterwards, and the same spurious correlations occur.

        • Martha (Smith) says:

          Maximilian,
          I’m having trouble understanding your second scenario. To talk about “the same spurious correlations” occurring, you would need each of the 100 independent researchers measuring exactly the same 100 variables as in the first scenario, but the description of scenario 2 says “state a hypothesis beforehand that something is related to cancer and measure and test it afterwards”, which sounds like the 100 independent researchers are not all measuring the same thing.

          • Maximilian says:

            I think it does not matter if they measure the same thing, or different things, or some the same some different. If you assume all effects are not there in reality, you get 5% of the time false positives.
            The same as if you measure all variables in one study and hypothesize after the data is known.

            • Martha (Smith) says:

              “I think it does not matter if they measure the same thing, or different things, or some the same some different. If you assume all effects are not there in reality, you get 5% of the time false positives.”

              Sounds like you misunderstand. One needs to be careful to understand precisely enough what “5% of the time” refers to. It means that if you are considering just one hypothesis test, and if you do that hypothesis test (measuring the same thing each time) on an extremely large number of samples from the same “population” (e.g., people with a particular type of cancer and given a certain treatment), and if the null hypothesis is true, then you could expect that roughly 5% of those samples would yield “false positives”. But if you were doing different tests (or measuring different things) on different samples, then the reasoning behind the assertion of 5% false positives would not apply, since the reasoning assumes the same hypothesis test and the same thing measured in all samples considered.

              • Maximilian says:

                Yes, and this definition is true for every hypothesis under consideration. So if there are no true effects, the cance that a particular result is a false positive is 5% – no matter if it is a repeated measure of the same variable in the same population or something completely different.

                If you have 100 different dices, each with a probability of 1/6 to get a 3, you certainly would agree that it does not matter if you roll the same dice again and again or 100 different dices – the frequency of 3s will be the same.

                Now, if you test the same hypothesis from the same population again and again, or different ones from different populations, as long as the treshold for the p-value remains 0.05, the frequency of type-1 errors will be .05.

              • Martha (Smith) says:

                “So if there are no true effects .. .”

                To some extent this is OK — but the assumption that there are no true effects is pretty iffy.

                In addition, when doing multiple hypothesis tests, we need to look at the probabilities of getting different combinations of “false positives”. This gets complicated, especially if the effects being tested for are not independent, which is common. The situation of throwing one die 100 times vs throwing 100 dies separately is a situation where in both scenarios you have independence between events. In real life situations where people do hypothesis tests, there is usually dependence between the events involved in the hypothesis tests — and different types of dependence in different situations. So the example with dice is not representative of the situations where hypothesis tests are typically used.

    • Andrew says:

      Maximilian:

      You write, “I never heard a compelling reason why hypotheses have to be stated before the data is known.” I agree. In most of my applied work, the hypotheses are not stated until the data have been analyzed. Even when I have a general scientific hypothesis ahead of time, the specific statistical model or “hypothesis” that is fit or tested will be constructed in light of the data.

      My problem is not with “hypothesizing after results are known” but rather with the use of null hypothesis significance testing when one has hypothesized after results are known. My solution to the problem is not to stop hypothesizing but rather to stop null hypothesis significance testing.

      This is one of my problems with the whole pre-registered replication movement. Pre-registered replications are fine, but I think they should be accompanied with stronger theory, measurement, and data analysis. If we stick with the status quo of empty theory, crappy measurement, and null hypothesis-based data analysis, than any benefits of preregistration will only be very indirect: we can expect repeated failure which maybe will motivate better theory, measurement, and data analysis. So I’d like to cut out the middle man and go straight there.

      • Maximilian says:

        Dear Andrew,

        thanks for the reassuring reply. At my university, what most people teach and think is that it is one of the deadly sins of Psychology to state your hypothesis after the data is known, so I am happy to hear I am not the only one who thinks this is not really important (or at least there are much more important things).

        I agree that the whole paradigma of null hypothesis significance testing is the real problem here!

        Also I think what you said about replication studys is a good point – it would be much better if researchers see them as gathering more data to be able to estimate the magnitude of an effect more precisely, than the attempt of finding an effect again of which we already know it is certainly there.

Leave a Reply