Why I no longer use the term “cargo-cult science”

In our new paper, “Interrogating the ‘cargo cult science’ metaphor,'” Megan Higgs and I write:

Over the past fifty years, the term “cargo cult” has been used to describe the actions of scientists who appear to follow forms of scientific inquiry but without the understanding and self-criticism that are essential to real scientific progress. The term has served a useful role by providing a short and catchy label to something that is otherwise difficult to explain. However, the term is also fraught with historical and cultural baggage and, in our opinion, encourages crossing of a subtle line between criticizing discipline methodological norms and criticizing the individuals currently carrying out those norms as part of a complex and context dependent social process. We find that carefully interrogating the term itself holds some important lessons for improvement in the science reform movement.

We conclude:

Better practice in quantitative research is not just about improving our use of statistical tests (advice akin to saying “change the shape of the headphones” or “build bigger runways” in the cargo cult narrative) or even abandoning tests entirely (“fake headphones and runways serve no purpose at all”); it also involves studying real effects, controlling and adjusting for variation, and carefully integrating statistical models with scientific models. It involves a deeper understanding of available methods and when they are well aligned with the bigger goals of the science. In today’s scientific culture, long-established principles of measurement and statistical design are often forgotten, or inadvertently ignored, because statistical significance and publication can be bought with the cheap coin of uncontrolled statistical analysis.

Improving statistical practice can be expected to reduce the rate of absurd claims that appear to be supported by data, but the real gains should arrive indirectly by incentivizing researchers to put in the effort to improve design, measurement, data collection, and the mapping of scientific theories to realistic models of data. Those pushing science reform efforts could benefit from respecting these challenges, rather than labeling other scientists, or their practices, as “cargo cult” and offering easy diagnoses that, while well-meaning, miss the mark.

The full paper is here. Also relevant is this paper on forking paths and workflow in statistical practice and communication.

44 thoughts on “Why I no longer use the term “cargo-cult science”

  1. I own (and play) two acoustic guitars made in Bozeman. I cherish them. I also own Lakoff&Johnson’s ‘Metaphors we live by’, purchased and read at the time of publication – I found it worthless then, and have yet to find a reason to change my mind about it. Sorry. Beyerstein, sadly, is no longer with us. Daston yes, Gigerenzer no (weird, isn’t it?) Gimme Feynman anytime.

  2. Three manuscripts that may explain why I no longer like terms such as “the incentives”, “publish or perish”, and “the system”:

    1) I wrote a manuscript titled “The natural selection of bad psychological scientists” which can be found on SSRN that sort of presents a different version of Smaldino & McElreath’s 2016 paper titled “The natural selection of bad science”.

    2) I also wrote a manuscript titled “Things I have wondered (so far)” which can be found on SSRN that sort of wonders about certain things, one of which is the whole “publish or perish” stuff that still makes little sense to me.

    3) The final thing I wondered in that latter manuscript is whether psychopathy might be present and prosperous in present-day psychological science. I subsequently wrote an extensive version of that wondering which can be found on SSRN with the title “Why psychopathy might be present and prosperous in present-day psychological science”. One of the things it wonders and ponders about is whether vague rules and such things might provide certain types of “scientists” to prosper.

    I hope it’s okay to share these, I have done so before here. I kind of don’t want to keep mentioning to my own manuscripts but I get so annoyed with several versions of explaining problematic issues that I long for some other version and some other variables (e.g. personalities of scientists) to get some more attention. It is my hope that especially the last manuscript mentioned here hopefully includes many useful references for those who want to think some more about other things than “the incentives”.

        • Quote from above: “He recycles this post all the time.”

          Is that about me posting the titles of my manuscripts? If so, I try to not do that but whenever I read something about how we should not talk about individuals but about the system or the incentives or whatever regarding problematic issues I feel the need to add my 2 cents. Hope that’s okay, but more than that I hope it’s clear that it may (at least in some cases) be a result of something somebody writes. Maybe I should stop reading stuff, that may be a possible solution for me possibly mentioning my manuscripts too much.

          As a side note: the manuscript about psychopathy has been posted exactly 1 month ago if I am not mistaken. I am not sure how many times I have mentioned that manuscript, but my guess is about two times or so…

  3. @A.A.A. “(g) the results do not support the hypothesis, but the experiment now appears to be an inadequate test of it. Apart from providing dull reading, papers with such conclusions share another feature: they contain an implicit promise of more along the same lines in the future. They make one wish that at least some writers, faced with the decision of whether to publish or perish, should have seriously considered the latter alternative.”
    (Tulving&Madigan,1970,p.442)

    • I quickly browsed through the paper by Tulving & Madigan (1970), and what they write in their conclusion “We have hundreds and thousands of little facts, we can make quantitative instead of qualitative statements, we can talk about all kinds of fine details in experimental data and characteristics of underlying processes- but the broad picture we have of human memory in 1970 does not differ from that in 1870” reminded me of a paper by Koch (1981) titled “The nature and limits of psychological knowledge: Lessons of a century qua “science””.

      As far as refraining from publishing, I have wondered several times whether I should stop publishing stuff. In fact, I think in one of the manuscripts I mentioned I explicitly wonder about how one is even able to determine whether one can correctly determine whether something is worthwhile to suggest as reading or learning material for a curriculum for example.

      It seems to me that it might be hard, if not impossible, to determine whether one produces manuscripts worthy of being published and read. I haven’t figured out how to do that, which is why I am happy to “merely” post my manuscripts as “pre-prints” and why I try to be very careful in how and what I state in the manuscripts. I think that might be appropriate and suitable in my case. I hope more capable and knowledgeable people can perhaps do something useful with the manuscripts I posted in some way, shape, or form.

  4. “The implication seems to be that the members of the cult did not possess the type of
    integrity scientists are expected to operate under, they didn’t work hard enough to
    bend over backwards to show how they might be wrong, …”

    I think this is taking the metaphor too seriously. The correct level of reading seems to be that both groups – scientists and islanders – are missing *something* that existed in the original cultural context in which their methods were developed. There is no reason this *something* needs to be the same thing in each case.

    Feynman explicitly suggests that in the case of the scientists, this may be a type of epistemic integrity. I think the obvious thing missing in the case of the islander is, as you also note:

    “the members of Feynman’s cargo cult lacked access to various aspects of modern science and technology”

    “[…] all the honesty, openness, and scientific integrity in the world wouldn’t have allowed them to leap the technological gap”

    This is just so obviously true that I think this ought to be the way we read the original speech too! There’s no way Feynman was seriously suggesting that, had the islanders been a little more honest with themselves, the planes would have landed. I mean, come on!

    I do agree that talking about “social rituals” is perhaps more useful. And I don’t like the Feynman speech – when I was in high school, I read the Feynman book, spent hours trying to find the mentioned rat study, realized it didn’t exist, and became skeptical of his claims more broadly. It has always bothered me that one of the canonical texts in scientific integrity seems to invent a study from nowhere.

    But, I mean, if we’re going to be talking about scientific integrity too, I think we can be better than Feynman. I don’t think this paper contains an honest portrayal of the beliefs in the original speech. At the very least, there is a much more charitable – and tbh plausible – explanation that gets totally discarded.

    • I read the blog post and the article the other day as well. I liked the blog post, because this now allows me to go on a meta tangent in my teaching that Feynman should have tried more “utter integrity” himself before repeating stuff he (likely) saw in Mondo Cane. But after reading more, the evidence is still pretty clear that cargo cults existed, even if the scenes he mentions are exaggerations.

      I didn’t find the Gelman & Higgs article convincing. The idea that you can go through the motions without making progress and need to be honest with yourself so you change course is a good one and cargo cults are a good metaphor for it. Metaphors are never perfect, but you don’t need to believe they could have built an atomic bomb/made the leap. But e.g. the John Frum movement in the link below could spend the time they spend on marching on economic activity that would allow them to buy an outboard motor themselves.

      Also this: “there is no reason to believe they were being dishonest”
      Chief Isaac: “John moves from America to Yasur and back, going down through the volcano and under the sea”
      https://www.smithsonianmag.com/history/in-john-they-trust-109294882/
      For me this a reason to believe, some people were dishonest, at least in Feynman’s sense of fooling themselves.

  5. If the following is useful please let me know. I started reviewing the paper a bit, and got to page 6 or so. I think some more general restructuring of the paper is likely possible, but I did not want to try and see if and how before sharing what follows to see if my comments make any sense and are appreciated. Here is an open review up to page 6 or so of “Interrogating the “cargo cult science” metaphor”.

    General remark “Cargo Cult” seems to me is sometimes spelled differently (cargo cult, cargo-cult,Cargo Cult) which I always try to avoid personally.

    p. 3 There seems to be an extra dot (.) and space between “(…) explain it. . In summary, the idea is to try (…)”

    p. 3 The section “Setting aside questions of historical accuracy, the (…) by building an atomic bomb.” seems very similar to the section “The members of the cargo cults may have lacked access to various (…) leap the technological gap.” on the same page. They also almost immediately follow each other. Perhaps one of these sections can be left out. I think the entire paragraph starting with “Feynman conveys something (…) atomic bomb” can perhaps be left out, or some sentences of that section transferred to the starting paragraph below the heading “Problems with the metaphor”

    p. 4 “We assume that the historical reasons for cargo cults are complicated (political, social, and cultural). Likewise, we assume reasons for researchers engaging in practices labeled as “cargo cult” are complex; “

    I wonder whether the “historical reasons for cargo cults are complicated” if “The analogy is to groups of people living in isolated Pacific islands after the end of the Second World War who apparently followed ritualistic practices in an unsuccessful attempt to replicate the cargo deliveries that had been delivered by the Allies when using these islands as airstrips during the war.”.

    What’s (politically? Socially? Culturally?) complicated about the attempt to replicate certain things to achieve a desired result. I wonder whether you might be attempting to make a (unnecessary?) appropriate connection to “complex reasons” for researchers engaging in practices labeled as cargo cult.

    p. 5 “This is inevitable and, to a large extent, desirable, in that it would be a great loss if serious research could only be conducted by teams with statistical expertise. It would be a mistake to propose a “science reform” that would require innovative statistical design, data collection, or analysis for each new study.”

    I wonder whether you could write this more optimally. To me, it sounds like you equate “statistical expertise” with “innovative statistical design, data collection, or analysis” (emphasis on the “innovative” part). Depending on what you would want to state, I would perhaps think about re-writing. Maybe something like:

    “This is inevitable and, to a large extent, desirable, in that it would be a great loss if serious research could only be conducted by teams with statistical expertise. It would be a mistake to propose a “science reform” that would require statistical experts to be involved, or that would require innovative statistical design, data collection, or analysis for each new study.

    p.5 The last two paragraphs could perhaps be left out, or sections moved over to the next heading “Moving from cargo-cult” to more specific statistical concerns”. I think the topic and flow is lost with these last two paragraphs as it stand now. They also seem to not add much when quickly reading.

    The final sentence of the paragraphs above these last (possibly unnecessary) paragraphs “Labeling the people or practices as “Cargo-Cult” is unlikely to be productive.” also forms a nice segue to the next heading in my view.

    p. 6 “Gigerenzer’s article is focused on the “null ritual” of “the magical 5% number” and “wishful thinking about …the p-value,” Perhaps include year (2004) behind “Gigerenzer’s”. He has written more articles of course, and I would be as clear as possible that you are referring to the 2004 article (I assume).

      • Ehm no it’s not, I am a real person. I just thought to provide some feedback, but realized later that the paper seems to already be published so several of my points are not really useful anymore. Curious to know why you thought it was AI or an experiment…Also curious to know why you think I should switch from SSRN to a blog platform like substack?

    • Wow, this is legit. When you pointed out the varying punctuation and capitalization of “cargo cult”, I thought maybe the linked paper was a manuscript, but it appears to be the final published version. This is especially glaring on the first page, where the title and the first sentence of the article use different capitalization. I guess the editors and proofreaders (if they had any) were asleep at the switch.

      • Thank you for this comment. It confirmed my retrospective analysis that almost immediately after I started to see some possible errors (such as the extra space and dot, and such as the possible almost similar sentences depicted closely together) I likely automatically thought it was a manuscript or pre-print. It also confirmed why I don’t take “official” publishing very seriously anymore, at least not as an automatic starting point in evaluation.

        The entire thing reminded my this morning of some idea I posted on this blog a few times now I guess, which is to have some sort of pre-print place where people can upload reviews that are also posted separately on this pre-print server. This in turn makes it possible for the authors of the reviewed paper, or the reviewer themselves, to cite the review for instance when revising the manuscript. And it allows for reviewers to list their reviews on their CV, and it allows for evaluation committees to inspect the reviews and to in turn use this information to evaluate a candidate.

        Take my review up to page 6 for example. The authors could maybe correct the extra space and dot thing, and correct the spelling of cult cargo to make it uniform across the manuscript. If that’s all they took from my review, they could perhaps thank, and link to, my review in a note that is sometimes used like “we would like to thank reviewer A. A. A (2025) for notes concerning spelling corrections” or something like that. I wouldn’t take up that much space in the reviewed manuscript.

        And if they agree with some more critical feedback (like comments about the appropriateness of certain comparisons or metaphors) they could re-write some sections and somehow refer to that review that pointed that out in the discussion section, or in a footnote, or something like that. In certain cases when the review is so useful and/or critical and/or extensive it may even be possible to include the reviewer as an author on the new version.

        Maybe that’s something Mr. Gelman, and/or others, can further think about. I think it also is in line with some previous posts on here that mentioned how some reviews are worthy of being published, and things like post-publication review, etc. If the paper discussed here would have been published at such a place, the published paper could now have been updated with little effort I would reason…

        • Perhaps Mr. Gelman and/or others can also think about how several (arguably) scientifically problematic processes and issues regarding “peer-review” seem less likely to occur using this format of publishing and reviewing papers.

          For instance, the decision to publish a paper is not heavily dependent on the reviewers’ judgement and decision as is commonly often the case at other journals. This, for instance, makes it less likely that a certain reviewer can “block” an author’s paper because it might be critical of the reviewer’s own work. Other forms of “peer-review” abuse such as a reviewer attempting to increase the citation count of their own work by recommending their own work as references to be added seem also less likely for the same reason.

          Just some additional things to possibly ponder for those interested in this whole publishing and reviewing thing in science and/or how to possibly improve problematic issues.

  6. “The implication seems to be that the members of the cult did not possess the type of integrity scientists are expected to operate under, they didn’t work hard enough to bend over backwards to show how they might be wrong, and perhaps they ignored information given to them about the shortcomings of their methods.”

    Andrew:

    Feynman implied no such thing about members of the cargo cult! His negative comments about lack of integrity and so forth were specifically and explicitly about “what is generally missing in Cargo Cult Science”— not the South Sea islander cargo cults of his metaphor.

    John

  7. Jcp, John:

    You might be right about Feynman’s intentions. Megan and I were concerned not so much about exactly what Feynman said in that speech but rather with how terms such as “cargo cult science” and “cargo cult statistics” are used today. As explained in our paper, we felt that the current use of this term has some problems which can be seen by looking at the analogy more carefully.

  8. Andrew:

    Understood. Misuses and quibbles over literal interpretations are the common fate of all good metaphors. Like forking paths, though, cargo cult was and remains a great metaphor! Just a metaphor. But a great one.

    Give Feynman a hug.

    John

  9. A better analogy is “bizarro science”. By flipping the “hypothesis to be nullified” from what is predicted by the theory to something expected to be false to begin with, all the incentives to do a good job become inverted. Ie, the less careful the study, the more likely you are to reject the default nil-null null hypothesis due to some confound.

    The problem (using Feynman’s definition) with cargo cult science is that it fails to produce the promised benefits, and of course the opportunity cost. Bizarro science is far worse because it produces actively misleading results.

    I doubt the pacific islanders in question would continue with such practices, it seems to require a large bureaucracy.

  10. Thanks Andrew! Probably should have mentioned that I did really like the second half of the paper. And I do think people – not Feynman, but other people in the years since – have certainly used this metaphor with the exact offensive implication John and I claim Feynman was innocent of, so I agree there are good reasons to discard it for something more appropriate.

    Aside: I think it’s a little funny how this exchange proves correct this other line of yours:

    “A focus on personal character can quickly lead to feelings of
    disrespect and defensiveness which may take focus away from understanding the
    underlying reasons for problematic practices.”

    Maybe it doesn’t exactly hold, because John and I wouldn’t have said anything if we thought the accusation were fair. And also Feynman is dead, so maybe we can load all our sins on him like this and beat him up like hes our sacrificial goat.

    But still, there’s some irony in how the response to the paper partially proves the paper itself correct – even mild personal criticism of (dead) third parties seems to really yank focus away from the main aims of a paper / criticism / whatever.

  11. You note the real gains will come from incentivising the good practices, improving design, measurement, collection etc. I might also add ‘choices of what to study’. Generally I think your point is very good/important and reminds me of how easy it can be to criticise, but producing really high quality or valuable research is of course very difficult.

    One thing I think is sometimes missing on this blog are exactly those examples of things you think are really valuable or good. Examples can be very helpful! Often the research on the blog is usually critiqued whereas only fiction gets the positive remarks! I’d love to see more things where you ‘critique’ a paper showing how it did various difficult things well or why you think it should be considered a success.

    • Jamie:

      Hey, pay me enough and I’ll blog on just about whatever you want! Seriously, though, I do blog about research that I like. Indeed, my blog post from yesterday, “Exposing omitted moderators: Explaining why effect sizes differ in the social sciences,” falls into that category!

      • Haha, well I HAVE paid for your books so that’s something :-D

        That’s true, though the critiques are often point by point about why something is so bad or messed up but we don’t often see why precisely something is good, more just that it seems interesting or something. Just an idea of what would be cool to see, I can’t back it up with the big bucks…

  12. I’m sorry, but I feel that this has gotten a little out of hand – ‘Judith Butler syndrome’ I call it. I had my first crush on a girl at the tender age of nine. Nowadays this wouldn’t do – rather than saying I’m a boy who likes girls I’d have to ‘identify’ as a “cis-gender heterosexual male”. This gets on my nerves. Should any Polynesians feel microagressed by the term Cargo-cult Science, more’s the pity. And please look up ‘niggardly’ in a dictionary (preferably the ‘American Heritage Dictionary’ – which, BTW, has nothing to do with the Heritage Foundation – thank your favourite god[s] for that). All of which makes me think of an old Herman (Jim Unger) cartoon: “I’m not Stone Age, I’m Bronze Age.”

    • Olaf:

      Is there something particular about our article that you disagree with, or did you just want to vent? If the latter, I recommend twitter and 4chan, where–unlike in our article–you might find something about the following topics mentioned in your comment but not in our article: Judith Butler, crushes on girls, the terms “identify,” “cis-gender,” “heterosexual,” “microaggressed,” and “niggardly.”

      I guess what I’m saying is, it seems that our article triggered you into unloading a lot of things that have nothing to do with what we were writing about.

      • I suppose it is well known that the term, “niggardly,” unfailingly gets the speaker/writer into trouble; however, here is some of the history.
        ———————————————————————————————————-
        https://en.wikipedia.org/wiki/Controversies_about_the_word_niggardly
        Niggardly (noun: niggard) is an adjective meaning ‘stingy’ or ‘miserly’. Niggard (14th C) is derived from the Middle English word meaning ‘stingy,’ nigon, which is probably derived from two other words also meaning ‘stingy,’ Old Norse hnǫggr and Old English hnēaw.[2] The word niggle, which in modern usage means to give excessive attention to minor details, probably shares an etymology with niggardly.
        ———————————————————————————————————
        “On January 15, 1999, David Howard, an aide to the mayor of Washington, D.C., Anthony A. Williams, used “niggardly” in reference to a budget.[8] This upset one of his black colleagues, who misinterpreted it as a racial slur and lodged a complaint. As a result, on January 25, Howard tendered his resignation, and Williams accepted it.”
        ——————————————————————————————————–
        From then on, every explanation regarding etomological distinctions made things worse. Although I have never seen this in my lifetime, I would not be surprised if someone takes offense at the phrase, “He chewed it down.”

      • You asked: “Is there something particular about our article that you disagree with”

        Erm, well, “fraught with historical and cultural baggage and, in our opinion, encourages crossing of a subtle line between criticizing discipline methodological norms and criticizing the individuals currently carrying out those norms as part of a complex and context dependent social process. We find that carefully interrogating the term itself holds some important lessons for improvement in the science reform movement.”

        My initial reflex was “now he’s going all Gadamer”.

        • Olaf:

          I have no idea who or what is Gadamer. I will say that I stand by that passage from our article. What we say there seems reasonable to me!

  13. I’ve used “cargo cult” as way of meaning “people who blindly follow the forms of a discipline, but do not understand the substance behind the forms and why they are meaningful for the discipline, mistaking thinking that the form is the substance”. I’ll add to the group saying that it’s a useful metaphor, even if its origins have some fiction and colonialism in them. Would you object to the use of “voodoo” to mean something like “manipulating a representation of a thing with the mistaken idea that somehow changes the thing itself”? (e.g. “voodoo economics”). The “voodoo doll” as we know it is similarly a product largely of popular culture and myth-making, and colonialism. But people know what it means as a metaphor.

  14. The article’s recommendations are very much what Feynman means by “utter scientific integrity”. So, the all the article does is take issue with the term “cargo cult science”.

  15. Andrew, in the article you write that the cargo cult metaphor ‘encourages crossing of a subtle line between criticizing discipline methodological norms and criticizing the individuals currently carrying out those norms as part of a complex and context dependent social process.’

    I think it’s fair to say that this basically means that [in your view] we shouldn’t blame any individual (it’s not a matter of, say, a lack of scientific integrity; it’s “the system”). Assuming this is a correct representation of your view (apologies if otherwise), I disagree for two reasons.

    First, if the focus is on “the system”, then the cargo cult metaphor seems very apt to me. After all, cargo cults are social movements; they are about group behavior, not about individuals. (I guess the word “integrity” is why you connect it to individuals, but I think this can just so easily be understood as the integrity of a scientific community.)

    Second, and more fundamentally so, I just don’t think that “the system” (e.g., incentives) frees us from all and any individual responsibility (and I think this is a pretty generally shared view). Within the same system, some people cheat, others do fantastic research. We are responsible for our actions. Furthermore, while I think it’s now broadly understood that science works great (generally speaking) because of how it works as an institution, not because scientists are in any way better than most, of even the average, person, a scientific community cannot function properly if it’s entirely made up of people who care first and foremost about their careers and not science (as more formally argued in this paper: https://scholar.google.com/citations?view_op=view_citation&hl=tr&user=bhs1owsAAAAJ&citation_for_view=bhs1owsAAAAJ:MXK_kJrjxJIC&trk=comments_comments-list_comment-text). So, while I would agree that we should not reduce this to a problem of a few rotten apples (“the system” does matter), we also should not absolve scientists entirely from their individual agency and responsibility.

    • Quote from above: “We are responsible for our actions.”

      I have wondered about tenure. Couldn’t that be seen as some sort of a “systemic” influence that should allow for academic freedom in service of the common good (or something like that) for tenured scientists? And if so, can anybody explain to me why and how “the incentives” or “the system” should be specifically be changed or aligned or whatever to improve matters so these tenured scientists don’t sub-optimally conduct and publish research?

      And what about the influence, responsibility, choices, etc. of tenured scientists in their roles as mentors, teachers, reviewers, editors, etc.? Shouldn’t their tenure allow for academic freedom in service of the common good so they can make sure things run as they should in science? What have all these tenured scientists been doing if things are so problematic? What even is (the point of) tenure?

  16. I read, I considered, l still used “carco cult science”. It’ll be around longer than any of us. Longer than the widespread memory of where it came from.

    • In his book about his fraud, psychological scientist Diederik Stapel wrote that “Every psychologist has a toolbox of statistical and methodological procedures for those days when the numbers don’t turn out quite right. Like everyone else, I was adept at using these tools to erase the less attractive findings and polish up mediocre results.” (p. 117). Cargo shorts are quite useful for keeping tools so Wikipedia says.

      Why not combine this all to propose “cargo shorts science” instead of “cargo cult science”! It has a “loose fit” which might be appropriate to describe poor statistical models, it has lots of pockets to keep and hide various “statistical tools” such as questionable research practices to use when needed, and overall it just comes “short” from a scientific perspective: “cargo shorts science”.

  17. Metaphors We Live By is more relevant than ever before. This is particularly apparent from some on this thread, lacking rhetorical awareness, who presume the objection is aimed at Feynman and whatever may have been his original intent in the paper.

Leave a Reply

Your email address will not be published. Required fields are marked *