Uri Simonsohn and I spoke here:

This workshop will bring together researchers, journal editors, publishers, funders, and scientific association leaders to identify practical, forward-looking strategies for strengthening data integrity and transparency in the social and behavioral sciences. Participants will explore innovative tools and frameworks to detect and prevent errors, promote accountability, and reinforce public trust in research. Discussions will also consider how journals, institutions, and professional societies can adopt fair, sustainable practices that support scientific rigor while ensuring accessibility for researchers across many contexts and settings.

I brought up some relevant points, including:

1. The science-reform movement as an awkward alliance between reformers (who anticipated that failed replications will cause people to move away from some bad published ideas) and status-quo people (who anticipated that successful replications would validate various now-controversial studies in the past).

2. It’s not always clear at first that a paper is bad, but then in retrospect its problems jump out at you. The analogy I gave is that Arthur Conan Doyle was fooled by photos of garden fairies that, years later, were obvious fakes. This is one rationale for post-publication review.

3. We should consider variation when hypothesizing effect sizes, and this connects to the point that researchers and the public should be more accepting of uncertainty. This is really the most important point. Later in the conference the health economist Jay Bhattacharya discussed the problem that people don’t know whether published research is true. It’s a good point, and it leads to the next step, to move beyond the expectation that research results should produce certainty. Even the cleanest and best study is only telling you about some set of people under some conditions at some particular time. Future effects on new people in new settings will differ.

4. The Bayesian cringe.

5. The role of the field of statistics in improving science (multilevel modeling!), and at times making science worse (null hypothesis significance testing!). As I said to Uri, I do think we’ve developed a better statistical understanding in the past fifteen years, and this has allowed us to understand and address replication concern in ways that were not done in previous decades.

But there were some other things I meant to talk about but I never got around to saying:

1. My frustration that many of the people who promote bad research don’t seem to even care about the work that they’re promoting. For example, consider the physicist who pushed the ridiculous claim that scientific citations are worth $100,000 each, or the biologist who pushed the ridiculous claim that chess players burn 6000 calories per day. If they really cared about these things, they could try to study them! For example trying to trace where this $100,000 is going to, or studying the variation in the value of paper. Or trying to understand physiologically where those 6000 calories were going. But nooooo . . . they just want these B.S. factoids. Or the people who studied ovulation and voting, but got the dates of ovulation wrong. So often it seems that the critics care more about the topic than the people who are out there pushing these claims.

2. The role of the National Academy of Sciences. The Academy is sponsoring this workshop, and the workshop has gone well, but I also wanted to point out that the National Academy of Sciences is part of the problem too! Their journal has published some notably bad articles (air rage, himmicanes, ages ending in 9, etc etc). I have no reason to believe that PNAS is worse than other journals, but it does get some attention.

3. The problem of junk science as a betrayal of trust. Later at the conference, the political scientist Skip Lupia made the point that research is expensive and it’s the responsibility of the academic community to justify this to the taxpayer, especially in a modern information-rich environment where many people might feel that they don’t need academic institutions at all because they can find everything online. And I agree with this. But, even beyond waste of resources, it seems to me that when credentialed scientists promote junk, this degrades the reputational coin of the realm. As it should be. Every time a researcher at Harvard or Stanford or the University of California or wherever is promoting ridiculous work, every time an academic podcaster plays the promotion game, etc., this does its part to discredit the scientific enterprise. And this makes me mad.

I guess I can see why that last point never came up in the discussion, because I expect that everyone in this meeting is, like me, incensed by that sort of scientific careerism. So I didn’t need to say that.

I do like the idea of replication being a norm.

For example, imagine a world in which, when this psychology professor tells his Stanford class that chess players burn 6000 calories per day, that some student would raise their hand and ask, “Where did that number come from?” And then, if the professor were to supply some reference or rationale, another student could ask, “Is there any outside confirmation about that claim?”

I don’t know how the professor would answer such a question. Maybe he’d give another reference, maybe he’d say he doesn’t know, maybe he’d just ignore the question and move on . . . there are many possible responses. The real point is to set up the expectation that there be a response. The goal is to move beyond the pattern of strong claims supported by vague references. When you look carefully, you’ll often find that the evidence claimed in support isn’t always there.

Beyond the direct value of the replications themselves, there are benefits from thinking about replication, in part because it moves you to think about evidence and to think about how the conditions of an experiment can vary. If, instead of thinking of that $100,000 per citation or those 6000 calories as cool numbers, you think seriously about their variation and you think seriously about replication, the claims themselves will crumble. And then, pushing it back one step, maybe you’d think twice about promoting those sorts of stupid claims in the first place.

So, I guess the thing I’d like to have added to the discussion is a clearer discussion of the links between the procedures of science and science reform (publication, replication, etc.) and the particular claims being made.