John Protzko, Jon Krosnick, Leif Nelson, Brian Nosek, Jordan Axt, Matthew Berent, Nick Buttrick, Matthew DeBell, Charles Ebersole, Sebastian Lundmark, Bo MacInnis, Michael O’Donnell, Hannah Perfecto, James Pustejovsky, Scott Roeder, Jan Walleczek, and Jonathan Schooler write:
Failures to replicate evidence of new discoveries have forced scientists to ask whether this unreliability is due to suboptimal implementation of methods or whether presumptively optimal methods are not, in fact, optimal. This paper reports an investigation by four coordinated laboratories of the prospective replicability of 16 novel experimental findings using rigor-enhancing practices: confirmatory tests, large sample sizes, preregistration, and methodological transparency. . . . [italics added]
Don’t get me wrong, I’ve got no problem with large sample sizes, preregistration, and methodological transparency. And confirmatory tests can be fine too, as long as they’re not misinterpreted and not used for decision making.
My biggest concern with the italicized bit is that I’m kind of concerned that the authors or readers of this article will think that these are the best rigor-enhancing practices in science, or the first rigor-enhancing practices that researchers should reach for, or the most important rigor-enhancing practices, or anything like that.
Replicability is great, and for the present discussion (although not in general) I’m ok with roughly equating “rigor” with “replicability,” so that “rigor-enhancing practices” would be those that make a study more likely to be replicable.
What, then, are the first steps that I would recommend to make a study more likely to be replicable? Here’s my quick list, in approximately decreasing order of importance:
1. Make it clear what you’re actually doing. Describe manipulations, exposures, and measurements fully and clearly. This is related to the “methodological transparency” of Protzko et al., but here I’m not talking about statistical methods, I’m talking about scientific methods. What exactly did you do in the lab or the field, where did you get your participants, where and when did you work with them, etc.?
2. Increase your effect size, e.g., do a more effective treatment. This might sound kind of obvious, but consider all the studies where, from theoretical grounds, one would expect very small effects, for example studies of subliminal messages (or notoriously, here).
3. Focus your study on the people and scenarios where effects are likely to be largest. For example, in a education experiment, it could be that the best-prepared students don’t need the intervention, and the students who are the least well prepared ahead of time can’t get much benefit from it; thus, to maximize the average treatment effect in your study, you should perform it on students in the middle of the range. Similarly, try to set up the conditions of your experiment so that the treatment effect will be as large as possible.
4. Improve your outcome measurement: a more focused and less variable outcome measure should result in a lower standard error of estimation.
5. Improve pre-treatment measurements; adjusting for these in your analysis should reduce your uncertainty in estimating average effects. Often a good way to improve outcome measurements and pre-treatment measurements is to take more measurements on each person, either asking more questions at the time of the study or doing more followups.
6. The methods listed in the above-linked article: “confirmatory tests, large sample sizes, preregistration, and methodological transparency.” Large sample size is ok, but in general I think it makes sense to get more data and learn more from each participants, rather than just dragging more people into a bare-bones study. As for the others: Sure, making preregistered predictions is fine, why not, but more as a way of uncovering potential problems with your study than by directly making it better. I don’t mind the four steps listed in in the linked article, but I really don’t like these as the first four characteristics—indeed the only four characteristics—of rigor in the social and behavioral sciences.
In short, there are ways of increasing statistical power, other than brute-force increasing your sample size.
These are general questions that come up in many studies; see here for a quick overview.
P.S. I’m a big fan of Brian Nosek and the whole replication movement. I wonder whether one of the problems with the above-linked article is that it’s a short piece with a zillion authors. The result can be a diffusion of responsibility where nobody gets around to checking to see if everything makes complete sense.
P.P.S. See here for comments from Nosek and a further discussion.
Writing comment – I think you’re referring to the non-italicized portion of the quote. Italics were added but not to the part you’re emphasizing.
Content comment – Most of your suggestions strike me as ‘understand your topic better’ as opposed to concrete design suggestions. ‘Do a better treatment’ or ‘pick a better population’ is going to lead to case-by-case design changes that reasonable researchers might disagree about. What if a different education researcher thinks that the intervention should work even for students who aren’t prepared? Or what if that hypothesis is wrong? I don’t disagree that they’re good ideas but it feels harder to write about. On the other hand, I can always make a general point of increasing my sample size or running confirmatory tests.
Alex:
Sure, “understand your topic better” for sure. But lots of researchers don’t even seem to be trying to get big effect sizes. For example, that study of subliminal smiley faces and attitudes on immigration, or the elderly-words-and-walking experiment: Sending a message that is imperceivable or hidden is pretty much a way to ensure a small or undetectable effect and in a way with no clear theory.
I’m not saying that researchers should never study manipulations for which there is no clear theory or where effects are expected to be small. There’s room in science for shooting in the dark, and there’s room for studies of small effects. I’d expect that such studies are less likely to be replicable, but there’s more to science than replicability.
My point is that if researchers study manipulations for which there is no clear theory or where effects are expected to be small, they shouldn’t be surprised when findings fail to replicate—and it would be a mistake for them to think they could take such studies, add preregistration and large N, and think that now they’ll be replicable.
Preregistration and large N aren’t bad ideas; in this case they should make it more clear that bad ideas are pointless. That’s really useful, actually! We just shouldn’t be implying that this will be giving replicability.
Yeah, that makes sense to me. I guess you might argue that a large N and preregistration make it more likely that such studies would find a null effect, and the null would be more replicable!
“2. Increase your effect size, e.g., do a more effective treatment.” . This is strange advice. If one knew in advance what treatment would be more effective, why do the study?
I think the second part of what you wrote for this item makes more sense: getting some understanding from the literature of what sort of effect sizes are reasonable to expect. A lack of this is a problem in biology and other fields: rather than think about effect sizes at all, some people glean from papers whether treatment X “has an effect” or not, blinding themselves to the possibility that applying treatment X will have an immeasurably small effect on their system.
Maybe 2 should be rephrased as “Think about effect sizes, and predict them as best you can beforehand.”
Raghu,
See my above reply to Alex. And, yes, there are settings where researchers are already doing their best to maximize effect size, and for those cases, sure, I’d recommend starting with steps 1, 2, 3, and 5.
“Increase your effect size, e.g., do a more effective treatment.”
Sorry, but this seems to be written by a data analyst who is not actually doing empirical work.
Isn’t the effect size the main purpose of a scientific investigation. If I make it big and see a big effect, have I learned anything.
How would I increase the effect size of genes, personality traits, or divorce, etc. etc.?
Increasing the effect size will increase robustness, replicability because it produces larger test statistics and smaller p-values, Bayes-Factors, or whatever inferential method you prefer. But reducing sampling error has the same effect. So, why do you downplay increasing sample size to reduce sampling error and advocate something that is not practical?
“Isn’t the effect size the main purpose of a scientific investigation”
No!
The first purpose is to establish that there is an effect! Many of the studies discussed here are trying to establish the presence and size of an effect – without even any fundamental reason to believe there is an effect, much evidence that the effect that has already been shown to exist! Not surprisingly, the relatively few that are tested often fail to replicate.
If you make it big and see a big effect you have learned something – you’ve learned where to start to assess the scope and range of the effect. From there you test the effect at larger and smaller scales to ascertain its scope.
Theres always an “effect”, if you don’t see one its only because your sample size is too small for your threshold.
The fundamental reason is that everything influnces everything else (in its future lightcone). Thats why once you get to sample sizes of 10-100k everything becomes “significant” at alpha = 0.05 so the threshold gets lowered.
So looking for the mere presence of an effect doesn’t tell us anything but how much money was spent, which is a proxy for a wealth/power-weighted collective prior.
Are you aware that you can set-up your NHST analysis in such a way that you will end up retaining (instead of rejecting) the null when the sample size grows large? (see concluding comment #6 on p. 40 here: https://doi.org/10.1007/s00407-022-00298-3)
Anoneuoid, are you aware that you can set up your NHST analysis in such a way that you will end up retaining (instead of rejecting) the null as sample size grows large? (see concluding comment #5 here: https://doi.org/10.1007/s00407-022-00298-3)
Sorry for the multiplicity, didn’t see my comment so I tried to repost it. Well, at least I corrected the number of remark I was referring to, it was supposed to be number 5. :-D
JR, retaining the null is always an error. The null is never true (outside some very limited circumstances, like perhaps “the difference in the mass of the electron and positron is precisely zero”).
See Raghu’s blog post link below. Everything affects everything else by some amount. What is relevant is always “how much”?
“That’s why once you get to sample sizes of 10-100k everything becomes “significant” at alpha = 0.05.” Not quite. We may run a one-sided test and get the direction wrong. Also, dependence may play out in such a way that rejection becomes more difficult rather than easier (for example in case of negative correlation between subsequent events).
Then run the other one-sided test, and you will get “significance” with sufficient sample size. Reality does not depend on what test you run. Or more accurately the test you run affects the tests run and data collected in the future, but it will still not be exactly zero with sufficient sample size.
I will need to read the paper closer, but can you quickly explain why I would want this? I don’t see it from skimming the paper.
“Then run the other one-sided test, and you will get “significance” with sufficient sample size.” That would be incorrect according to significance test logic; you are not supposed to pick the test after having seen the data. (I know that people do this but anyway, Bayesian statistics should not be criticised either based on designing the prior after seeing the data so that you get your favourite outcome.)
While there is 100% chance of some effect, a priori there is 50% chance the effect is in either direction.
Imagine rather than doing a test, “god” (or whatever omniscient entity), tells me the effect is positive. Ie, there is 100% certainty, something no statistical test can ever achieve. What can I do with that info? That is barely any improvement over what I knew a priori.
Instead, tell me the effect is an increase of say 8-12%. Now there is maybe something that can start narrowing down the possible explanations. But even that is extremely vague. What I want to know is the dynamics of the relationship, and how it varies under different circumstances.
Eg, Wansink discovered people reported feeling less hungry on average after eating at the buffet. This yest provides no useful information whatsoever.
Instead they could have monitored hunger before, during, and after eating for another 48 hours. Perhaps check what the curve looks like for a high-fat vs high-carb meal. You could check blood sugar levels and see how that relates. Now we are starting to get enough info to guess at some quantitative theories about hunger.
Whether x increases/decreases y (only on average, not even reliably!), is just not fertile grounds for science.
Everything affects everything else.
I think Andrew referenced experiments. The examples you gave are non-experimental by nature. Then, you’re obviously right, points (2) to (5) do not apply.
Finally, I think there is no need to recommend larger sample sizes. Even my math-averse family knows that, researchers even more so. Mentioning it for the sake of completeness is fine, but front and centre, that’s surely not necessary.
Ulrich:
I do empirical work; see here for some examples.
Regarding your second paragraph: yes, it depends on context. Sometimes researchers are testing manipulations that can be changed to be more direct and focused (as in my suggestion 2 above); other times it will make sense to improve measurements (my suggestions 4 and 5 above) so that larger effects will be observed. I’ve seen many examples where outcome and pre-treatment measurements are sloppy, which leads to smaller effects and higher noise.
Finally, increasing sample size is fine, but not if researchers use this as a way of chasing statistical significance without understanding. Recall that the original topic here is replicability, so the goal is not just to get a statistically-significant p-value, it’s to find results that will appear in future experiments.
Maybe it is difference of experience, but I think the advice here is spot on: in many fields the key to designing good experiments is to work hard to optimize effect sizes (as well as to reduce noise).
You *can* optimize the *observed* effect different factors through clever experimentation. For example, the famous Millikan Oil Drop study used ingenious instrumentation to make the charge on a single electron directly observable. Millikan could not change the charge on a single electron, but he could setup a system in which other effects were cancelled out and the effect he was studying was magnified. Microscopes magnify effects; PCR turns minute quantities of DNA into massive quantities of DNA (and, for qPCR observable changes in florescence). Once I (finally) learned about effect sizes it changed entirely how I understood progress in science: most of the big advances turn directly on new techniques/instrumentation that transform previously miniscule effects into large effects.
In my lab, once I started thinking about this we started working hard to maximize effects. I study learning and memory. I realized that we could optimize the training protocol to ensure that every single animal has a very strong memory trace. Then, for measuring changes in gene expression we maximized effects by working out how to isloate just memory-related neurons (trying hard to exclude glia and the many neurons in the same region of CNS that were not encoding the memory). Note that’s not a reduction in sampling variation (though we work really hard on that as well)–when we compare gene expression levels in *just* memory-relevant neurons to controls we get a much bigger difference than a comparison to a mixture of memory-relevant and memory-non-relevant— isolation down to the system components most affected is a key strategy.
I feel like whatever the field of study, there are strategies for *experimental* effect maximization….. I’d be curious to hear specifically why/where this is not possible.
In my particular field (learning and memory), there is a bunch to do to maximize effects. In behavioral experiments we’ve worked hard to maximize our training protocol so that every animal strongly encodes the memory. When tracking changes in gene expression we specifically isolate neurons that help store the memory rather than just whole regions of CNS. So
I agree with these comments and would put them in the category of “know what the hell you are doing with your research” Too much of the research is being done quickly and not with much thought behind it. Doing clever studies instead of well thought out and designed studies seems to be the norm in many areas of psychology. I think Nosek et al are trying to come up with ways that those who do an apparently value that science actually show it is valid. I agree with you that the best thing is to just do thoughtful well constructed science that takes some thought to design well.
First off, note that word, replicable is not in the spellchecker of this blog. More importantly, it seems to me, that besides replicability [ note, it too is not in the spellchecker of this blog] the notion of generalization is often overlooked. Too often, a study is done using American undergraduate psychology students and the results are then extrapolated to the world at large. The other day, a study by Francesca Gino at Harvard was discussed in detail in this blog because of certain suspicious anomalies; however, even if that study were completely legitimate, is it warranted to assume it applies to the Ivy League, universities all over the land, left-handed shortstops?
Spell check is part of the browser software rather than the blog. Eg, chrome, firefox, etc.
There is probably a way to use a better dictionary than the default or at least add custom words.
Yes, too often people are unwilling to say that brains are brains and that sampling doesn’t matter as long as within the volunteers you randomize.
I worry that lots of people are already doing #3, but implicitly, and so we have entire bodies of research lit where the effects are presented as broadly generalizable but are actually conditioned on a narrow range of subject or stimuli space (so #2 and #3 seem related).
Maybe the advice should be to make identifying the participants and the stimuli that will lead to the biggest effect an explicit part of your planning process, and consequently your reporting process.
Yes, good point.
Thank you for posting this! I too worried that Andrew was basically promoting “study the regression to the mean”! In business books wizards will improve poor performing stores but really they’re just benefiting from natural variation over time (unless the stores are chronic underperformed). But I do see merit in the idea of “make the effect as big as possible” then explain the who or what that possible big effect applies to. At least you then have a likely ‘effect ceiling’.
On another blog (which Andrew reads), I and others had some depressing exchanges with a then-front-pager, a PhD candidate in psychology, about generalizability a few years ago. She kept waving away the “technical” statistical issues and insisting that the “insights” gleaned from narrow studies (also underpowered, of course) were nonetheless “fascinating.” More or less Upton Sinclair’s aphorism as applied to graduate students, I suppose.
I work in a domain where often we need to test then roll out a program more broadly. Some of the suggestions here will decrease “replicability” in the sense that the program won’t scale during the roll out (eg focusing the test on the people and scenarios where effects are likely to be largest). Perhaps this not a problem of replicability but rather external validity.
What the effect size is in the groups where the effect is smaller should be the next question to study. But I think that something to think about is doing studies on groups instead of slicing and dicing after the fact.
Also it’s ok to do inference on small samples without expecting or demanding “statistical significance” or the equivalent, just accepting the uncertainty that’s there, making decisions being aware of the uncertainty you have, and, as you say, designing the next experiments to reduce that uncertainty.
why are you so confident that your steps #1-5 are all better at increasing replicability than the qualities proposed by the authors? after all, they have empirical evidence showing that these qualities are effective in increasing replicability. is there any evidence that having researchers collect more observations per study participant or selecting a sub-group who have the highest likely effect sizes increase replicability? it seems like these might decrease replicability; unlike pre-registration which ties researcher’s hands, they give researchers more “degrees of freedom.” if you ran a similar experiment, and told 4 labs to replicate 16 different results and instructed each lab to pick a sub-group they think is most likely to have large effects, my guess is the results would be less replicable, not more, especially if labs are not required to preregister.
Sam:
You make a good point. Without preregistration or something like it, clueless or opportunistic researchers (take your pick!) can obliviously or intentionally use forking paths to find “statistically significant” comparisons amidst noise and then publish unreplicable results. Doing a higher-quality experiment won’t remove that possibility.
So here’s what I think: Preregistration, large sample sizes, etc., won’t make a bad study replicable; they’ll just make it easier to see that the results cannot reliably be replicated. To make a study replicable, I think you typically need some combination of steps 1 through 5 of the above posts. But I agree with you that only doing steps 1 through 5 will still make a study vulnerable to forking paths.
In my defense, I also include step 6 in my above list! So I guess what’s most possibly misleading is my title. To clarify, I will change “it’s not preregistration or increasing the sample size” to “the first steps are not preregistration or increasing the sample size.”
I ran into this today. Seems somewhat related
https://bpb-us-e1.wpmucdn.com/blogs.roosevelt.edu/dist/b/153/files/2012/02/William-S-Gosset-and-Experimental-Statistics-Ziliak-JWE-2011.pdf
Did not know that student had such close correspondence with and frequent friction against Fisher.
All this stuff was covered in the experimental design text books that I read at outset of my statistical career in 1973. Does nobody read these books any more???
My concern with the paragraph in italics is the emphasis on methodology. But I guess that’s what academics think about most.
Let’s stand back and think about this. We do an experiment several times using an identical protocol, including an identical design and identical methods, but get different results. So, what has changed? Has the protocol changed? No! It’s the data that is different.
If you want to understand why an experiment doesn’t replicate you need to forget about methods, put on an overall, or lab coat, and get involved in the data collection. Then everything will become clear.
Unfortunately, from an academics point of view, you can’t publish a paper that says our studies didn’t replicate because we f—–d up the data collection, but now we know what we did wrong and our future studies will be ok.
Did you mention randomisation? Not sure. It is very important, both when laying out the experiment/study but also when collecting the data.
I used to say to my (statistically illiterate) bosses that a study that is well designed and executed can often be interpreted without any analysis but that no amount of analysis can save a study that is badly designed and/or badly executed.
Peter:
Unfortunately, yes, I think that very few people read those old-school experimental design textbooks. Look at all the literature on A/B testing where researchers are reinventing the wheel.
Can you explain how 4-5 above in your list of recommendations would work for a typical educational study?
4. Better pre-tests.
5. Better post-tests.
Those will be difficult, because education research is one area where people have already worked very hard on measurement.
True, we have a community around educational measurement (though I had to teach the notions of reliability and validity to my M.A. thesis advisors who had a cog psych education!). There are a few practices educational researchers use but there’s probably a bit of room for growth. We typically rely on pretests and posttests for our smaller experiments built on intuitions and may not necessarily apply psychometrics principles in all cases.