Economics professor Abel Brodeur writes:
Replication is key to the credibility and confidence in research findings. Through their attempt to falsify past evidence, replication efforts contribute in essential ways to the production of scientific knowledge. They allow us to assess which findings are robust, making science a self-correcting system when they are not, with major downstream effects on policy-making. Despite these benefits, reproducibility and replicability rates are surprisingly low, and direct replications rarely published. Addressing these challenges requires innovative approaches in how we conduct, reward, and communicate the outcomes of reproductions and replications.
That is why we are excited to announce the official launch of the Institute for Replication (I4R), an institute working to improve the credibility of science by systematically reproducing and replicating research findings in leading academic journals. Our team of collaborators supports researchers and aims to improve the credibility of science by
• Reproducing, conducting sensitivity analysis and replicating results of studies published in leading journals.
• Establishing an open access website to serve as a central repository containing the replications, responses by the original authors and documentation.
• Developing and providing access to educational material on replication and open science.
• Preparing standardized file structure and code and documentation aimed at facilitating reproducibility and replicability by the broader community.
How I4R works
Our primary goal is to promote and generate replications. Replications may be achieved using the same or different data and procedures/codes, and a variety of definitions are being used.
While I4R is not a journal, we have an editorial board which actively recruit and select replicators. We currently have an editorial board for economics and finance, and political science. Our scope will be changing over time. For 2022, we are focusing on the following journals:American Economic Review (AER), AER: Insights, American Economic Journal: Applied Economics, Economic Policy and Macroeconomics, American Journal of Political Science, Canadian Journal of Economics, Economic Journal, Review of Economic Studies and State Politics & Policy Quarterly. We will also reproduce and replicate a limited number of studies in other leading outlets.
Once a set of studies has been selected by I4R, our team of collaborators will confirm that the codes and data provided by the selected studies are sufficient to reproduce their results. Once that has been established, a team of editors will recruit replicators to test the robustness of the main results of the selected studies. For their replication, replicators may use the Social Science Reproduction Platform or our template which is available here (link to be provided). They may also decide to remain anonymous. The decision to remain anonymous can be made at any point during the process; initially, once completed or once the original author(s) provided an answer. See Conflict of Interest page for more details.
Once the replication is completed, we will be sending a copy to the original author(s) who will have the opportunity to provide an answer. Both the replication and answer from the original author(s) will be simultaneously released on our website and working paper series.
We will provide assistance for helping replicators publish their work. Replicators will also be invited to co-author a large meta-analysis paper which will combine the work of all replicators and answer questions such as which type of studies replicate and what characterizes results that replicate. For more on publishing replications, keep reading!
We need your help
I4R is open to all researchers interested in advancing the reproducibility and replicability of research. We need your help reproducing and replicating as many studies as possible. Please contact us if you are interested in helping out. We are also actively looking for researchers with large networks to serve on the editorial board, especially in the field of macroeconomics for economics and international relations for political sciences.
Beyond helping out with replication efforts, you can help our community by bringing replication to your classroom. Our collaborators at the Berkeley Initiative for Transparency in the Social Sciences (BITSS) are excited to announce the launch of the Social Science Reproduction Platform (SSRP), a resource for systematically conducting and recording reproductions of published social science research. The SSRP can be easily incorporated as a module in applied social science courses at graduate and undergraduate levels. You can learn more about the SSRP in this blog post.Want to get involved? Have thoughts? Want to replicate a study, reach out to us! If you’re an instructor, use the SSRP in your class!
Where to publish replications
Incentives for replications are currently limited, with a small number of replications published in top journals. Moreover, reproducing or replicating others’ work can lead to disagreements with the original author(s) whose work is re-analyzed. One of I4R’s main objective is to address these challenges and help researchers conduct and disseminate reproductions and replications.
As a first step to better understand publication possibilities for replicators, our collaborators (Jörg Peters and Nathan Fiala) and the Chair, Abel Brodeur, have been contacting journal editors for top economic, finance and political journals asking them whether they are willing to publish comments for papers published in their journal and/or comments on studies published elsewhere. The answers are made publicly available on our website. We also highlight special issues/symposiums dedicated to replications and journals which strictly publish comments. Please contact us if you want to advertize other replication efforts or special issues related to open science and replications.
We will continue developing new and exciting features based on input from the community. Do not hesitate to reach out to us!
https://i4replication.org/contact.html
I’ll just add that, yes, replication is important; also, all the replication in the world won’t save you if your measurements are crappy of you’re not studying something real. Honesty and transparency are not enuf. Replication’s important in an indirect way, in that the prospect of replication should motivate more careful work and should weed out some of the worst stuff. But, as always, I wouldn’t want to leave people with the impression that, if they just add reproducibility and replication to business as usual, that all will be well. Taking this seriously will require abandoning or revamping some entire subfields of social and biological sciences.
”Through their attempt to falsify past evidence, replication efforts contribute in essential ways to the production of scientific knowledge.”
Where did the long quotation from Economics professor Abel Brodeur come from? The reason I ask is that the second sentence, repeated above, is an astonishing one. Is that what replication efforts are in Economics (i.e. “attempt(s) to falsify past evidence”)!? Perhaps that’s why the whole replication “Industry” is a bit of a mess.
Googling bits of the quotation in the top post simply leads back to the top post.
I found the following sentence on the Institute for Replication website :
”As falsification checks of past evidence, replication efforts contribute in essential ways to the production of scientific knowledge.”
https://i4replication.org/index.html
Which is a little different – the first “falsification” phrase is rather meaningless and so one glazes over, or assigns one’s own (reasonable) meaning, when reading the sentence.
I was taught that what separates a researcher from a true scholar is the latter’s concerted efforts to undermine their own evidence. A researcher whose paper merely reports results is a technician; a researcher who spins results is a salesman. A scholar is a researcher who welcomes all legitimate, civil criticism. A scholarly paper contains the tool chest for dismantling its result, if at all possible.
I do get that the word “falsify” comes off as aggressive. Until you realize that the word is part of a core tenet of science: a theory is only scientific if it is “falsifiable.” In that context, “attempt to falsify past evidence” is synonymous with subjecting research to scientific rigor. I was going to say that falsify is actually a neutral scientific term, but that’s not quite right. We use terms like “falsify” and “debate” and “skepticism” and “dispute” routinely, not because they become neutral in a scientific context, but because science is adversarial by design. Hell, when we repeat the cliché that “science is self-correcting,” we’re really saying “scientists correct each other.”
I agree with you Michael. The first step of falsification of an interpretation/hypothesis comes from the researcher(s) themselves before submitting the work. However that’s not done by “replication efforts” (the researchers should have done that to their satisfaction) but by considering alternative explanations for their interpretations and exploring these with new experiments/analyses.
I agree with you also that “falsification” is one of the particularly useful concepts in science. It was an excellent insight of Popper’s that interpretations/hypotheses should be falsifiable and most researchers incorporate that implicitly in their research.
I have two problems with the quotation in the top article. Firstly replications aren’t done in an attempt to falsify past evidence. If they’re done as part of the “Replication Industry” they’re done to assess whether a study is reproducible – perhaps to determine whether that line of research has a solid foundation in respect of that study – perhaps to make a more general point about the reproducibility of research in some area.
In the normal course of science, reproducibility is done in a more passive (“by the way”) manner. If someone determines a crystal structure of the Covid protease, this study is likely to be replicated because someone else wants to see whether they can determine the structure of the enzyme with an inhibitor (aka a potential drug) bound into the enzyme. In doing so they first of all assume that the first study is correct and then they will, for example, repeat the crystallization of the enzyme in the presence of the drug to determine the structure of the complex. In doing so they have effectively reproduced the first study in the context of their own aims.
Most researchers attempt reproductions in this way, mostly with a positive intent of incorporating it into their own work – in general we don’t set out to falsify others research at all (unless perhaps if we happen to hate those researchers!).
Secondly, falsification a la Popper doesn’t occur at the level of evidence (as in “attempt to falsify past evidence”. It occurs at the level of interpretations and hypotheses.
“we don’t set out to falsify others research at all”
In other fields (physical and life sciences) it’s not necessary to specifically attack other research because in other fields the failure rate is so low.
In social science, psychology, behavioral economics and such, when studies use the NHST paradigm incorporating the myriad problems that commonly come with it – all of which come *after* highly questionable assumptions about what’s going on in other people’s heads – the failure rate is incredibly high. Furthermore, since a substantial number of researchers simply don’t want to be bothered to recognize the problems with the NHST paradigm, the only way to reject research using it is to repeat the experiment and show that it fails replication.
Hell, when we repeat the cliché that “science is self-correcting,” we’re really saying “scientists correct each other.”
Yes – again I think it’s a little more passive tho. Science is “self-correcting” in the sense that useful, important, reliable results are cited and built on and rubbish, unimportant stuff is not. So the cream effectively becomes apparent and rubbish is ignored. Incorrect interpretations/hypotheses may ultimately be shown to be incorrect (aka “falsified”) but in many cases if incorrect interpretations/hypotheses are in unimportant areas then they’ll simply be ignored.
It’s quite like the difference between a market economy and a planned economy. Science works by letting things proceed according (in the past) to the personal interests of scientists aligned with what they can get funded, with lately more direction from above in the form of consideration of societal/commercial aims (Man on the Moon, March of Dimes, War on Cancer, Human Genome Sequencing etc.). The inherent interests/values and aims of the participants drive it forwards and it works because there is an external reality that research aims to uncover (this is tough in Economics since the external reality is an ever-changing one, requiring consideration of the historical dimension). Because of its market nature and an external reality, science can accommodate some level of chicanery and general Fayerabendian practices :)
A planned economy approach is horribly inefficient – in science this might be the philosophy of “reproduction”; e.g. we make sure everything published is reproducible before proceeding. It doesn’t work.
To pursue the analogy with modern market economies – these are susceptible to gaming (monopolies; destruction of worker communalism; excessive patent protection; price fixing/cartels and so on) and so some degree of regulation is required. This seems to be the case in modern science which is susceptible to personal and corporate gaming (more so than in the past IMO – e.g. self-citation; citation circles; image-manipulation; honorary authorships; plagiarism; publication monopolies; agenda-led misinformation) and so a broad watch over scientific practices is useful (Pub-Peer; blogs like this perhaps).
I think I know what you mean by replication not helping when “measurements are crappy or you’re not studying something real,” but I’m not sure. I can think of three general scenarios where honest research results in bad work that undermines the empirical basis for a particular field or subfield:
1) Researchers design studies without a clear and static hypothesis. They measure many potential outcomes, and then get excited over the one that appears meaningful, even though something would’ve looked good by chance regardless.
2) Researchers, out of ignorance, conduct studies with significant flaws, such as a standard of evidence too low given the implausibility of the hypothesis, noisy measures, inappropriate analytic methods, non-representative populations, poor analogs to real situations, overreaching conclusions, etc.
3) Good research with legitimately unsupportive results, or small p-values, or effects too small to be deemed “meaningful,” or with hypotheses or conclusions that fail to strike journal editors as “exciting” or “important” or “novel” enough, etc., is never reported due to publication bias or the fear thereof, making the relatively small number of chance results a disproportionate part of the literature.
The bad papers in scenarios 1 and 3 are likely to be shot down by replication attempts. But a replication won’t necessarily weed out bad papers in 2, if the replicators credulously accept and repeat the original bad premise, ungeneralizable methods, unreliable measures, ridiculous interpretations, etc. Is this what you mean? That replication will help when the only problem is selection bias (either among a single study’s outcomes or across all studies in a field) but not when the study itself, or its interpretation, is flawed?
Also, which “entire subfields of social and biological sciences” are problematic in your view? As the kids say, spill the tea!
> Replication’s important in an indirect way, in that the prospect of replication should motivate more careful work and should weed out some of the worst stuff.
That might wishful thinking like it was here 35 years ago – https://statmodeling.stat.columbia.edu/2012/02/12/meta-analysis-game-theory-and-incentives-to-do-replicable-research/#comment-73427
Depending on the topic, studies can be similar enough to be considered replications (why I often wrote assess replication first and then if adequate consider combining).
Ironically, the Director of the Replication Institute is best known for work that is replicable but based on wrong assumptions,
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3848786
and the director of the Berkeley Institute for Transparency in the Social Sciences who together with another member of his institute is in the Replication Institute’s Board of Directors is known for reacting to a replication of an article of his that contained flaws that were never corrected in the journal with the infamous Worm Wars: https://www.columbia.edu/~mh2245/w/worms.html