E. J. Wagenmakers writes:
Together with several colleagues we just launched an open-access journal, the Journal of Robustness Reports. The journal mission is summarized here:
The vast majority of empirical research articles report a single primary analysis outcome that is the result of a single analysis plan, executed by a single analysis team (usually the team that also designed the experiment and collected the data). However, recent many-analyst projects have demonstrated that different analysis teams generally adopt a unique approach and that there exists considerable variability in the associated conclusions. . . . In order to showcase the practical feasibility and epistemic benefits of this approach we have founded the Journal of Robustness Reports, which is dedicated to publishing short reanalyses of empirical findings. . . .
I hope you like the idea — it reminds me of this paper we did a while ago.
I hope many researchers will be interested in contributing, now that they know the journal exists.
Here’s some more detail on what they’re looking for:
Robustness Reports in the Journal of Robustness Reports consist of the following sections:
1. An Abstract section that does not exceed two lines in print. This is equivalent to 180 characters (including spaces).
2. A Goal section, which outlines the question that the reanalysis is trying to address.
3. A Methods section, which provides background information about the reanalysis.
4. A Results section, which describes the main outcomes of the reanalysis.
5. A Conclusion section, which compares the results from the reanalysis to those from the original analysis and assesses the degree to which the reanalysis corroborates or undercuts the conclusions from the original analysis.
6. An Acknowledgments and Disclosures section, which may consist of the following subsections:
• A Reproducibility subsection. The purpose of this subsection is to declare whether or not the reanalysis team was able to reproduce the original analysis, along with an explanation in those cases where this proved to be impossible.
• A Code and Data Availability subsection. At a minimum, this subsection must include a link to a public FAIR repository that contains the code and data used for the reanalysis. All reanalyses must be fully reproducible. The Robustness Report authors might choose to supply the original findings with additional data, which also need to be made publicly available in the same FAIR repository. The repository may also contain a more extensive version of the Robustness Report.
• An Author Contribution subsection. For multiple-author articles, this subsection may be used to document the contributorship of the different authors.
• A Funding subsection.
• A Conflict of Interest subsection.7. A References section.
JRR limits Robustness Reports to only 500 words (excluding the Acknowledgments and Disclosures section, the References section, figure/table captions, and the title page) and one display element (table/figure), with additional material presented as online supplements. A constructive response of the original authors and the editorial summary are also limited to 500 words and one display element, but need not feature the sections outlined above. The length restrictions are in place because one of the goals of JRR is to convince mainstream empirical journals to adopt the format and arrange their own robustness reports to accompany articles of substantial interest — an investment of 500 words is minimal and similar to that of a comment or a letter to the editor (which which many journals are already familiar).
500 words isn’t a lot! It’s shorter than many of our blog posts. For example, this post from a couple years ago, “I’m skeptical of that claim that “Cash Aid to Poor Mothers Increases Brain Activity in Babies,” seems like it would fit into E. J.’s new journal–but it’s a lot more than 500 words, and I’m not sure what I’d want to cut! I guess I could extract a 500-word summary from it and stick everything else in an appendix . . . would that work?
Hello Andrew. Interesting question at the very end! I think it could work as you suggest. If you feed the post to ChatGPT and ask it to give a 500-word summary, it may do a good job already :-) And yes, there is always the online supplements. As we suggest in the edItorial, exceptions to the rule are possible, but I’d like to see whether the 500 wrd limit is feasible. If it turns out it does not work well then we can relax the word count restriction.
E.J.:
I don’t need a chatbot to write a 500-word summary! It’s easy enough for me to directly extract 500 words from my post to summarize it. The difficulty is that the actual content of the replication cannot be described in 500 words and one figure. The content includes all the models I fit, along with why I decided to fit them. With a requirement that it be no more than 500 words and one figure, the paper will necessarily be incomplete or else it will be split in two parts (if I put everything else into the supplement). I can try it for that example and see how it goes.
I’m not saying that your 500-word limit is a bad idea; it will just restrict the replications you publish to be in three categories:
1. Clean and simple analyses that can be explained and presented in 500 words.
2. Incomplete analyses that only present a fraction of what was done, whatever can fit in that window.
3. Analyses that are split into the main article and the supplement.
Maybe that will work; I have no idea!
This idea is long overdue, but better late than never! May it take off quickly!
I’m genuinely baffled. Why 500 words? I always thought journals had these procrustean policies because of print and to some extent reviewing. I clicked through and found this:
Is the idea that you only ask the reviewers to review the 500 words? If not, where’s the saving by putting everything in an appendix?
Andrew—I’d guess one figure means you can pack in six microscopic subfigures, just like in bio journals that restrict number of figures. I’m writing a paper with biologists now targeting one of the Nature journals (not my call—I would never voluntarily review for or submit to a pay-to-publish journal left to my own devices). The process my colleagues follow for all of their papers is to first generate the max of 4 figures, each with about half a dozen subfigures. Tomorrow is like (threw that in for our grammar critics) our fifth meeting on the figures and not a word of text has been written, even the captions. Later, we’ll fill in the captions, title, abstract, references, and the few hundred words allowed around that. Then we’ll be super cagey about how much we reveal in the appendix, forcing the reviewers to ask for explanatory graphs that we already made but are afraid to share for fear of griping. If you can’t tell, I really dislike this furtive process as a way to communicate science.
P.S. The good news is that it’s a proper open access journal.
Hi Bob. Yes you found the reason why we want to have 500 words. This is uncharted territory to some extent, and we will see where we end up — we are willing to update our beliefs. We are also willing to make exceptions. But the main goal is to make the publication of reanalyses mainstream, and if we have 4000 word contributions then it will make it easy for journals to ignore the idea. Imagine you run PNAS or NEJM, and you have just accepted a shiny new paper that reports a really impactful result. Arranging two 500 word commentaries with reanalyses is simple, fast, and fun; try this with two 4000 word commentaries and it becomes complicated, slow, and less fun. At least that is what editors might think. But we will see how things unfold! Who knows, we might end up accepting “brief reports” (the 500 word format) and “regular reports” (no word limit). I think this is one of those cases where we just have to get started and see.
Two quotes: one from blog post and one from the above reply:
“The length restrictions are in place because one of the goals of JRR is to convince mainstream empirical journals to adopt the format and arrange their own robustness reports to accompany articles of substantial interest (…)”
“But the main goal is to make the publication of reanalyses mainstream, (…)”
I looked up some definitions of “scientific journal” and it usually goes something like “A scientific journal is a periodical publication that disseminates original research findings, reviews, and theoretical discussions in various fields of science. These journals serve as a platform for scientists and researchers to share their work with the global community, allowing for peer review, critique, and validation of results.”
I wonder how this definition relates to the two quotes. I also wonder whether the goals “to convince mainstream empirical journals to adopt the format” and “make the publication of reanalyses mainstream” are appropriate concerning, and are in line with, the general and most important goal and function of a scientific journal.
Regardless, should we in light of this new proposal and the two quotes add some things to the definition of scientific journal as options for people to be aware of? Apparently, if I understand things correctly here, it seems to me that some people might think it is appropriate that a journal can have certain goals that I thought were limited to the content and papers in that journal. But now I wonder whether certain people think a journal itself, and not merely the papers in the journal, can have the goal and function of promoting certain ideas and nudging other journals to adopt these ideas.
I think this might be yet another great addition to my possible new journal titled “The Journal of Critical Examination of Recent Proposals for Social Science Reform” alongside, for example, that redefine statistical significance paper from a few years ago that Mr. Wagenmakers and his colleagues might know about as well.
The goal of “The Journal of Critical Examination of Recent Proposals for Social Science Reform” is to highlight certain recent papers or proposals that aim (or claim to aim) to improve matters, but do so in a very critical manner that is often absent in these papers and proposals. The main goal of my possible new journal is to think two steps ahead (or something like that), and wonder how certain proposals might produce scientifically undesirable effects somewhere down the road.
Take this recent journal proposal by Wagenmakers and colleagues for instance. We recently had a discussion on here where this whole many-analyses thing was talked about as well (I want to link to it but that often results in my comment not getting posted). Now, perhaps these type of many-analysis studies are designed in such a way that they almost always show that many researchers choose different analyses (see the recent discussion on here). Or maybe these type of studies stop at showing that many researchers analyzing stuff leads to different analyses and/or conclusions, and not take it a step further by showing how some of these analyses are better than others, etc.
This kind of critical examination of (what I assume might be part of) the reasons for this publishing reanalyses journal might be part of a more extensive paper about this publishing reanalyses journal for my new possible journal. It could further talk about how these many-analyses papers, and now this journal proposal, might pave the way for building an image that only when many researchers analyze (and gather?) data, the analyses and “collaborative” conclusions can be trusted. Or something like that.
My new possible journal titled “The Journal of Critical Examination of Recent Proposals for Social Science Reform” is expected to be a welcome addition to all the new journals and initiatives that have popped up in recent years, most notably from certain “centers” and “societies” and from “open science” or “collaborative” and “transparent” scientists (who sometimes do not really do as they say). In fact it is reasoned that just keeping an eye on what this group of “collaborative” people propose might provide a steady stream of possibly useful papers and proposals for my new possible journal.
I am not sure if I will start this “The Journal of Critical Examination of Recent Proposals for Social Science Reform”, but in the mean time I am merely writing about this stuff on this blog in the hope it contributes something useful in some way, shape, or form…
I am also pondering whether I should propose a new part and/or new field of metascience titled “meta-metascience”. It might be about how metascience can have similar problematic issues as regular science concerning things like corruption, flashy studies, short-term thinking, manipulation, flawed design, flawed conclusions, etc.
Meta-metascience can, for instance, wonder whether the idea (or attempt or wish or whatever term is most appropriate) by Wagenmakers and colleagues “to convince mainstream empirical journals to adopt the format and arrange their own robustness reports to accompany articles of substantial interest (…)” might be based on too few and/or too flawed studies to even warrant such a proposal at a singular journal, let alone many journals. In that way meta-metascience might make clear that similar processes in metascience might be happening at this point in time regarding this proposal as has been criticized in the past regarding regular research.
Meta-metascience can also, for instance, mention the possibility that metascientific studies can be flawed, or can be influenced by corruption, or can be consciously or unconsciously designed to find certain things that might in turn be useful for certain things that might follow later, or can result in proposals that might not think well enough about possible scientifically undesirable effects.
For instance, meta-metascience can discuss how studies and papers about 1) replication prediction markets, 2) many-analyses, 3) crowdsourcing science, 4) redefining statistical significance, and 5) constraints on generality may all paint a certain picture and/or may have been designed to move in a certain direction and/or may all result in certain subsequent proposals that may be scientifically damaging in the long term. Maybe all these kinds of research, conclusions, and subsequent proposals might ultimately result in ever more bureaucracy, more centers popping up, and more control and direction. Who knows. The main function of meta-metascience might be to make things like this clear, and to point things like that out.
Alexander:
Yes, this has come up recently on the blog:
– https://statmodeling.stat.columbia.edu/2024/09/26/whats-the-story-behind-that-paper-by-the-center-for-open-science-team-that-just-got-retracted/
– https://statmodeling.stat.columbia.edu/2024/04/08/people-have-needed-rituals-to-turn-data-into-truth-for-many-years-why-would-we-be-surprised-if-many-people-now-need-procedural-reforms-to-work/
– https://statmodeling.stat.columbia.edu/2024/03/27/the-feel-good-open-science-story-versus-the-preregistration-who-do-you-think-wins/
– https://statmodeling.stat.columbia.edu/2023/06/22/here-are-some-ways-of-making-your-study-replicable-no-its-not-what-you-think/
Thank you for the reply and the links to previous (meta-metascientific?) topics and discussions on this blog.
A possibly additional relevant 5th link of a recent (meta-metascientific?) post and discussion about many analyses might also be appropriate here:
https://statmodeling.stat.columbia.edu/2025/03/25/lets-analyze-how-we-analyze/