Suppose you and a friend witness a car crash. After a while, the police arrive. One officer turns to you and asks, “How fast were the cars going when they smashed into each other?” Another officer speaks to your friend and asks, “How fast were the cars going when they hit each other?”
Could the phrasing of those questions influence the speed estimates you and your friend provide?
A classic psychology study suggests it can. . . . Loftus and Palmer (1974). Participants watched short videos of car crashes and were later asked to estimate the vehicles’ speed. Those who were asked how fast the cars were going when they smashed into each other gave higher speed estimates than those asked how fast they were going when they merely hit. A week later, participants in the “smashed” condition were also more likely to falsely recall having seen broken glass at the crash site. . . .
This study helped launch an entire research tradition on the malleability of memory . . .
Now comes the plan:
Despite its iconic status (and considerable citation count), we still don’t know how large or reliable the effect is by today’s standards of research transparency, statistical power, and replication.
That’s a gap worth addressing–not to question the legacy of the original study, but to better understand its robustness.
Where I disagree
I kinda see what Zwaan’s saying, but I have two concerns with how he frames the problem:
1. He writes, “we still don’t know how large or reliable the effect is by today’s standards of research transparency, statistical power, and replication.” I think he could say, “we still don’t know how large or reliable the effect is,” full stop. When it comes to publication practices, standards of research transparency, statistical power, and replication have changed a lot in the past fifty years. But an effect is an effect, and it’s a legitimate question: How large or reliable is the effect?
Also, I’d ask, “How large or variable is the effect?” because I find it more useful to think in terms of variability (which is defined in terms of the underlying process) rather than reliable (which is more of a statistical concepts), but that’s a minor point.
2. He writes that this is worth studying “not to question the legacy of the original study, but to better understand its robustness.” I don’t get this.
First, if it turns out that, under a variety of conditions, the result does not replicate, then, yeah, it should call into question the legacy of the original study! Just as, if the results do replicate, this project would support that legacy.
Second, if you say that the goal is “to better understand its robustness,” you’re already implicitly accepting that the original published result was correct, that it uncovered a real effect. And maybe it did! But, if it didn’t, then it’s not an issue of “robustness”; it’s a problem of analysis and interpretation of the original data.
Again, I’m not saying that I think the Loftus and Palmer (1974) study was no good, or that I don’t believe its results, or that I don’t think it will replicate, or whatever. I have no idea. My only point here is that, if you’re going to think about a replication, either a hypothetical replication or, in this case, a series of real replication studies, you should be open to the possibility that the original study was flawed and the original study was mistaken. I’m not saying you should assume this, just that you should allow it as a possibility. And I don’t think Zwaan is allowing that in his framing above. I think he’s being too deferential.
What’s happening next?
Zwaan and his colleagues are organizing a large-scale replication study, and the author of the original 1974 paper is still around and, according to Zwaan, “has expressed enthusiastic support . . . and is willing to help us reconstruct the stimuli as faithfully as possible.” That sounds great.
P.S. Lots of people seem to be annoyed at this play to replicate Loftus and Palmer (1974). I don’t get the annoyance. There are lots of psychology research projects going on; why not replicate a very influential study from fifty years ago? This sort of replication seems a lot more valuable than one more study of paranormal abilities or power pose or whatever. The point is not to claim that this replication study would rock our world; it’s just a potentially valuable scientific study. I feel like this replication study is being held to a higher standard than run-of-the-mill psychology studies. There’s lots of research going on, and I think that replicating influential experiments of the past is a good idea.
I wonder if Zwaan was only hedging against the potential fallout from questioning some core research — by ‘core’, I mean that much other research depends on this paper. I would make the same call, to try and avoid offense before the replication is done.
I also agree with the premise that the research would benefit from a replication study, based on what I see in the paper. ScienceDirect gives the number of citations as 1454, so replicating this paper sounds like a good investment. From the text, I glean n = 45. This simply does not cut it in a multi-arm trial (or in any trial, really). This is not meant as a criticism of the original authors — we all gotta work with the resources we have. But, as it is their privilege to publish an n = 45 study, it is my privilege to wonder if the results hold up in real life.
Quote from above: “ScienceDirect gives the number of citations as 1454, so replicating this paper sounds like a good investment.”
What would it say about the state of (parts of?) psychological science when this large-scale replication project would find that the results of a paper that has been cited 1454 times could not be replicated (whatever that exactly means or however that exactly is determined)?
I wondered about this sort of thing back in 2017 on this blog. I am linking to it here because it also contains a more detailed description of the idea I posted back in 2013 linked to elsewhere in the comments here about performing replications and working together in a different manner. One that makes a lot more sense to me, also with regards to replicating a study that has been cited 1454 times decades later:
https://statmodeling.stat.columbia.edu/2017/12/17/stranger-than-fiction/#comment-628652
Maybe it could help when folks with some “status” or “seniority” like Zwaan, or Nosek, or Lakens could talk about this idea some more. That is if they like it. If I remember correctly, I think I e-mailed at least most of them about it at one point in time over the years…
Words, and the way things are phrased, could play a role in many things. I wonder whether the scientists can investigate this even more, for instance by incorporating the following different ways to ask for collaborating scientists regarding this possible project. Here are five options that phrase certain things in a certain way, that may influence the decision of the scientists that are being asked or their collaboration:
1. Do you want to collaborate with a large-scale replication of Loftus & Palmer (1974)?
2. Do you want to collaborate with a large-scale replication of Loftus & Palmer (1974) where we will kind of just decide on some random amount of participating labs, or see how many will join, and not worry too much about costs and benefits of this all?
3. We think Loftus & Palmer (1974) should be replicated in a large-scale replication, and we have some sort of “status” or “seniority”, and we think it’s perfectly fine to propose this and ask for attention for this. Do you want to collaborate with us?
4. Do you want to collaborate with a large-scale replication of Loftus & Palmer (1974) where the results might be mentioned in newspapers or on TV, which may involve reporters asking us questions, which may then provide us with the opportunity to mention certain things or talk about certain other things (like our new website we just started to facilitate our project, or our new journals that is going to publish this all). Do you want to collaborate with us?
5. No matter the results of a large-scale replication study, it could spur on numerous new studies and enthusiastic follow-up studies concerning the specific topic or specific design we have picked out to be replicated in a large-scale replication study and are asking others to join. Do you want to collaborate with us?
For those that like to work and replicate and decide on things in a different manner, here’s an idea from 2013. Also check out the later comment from 2017 where a blogpost by Zwaan from 2017 is mentioned about what is there termed concurrent-replication:
https://groups.google.com/g/openscienceframework/c/2nhHMdGGhrw
For this:
> That’s a gap worth addressing–not to question the legacy of the original study, but to better understand its robustness.
see this:
https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
The whole document is really well done. There’s a section called “Overuse of em-dashes” that says,
It also mentions that OpenAI recently managed to contain the em-dashes recently and it’s mostly a Claude problem these days. I don’t know when the quote was written.
There’s another section, “negative parallelisms,” which says,
It contains a whole subsection titled “Not just X, but also Y.”
The word “contain” was used as a synonym for “limit” [the em-dashes] and then in the last sentence, it is a synonym for “consists of” or “includes.” Chat GPT assured me that
“So, if you encounter the word contain in a particular sentence, the surrounding words usually make clear which meaning is intended.”
“That sounds great.”
https://theindependentcritic.com/memento
Just like in the movie “Memento” this let’s replicate stuff from decades ago is doing things backwards in my view.
If you’re not careful you may not only be replicating a “classic” study, but you facilitate replicating the “classic” accompanying process of countless bad follow-up studies, poor reasoning, and lack of proper theory testing and formulating.
The idea linked to elsewhere in two places here in the comment section is aware of this possibility and tries to prevent this by explicitly connecting follow-up research to an initial replication of some “classic” finding (should one even want to replicate such a “classic” finding).
There have been at least two published replications of Loftus & Palmer (1974).
https://pdf.erytis.com/ssh/SSH.9104.pdf
https://www.researchgate.net/publication/308325273_Appraising_Loftus_and_Palmer_1974_Post-Event_Information_versus_Concurrent_Commentary_in_the_Context_of_Sport
One of the replications found a significant effect, the other did not.
I checked out the Zhou (2026) paper which has actual links to the videos used. I noticed the car seems to have side doors and window removed prior to hitting the other car, and then flips in an incredibly short 5 second clip. If that is all correct, it seems possible that certain characteristics of this particular video might facilitate the effect, one being the absence of windows which can facilitate thinking they must have been broken due to the impact and another thing being the actual word “smashed” in relation to “glass” as these words might be readily associated in one’s mind.
Next to this, it seems possible that technically there could be broken glass present in the short video. It’s pretty bad quality and incredibly short but I pauzed the frame of the video of the smash condition playing at 0.25 speed and at around 0.17 seconds I can see some stuff flying around which might be parts of the car lights which may contain glass (?) and at 0.17 or 0.18 seconds I can see something in the side window which might be a hole or crack in the side window.
Those are only a couple of several replications. The papers themselves cite others.
I will note that the non-replication (Goldshmied) is an extension and generalization at best. There is no effort to do an exact replication. And it’s not a great extension at that. They have a good idea to extend the finding to sports (hockey hit) but then take specific technical details from L&P, like the word “smash”, which in the context of hockey just isn’t really a thing. So, not surprised it doesn’t replicate. While “bump” is lighter in hockey “hit” is typically for harder checks. “Smash” has other connotations in human collisions that don’t exist for cars. As I read further it appears that “smash” did have an effect but it was to reduce the rembered impact. There are several other changes that mitigate the effect but point to non-generality, which goes to Andrew’s point.
BTW, the Goldshmied study has a horribly reported method. The variables are introduced prior For example, there is no explicit explanation of what the questions were to assess things. When studying something that’s related to leading questioning you need to show the questions. Also, if you want to see a horrific data analysis and interpretation or need one for a class on outliers and what they really mean please make sure you download it.
I like to think LLMs use em-dashes because I use them. Maybe yhe LLMs recognize quality when they see it, and have decided to emulate my style.
Seriously, I kinda resent that I will have to change my writing style in order to not be mistaken for an LLM.
Andrew may have a similar complaint, since he very frequently uses a “not x but y” construction, saying things like “I’m pointing out these errors not to criticize the authors but to point out that we can all learn from our mistakes”.
There is nothing wrong with em-dashes when used correctly, and nothing wrong with “not x but y”.
As for the content of the writing; I don’t think the effect being tested could plausibly be exactly zero, so the issue id the sign and magnitude of the effect, as well as its persistence (it couldn’t stay exactly the same with time, maybe the effect is bigger or smaller than it used to be). It will be at least somewhat interesting to see the result.
Right, what would it even mean to be exactly zero? Somehow, over time, if there is some imbalance in responses, the universe will compensate to bring the imbalance closer to zero, such that over an enormous number of replications the responses will average first 0.0x and then after a while 0.00x and then 0.000x and then 0.0000x and eventually 0.000000000000000000000000000000000000000000000000000000x and soforth?
Does anyone really believe that ANYthing has zero effect to even 1000 decimal digits? 10,000? 100,000,000?
Is there any experiment we could conceivably do which could even establish 100 digits of accuracy much less 1000 digits of accuracy in any measurement?
The electron dipole moment (the non-roundness of the charge distribution) is zero to around 30 decimal places. That seems to be the limit of precision right now, though the physicists are working on making the bound even tighter. The theory predicts exactly zero. They are trying to break the theory, but it is stubborn.
https://www.quantamagazine.org/the-electron-is-so-round-that-its-ruling-out-new-particles-20230410/
Wikipedia says otherwise:
https://en.wikipedia.org/wiki/CP_violating_moments
Sorry, I read my sources wrong. The current bound is consistent with zero, but the precision isn’t high enough to rule out the standard model. Other theories have been ruled out, but we need to go 8 orders of magnitude further to get to the level of precision required to rule out the SM. Every few years they improve the bound by an order of magnitude, so give it a couple decades.
Yeah, when it comes specifically to particle physics I could imagine a few kinds of exact zeros. It’s the exception that sort of proves the rule. Can you imagine the wording of a question having zero effect to 30 decimal places? Note there are only ~8B people on the planet, so an exact measurement of every single person would have at most 9 ish decimal places. Can you even imagine asking 8 B people a question, with 4B getting one wording and 4B another, and then the number who respond in a certain way is *exactly the same number down to the individual person* across those two arms?
How about there’s some kind of repeated experiment so everyone gets 1000 questions over a 10 year period and the answers are *exactly equal* to 12 decimal places?
no
As a fellow user of em dashes I cling to a hope that the conversation will move from “was this written with AI?” (impossible to determine with certainty) to “is this written well?” (subjective but ultimately more useful).
I read somewhere that the latest LLM version now avoid dashes (and some of the more obvious LLM-isms). So we can go back to using dashes as needed, but need to note that the LLM writing detectors need to be updated.
This seems like it’s part of a trend that fundamentally narrows the massive amount of replication work that has long been part of scientific research. This is considered an important study because there are 5 decades of work on false memories that were inspired by it. “Inspired” does not mean taking the results on blind faith. It means researchers designed other studies that measured false memory effects. Those are replications. As a goal of this current replication “robustness” is not shown by whether or not the initial paradigm identically replications. Robustness requires quantifying false memory effects across a range of experimental paradigms, NOT merely replicating the initial paradigm.
This reminds me of “The most replicated finding in drug abuse science”
https://drugmonkey.scientopia.org/2014/07/08/the-most-replicated-finding-in-drug-abuse-science/
If false memory effects differ in magnitude across experimental paradigms (they do), then a replication-focused study should look at the experimental variants that are most likely to explain those variations and run them as part of a single study. Some of that does seem like it’s part of their study plans, but then why call this a replication of an old paper rather than at attempt to harmonize factors that led to variations across decades of research?
Quote from above: “(…) , but then why call this a replication of an old paper rather than at attempt to harmonize factors that led to variations across decades of research?”
Here are a few options:
1. Replications are (or should we say were) hip and gather attention
2. Replications have been advertised and promoted to be good and useful, so collaborating scientists might be more willing to join in (we’re so helpful and all about replications!)
3. Replications are (or should we say were) hip and gather attention so it might be easier for people to focus on that term to get money (perhaps some of that money will cover the organizer’s particular university’s electricity bill via overhead costs or something like that), attention in the media, etc.
4. Replications are (or should we say were) hip and gather attention so that might provide the orginazing party to promote some other things, or mention some other things, or direct certain things in a certain direction whilst focusing on “a replication of a classic study”
5. The scientists enthusiastically organizing this all and asking for attention and participation in this all didn’t really think things through regarding replicating research that has been published decades ago and has been cited lots of times. Perhaps they have never really thought about the kinds of things you wrote in your first paragraph
I couldn’t help but notice that this “most-replicated-study” blog post by drugmonkey is the quintessential argument in favor of piranhas.
The drugmonkey first states that the findings have been repeatedly replicated, then lists 24 different minutiae that have to be addressed for the experiment to work as intended, including “Dirty/clean labcoat (I kid you not).” [parenthetical in the original]
Commenter Storno spots the problem, “What is the point of replicating a study that only works in an extremely narrow range of circumstances?” Which of course just pisses off the drugmonkey.
I’m curious which papers from 2026 leveraging “today’s standards of research transparency, statistical power, and replication” Zwaan and colleagues would expect to replicate in 2076? Would they set aside money to fund a 50 year anniversary replication of LLM persuasion published this year? If not, why do we publish, read and debate these effects?
Is the goal of social science to identify the narrow range of effects that are culturally and temporally invariant or is it something else?
Quote from above: “P.S. Lots of people seem to be annoyed at this play to replicate Loftus and Palmer (1974). I don’t get the annoyance. There are lots of psychology research projects going on; why not replicate a very influential study from fifty years ago?”
Lots of things to be annoyed by, one thing being that we are all providing feedback here in the comment section for a project which might be used by the organizers and scientists thinking about and performing the study (if it actually is going to be done still). This may end up improving crucial things of the design, but they will only get the rewards from that. Not any of the commenters here. That’s also one of the most annoying, and in my view un-scientific, things I dislike about peer-review in general.
And this process seems to also be present in a certain way with more recent things like talking about a project on social media or the posting of a pre-print. The work of people with “status”, or “seniority”, or those that somehow have many followers, or those that say the “right” things on social media, receive lots of attention, feedback, tips, and help. Then they design, write, and publish and will get most, if not all, of the benefits from it. In an extreme form it’s perhaps like collaborating with a bunch of people and then listing all co-authors alphabetically, except the first “I shout the loudest on social media” author.
Perhaps there are many scientists out there who could, and would, do much more useful and better things with the resources that these kinds of large-scale replications by certain scientists of decades old studies use. But they may never even have the opportunity, or even when they would, they will never receive the attention they, and their work, deserves. That’s one of the things that annoys me as well.
Anyway, I’m pretty tired of it all. For some more general reasons for this, which may help with understanding certain things, check out some stuff I wrote like “Psychological Science Replicates Just Fine, Thanks”, or “The Natural Selection Of Bad Psychological Scientists”, or “Things I Have Wondered (So Far)”, or “Why Psychopathy Might Be Present And Prosperous In Present-Day Psychological Science”, or “Pre-registration, Grocery Lists, And Particular Pre-registration Issues” which can all be found on SSRN.
Aaa:
Why do you declare so confidently that the organizers and scientists thinking about and performing the study will be the only people who will get the rewards from it? The purpose of the study is to add to scientific understanding, and the benefits should to future researchers, as well as, ultimately, the general community. I don’t know that the people doing the study will get any rewards from it at all! If they’re tenured professors, they’re already going to get paid, and their main reward would be the satisfaction of a job well done and the ability to contribute to the common good. I guess some of them will get some financial resources such as summer salary and research support, but the main value of this is to enable that the research is done.
Sentences like “the purpose of doing a study is to add to scientific understanding” and “the benefits go to future researchers and even society as a whole” are too vague and hard to prove or disprove in my view and reasoning. The also seem to leave out the issue of limited jobs and money for limited scientists, and it seems to leave out other possible options to then compare this sitation to. And, even if these types of projects might be beneficial for the general community, they could still benefit certain individual scientists more than other scientists from this project. That’s all part of what is annoying to me. It’s also why I like my idea linked to elsewhere more, as I reason that with small groups of researchers each replicating each other’s work, things are possibly, or perhaps even likely, more balanced when it comes to things like receiving credit or money.
I reason there are likely more immediate and more clear possible benefits of these types of projects that might be disproportionately, and even unfairly, distributed amongst the collaborating scientists. For instance, an established professor might be the one being interviewed after such a project might be mentioned in a newspaper, and this might provide them with the opportunity to directly or indirectly promote their website or own platform. As a side note and example, it might be amusing to note that I am technically the first author of that 2015 Open Science Collaboration Psychology Replication project because of my name and the alphabetical order of the authors. But I wasn’t contacted to speak about the project. Not that I should have been, I only coded and verified things if I remember correctly, but it’s perhaps a nice example to use here to point to some things. It seems to me that certain other people, perhaps most importantly Mr. Nosek who were asked to talk a lot about the collaborative project. That in turn, may have provided him with a bunch of opportunities for promotion and who know what of his OSF platform, or it gave him attention and recognition that may have been beneficial for many things (e.g. becoming a professor, gathering more grant money, being asked all the time to talk about certain topics like replication). Now one could argue that that has benefitted the community of scientists, or perhaps even society, but I don’t necessarily agree (see for instance my manuscript titled “Psychological Science Replicates Just Fine, Thanks” or “Pre-registration, Grocery Lists, And Particular Pre-registration Issues”). But it’s all hard to know and determine, and it likely depends on perspectives and time periods and comparisons to other options and possibilities and who knows what else. I mean from a certain perspective even fraudsters like Diederik Stapel may have indirectly caused certain positive things to happen for the scientific community or even society at large.
But, to name one more direct and clear possible benefit of these kinds of large scale collaborative projects: the money. If such a large scale project is being financed, does more money go to the organizing scientist? Does Zwaan for instance in this case here ask for money for his big replication project, and could his university then take a large percentage of this money for “overhead costs”? I don’t know how that all works from experience, but I’ve come across some information about this kind of stuff and I always wonder how that would go with these types of large collaborative projects.
Then, there is the other more direct and clear possible benefit of being first author as an organizing scientist. Not sure how this will go here, and this sometimes differs it seems, but that could be another example of the kinds of benefits I am thinking about here. Isn’t that seen as beneficial for one’s CV and promotion opportunities and things like that? And on top of that, such a large-scale project might even find it’s way in some high impact journal, which might then be even more beneficial concerning certain things.
Then there are many other things that annoy me of these kinds of projects. One being the role of peer-reviewers (especially when it’s a so-called Registered Report) and the possible group processes in decision making concerning design and things like. I think these might work sub-optimally and not in line with how I view science should work with these types of projects. My linked to idea of replication in small groups elsewhere in the commments here tries to optimally deal with such possible issues by having each researcher design their own study which the group then all replicate. One of the ideas behing that format is that that the individual researchers can ask for, and use the advise of the others, but aren’t impliclty or explicitly pressured to use it or go along with some design idea (like might be the case with peer-review or group decision processes).
I’ll stop now, but refer to my manuscripts mentioned previously for some more information about this all, and many more reasons why certain things annoy me.
A new day! To add some more things to ponder in light of some very general view or assumption that this kind of large scale replication project is beneficial for the community or even society at large:
What if this replication project will be performed as a so-called Registered Report, would that necessarily be beneficial for the community or society in general? Did you know that, apparently, some Registered Reports have not made, or perhaps even still do not make, the protocol or registration publicly available (see Hardwicke & Iaonnidis, 2018; Chambers & Mellor, 2018; Chambers & Tvavella, 2022). This might be one thing to be aware of before thinking very generally that some research project is beneficial, also for instance in light of “today’s standards of research transparency”.
And if this project is performed as a so-called Registered Report, does this directly or indirectly kind of also support or advertise this particular publication format? Or will it when it is written about in a newspaper or some associated scientists is being asked about it? And is that a good thing for the community of scientists or society at large? If I am not mistaken, Registered Reports is a format where not only non-transparency of crucial research information can exist, but peer-reviewers seem to be given a possibly very influential role in the actual design of the study. Imagine how that could go in light of a large scale collaborative project. Could that mean that hours of collaborative decision making by the collaborating researchers could be implicitly or explicitly thrown overboard by some (anonymous?) reviewer?
Or could this result in a situation where the opinions and suggestions of the peer reviewer will be given way more, or too much, weight because of possibly wanting to please the reviewer to just get accepted and published (see Binswanger, 2014). Or, could this particular format provide a reviewer with some possible ways to nudge the collaborative project in a certain direction, or perhaps even sabotage things by purposely suggesting X, Y, or Z? Would that be good for the community of scientists or society at large?
Talking about society at large, does the Registered Report format kind of emphasize and underline the role of peer review and journals or perhaps making them even more powerful and important and influential? Would that be good for the community of scientists or society in general? Will members of society have to pay money to even be able to read and look into the final paper of that paper, and is that good for society?
So much annoyance about so many things! But, perhaps more importantly here, it might show how some things can also be seen from a different perspective.
“Sentences like ‘the purpose of doing a study is to add to scientific understanding’ and ‘the benefits go to future researchers and even society as a whole’ are too vague and hard to prove or disprove in my view and reasoning.”
That’s because they’re inductions (specific to the general) and not deductions. So, yeah, you can’t prove them, but that’s because you can’t prove inductions, not because the claims themselves are unreasonable (even if you find them contestable).
Quote from above: “That’s because they’re inductions (specific to the general) and not deductions. (…)”.
I am no expert on logical reasoning but I think these kinds of sentences you are talking about are just statements, which can be true or false (see page 25 of Dowden’s Logical Reasoning linked to below). I don’t think these sentences followed as a conclusion from certain premises or someting like that, and I don’t think they are part of some sort of inductive or deductive reasoning in that strict sense.
From the dictionary “prove” can be defined as “to establish to truth or validity of (something) by the presentation of an argument or evidence” and “to demonstrate the reality of (something)”. I reason that some sort of general assumption or statement like “the benefits go to future researchers and even society as a whole” in light of the replication project is kind of hard to prove or disprove, or in other words, to establish the truth or validity of, or to demonstrate the reality of. This could, for example, be because there might be different perpectives about what is beneficial, or there could be different times at which one determines whether a project has been beneficial (e.g. now or in 50 years), or there could be different things for which it has or has not been beneficial, etc. Or, perhaps the purpose of this project, or any other project, isn’t necessarily to add to scientific understanding (e.g. look at studies performed by fraudsters), but that may be hard to prove or in other words demonstrate the reality of or establish the truth or validity of at certain points in time.
B. H. Dowden “Logical Reasoning”:
https://www.csus.edu/faculty/d/dowden/_internal/_documents/logical-reasoning.pdf
There are answers to these questions! To make this concrete, physics has both large AND small collaborations.
1. The Higgs boson paper (thousands of authors) has the collaboration listed as the author. The individual contributors are listed by university in an appendix. There is a “publication committee” within the collaboration, and the chair of that committee is the one that you email with questions. This person changes over time. https://www.sciencedirect.com/science/article/pii/S0370269312008581
2. There are ways of documenting who did what in a collaboration, called the “CRediT system.” You get assigned a role saying what you did, like “data analysis” or “writing”.
https://publishing.aip.org/about/news/aip-publishing-announces-implementation-of-credit-system-helping-recognize-individual-contributions-to-scholarly-work/
https://publishingsupport.iopscience.iop.org/questions/credit/
3. There are gauntlets of internal peer review to go through before the paper even leaves the collaboration.
https://cms.cern/content/publishing-analysis
Scientists from rival collaborations are asked to do external peer review. (ATLAS and CMS are the two rival collaborations at CERN.)
https://hdsr.mitpress.mit.edu/pub/1lhu0zvn/release/4
4. CERN deals with the money itself. The researchers are paid by their own institutions.
Quote from above: “There are answers to these questions!”
I am personally not interested in physics comparisons, I know only some things about psychological science type research like what the replication project is about if I understand correctly.
I dislike research bureacracy (see Binswanger 2014), and I dislike a lot of group processes in science a lot as well, and I dislike traditional peer review, and I dislike all these types of author credit systems and similar countless proposed ways to improve countless aspects of doing research or writing stuff.
That is all.
I am now going to listen to Getdown Services’ “Dog Dribble” to make some things make sense again…
“I dislike research bureacracy (see Binswanger 2014), and I dislike a lot of group processes in science a lot as well, and I dislike traditional peer review, and I dislike all these types of author credit systems and similar countless proposed ways to improve countless aspects of doing research or writing stuff.”
Well, other people think these are necessary to varying extents (especially relating to the amount of resources necessary to perform certain experiments/calculations) and/or like them.
That is all.
Andrew:
In a narrow sense, AAAnoymous is correct. Other people cite papers, not comments in blogs. Tenure and promotion committees look at your publications and h-index. They don’t look at comments you left on a blog.
There are very few people who have a CV like yours and very few people who are cranking out 30 papers/year outside of big AI labs (e.g., Dave Blei’s). On the other hand, I think you have one more promotion step to go outside of admin, which is a university professorship. The sure way to get one of those appears to be winning a Nobel (memorial) prize. The only way you can get a meaningful raise is to get a competing job offer from Harvard and Stanford.
As a result of a recent new blog post by Zwaan about this project dated oct. 15th 2026, I just checked out this blog and the comments here again. More specifically, I checked out the blog post by Zwaan dated sept. 8th 2025 for which a link was added later here in this blog post (see comment by Mr. Gelman dated oct. 7th somewhere in the comment section here). I noticed the following three sentences in that Zwaan blog post dated sept. 8th 2025:
1. “We don’t yet know whether the grant will be funded—and the project depends on that funding for coordination.”
2. “Share the data: We’ll take care of coordination, data cleaning, and meta-analysis.”
3. “Get credited: All contributing sites will be acknowledged. Substantial contributors will be offered co-authorship.”
So, it seems Zwaan is asking for money (which I wondered about, see one of my previous comments here) which makes me wonder how that works with money distribution and things like that. Is Zwaan, or his university, able to take a considereble and disproportionte percentage of the received grant money for this large-scale “collaborative” project for “reasons” (e.g. coordination costs? overhead? whatever term looks fancy enough)? And, given the emphasis on “transparency” would it be important or useful, especially concerning these kinds of large-scale “collaborative” projects, to explicitly and clearly and transparently provide a nice overview of exactly how the money has been used?
One more thing I dislike about these large-scale “collaborative” projects is the possible ever more clear distinction and separation of tasks and roles and responsibilities. The sentences in Zwaan’s blog post about data collection and processing is something I, for instance, worry about. Before you know it, some people might even suggest collecting data through some sort of centralized computer or lab or process, where participating labs have little control and insight concerning their individual lab’s data collection and processing. And even when the separate labs would actually still collect their own data, the subsequent task division of data processing in these kinds of large-scale “collaborative” projects might involve too much risks and might involve opportunities to deny responsibility for any possible errors or even fraud by almost all labs and people involved. Also see Steen (2010) “Retractions in the scientific literature: do authors deliberately commit research fraud?” where the author hypothesized that fraudulent authors diffuse responsibility across many co-authors and where the author found that fraudulent papers had more authors than erroneous papers.
I am undertanding things correctly here that participating as a lab by collecting data for this “collaborative” project might not even necessarily result in being a co-author but merely being “acknowledged”? If so, it perhaps nicely underlines some of what I wrote about benefits for some and not others, and how this might be disproportionate in these large-scale projects, etc.
So much to be annoyed by that I think I have moved on from annoyance to something else…
When you say “Rolf Zwaan writes”, did he write to you directly or is this a blog post/article from somewhere? If the latter, can you link the full post? I’d love to read more, but can’t find it with a quick Google search.
It was a blog post and I added a link; thanks. He also has a couple of followup posts there on the topic.
If you need to search for something like this, quote a sentence in it. For example, it’s the first hit to this query, ["Suppose you and a friend witness a car crash. After a while, the police arrive."]. Include the quotes, don’t include the braces.