Adam Marcus and Ivan Oransky write:
Twice during his Senate confirmation hearings at the end of last month, Robert F. Kennedy Jr., America’s new health secretary, brought up a peer-reviewed study by . . . Anthony Mawson, an epidemiologist and a former academic who has published several papers alleging a connection between childhood vaccines and autism. (Any such connection has been thoroughly debunked.)
But wait! you might ask. If this has been pre-debunked, how did it appear in a peer-reviewed journal? Marcus and Oransky explain:
[The paper] to which Kennedy was referring, appeared in a journal that is not indexed by the National Library of Medicine or by any other organization that might provide it with some scientific credibility. One leading member of the journal’s editorial board, a stubborn advocate for using hydroxychloroquine and ivermectin to treat COVID-19, has lost five papers to retraction. Another member is Didier Raoult (whose name the journal has misspelled), a presence on the Retraction Watch leaderboard, which is derived from the work of a nonprofit we cofounded, with 31 retractions. A third, and the journal’s editor in chief, is James Lyons-Weiler, who has one retraction of his own . . .
Perhaps a scientist or politician—and certainly a citizen-activist who hopes to be the nation’s leading health-policy official—should be wary of citing anything from this researcher or this journal to support a claim. The fact that one can do so anyway in a setting of the highest stakes, while stating truthfully that the work originated in a peer-reviewed, academic publication, reveals an awkward fact: The scientific literature is an essential ocean of knowledge, in which floats an alarming amount of junk.
Yeah, someone had actually pointed me to that paper a couple weeks ago and I responded that it’s not worth looking at something published on a propaganda website. With enough effort it should be possible to find the flaws in the paper, but I don’t want to be drawn into a game of whack-a-mole, as unscrupulous and clueless people (it can be hard to distinguish between the two; recall Clarke’s law) can produce b.s. faster than I can clean it up.
At this point, you might be tempted to say, “Hey, stop gatekeeping, man! Just cos something is published in Pravda, that doesn’t make it false. Evaluate the claim on its own merits.” To which I’d reply: sure, but in this case the box is wide open. I take a paper published in a propaganda website no more seriously than an unpublished paper on Arxiv, or Vixra, or some student’s term paper, or this stuff.
The point is, if you want to be open-minded, there’s no good reason to start with work promoting a politically loaded, much-debunked idea and published on a propaganda site.
Unfortunately, the legitimate peer-review system also produces its share of duds, including papers by decorated academic researchers publishing in top journals–that last guy is a Professor of Ethics, if you can believe it! The situation is grim.
I have a sort of optimistic view of future research in which everything is chucked up on arXiv or similar for a small fee, and other scientists/patient advocates/etc comment their concerns about the article’s flaws and suggestions for making it better.
Authors then respond to concerns, possibly updating their manuscript when warranted, and it gets a new DOI. The “publisher” (using that term loosely) keeps the most obviously misleading crap off the repository, and mostly just makes sure studies have been registered with IRB, etc. Journals would still exist but mostly to curate high quality research with paid peer review.
I haven’t fleshed out this idea fully, but it doesn’t take long to find (peer-reviewed) borderline fraud out there in most journals, so I don’t think it can be much worse than the alternative.
Some journals already have processes very similar to this, F1000Research for example.
I agree, and have used F1000 in the past. It’s a great system, although I will say that the ease with which you can identify and solicit sympathetic reviewers is a little concerning.
Yes, that’s true, although I think that is counterbalanced to some extent by the reviews being signed and open. If I am interested enough to read all the review reports, then I will probably also be interested enough to notice whether the reviews are at all thoughtful and seriously engaged.
Robin –
Check who’s heading HHS, NIH, and the FDA. Their appointments stem largely from a growing belief that peer review is, at best, unreliable and, at worst, “pal-review.”
We’re seeing a shift away from peer-reviewed publications toward questionable alternatives, as evidenced in this example:
Flawed Science, Bought Conclusions: The Aluminum Vaccine Study the Media Won’t Question
https://www.trialsitenews.com/a/flawed-science-bought-conclusions-the-aluminum-vaccine-study-the-media-wont-question-aaec2793
Here, Kennedy seeks to replace a peer-reviewed article with a Kennedy-reviewed version, arguing that discrepancies between raw data and study conclusions stem from a nefarious big pharma cover-up, despite those discrepancies being artifacts of necessary adjustments for confounding variables.
As flawed as peer review can be, I’ve always asked its critics: Would we be better off without it? To some extent, we’re now running that experiment, and my Magic 8-Ball says, “Outlook not good.”
While I believe there are better alternatives to the current peer review system, and maybe just chucking everything up on arXiv may be a part of better alternative, I’m struggling to stay optimistic about the broader direction we’re heading.
In all fairness, on the positive side, it seems that Kennedy’s tweet about the links of that study to the big pharma conspiracy now has a “community note” attatched.
https://x.com/SecKennedy/status/1951423851314946083?t=K00m7MkI5if2CJTH8zJ_RQ&s=19
I’d still prefer peer review as a highly imperfect standard, but at least where we’re going might be less sub-optimal than it could be.
Yes, we would be far better off with no peer review (in the institutionalized “read a paper” sense).
Actual peer review is redoing the experiments, but people think this is unnecessary because of “peer review”.
Besides suppressing replications, it also suppresses novel ideas in favor of slight “safe” modifications to the status quo.
Thus, it impedes progress via nation-scale generation of misinformation and stifling original thought.
The proper quality control is independent replication and predictive skill.
I do agree with you here, and must admit that, for all its faults, my manuscripts are often improved by peer review (and, similarly, the process of rejection, revision, and closer examination).
At the same time, peer review isn’t much of a guarantee of anything, and I’ve often received better advice from someone reading my paper and mentioning a potential flaw to me than the unpaid peer reviewers giving a cursory read and halfhearted whine about p-values.
There is an ongoing debate in metascience about the purpose of self-reflection when it’s being disingenuously weaponised by charlatans like RFK Jr, Robert Malone, and Vinay Prasad.
Robin –
“Yes. A follow-on question after ‘What would you have replace peer review?’ would be ‘Hasn’t the process of peer review significantly improved the quality of your work on occasion?’
If the answer to the second question is ‘no,’ it may well be to some extent because of crappy peer review. Certainly, we’ve all seen reviewer comments that seemed to reflect a lack of understanding or some form of professional gatekeeping. But I’m still reflexively skeptical of anyone who says that peer review has *never* led to improving their work. A blanket statement like that suggests to me that someone isn’t a very engaged researcher, or that they’re pushing an agenda.
So it goes back again to what is less suboptimal. Assuming much peer review is crap, and some improves the quality of work, is there a net benefit or loss? I’m inclined to say that in my limited experience, the net cost is fairly minimal: time lost, aggravation, opportunity cost. But I don’t see how in the end those negatives lead to significantly less quality output. I do think the benefits lead to significantly more quality output.”
There are many papers that try to assess peer review, I have never seen one come to the conclusion it is a net good. Please share it if you find one.
And usually it is the people who rose to the top who feel able to speak out:
https://pmc.ncbi.nlm.nih.gov/articles/PMC1420798/
Getting useful feedback on your work is great, no one disagrees with that.
Joshua, I expect most scientists in my research areas would say that peer-review has improved their manuscripts – both Robin and I here have said as much. Likewise most scientists (probably all, actually) in my research areas can provide anecdotes of how the peer review of one of their manuscripts has driven them to a state of powerless frenzy. But I could say the same about interactions with my pet insurance provider. I have yet to see an idealized replacement for peer review that didn’t seem to have fundamental flaws, although there continues to be modifications that improve the process. The bottom line IMO is that there should be some notion of “minimal quality” as a starting point for assessing the value of published research, even if sometimes peer-review gets this wrong.
I think my views on the value of peer review are widely shared – I think by you also, so I’m certainly not disagreeing with you! I spent 20 minutes on google scholar searching and scanning papers on peer review (best to use “peer review” as a search text and limit to “since 2021” since there are gazillions of papers). Here’s a tiny selection of what I found (haven’t provided links but these are easily found by googling):
Lockwood 2025, Why I value peer review
Roberts et al. 2025, The Peer Review Process: Perspectives of Reviewers
Berg et al 2024, Peer review: the imprimatur of scientific publication
Morley and Grammer 2021, Now More Than Ever: Reflections on the State and Importance of Peer Review
Lockwood 2025, Why I value peer review
Roberts et al. 2025, The Peer Review Process: Perspectives of Reviewers
Berg et al 2024, Peer review: the imprimatur of scientific publication
Morley and Grammer 2021, Now More Than Ever: Reflections on the State and Importance of Peer Review
I couldn’t get access to the first, but there is no evidence in favor of peer review in the others. I even clicked through to some of the other refs. A few talk about how the reviewers can learn from reading the papers, and feel good about participating in a community. There were a few studies that reported it slightly increases the readability (eg, 3.5 -> 3.75 on a five point scale). But that is not the validity of the methods/analysis/etc for which peer review is supposed to provide quality control.
Instead they say there is no evidence, but they know it works based on some kind of non-empirical faith (like described in the Richard Smith paper I linked above).
Why is there not a single study showing peer review is an effective QC method?
This one pretty much sums it up:
https://physoc.onlinelibrary.wiley.com/doi/10.1113/EP092108
Contrast with the Royal Society motto: Nullius in verba (“take no ones word for it”). Science is about not even trusting yourself, let alone other people.
Anon
I don’t trust myself, nor anyone else, as the quote you provide says. It is necessary for science but also for life. However, it is sad, not something to be celebrated. Without any trust, life is bleak. I don’t think science is any different – a scientist needs to be skeptical and constantly require evidence and question that evidence, but I wouldn’t characterize that as distrust. If we trust nobody, it should also mean we trust no evidence, as increasingly “evidence” can be manufactured, mis-represented, or otherwise deceptively portrayed. So, it may be a necessary survival mechanism to distrust everything and everyone, but where does that leave us? Wouldn’t it be nice to feel that there are some things or some people we can trust? Trust need not be blind or immune to questioning, but I’d rather start from a position of trust until distrust is earned – reputation matters. If we start from a position of distrust, then I don’t see how trust can ever be earned, and that seems like a dismal conclusion.
It’s not clear what point you’re making Anon. My post was addressed to Joshua and was elaborating on the fact that it seems to be widely appreciated at least in the “hard sciences” that peer review is a valuable if somewhat annoying process, can help to improve the quality of manuscripts, and (my opinion) that some of basic quality control (e.g. peer review) is required as a first step in the potentially long process of establishing the value and contribution of a scientific paper.
That’s the point of the very quick trawl through recent papers on “peer review” that I posted a selection from. Some relevant quotes in support of the point I was making:
From a survey of reviewers reasons for agreeing to review in Roberts et al 2024:
83% of reviewers surveyed stated that one of the reasons for agreeing to review is: “I enjoy the opportunity to help improve a paper”
…and so on. I think those are quite widely held views.
It’s a dismal parsing of the “Nullius in verba” quotation to consider that it equates to “trust no one”. If I was going give it a contemporary translation it would be “show us the evidence”. It’s difficult to imagine how one could proceed in life (or science) without taking what might occasionally be the risky step of trusting others and as Dale relates, being skeptical doesn’t equate with being distrustful. I trust all of my scientific collaborators for example, and I think I have good reason to do so. There are many researchers in the fields I work in that I have a high degree of trust towards – I trust their competence and integrity for example – and I know that it’s possible for me to do so while at the same time being aware that I might occasionally be skeptical of some of their results or interpretations.
IMO one of the reason that US sociopolitics is in such a mess right now including the dismal atttacks on science intertwined with antiscience conspiracy theories is that many people have been suckered into adopting distrustful attitudes against individuals and institutions that they might otherwise have little reason to distrust.
I actually looked at those links 3-7 to see the tremendous benefits to write my last post. That is a very misleading line. None had to do with valid logic/etc as the reader would expect. Also why does you post leave out which paper it came from?
The clear point, that everyone who looks at peer review from an honest, scientific perspective comes to is: 100% of the evidence shows it does not have the claimed benefit.
Literally 100%, there is not a single study ever published, showing it does this:
These types of beliefs that exist despite being contrary to all evidence are actually pretty common in biomed. Meanwhile, replications and quantitative predictions are rare. So I dunno who got to group it with “hard science”, but it doesn’t meet any criteria I would use.
It is one of those topics where it is time to switch to studying why a group of people share a delusion, to better understand human nature, at this point.
Anon, you’ve interjected into a thread in which I was largely agreeing with Joshua that peer review likely has a benefit on improving the quality of published work overall and was elaborating on the evidence that that seems like a widely held view, at least in the research areas I’m familiar with.
Your interjection is again missing the point although you’re welcome to insist on some requirement for some published studies to validate what are often subjective views (“my manuscripts are improved by peer review”) but which if it was required could be validated objectively (see next paragraph).
Robin, myself and some of the authors on the papers I linked to consider that their papers can be improved by peer review, and in my experience that’s a common viewpoint. Of course, if one felt determined to have some objective assessment of this (I expect this already exists) there is an increasing evidence base that one could use to pursue this. Nature now publishes peer reviews and peer-review history (reviews, authors responses, editors inputs) along with articles, and eLife, Royal Society (your favourite!), F1000, PLOS, BMC public health and probably others, do the same in various forms.
So one could easily assess the effect of peer review on paper improvement in these journals. You could do it!
Yes, I agree that peer review is a valuable part of scientific publishing – that certainly seems the case from my biomed perspective. What hasn’t been mentioned is that it has a substantial positive effect on the pre-submission parts of science. If you want to publish in a decent journal you know the quality of your work needs to be high – obvious things like appropriate controls and statistical power, additional experiments and analyses done if potentially non-obvious interpretations are made and so on should be inherent in the research ethic of researchers, but the prospect of peer review does make one think about how the work will appear to someone not part of the research effort – e.g. “is this interpretation likely to be accepted as it stands or should we really try to get that additional experiment to work?”
Also agree with Robin that manuscripts are often improved as a result of peer review and I usually find reviewing papers rewarding. In my experience the most frequent positive effect is in clarifying elements of the presentation. So as we becomes more and more familiar with the manuscript that we’re struggling to prepare it’s easy to lose sight of potential confusions and potentially ambiguous descriptions or some lack of clarity in figures that could be improved by annotations and so on, and reviewers are usually pretty good in helping with these.
It’s difficult to imagine what things would be like without peer-review! There does need to be some element of quality control and the present use of arXiv and biorXiv are great in their present form/role but would only go so far if they became the final location of all undifferentiated stuff. I expect that at the “top end” of science (again from a biomed perspective) where researcher are strongly invested in finding things out that will be useful, things might not change so much and journals (if they still existed) would continue to use some sort of selection for what they accept – funders and other interested parties like the pharma industry would likely enforce this. I expect we’d also see a massive and growing sea of self-serving and agenda-led junk and the “bottom end” of the “research” effort.
Chris:
I think that if peer review is not idealized as it is by Gladwell, NPR, etc., we can step back and see its virtues. One challenge is that, with more and more researchers writing more and more papers (even before the horrible new development that you can now push a button and produce an entire new paper made out of plagiarized material), there has been more and more demand for reviews. That’s one reason why I think it would be better to have more of the review effort devoted to papers that people are actually going to read.
Meanwhile, though, yeah, my colleagues and continue to write papers, submit them to journals, go through elaborate efforts to respond to review comments, etc. It doesn’t seem like a sustainable system but we still spend a lot of time within it.
Andrew, in my research area (biomed) there are good journals and crappy journals and we can just ignore the latter. I publish maybe 2-3 papers a year and it seems reasonable for me to review maybe 5-6 papers a year. You can simply ignore review requests from crappy journals – that seems to be a widespread approach – why not concentrate on what are likely to be quality papers from good journals in an area of your expertise/interest – the review task then becomes quite pleasurable in my experience. That’s not to say that occasionally a decent paper doesn’t end up in a not-so great journal, but we have pubmed and google scholar to help with that.
If crappy journals receive the plagiarised garbage you refer to then they either identify this through plagiarism checks or they don’t – either way it doesn’t matter that much. Usually in time the journal will become recognised as being particularly rubbish for accepting junk (and as sometimes happens the editoral board might resign or the publisher may shut it down). If junk does get published it’s likely to have little effect on the progression of scientific fields, although of course it might be of value to science misrepresenters and conspiracy theorists. Funders that funded the junk are rather likely to be a little more careful with their funding and governments that have some aspirations that their research base is of a decent or growing quality might well choose to impose sanctions on their dodgy researchers – or maybe not, but that’s their lookout.
That’s a broad and (and elitist!) view that recognises that like everything else in life there is a hierarchy of quality and one may as well engage with and nurture the good stuff. It takes account of the likelihood that the large majority of scientists (I can only talk from a biomed perspective involving US and UK government/charity funded research) are good at what they do and want to find stuff out. There will always be cheaters but they bad ones seem to be increasingly identifiable. Sadly as with the subject of this thread there are also dismal efforts to make some of the “cheating” approaches a new “gold standard”.
Robin:
I agree, and this connects to my proposal to turn journals into recommender systems.
Andrew: I am only just reading this for the first time but quite like the idea. I would be worried that important papers might get glossed over for many years without review, only to get picked up after the authors have forgotten what they did, but I guess you could solicit journal interest as well.
Robin:
Yes, I was imagining that you could “submit a paper” to a journal by pointing to the online preprint and writing a one-page letter explaining why you think it would be good for the journal.
The big worry I have about these open and voluntary peer review models is that I’d expect people to preferably read and review big name papers, maybe with a positive bias because they may depend at some point on these big names. Young authors not associated to big names may have a hard time finding reviewers (and readers), but they would need them the most.
Exactly, or only papers are reviewed that receive lots of attention from certain (groups of) people on (a-)social media, etc. etc. etc. It’s all likely to be(-come) a conceptual replication of several current problematic issues, and/or it introduces new problems.
Once more I would like to refer to an idea I have posted here a few times, and is (or was) currently (somewhat) executed by JOIBS which is some journal on Researchers.One (or something like that). Anyway, the general idea is as follows:
A paper will be peer reviewed, and the reviews, however positive or negative, will be published online alongside the reviewed work. The purpose of this format is to encourage open and thoughtful evaluation of works of empirical behavioral science during and beyond the initial review process.
An important feature of the idea is that the reviews are published separately and non-anonymously. Publishing reviews non-anonymously and separately opens up several options, and could lead to several positive consequences. For instance:
– A reviewer could list a published review on their Curriculum Vitae. Publishing non-anonymous reviews provides the option to depict a separate list of “Published Reviews” on a Curriculum Vitae
– Employers, members of PhD committees, members of hiring committees, etc. can use published reviews as an additional way to assess and evaluate a candidate which has listed their published reviews on their Curriculum Vitae
– Several (arguably) scientifically problematic processes and issues regarding peer-review seem less likely to occur using this format of publishing and reviewing papers. The decision to publish a paper is not heavily dependent on the reviewers’ judgement and decision as is commonly often the case at other journals. This,
for instance, makes it less likely that a certain reviewer can block an author’s paper because it might be critical of the reviewer’s own work. Other forms of peer-review abuse such as a reviewer attempting to increase the citation count of their own work by recommending their own work as references to be added seem also less likely for the same reason
– The publishing of the review means that a review can be cited by the reviewer, the authors of the reviewed paper, or others reading the review (for instance, in addition to the reviewed paper). Interesting ideas for follow up studies provided by a reviewer in a published review can, for example, be added to or discussed in the discussion section of the reviewed paper by the original authors in a revision which includes a citation of
the review
– A review can perhaps even lead to co-authorship concerning the paper under review and/or a future paper. If a review catches big mistakes, or provides tremendous improvements of the reviewed paper, the reviewer might have contributed substantially to the final version of the paper. In that case, the authors of the reviewed
paper might decide to ask whether the reviewer wants to become a co-author on the reviewed paper. Or, the reviewer might be asked to contribute as a co-author on a next project. In this way, a review might lead to earning co-authorship.
Anyway, I hope it’s okay to share for the Nth time on here. I tried really hard not to, but given the several recent comments about this publishing and peer-review thing I couldn’t resist. I hope it might be interesting or useful for those thinking about this all.
“The point is, if you want to be open-minded, there’s no good reason to start with work promoting a politically loaded, much-debunked idea …”
…. obviously true, and is a common tactic across the political spectrum in government, media, academics, punditry, etc.
and the obvious point here was really to single out RFKJr and hammer him specifically.
Rohrbach,
RFK does not need me to hammer him; he does just fine on his own by promoting junk science. Given his powerful position in government, he deserves to be hammered on this one. Malcolm Gladwell annoys me too sometimes, but he’s not making policy.
Given that Andrew is still in France, his criticism of Didier Raoult strikes me as dangerous or an act of courage:
https://en.reseauinternational.net/les-marseillais-sortent-dans-la-rue-pour-soutenir-le-professeur-didier-raoult-et-en-finir-avec-la-fermeture-autoritaire-des-restaurants-et-bars/
“The Marseillais take to the streets to support Professor Didier Raoult and put an end to the authoritarian closure of restaurants and bars”
Didier Raoult is really prolific. From https://en.wikipedia.org/wiki/Didier_Raoult
“Raoult has more than 2,300 indexed publications…He is one of the 99 most cited microbiologists in the world and one of the 73 most highly cited French scientists…is also one of the 7.3% most self-cited authors, more than 25% of his citations coming from papers he co-authored.”
Yep, you can find something in the medical literature to support pretty much anything you want. You learn this very quickly when trying to make a quantitative model. Then the impossible concentrations, rates, and such start appearing when working out the consequences (eg, entire cell surface must be devoted to a single receptor).
I guess someone needs that experience a few times to deeply understand that no bunking or debunking has actually been performed for the vast majority of topics. It only appears so at a very superficial level.
Further, there is “a connection” between vaccines and autism, but it may be negligible or otherwise uninteresting (eg, doctors are more/less likely to give an autism diagnosis to vaccinated children).
The claim there is no connection is not plausible, since we know everything is correlated with everything else and collectively caused by all events in its past lightcone. And I wish we could take that as shorthand for “negligible”, but then you look at the statistical tests used for the bunking/debunking and see exactly zero correlation *is* what they mean.
Anon:
I agree that everything is correlated with everything else. But beware the one-way-street fallacy. If someone makes a claim based on bad evidence (or even fake evidence) that X and Y have a positive correlation, the appropriate response is not to say that the correlation between X and Y is zero. But it’s also inappropriate to say, “hey, they might be positively correlated,” without also saying, “hey, they might be negatively correlated.” Or, more to the point, the correlation could vary based on the scenario.
There is nothing so absurd that a philosopher has not already said it.
-Cicero, “Concerning Divination”
Millions of pages,
filled with writings by fools and sages.
Writings of different ages,
that may be more like boxes, and act like cages.
If you can find evidence for anything,
in the many pages ever increasing,
perhaps it’s better to look for something,
in the few words depicted in a scribbling.
This is a totally fair analysis because 1) it takes a lot of time, and 2) many vaccine skeptics might not be satisfied and then request analyses of other studies; however, if no serious researcher ever looks at the vaccine skeptics’ main citations, then they’ll use that as evidence that they’re unfairly not being listened to. It seems it would be a significant value for a serious researcher to review at least one of the citations; after that, declining further analyses of citations due to point 2 above seems more than reasonable.
Better than “You can cite peer-reviewed research in support of almost any claim”, you can cite the same peer-reviewed research in both support of a claim and support of its contrary. Comparing children’s performance on addition and multiplication facts before versus after the coronavirus pandemic, the “group before the coronavirus pandemic was more accurate in addition compared to the group after” (see the caption of Fig. 1), but the Pre-Coronavirus group was more accurate in multiplication than in addition compared to the Post-Coronavirus group (see the graphical representation in Fig. 1) [See https://doi.org/10.1016/j.cogdev.2025.101562 if you have US$ 24.95 to waste].
This reminds me of an idea I once had where I would take a few random social science papers and rewrite the (crucial parts of the) introduction section by taking the sentences of the original paper’s introduction and writing down the opposite of these sentences and statements.
I then would try and find papers and sources that supported this now opposite and flipped introduction so to say to be included as references. I wonder to what extent if I would be able to complete these new opposite introductions in this manner.