Dorothy Bishop shares this hilarious/horrible story from neurosurgery researcher René Aquarius:
Mid November 2023 I received a request to peer-review a manuscript for a special issue on subarachnoid hemorrhage for the Journal of Clinical Medicine, published by MDPI. . . . I ended up recommending rejection of the paper . . . My biggest gripes were that the authors claimed that data were collected prospectively, but their protocol was registered at the very end of the period in which they included patients. In addition, I discovered some important discrepancies between protocol and the final study. . . . The biggest problem was the lack of a control group. . . .
OK, so far so good. Maybe a mistake to review this paper for free, but this sort of thing can be a learning experience—even a really bad paper can give you some sense of what people are working on.
The paper had two reviews, and the other reviewer was positive, so the journal split the difference and asked for revisions. Aquarius tells the story:
Revisions were quite extensive . . . and arrived only two days after the initial rejection. . . . before I could start my review of the revision, just four days after receiving the invitation, I received a response from the editorial office that my review was no longer needed because they already had enough peer-reviewers for the manuscript. I politely ignored this request, because I wanted to know if the manuscript had improved. . . .
The manuscript had indeed undergone extensive revisions. The biggest change, however, was also the biggest red flag. Without any explanation the study had lost almost 20% of its participants. An additional problem was that all the issues I had raised in my previous review report remained unaddressed. I sent my newly written feedback report the same day, exactly one week after my initial rejection.
When I handed in my second review report, I understood why I initially got an email that my review was not needed anymore. One peer reviewer had also rejected the manuscript and had concerns similar to mine. Two other reviewers, however, accepted the manuscript. . . . because my vote was now also cast and the paper received two rejections, the editor couldn’t do much more than to reject the manuscript, which happened three days after I handed in my review report.
Then things got worse:
In December, about a month later, I received an invitation to review a manuscript for the MDPI journal Geriatrics. You’ve guessed it by now: it was the same manuscript. . . . again, transformed. It was now very similar to the very first version I reviewed. Almost word-for-word similar. That also meant that the number of included patients was restored to the initial number. However, the registered protocol that was previously mentioned in the methods section (which had led to some of the most difficult to refute critiques) was now completely left out. The icing on the cake was that, for a reason that was not explained, another author was added to the manuscript. . . .
There was no mention in this invitation of the previous reviews and rejections of the same manuscript. Although one might wonder whether MDPI editors were aware of this, it would be strange if they were not, since they pride themselves on their Susy manuscript submission system where “editors can easily track concurrent and previous submissions from the same authors”.
Because the same issues were still present in the manuscript, I rejected it for a third time on the same day I agreed to review it. In an accompanying message to the editor, I clearly articulated my problems with the manuscript and the review process. The week after, I received a message that the editor had decided to withdraw the manuscript in consultation with the authors.
And, the unsurprising conclusion:
Late January 2024, the manuscript was published in the MDPI journal Medicina. I was not attached to the manuscript any more as a reviewer. There was no indication on the website of the name of the acting editor who accepted it.
The most interesting part of the story is that the publisher went through all these steps of reviewing and revising.
If they just want to make money by publishing crap, why bother engaging outside reviewers at all? I’m not really sure.
My best guess is that this is an interaction of the many actors in this system. The owners and managers of the publishing company are presumably mostly motivated by the money, but maybe they are also doing this in part to advance science, as they see it: publish anything, let a thousand flowers bloom, etc. But even if the incentives for the journal are purely financial—publish as many papers as possible, collect the publication fees, cash the checks, buy yachts NFTs—, they’ll still want to preserve some legitimacy, in part because if the journals are de-listed and discredited, it will be harder for them to get masses of submissions and special issues, and in part because the publisher may have a goal of selling themselves to a bigger fish (as here), in which case, again, they don’t want to be running too transparent of a publication mill of this sort.
Soooo . . . to maintain some credibility as a journal, they want to do some editing and reviewing, which means they need the participation of some editors and reviewers. Presumably the editors know the score, but they may still feel that publication in this journal is a way to get important work out there, and it shouldn’t be too hard for these editors to find reviewers who will be sympathetic to the papers being handled.
Putting it all together, the journal has some aspects of a scam but it’s not entirely fraudulent. Some reviewing and editing is done, just in the context of the larger goal which is to publish as many papers for money as possible. But there’s a balance.
My own story
The publisher discussed above is the notorious MDPI—check out its wikipedia article for lots of background.
I’ve published with MDPI! Twice. The first time was before I’d heard about the journal’s reputation. The paper is here. I just looked it up on Google scholar and it has 550 citations, so it did get out there a bit. This is not to say that publishing in Entropy was the best choice—maybe if we’d done the extra work and published in a conventional journal (typically a grueling review process which leads to an improvement in the quality of the paper but at large cost in time and effort) it would’ve had a larger impact—; that said, I see the appeal of the publish-in-a-journal-that-accepts-everything strategy. The many hurdles involved in conventional journals adds lots of stress and uncertainty to traditional publication. On the other extreme, we could just publish everything on Arxiv and bypass the journal system entirely—that might be fine too; I don’t really have a sense of how much more a paper will get read and taken seriously if it has some sort of official journal publication.
My second MDPI publication came this year—I received a personal request to submit a paper as a favor for a special issue that was being organized in someone’s honor, and I published this article, which is a revision of an unpublished manuscript I wrote with the late Keith O’Rourke. When I posted on that paper, we had some discussion in comments regarding the MDPI thing.
I share these stories just to give an example of how legitimate papers can get published in what are sometimes considered predatory journals. It’s always a mix.
The Susy tale is amusing and frightening. But not nearly so frightening as the incoming Trump Administration’s determination to do away with vaccinations and fluoridation. How in the world did this happen? I refer back to the wisdom of my former colleague, Peter Armitage: “Always shoot a dead horse because you can never be sure it is dead.”
The danger of that strategy is that you tell people there is a dead horse there and some of them will go and poke it. You also expend your limited time in front of an audience trying to suppress bad stuff that other people want you to talk about, not spread good stuff that you think will be most effective or useful for your audience.
But lots of people spent the past 15 years trying to get mainstream institutions to take the anti-vaxx movement seriously as a threat to public health.
Sean:
I think part of the problem was the persistent storyline—persistent, even though not supported by data—that anti-vaxxers were rich liberals. It seemed to make lots of people happy to think this. My impression was that the mistaken anti-vaxxers-are-rich-liberals story was pushed by conservatives (for whom this was just another examples of silly liberals), liberals (who often enjoy poking fun at the charming foibles of their group, as they see it), and center-left types who could demonstrate their independence by criticizing the far-left conspiracy theorists.
So he effort you mention of the past 15 years, trying to get mainstream institutions to take the anti-vaxx movement seriously as a threat to public health, may have been misdirected, to the extent that it was aimed to correct the views of rich liberals, rather than seeing it as an anti-political movement that was, until recently, neither of the left nor the right.
There were definitely people in the USA who found it useful to say “sure the right has creationists and climate change denialists but the left has health woo and environmentalists I disagree with so bothsides” (there has been a realignment and now conspiritualty tends to align red in the USA). But someone like Orac who specializes in this issue would be better than me on why it was hard to get institutions to take the threat seriously.
Back to Paul Alper, a majority of parents in many parts of the USA and Canada are antivaxxers and when a policy gets majority support it can get governments trying to enact it.
The percentage of children who are getting vaccines on schedule correlates with income (https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2817476), but this is partly because lower income parents have less access to health care, not because they are anti-vaxxers. Sadly enough, there are too many rich liberal anti-vaxxers (think Marin County, across the bay from San Francisco). Some of these just want a free ride for their kids from herd immunity, but some are indeed anti-vax.
John G Williams: I believe that a key anti-vaxx strategy is to get people to not vaccinate their children whatever it takes to prevent that, much like as environmentalists we want people to burn less fossil fuels whatever it takes to get them to burn less. So the people who adopt the full antivaxx ideology are a minority of the movement, but they just have to persuade parents one of “its annoying” “its dangerous” “your kids will be safe without vaccines.” And from January 2025 they can make it harder to get vaxxed in the USA or easier to avoid vaccinating your children in the USA.
I was told the other day by someone (who isn’t really interested to put much thought/research into such things) that vaccines aren’t meant to prevent infection or transmission.
So the idea that herd immunity is not a reason for vaccinations (first seen due to the very successful covid vaccines) is trickling down. Are there any recent surveys?
What is the current public health motivation?
I also wonder if people are aware of what happened after rinderpest (cattle measles) eradication (which was accomplished by *ending* mass vaccination, btw). Now the cattle are getting goat measles due to loss of cross immunity, which is more deadly for cattle. The same type of thing is expected if (human) measles was ever eradicated.
What vaccines are intended to do depends on the vaccine, as I think you know. Vaccination for smallpox was pretty good at preventing infection, and so transmission. The flu vaccine is mostly good at reducing your odds for getting seriously sick. And, the idea that herd immunity can protect your kids without exposing them to the very small risk of harm from vaccines dates back to before Covid. The measles vaccine is a case in point.
Well, I’ll stop there but the “new” idea is that vaccines do *not* provide herd immunity. It is also interesting that, when challenged from outside the echo chamber, now the rhetoric allows to distinguish between different vaccines.
The public justifications has changed from:
A) Eradicate the disease to benefit all future generations
B) Herd immunity to protect those around you with weakened immune systems
C) Reduce the chance of severe illness in the vaccinated individual
The original public health justification is gone.
The standards are dropping lower and lower, which is a general phenomenon across medicine. Prices/costs/risks are skyrocketing while the benefits dwindle away to nothing. As is expected when the intellectual environment becomes polluted with NHST-generated misinformation.
People with low standards make much better healthcare consumers, and are the reason life expectancy peaked about a decade ago.
Why bother with review if the only point of the journal is to churn money? Well, the review is part of what draws in the revenue. If the authors are climbing the tenure ladder, the publication doesn’t count for much, if anything at all, if it isn’t in a peer reviewed journal. At least that’s the case in medicine. So if they don’t at least go through the motions of review, authors won’t submit, and the money will not flow.
I don’t find the question in your title puzzling, for the reasons you wrote in the post. If you want to publish a scam journal, it’s best for you if it’s not an obvious scam, lacking peer review at all, because then the journal couldn’t be on lists of “approved” peer-reviewed publications. Plus, no one would submit papers to it. You then have to have mediocre peer review, but this doesn’t involve getting only mediocre reviewers — that would take work to ensure — but rather casting a wide net that includes mediocre reviewers. The net will snag excellent reviewers as well, like René Aquarius in the example in the post. Similarly, a wide net for mediocre papers will catch good papers also, like Andrew’s.
1. First of all, MDPI sounds like bad news. I would be leery of publishing there. Or being a reviewer for them. What do you expect if you are dealing with bottom feeders? Who has time for that anyways? I mean, I didn’t do anything good enough for Science or Nature, but I pretty much had no problems at ACS journals, which are pretty legit. I was even a little leery of Elsevier, at least of some of their vast array of stuff (is/was a spread).
2. I think this is sort of like the psychology experiment where they’re not actually testing the subjects but the proctors who electrocute the subjects. I.e. I’m actually more interested in the actions of the reviewer than the crap paper. I mean…who sends in an unsolicited review? It’s one thing to maybe have some prurient curiosity to read the revised paper. But to proactively send in an extra review after being told, no? I mean…sheesh…don’t come whining to me about wanting to be paid for peer review, then! And FWIW, I see a lot of reviewer questions (similar complaints, stories, questions) on Academia Stack Exchange, where the reviewers see themselves as gatekeepers that must be appeased no matter what. (Which is not true, they are assistants to the editor!)
3. FWIW, I’m not some super scientist or paper writer, but I had 5 papers where I was first author (in decent science society journals). And every single one was accepted without revision. I still remember my PI (who was corresponding author) coming into the lab and asking how we were supposed to respond to revisions when the communication said “accepted”. I just looked at in and told him, accepted means accepted. Why should you expect changes? He said it was the first time he’d ever seen that and he had 100+ papers as a PI (and 200+ as a journal subeditor) and had never seen it before.
The funny thing was he had told me the paper, my first, would be flat rejected and I told him to send it in anyways. He never screwed with me after that! Granted he was “hands off” anyways. But I just wrote them, let him look at them (and had no editorial changes from him, even) and sent them in. So…I sort of wonder what makes people think this stuff is so hard.
Maybe I lost out by not experiencing reviewer bunfights? Missed out on a key aspect of academia? Actually, after getting used to just getting accepted, I might have been annoyed to have to negotiate with some gatekeeper reviewer fussbudget. I would have just told the editor take it or leave it (since he’s the gatekeeper, NOT the reviewer). And if there was some long correspondance developing just shifted journals. I’m sort of in shock when people tell me they spend a year or two getting something published.
Are they bad writers? What causes that. I mean sure if someone tells me I made a mistake, I’d fix it. But not if they tell me to change the scope (do more experiments).
Just write clearly, and don’t exaggerate. Heck, you can even make some comments about possible interesting aspects if you do it towards the end and caveat it (“may be relevant in the production of X” rather than “we invented a new method of making widgets”.) The key is really that you are communicating to the reader some why they should care and some context…not overselling. Oh…and be honest. If you dropped a sample on the floor, so be it. That’s why there’s 9 data points out of 10 started. Que Sera Sera (Doris Day).
I did hear one funny thing about one of the papers…where I had very bluntly corrected a mistake from a premier industrial lab and a near Nobel PI. It wasn’t the main content, but was sort of encompassed by my research, so I went ahead and “corrected the record”. My PI was a little leery about it…that was not how he rolled. But I just said it’s my paper and I want to explicitly quote something needing correction. And we sent it in. It flew through. But years later I was at a conference and a BSD (call him BSD2) came up to me and said they sent him the paper when it got two flat accepts and a flat reject (from the near Nobel PI, call him BSD1). But I had never even heard about it before that, nor had my advisor. Editor just handled it and we never even saw or heard about the bad review.
4. I do think that medicine is sort of a social science. Maybe a good comparable is education research. Lot of low N studies, lot of confounding variables, and human subjects. I think chemistry is a lot better in general. I mean there’s some questionable stuff that tries to be device-y and get into Science/Nature. (E.g. https://en.wikipedia.org/wiki/Plastic_Fantastic.) But meat and potatoes chemistry by an American R1 uni? That stuff is fine. And it’s not written in Chinglish either.
Anon:
Regarding your point #3: In my experience, it varies. Some of my favorite papers were accepted immediately with no changes, others were clearly headed for acceptance but needed minor adjustments, others required many exhausting rounds of reviewing, and others were flat-out rejected and I had to send to many journals before they ever appeared.
Also I’ve had some weird experiences:
– A paper that was clearly outstanding but which got dragged through multiple rounds of revision because, it seems, one of the referees hated my coauthor’s guts and so just kept throwing in new objections in order to drag things out. If I’d had any idea, I would’ve just sent it to another journal.
– Another awesome paper, ultimately one of my most cited, where the reviewers were giving me a hard time for no good reason, and the editor solved the problem by slotting it as a discussion of another paper that happened to be scheduled to appear in the journal. My paper was not a discussion of that other paper, but that was a way for it to get published.
– Papers with coauthors who passed away or who otherwise could not participate in the revision process.
– A book review that was solicited by the journal, I wrote it and sent it in, then they made weird changes to it without asking me. I objected to the changes, and the paper never appeared in the journal.
– A different paper that was solicited by a different journal, I wrote it and sent it in, then they told me I had to pay a publication fee. I said no way so it never appeared in the journal.
– A paper written by a coauthor, without my name on it, that was basically him rewriting stuff I’d told him. I only found out about this years after it had been published. I guess he wanted a single-authored statistics paper on his C.V.
– One day, many years ago, I received a postcard in the mail from someone asking for a copy of my paper that had appeared in the Journal of Cerebral Blood Flow and Metabolism. I was like, what? I assumed this must have been some other Andrew Gelman–what do I know about cerebral blood flow and metabolism?–but actually it was me! It was a paper written by some colleagues at a place where I worked, and I’d participated in both the research and writing of the article. I’d lost touch with the people in that lab, and I never knew the article had been submitted, let alone accepted.
– Over ten years after graduation, I wrote up my undergraduate thesis as a research article and sent it to a political science journal. It was rejected–with helpful comments, but I didn’t feel like putting in the effort to revise it and go through the review process. So I adapted it again and included it as a chapter in a book I was editing. In the meantime I blogged about it, and I was contacted by someone from an Italian social science journal that I’d never heard of who wanted to publish it. I figured, why not, this could reach more readers.
– Some colleagues and I wrote a paper, sent it to a journal, and got reasonable review comments. But it was possible to reply to the comments, because in the meantime one of the coauthors had become very difficult. You can’t really revise a paper if the authors can’t agree on it. So we left the paper unpublished and just did more work on the topic, publishing a later paper with a different mix of authors and citing that unpublished work.
– When I wrote my zombies paper, I had a hard time even getting it published on Arxiv. Eventually they agreed to publish it on Halloween, or April Fool’s Day, or something like that. I never published it in a journal, but then one day I was contacted by a publisher of a college writing anthology and they asked if they could include my article. I said yes, and I clarified that the article was not really coauthored by George Romero.
– I’ve even published in PNAS! So all things are possible.
Good job. You’re the real deal. Tenured prof at a (top) R1 school. Glad you interact with the hoi polloi as well.
It makes sense given the broad extent of things you’ve worked on, that you can contribute some special thing ad hoc when needed for a conference or the like, just out of old inventory almost. Not necessarily “next man up” in a research program (me slogging along doing experiments and writing papers right after an LPU was finished).
I do sort of roll my eyes when some NY Yankee librul tries to take me to task and say that I don’t know what “real scientists” are like, since I’m just a Philistine bizness weenie. Been inside the belly of the beast. Got the union card. Dated a female “name” professor. Enough so I don’t have to be impressed with the whole creative-class, NPR-breathy-voice, capital-S-Science reverence. ;)
“I do sort of roll my eyes when some NY Yankee librul tries to take me to task and say that I don’t know what “real scientists” are like, since I’m just a Philistine bizness weenie. Been inside the belly of the beast. Got the union card. Dated a female “name” professor. Enough so I don’t have to be impressed with the whole creative-class, NPR-breathy-voice, capital-S-Science reverence. ;)”
Oh, dude. The bragging is getting old.
You’re right.
Andrew wrote, “I was first author (in decent science society journals)” and that puzzles me. According to what I recall, some disciplines/journals list authors alphabetically, some list the primary author first and some list the primary author last. What I do not recall is which disciplines/journals those are, respectively. Moreover, has there been a shift as time moved on? If so, when?
Paul:
No, that sentence was written by an anonymous commenter, not me.
I wrote it. The comment and uh the papers. ;-) And it absolutely was the norm in chemistry, in the US, a few decades ago, for the first author to be the main experimenter and usually also the actual writer of the text.
Yes, I’m aware that some disciplines (math?) are alphabetical. But honestly, that was/is a whole different culture from what I saw. I’m not sure how you’d measure it, but if anything, I suspect across the whole universe of academic publishing the alpha lists are more the minority, although common enough that I had heard of it.
In medicine, the first author is generally the one who contributed most to the work being presented, and typically also did most of the writing. The last author position is sometimes reserved for a “senior author,” who might be the PI of the lab that hosted the work or another work group leader. These are kind of squishy guidelines, and sometimes the author team does not initially have a consensus view of who belongs where in the ordering, leading to, depending on the work group climate, conflict or constructive negotiations.
A few medical journals require each author to describe their actual contributions to the published work, and include that in the publication itself. For my part, I think that’s the better way to go.
I don’t know if this is elitist or what, but I’ve had serious concerns about MDPI journals in healthcare for several years now and generally avoid citing papers published there or reviewing for them. It doesn’t help their case that virtually every time I skim an article I see in an MDPI journal, the methods are lacking and the choice of analysis often suspect.
Some senior researchers have claimed to me that they publish with MDPI as a last resort because their paper isn’t accepted elsewhere (e.g., if it’s showing null findings). However, plenty of journals accept null results on principle, and are better options than a clearinghouse designed to churn out resume padding. I suspect many people publish in MDPI because they would rather deal with the minor hassle of continuously flipping the same paper than the major hassle of reanalyzing/editing.
The real problem here is that once people see papers by Andrew Gelman in Entropy, they will start thinking that that journal and by extension the publisher are credible places to publish in. By publishing there, Andrew is helping MDPI to become credible.
At the same time, it is true that mainstream publishers like Elsevier will publish articles in their top journals that are just total and utter garbage (e.g., I believe Elsevier even has a journal called Homeopathy). Springer or Wiley do the same (wasn’t there a huge number of retractions of papers recently from one of these publishers?).
So one could argue–what does it matter that MDPI looks fishy? They’re just trying to get to the top table, where others got there before them. One could even argue that, to level the playing field between the west and developing countries, we should do everything we can to help them get there.
What I would do in Andrew’s place, if I wanted to have a legitimate published paper in a proper journal but also want to by-pass the hassle of a detailed and sometimes unfair peer review process, is to start my own journal and call it Working Papers or Technical Reports in Statistics or something; have it published by Columbia University Press or some such entity so it’s clear it’s not a home operation. This way the paper gets a doi and can be cited, but we are not feeding articles to publishers who are adding so much noise into the literature, and harming young scientists careers by making them think they can publish in those venues and build a career, given that Andrew published there. But such a journal would also make it clear to the reader that the often painful and noisy peer review process was by-passed (which, if I understand correctly, is Andrew’s main goal here in publishing in Entropy).
My own policy is to never publish in such journals. I won’t ever publish in places like Frontiers either (I have published there, but that was before I understood what Frontiers was doing). This is mainly because all my paeprs are with students or postdocs and it’s my job to make sure they get articles published in credible journals. If there is a case where I can’t be bothered to get reviews, e.g. from people who wouldn’t even understand what I’m writing, I just put up my work on arxiv or just online somewhere and am done with it.
Also, Andrew, when MDPI “invites” you to submit a paper to a special issue, they are counting on the fact that massaging a scientist’s ego and giving them external validation will usually get the desired result. One has to learn to not fall for that kind of flattery.
Shravan:
I agree that a good argument could be made that I should not have published that second article in the MDPI journal, given that I knew what they were. Just to be clear, though, the reason I submitted it was not because of my “ego” or “flattery,” it was because they were asking me in a nice way and I wanted to be helpful. And to this day I do not know whether this was a scam or whether the people who asked me were entirely sincere.
Again, I agree with your larger point, in that getting someone to contribute as a favor could be emotional manipulation of a sort; I just wouldn’t call it ego massage, external validation, or flattery.
Also, yes, I do put unpublished papers on my website, right here, and I can also publish things on Arxiv. I don’t always get around to putting things on Arxiv, and I probably should. One question I have, and I really don’t know the answer to this one either, is what benefit there is to science for me to publish in journals compared to just posting on the preprint server.
The standard list is probably
1. They are ARCHIVED. On acid free dead-tree paper (which can last up to 1000 years). In multiple university libraries. Yes, the actual brick buildings with bound journal volumes. This is MUCH better than ephemeral electronic records. Heck, I trust microfiche more than I trust the Wayback Machine!
2. The editorial process (to include peer review, copy editing) improves the quality of your paper (logic, presentation, clarity, rigor, correctness, etc.)
3. The pressure to satisfy [2] induces you to vet your own work more, even if actual changes from [2] are not made.
——-
Now, these standard rationale may not be sufficient for you. And if this is your uncertainty on the value (if it’s worth it), then fine. But you at least know the arguments, right?
Anon:
1. I think that most journals are no longer printed.
2. Yes, agreed. The problem from the other direction is that the review process is a pain in the ass, both in terms of paperwork (from registering with the journal website, all the way to filling out a zillion forms at the other end) and in substance (because something like half the review comments will be flat-out wrong).
3. Agreed.
Also:
4. Publication in an official journal can be useful to collaborators and students.
5. If something is in an official journal, it might be taken more seriously by some readers, who might feel the need to account for it in a way that they wouldn’t, if it were just an arxiv paper.
6. Desire for “closure” on the project, force of habit, etc.
[1] Ugh. I just researched this and it says ACS doesn’t print stuff since 2012. Yikes. I really didn’t know.
Oh well…I still drive a stick shift, too and was in shock when I bought my first new car in 20 years (in 2015) and found out that most cars don’t even offer “standard” as an “option”.
:(
As a reader, if I come across something interesting on a pre-print server I generally assume it’s not the finished version and so try to find the version that’s been published in a journal. Then, if I can’t find it, I make one of two additional assumptions:
(a) If the pre-print is quite recent, I assume that it’s under review somewhere and will often avoid citing it until the final version is available.
(b) If the pre-print is quite old, I assume that the author or reviewers found some problem(s) with it and it died a quiet death.
So, from the perspective of a single reader, there is value in an author either publishing in a journal OR providing some sort of disclaimer in the pre-print saying that it’s the finished version and won’t be superseded by a published copy.
BTW, your article appeared in a special issue on “Bayesianism”. I find the title of this special issue really odd.
There’s a paper in that special issue on how to compute Bayes factors robustly for data that would conventionally need one-sample or two-sample t-tests. Also, one of the special issue editors has two papers in this special issue. Another editor’s name appears on one paper in the special issue. It is extremely unusual for an editor to publish articles in their own special issue.
It all looks fishy to me.
Max Shepsi,
I think a key factor will be who wrote the paper that is on an online server like arXiv. I’ve read papers by Andrew and Michael Betancourt even if they were just online versions. OTOH if some obscure Indian I don’t know with an unpronounceable name from some no-name university in the middle of nowhere wrote a paper and put it on arXiv, well, I might look at it to see if it’s interesting, but I wouldn’t start with the assumption that it’s worth reading. It may still turn out to be though.
And it has happened quite often to me that I read a published paper in a top journal by a famous author, and ask myself, is this for real or did this person just publish this paper to see if it could get past peer review? Check out my re-analysis of a paper in a top-class journal on the benefits of different kinds of breathing exercises from Huberman’s lab; the mismatch between the data and the paper’s claims is simply hilarious. See PubPeer. I felt it was below my paygrade to even re-analyze that data, but I got curious because of the impossible statistics reported in the paper.
Because of the many examples like these, it’s hard to take peer reviewed publications for granted, and I often assume that if someone put up a paper online (esp. someone I know), then I will want to read it anyway. In linguistics, one often used to just read unpublished manuscripts; in the pre-internet era, photocopied versions of a draft by a linguist were not at all uncommon as a subject for class discussion. It’s mainly psychology that is obsessed with external validation and stamps of approval–at least, that’s my impression.
That said, the German Research Foundation (DFG) has strict policies on what counts as scientific output, and they make a strict distinction between peer reviewed and non-peer reviewed stuff when evaluating grant proposals or the results of grant proposals.
Years ago, I send a manuscript to an interdisciplinary MDPI journal*. It hadn’t known the journal or the publisher before and it was pretty non-standard for my research. But the topic of the manuscript seemed to fit really well, the journal was ranked very highly in my universities publication norm system and several colleagues had published there, too. So, I assumed it would be a good journal. The experience was incredibly, uhmmm, ‘interesting’.
Review reports (2) came in quick, were a bit awkwardly superficial although not obviously non-sensical. But the r+r was supposed to be submitted incredibly quickly (I think it was 1 or 2 weeks). Initially, I chalked that up to cross-disciplinary differences in norms, as the journal also published stuff from technical fields, which tend to have faster turnaround times.
But one reviewer comment would require a reanalysis of the data if I wanted to address it adequately, which I didn’t have access to at the time due to travelling abroad. So, I requested an extension of the deadline by 2 weeks.
The response from the editor sent off all kinds of alarms with me. No extension given, but manuscript basically already accepted and looking to publish it quickly in ‘our open access journal’ (explicitly mentioned in this way). Basically, reading between the lines, the message was clearly: just do some minor performative changes, re-submit and pay our (substantial) open access fee, and we will publish your manuscript anyway.
That did not sit well with me at all, and resulted in a late-night hotel room google deep dive on the journal and the publisher. And yup, there were the predatory publishing horror stories. First time I ever retracted a manuscript. Co-author was not too happy with me for doing so given how well ranked the journal was. Neither was my supervisory, given that I was still non-tenured, and the publication would have been a “good” addition to my cv – and our departments publication output. But I really did not want to be associated with this kind publishing scheme. Still feel like a dodged a bullet. And thankfully, published the manuscript in different journal a few months later. After a more critical revise and resubmit process.
Sure, good research can get published in bad journals, and vice-versa a lot of nonsense gets published in “reputable” journals. But we don’t have to actively support these predatory business models in academia.
*Sustainability
Andrew, just out of curiosity of your position, would you submit to MDPI again if a similar proposition would to pop up again?
K:
No, I wouldn’t. If it were a request to do a favor in that way, I’d reply that I’d be happy to help out, but I’d rather not do that through that sort of publisher. The individual people whom I worked with in the process all seemed fine, but I agree that by sending them my paper, I was contributing to the larger problem.
Reviews don’t always improve things. For example Bishop’s paper on multiverse-type questions failed to cite Gelman’s well-known multiverse paper with Steegen et al., or the Simonsohn papers on a similar topic : https://journals.sagepub.com/doi/10.1177/25152459241267904
Anyone:
It’s hard to keep up with the literature, and the multiverse idea is obvious and timely enough that it makes sense that multiple researchers independently came up with it.
I just hope that future researchers cite all these papers, because they make different points.
Fair enough, just seems odd that someone who is passionate about the topic failed to find these papers years after they were published in journals on the same topic area. A mea culpa would be nice?
Anyone:
I guess it’s always possible to do the references more carefully. When writing my books I tried really hard to include all possibly relevant references to everything we did, but even so I missed some things.
Here’s something funny. I clicked on your link, which took me to the Bishop and Hulme paper, which in its abstract referred to the R package DeclareDesign.
That package name rang a bell–maybe I’d blogged on it? So I searched the blog for DeclareDesign and found a post from 2021 entitled, DeclareDesign, which was forwarding an announcement about that package from its authors, Graeme Blair, Jasper Cooper, Alexander Coppock, and Macartan Humphreys. There was also a link there in that post, but the link didn’t work. So I googled DeclareDesign all by itself, and this took me to their webpage, which includes a link to a book of theirs on the topic. I clicked the link and then clicked on references, and . . . the list of references doesn’t include my article or Simonsohn’s!
So that seems to be the reason that Bishop and Hulme didn’t refer to Simonsohn or me–they probably just worked from the reference list at DeclareDesign.
What seemed funny at first Blair, Cooper, Coppock, and Humphreys didn’t cite my paper on the multiverse, given that I know two of those authors personally, and they directly sent me the information about their package, so they knew it was a topic that interested me. I’ve written a lot about design in the social sciences, not just that multiverse paper! Even though they cite various technical papers of doubtful relevance such as “Existence of Unbiased Estimation for the Minimum, Maximum, and Median in Finite Population Sampling.” But, looking carefully, our multiverse paper, although highly relevant to the Bishop et al. paper that cites DeclareDesign, is not so closely connected to DeclareDesign itself.
For some reason I can’t respond to your comment directly Andrew, but I agree this sort of thing is amusing. I just find it strange that myself, as an outsider to psych, was aware of your and Uri’s papers, but Bishop wasn’t even in 2023/24. We all forget things for sure, but seems to me that if you are criticising others then you should also be willing to hold your own hands up (as you are, and occasionally blog about). Thanks for engaging.