Dale Lehman writes:
I missed this recent retraction but the whole episode looks worth your attention. First the story about the retraction.
Here are the referee reports and authors responses.
And, here is the author’s correspondence with the editors about retraction.
The subject of COVID vaccine safety (or lack thereof) is certainly important and intensely controversial. The study has some fairly remarkable claims (deaths due to the vaccines numbering in the hundreds of thousands). The peer reviews seem to be an exemplary case of your statement that “the problems with peer review are the peer reviewers). The data and methodology used in the study seem highly suspect to me – but the author appears to respond to many challenges thoughtfully (even if I am not convinced) and raises questions about the editorial practices involved with the retraction.
Here are some more details on that retracted paper.
Note the ethics statement about no conflicts – doesn’t mention any of the people supposedly behind the Dynata organization. Also, I was surprised to find the paper and all documentation still available despite being retracted. It includes the survey instrument. From what I’ve seen, the worst aspect of this study is that it asked people if they knew people who had problems after receiving the vaccine – no causative link even being asked for. That seems like an unacceptable method for trying to infer deaths from the vaccine – and one that the referees should never have permitted.
The most amazing thing about all this was the review reports. From the second link above, we see that the article had two review reports. Here they are, in their entirety:
The first report is an absolute joke, so let’s just look at the second review. The author revised in response to that review by rewriting some things, then the paper was published. At no time were any substantive questions raised.
I also noticed this from the above-linked news article:
“The study found that those who knew someone who’d had a health problem from Covid were more likely to be vaccinated, while those who knew someone who’d experienced a health problem after being vaccinated were less likely to be vaccinated themselves.”
Here’s a more accurate way to write it:
“The study found that those who SAID THEY knew someone who’d had a health problem from Covid were more likely to be SAY THEY WERE vaccinated, while those who SAID THEY knew someone who’d experienced a health problem after being vaccinated were less likely to SAY THEY WERE vaccinated themselves.”
Yes, this is sort of thing arises with all survey responses, but I think the subjectivity of the response is much more of a concern here than in a simple opinion poll.
The news article, by Stephanie Lee, makes the substantive point clearly enough:
This methodology for calculating vaccine-induced deaths was rife with problems, observers noted, chiefly that Skidmore did not try to verify whether anyone counted in the death toll actually had been vaccinated, had died, or had died because of the vaccine.
Also this:
Steve Kirsch, a veteran tech entrepreneur who founded an anti-vaccine group, pointed out that the study had the ivory tower’s stamp of approval: It had been published in a peer-reviewed scientific journal and written by a professor at Michigan State University. . . .
In a sympathetic interview with Skidmore, Kirsch noted that the study had been peer-reviewed. “The journal picks the peer reviewers … so how can they complain?” he said.
Ultimately the responsibility for publishing a misleading article falls upon the article’s authors, not upon the journal. You can’t expect or demand careful reviews from volunteer reviewers, nor can you expect volunteer journal editors to carefully vet every paper they will publish. Yes, the peer reviews for the above-discussed paper were useless—actually worse than useless, in that they gave a stamp of approval to bad work—but you can’t really criticize the reviewers for “not doing their jobs,” given that reviewing is not their job—they’re doing it for free.
Anyway, it’s a good thing that the journal shared the review reports so we can see how useless they were.
This is a standard aspect of NHST bizarro science. Collective opinion determines the scientific opinion rather than vice versa.
I mentioned it the other day regarding babies and pain until the 1980s. And more recently about the wholly implausible and unscientific claim that the vaccines had some meaningful effect on transmission.
Only when the vast majority of people experienced otherwise did “the science” (crappy studies that ignored testing rates) match reality. Then the confound reversed (vaccinated started getting tested more often), so they just stopped talking about it.
I didn’t read the above study but it sounds like a less polished version of the standard junk.
I bow to no one when it comes to following Alex Jones daily and his shameless pushing of antivax stuff. But I was unaware of any of the details being discussed today. However, I did find this by David Gorski and his posting is dated October 23, 2023–today!!
https://sciencebasedmedicine.org/msu-economics-professor-mark-skidmore-exonerated-over-his-retracted-paper-claiming-that-covid-vaccines-killed-278000/
It begins with, “If there’s one thing that I’ve learned over the last two decades dealing with antivax propaganda, it’s that bad papers written by antivax ideologues designed to promote a narrative that vaccines are dangerous and/or ineffective (but mostly dangerous) never die.”
Somewhere in the middle is, “So science-y! So statistics-y! So wrong!”
The article ends with “[Skidmore’s] cachet in the antivaccine movement has been bolstered because now not only does he have a narrative of having been “censored” when his original manuscript was retracted but he now has added a narrative of “exoneration,” even if it requires publishing in an antivax journal that, I daresay, before the pandemic he never would have considered, recognizing it for what it is.”
Paul,
There are a lot of people out there who are somewhere on the convex combination of deluded and political hackery, so it stands to reason that some of them will be tenured professors at respected universities. On the plus side, that guy doesn’t seem to have been involved in the Jumbotron fiasco.
Ideally, someone could create a special medical school with him, Dr. Oz, Dr. Anil Potti, Scott Atlas, Naomi Wolf, Andrew Wakefield, the Surgisphere team, that dude from Ohio State who keeps suing people getting papers retracted, etc etc etc, all in one place, to try to contain the damage.
Maybe I am the only one who thinks this, but I do not understand the problem with a picture of Hitler on the Jumbotron. It was a quiz question and the question was permanently displayed next to his picture. It was not promoting anything he did or said. I see no reason not to ask a question about him. Except, perhaps, that it might kill the mood to remind the crowd of a man responsible for literally millions of atrocities. On the contrary: I would love to see someone who did not know go to Wikipedia and read up on Hitler. Let them learn (or remember) everything he did. Could there be any more potent medicine against autocratic tendencies than a reminder of what happens when people vote far-right groups into power? What am I missing that makes people so upset?
Full disclosure: I have literally only attended a single sporting event in my life, so I have no idea what would be usually shown on the Jumbotron.
I had the same reaction. I do think it was a stupid choice given the realities of public reactions, but I hardly think it needed to be apologized for. I, too, would rather people remember and know than pretend it didn’t happen (which a disturbing number of people don’t believe it happened).
Raphael:
Follow the link. From the article by Dan McQuade:
I agree with McQuade. MSU is an educational institution, and here they have a great opportunity for some student group to create fun and informative video for the Jumbotron. They have a whole College of Communication Arts and Sciences! Instead they have some intern pulling up crap from Youtube? Really, what’s the point of that?
@Andrew:
I completely agree with your point. The misleading PR could indeed be called a fiasco. And posting videos without credit is unworthy of an institution of higher learning.
Two points on the McQuade comment:
1. McQuade’s main point of criticism was the presence of Hitler on the screen, not MSU’s handling of the matter:
“A video question about this is kind of weird; a Jumbotron Hitler image is unfathomable. I suppose this is a good rule of thumb: The larger the image of Hitler, the less appropriate it becomes. A Hitler question in a bar, or on a little trivia video on YouTube? I’d avoid it, personally, but it’s probably fine. But if you put Hitler on a 5,359-square foot Jumbotron, it is not right. You just shouldn’t do that. This is something to bear in mind, if you didn’t know before. Please don’t do that (put a giant Hitler image up) if you ever happen to find yourself in control of a big screen.”
Apparently some people find it offensive that Hitler is on the big screen, which is why this episode became a thing. I still fail to see what makes a trivia question about Hitler problematic.
2. I fail to see how McQuade would know that “the intern” (your words, not McQuade’s) never saw the video. He is quite adamant about this in his opinion piece:
“This confirms the obvious: Someone at Michigan State threw up a random YouTube quiz on the Jumbotron without asking the creator or even watching the video, and that led to Hitler showing up on the scoreboard.”
Maybe that someone watched the video but considered it to be unproblematic? Perhaps it is just lack of imagination on McQuade’s part? I think he made an unjustified leap to a conclusion.
I like the comments there where people mention they aren’t “biostats experts”. As if there might be some dark art method of transmuting internet survey responses into a measurement of mortality.
People really believe if they manipulate numbers just the right way they can skip working out the precise consequences of their theory and collecting the right data.
Also, I looked at the survey. It starts:
Q1 Have you had COVID-19?
o No. (1)
o Yes. (2)
How about “Unknown”?
Anon:
What we need are questions at the next level of abstraction. For example:
Direct measurement: Ask respondents to list everyone they know. Then track these people down, test them, and estimate whether they ever had a health problem from covid.
Indirect measurement: Ask respondents if they know someone who had a health problem from covid.
Super-indirect: Ask respondents, “If we were to ask you if you knew someone who had a health problem from covid, how would you respond?”
Super-super-indirect: Ask respondents, “If we were to ask you how you would respond if we asked you if you knew someone who had a health problem from covid, how would you respond?”
Etc.
Reminds me of Meehl:
https://www.jstor.org/stable/186099
Ask respondents to list everyone they know, along with how they think each of those acquaintances would respond if asked if they knew anyone who had a health problem from COVID.
I see this as a call to structure peer review through novel methods. Sketching out the ontic, theoretic, empirical, and practical aspects of research as well as their connections would be helpful. It seems like the construct validity issue from the paper above would be much easier to identify if we reminded peer reviewers of its importance. The lack of a meaningful and rigorous process permits laziness. In high-stakes domains like surgery and aviation checklists are a routine part of the job. In research, one might consider something like a checklist: a sense-making guide to promote discussion of key content.
How about author makes a list of assumptions they are making for each claim. Next the peer reviewers try to think of any others, then there is some back and forth about the final list. Maybe some assessment of the plausibility of each assumption can be included as well.
But really, in this case, the problem is that we simply have no method for really assessing vaccine safety. Stuff like VAERS, or people reporting side effects in an app, or asking people in a survey, are easily dismissed due to the dubious assumptions required. The RCTs are ended early, and exclude anyone most likely to die. So even 20% higher mortality rates are considered “not significant”. And I see no interest in remedying this situation.
It is very scary to think of how much damage the government and healthcare industry can do next time. Or even look at the ongoing reversal of life expectancy since Obamacare got so many more people on blood pressure meds, pain killers, antidepressants, etc. And the FDA overriding their own science committee to approve Alzhiemers drugs that don’t even seem effective according to the weakest standards.
Agreed! The counterfactuals underlying each decision in the decision tree of the research are critical and should generate limitations at each point.
The RCT errors/omissions in reporting could be handled by GRADE risk of bias (roB) tool, right? Seems like it’d be a start but I’m unsure how broadly it’s used even in medicine. Kind of a neat problem to think about how to persuade journals to adopt a core set of quality appraisal methods…
The political issues you mention are a whole other problem that would require a whole other kind of intervention.
I think it is more about “tell me what you are assuming so I can figure out how you think this data was generated.” Things that really work require a minimum of jargon. Like seeing a comet when it was predicted, or pulling a ship out of the harbor using your own simple device.
The superficiality of the reviews (and the astoundingly bad English) is just breathtaking. So this is what it’s like to publish in the journals from the Nature group of Springer?
What I could not find out was who the action editor was. This person was ultimately responsible for accepting this paper based on these ridiculous reviews. The journal carefully hides that information, even though they out the second reviewer.
I found the editor, someone called BIRLUTIU:
https://static-content.springer.com/openpeerreview/art%3A10.1186%2Fs12879-023-07998-3/12879_2023_7998_AuthorComment_V4.pdf
You can scroll down here and find Birlutiu
https://bmcinfectdis.biomedcentral.com/about/editorial-board
All review reports should be public.
Maybe all the respondents knew the same sick individual.