James Heathers wrote this screed (I mean that in a good way!) about the report by Harvard University on a business school professor who was involved in fraudulent research.
For some background, here’s a link to a news story on the Harvard investigation, more background is here, and here are some of my own reanalyses of those data.
From various discussions of this case, four points are clear:
1. There is no dispute that the published work in question failed to demonstrate the effects it purported to show. There were four published papers that were scientific failures—not in the good sense of reporting some failed experiment and trying to figure out what aspect of theory, experiment, measurement, or analysis went wrong, but in the bad sense of making a confident claim and saying it was borne out by experiment, even though it wasn’t.
To give an analogy to news reports, imagine three possible news stories:
(a) An investigation was performed into something suspicious, nothing came of the investigation, and the failure was reported.
(b) An investigation was performed into something suspicious, nothing came of the investigation, and no report was released.
(c) An investigation was performed into something suspicious, nothing came of the investigation, but it was reported as if the investigation was successful.
Option (c) is the worst! If a news organization is going to do option (c), they’d be better off not pretending to do reporting at all. It would be better for them to just run stories about fires and funny animals and other traditional local-news staples.
2. The erroneous published work had a wide audience. These papers and related work were published in top journals, they supported careers at leading research institutions, they were promoted by celebrities and featured in major media outlets.
3. There is no dispute that there was fraud in the data processing and analysis of these studies. The four published papers in question were not just a blight on science, they were an intentional blight on science. I don’t care so much about this—recall Clarke’s Law—but, sure, it’s part of the story.
4. There is some residual dispute about who was directly involved in the fraud. As Heathers puts it, there are various “unconvincing” but “technically possible” alternative explanations for how exactly the data were manipulated. In the words of the Harvard report, the research conduct was conducted “intentionally, knowingly, or recklessly.” I guess that, of those three, “reckless” is the most positive option.
In decreasing order of importance
Heathers also writes about a separate investigation by Duke University of another business-school professor involved in this research:
The dataset was massively and provably falsified. . . . There is not enough evidence to determine [who did it].
Ultimately, the who-did-it question is less important than the what-happened question and the how-did-this-stuff-get-promoted-on-NPR-etc. question. Here’s Heathers:
The question of “can the work be trusted?” is not a personnel matter. It is an “everyone in the scientific community” matter. It is an “are we ruining the legacy of the Enlightenment” matter. . . . But the real game is in the actual harms that come from putting out untrustworthy research. These start with the broken degrees or careers of junior collaborators whose efforts have been wasted. They continue through the expenditure of further thousands of hours, and millions of dollars, on pointless follow-up research. And then, sometimes, but more often than you’d hope, they kill people. . . .
As I said before, it’s rational for an institution to want to know [who was responsible for the research misconduct] . . . And, yes, I think it’s justified to keep this bit private if you want, because it IS a personnel matter.
But the work itself is an everyone-else-as-well matter. And sewing the two together just lets you hide the mechanics behind the HR, keep it all from public view, and avoid scrutiny.
As usual. For shame.
Here’s my restatement:
In decreasing order of importance:
(1) The error,
(2) How the error persisted,
(3) The research misconduct,
(4) Who did it.
Unfortunately, as Heathers says, the emphasis is often in the reverse order.
The only problem with this ordering is that it fails to hold anybody accountable. (4) is important because without it, it is hard to imagine much changing. I agree that it can be an unwelcome diversion from those other important questions, but I think responsibility is important.
Dale:
I agree that all four are important. I think #4 is least important, but the importance is not zero.
Also, good point about the difference between direct and indirect importance. In the moment, #4 is the least important, but my above analysis is static.
The linked page on Substack is no longer available.
Found the archive here: https://web.archive.org/web/20241216082155/https://jamesclaims.substack.com/p/the-gino-report-just-makes-me-angry
So he deleted it in the past month? Weird.
There may be a good reason to delete it. I don’t think it’s fair to the author to publicize a post he decided not to share.
He did decide to share it, because Andrew found it and the Internet Archive crawler of the open web found it. (Presumably, it was also emailed to all his substack subscribers). Then, he decided to stop sharing it; maybe he regretted it, or maybe he finally sold the piece to a magazine, but they requested he delete any drafts posted online. Or maybe Substack deleted it for reasons of their own (which he didn’t necessarily agree with). Or maybe someone pressed the wrong button somewhere and the deletion was intentional. So, in the absence of any other information, I don’t think it’s a moral wrong to post a link to an archived copy of a now-deleted public article.
Was it the author’s decision? Pages sometimes get removed due to lawsuits. I recall a recent Substack post about just that, which subsequently itself disappeared.
I need grammatical help regarding this sentence:
(c) An investigation was performed into something suspicious, nothing came of the investigation, but it was reported as if the investigation was successful.
Because of the tone and the implication of lack of success, shouldn’t that be
(c) An investigation was performed into something suspicious, nothing came of the investigation, but it was reported as if the investigation were successful.
The subjunctive in English is tricky, and perhaps already dead, and I am not entirely sure that I am correct.
https://en.wikipedia.org/wiki/English_subjunctive
Yes, you are correct. The subjunctive mood dictates that it should be “were” rather than “was.”
But I think this is a pretty common mistake in both written and spoken word, also among native speakers.
I’d say the critique of Harvard’s report here is something like a book reviewer who complains about a book not being about what they wanted to see addressed, rather than evaluating the book for what it is. Or maybe like pundits who want a big criminal case to be about deep social issues, rather than if the defendant did the deed. The Harvard investigation had the mandate to focus on items 3 and 4 (basically, guilty or not), not e.g. “how-did-this-stuff-get-promoted-on-NPR” – that’s someone else’s problem. The same way a criminal trial isn’t much about why social services failed a perpetrator. Don’t get me wrong, that’s a good and important question to ask, but it’s not the question before a court.