When I say “passive corruption,” I’m talking not about the people who directly cheat, but about those who know about cheating but don’t do anything about it, I suspect out of some mixture of the following motivations:
1. Not wanting to waste any more time or attention on bad work
2. Fear of the social or professional consequences of confronting cheaters
3. Concern that the general air of skepticism will spread to their own work.
I was thinking about this in light of Stephanie Lee and Nell Gluckman’s new article, “A Dishonesty Expert Stands Accused of Fraud. Scholars Who Worked With Her Are Scrambling,” following up on the story we discussed a couple days ago of the Ted-talking data fakers who write books about lying and rule-breaking. Lee and Gluckman write:
To Maurice E. Schweitzer, a University of Pennsylvania business professor, it seemed logical to team up with Francesca Gino, a rising star at Harvard Business School. They were both fascinated by the unseemly side of human behavior — misleading, cheating, lying in order to profit — and together, they published eight studies over nearly a decade. Now, Schweitzer wonders if he was the one being deceived. Gino is on administrative leave from Harvard amid allegations that research she co-authored contains fabricated data . . .
The revelations have shaken and saddened the behavioral-science community. . . . some are looking with suspicion at the dishonesty researcher they once knew and trusted, a deeply disorienting sensation. A prolific body of studies, a record of headline-grabbing results, a dedication to running experiments on her own: These once looked like the hallmarks of a model scholar. . . .
“There’s so many of us who were impacted by her scholarship, by her leadership in the field,” Schweitzer told The Chronicle, “and as a co-author, as a colleague, it’s deeply upsetting.” . . . By one count, she has 148 collaborators. According to her résumé as of August 2022, she has published more than 135 articles since 2007, many of them in the field’s top journals . . .
Kouchaki and Galinsky are among Gino’s most frequent collaborators, having worked on, respectively, at least 14 and seven papers with her. Neither responded to requests for comment. . . .
From late 2016 to 2019, Gino was the editor of Organizational Behavior and Human Decision Processes, where she characterized herself as a proponent of “open science” efforts to solve her field’s replication crisis. . . .
Gino’s dozens of papers about ethical leadership and workplace behavior, done with other scholars at elite business schools, led to countless speaking and consulting gigs . . .
In 2019, an outside team published a meta-analysis of studies about dishonesty, including several of Gino’s, and tried to obtain the original data for each of them. For 10 papers that listed Gino as the first author, the team doing the analysis reported being told that the underlying data was unavailable. . . .
[Gino’s collaborator] says he did with Gino what most academics do: trust each other. “I don’t tell my Ph.D. students, ‘Never plagiarize work, never make up data,’” he said. “I assume that’s obvious.” But in hindsight, he acknowledged that it would have been better to supervise the data collection more closely. “Clearly we need to be more vigilant and less trusting than we’ve been,” he said. . . .
Still others say they are reserving judgment until they finish digging into their past work. “I am waiting to learn more about this case,” Juliana Schroeder, an associate professor at UC Berkeley’s Haas School of Business, and a seven-time collaborator with Gino, tweeted over the weekend. “It is extremely concerning.” . . .
I have a problem with this narrative.
OK, just to be clear, I don’t have any problem with Lee and Gluckman’s reporting, nor do I have a problem with the unfortunate collaborators of the researcher who was making up data. I, too, have been in collaborations where I’ve never looked at the raw data and I was never involved in the data collection. It really is all about trust, and anyone can get conned by someone who is willing to lie about things. As we discussed in our earlier post, there’s this weird thing where many prominent cheaters don’t just cheat, they also seem to love to talk about it, writing books with titles such as “Evilicious: Why We Evolved a Taste for Being Bad,” “How We Lie to Everyone—Especially Ourselves,” “Is Everybody Cheating These Days?,” and “Why it pays to break the rules at work and in life.” These people either like to live on the edge and taunt the rest of us, or they sincerely believe that “everybody is cheating.” If you’re a cheater and you regularly lie to your friends and collaborators and you write books about how it pays to break the rules, then maybe you think that normies are just saps, the academic equivalent of tourists walking around in Times Square in Bermuda shorts with wallets hanging out their back pockets.
Ok, then, so here are my problems with the narrative of the behavioral scientists who now are discovering they were too trusting:
First, these people were doing all this research on cheating. They were collaborating with a researcher who was writing books and giving speeches on how everyone’s a cheater. So why would they think it’s ok to follow blind trust? It’s almost as if they didn’t believe their own research! In tech terms, they weren’t eating their own dogfood.
Second, this has happened before and I remember how it happened. Various past cheaters got tons of institutional support. Marc Hauser finally got kicked out of Harvard, but that didn’t stop superstar academic linguist Noam Chomsky from continuing to defend him, nor did it stop superstar academic psychologists Susan Carey and Steven Pinker from supporting his later iffy venture; see P.P.S. here. Brian Wansink got kicked out of Cornell but it took awhile, and, before that happened, various people were saying we should go easy on him because he was such a nice guy. The cheater was defended by the tone police. When the problems with Matthew Walker’s sleep research came up, the University of California didn’t care. Maybe Walker was a nice guy too. With Dan Ariely it was the opposite story. After his disappearing-data problem arose, I asked around informally within the decision science community, and it seemed that there was a consensus that this guy could not be trusted. But I guess this scuttlebutt didn’t reach the administration of Duke University, nor did it seem to concern the editors of the Wall Street Journal.
Anyway, here’s my point. These people were writing papers and books about cheating, they had cheaters in their midst, they still have cheaters in their midst. And that’s not even starting on all the bad research where there’s no data fabrication or outright lying, just producing useless unreplicable research using forking paths. It’s common knowledge in the behavioral science community that tons of crap is out there and will never be retracted, also lots of people don’t speak up when they are aware of dodgy behavior; see items 1, 2, 3 above.
So here’s the deal. I’m not saying that most or even many behavioral researchers are liars, cheaters, or frauds. What I’m saying is that this is an academic community that’s consistently looked away or downplayed lying, cheating, and fraud in its midst.
As I put it a year ago in my post, Should we spend so much time talking about cheaters and fraudsters?:
Science is kind of like . . . someone poops on the carpet when nobody’s looking, some other people smell the poop and point out the problem, but the owners of the carpet insists that nothing has happened at all and refuses to allow anyone to come and clean up the mess. Sometimes they start shouting at the people who smelled the poop and call them terrorists. Meanwhile, other scientists carefully walk around that portion of the carpet: they smell something but they don’t want to look at it too closely.
A lot of business and politics is like this too. The difference is, we expect this sort of thing to happen in business and politics: rulebreakers keep doing it over and over again. Science is supposed to be different.
P.S. I’m not saying that statisticians and political scientists have any moral superiority to psychologists and experimental economists. It just happens to be easier to make up data in experimental behavioral science. Statistics is more about methods and theory, both of which are inherently replicable—if nobody else can do it, it’s not a method!—and political science mostly uses data that are more public so harder to fake (exceptions such as Mary Rosh and Michael Lacour aside).
P.P.S. Here’s another example. The first edition of the Nudge book promoted the research of Brian Wansink, which they called “masterpieces.” Then after that work was discredited, the Nudge authors removed it from their second edition. Removing work that’s known to be fatally flawed—that’s a good thing to do. But the bad thing is that, in this second edition, there’s no mention of their mistake from before! They memory-holed their earlier cheerleading for work that turned out to be fraudulent. They’re not rewriting history exactly, but they’re framing things as if the error had never existed, thus losing an opportunity to confront their error. To connect this to the main point of this post, yes, they got fooled, but also they’re setting themselves up for future problems by looking away from the problem.
As with the City of Flint’s water, universities are sooner or later going to have to find institutional ways of dealing with this toxic stuff.
We could look to Harvard, where a critical mass of faculty scandals warrant a look a the big picture. This was a very incisive post.
This was already solved long ago: Nullius in verba.
Make direct replications standard before you count on it (conceptual replications serve a different purpose and don’t count). Ideally the replicators even have some degree of competion (or even animosity) with each other so there is incentive for it to fail. Then you can really trust whatever both agree on.
We have a relatively few cynical people taking advantage of the low standards used by a huge crowd inadvertantly generating vast amounts of misinformation and teaching it to each other.
Whether fraud or not knowing the important aspects of the experiments, it is same result in the end.
This is all very true. But there’s a real question of “whose job is it” to police researcher integrity? There are no resources or systems in place for this. Is it other researchers? Universities? Journals? Funding agencies? A 3rd-party ethics org?
There is a wide gulf between *suspecting* someone of fraud, and confirming/convincing other of it. Raising suspicions (to whom? department chair? journal editor?) will do nothing (but harm the informant politically) without strong evidence (didn’t Data Colada create 90 pages before going to Harvard?) and enough credibility an audience will be receptive. Calling a fraudster out in a meaningful way requires access, TIME, and usually, reputation.
Those likely to have the most access to a fraudster’s data and research process are likely to be those with the least power and credibility: grad students. In most cases no one would listen to them, and political cost of trying would be huge (at best, lost advocacy leading to failure on job market; at worst, kicked out of the program, losing health insurance, and having a “failure gap” on cv).
For junior profs on the cut-throat tenure track, investing the TIME to gain definitive evidence directly takes away time developing research needed to stay employed — one paper can be the difference between tenure and getting fired. Perhaps some high integrity APs have tried in the past and we just don’t know — their complaints never surfaced to wide-scale attention, and they got weeded out of the system for focusing on something other than advancing their own research (and likely made some enemies).
For tenured profs, many have been entrenched in the system so long by that point, it doesn’t occur to them to change it. Or, they don’t want to risk waves in the field they worked so long to be beneficiaries of. Or, they have a portfolio of friends/favors accumulated that would create cognitive dissonance to breach. Or, they are focused on making full professor or moving to a new institution. Or, maybe they got tenure because they succumbed to fraud themselves, and of course now can’t start outing it.
So it falls to a small number of self-appointed data cadets, who are secure in their employment and success, and can afford to spend substantial time and reputational capital weeding out examples of fraud. The privilege of having the opportunity to do this is very rare; and of course, the willingness to do it both admirable and a boon to science. But it feels like it will just be fraudster whack-a-mole until more systematic checks are adopted.
Perhaps the onus should simply be on direct collaborators to verify there is no fraud in their individual projects. No more “I never saw the data” — everyone has to be fully responsible. But there are so many places fraud could take place! Not just the data, but within the experiment itself (was there really no nudging by condition? was the confederate really hypothesis blind?). And given what a race publish or perish has become — let s/he with the most A pubs at tenure time win all — one can imagine how those who took the time to check over every bit of every collaborator’s work would get weeded out of the race, over those who took the calculated risk that the time saved by not doing so was more valuable than the risk that fraud would both be performed and caught.
With all of this, I do think it’s clear that the current system of default assumption (by universities, journals, collaborators) that researchers are 100% honest unless proven otherwise (through a long, difficult, and unlikely post-hoc process) is not working for science.
Perhaps researchers could just sign an honesty pledge at the top of their paper submissions. More realistically, perhaps having a highly skilled methods-and-data audit team that audits paper submissions at random might help keep more people in line. Like the IRS for researchers. My initial thought is that is should be paid for by the journals as part of the integrity value of the journal (and these journals make enough!) or a non-profit for an open access. Severe transgressions uncovered could be reported to the offending researcher’s university, in addition to blocking the paper.
It’s not perfect, and there is of course the potential for corruption and back patting. But surely fewer people cheat on their taxes than if there was no possibility of IRS auditing at all?
To get something like this to happen, journals would need to feel there was enough pressure to improve scientific integrity to motivate the expense. And of course, hiring would be hard! We might need to start a new degree program in data sleuthing…
This is an excellent comment, and the lack of “policing” or verification is important for uncovering incompetence as well as dishonesty. I disagree that journals should, or could, be investigators. One would think that funding agencies would be the entities that should care, and I’ve long been amazed that they are so passive — about this and many other things. (I haven’t looked up whether these cases involve NSF or NIH funding, so this is a more general comment.)
Sandra, Raghu:
Yeah, just to be clear, I don’t see any easy solutions. I’m pretty outspoken, but even for me, I will run into issues 1 and 2 above. And there’s also the flip side of this, which is that we’re typically so careful about not accusing researchers of cheating or malpractice, that it’s often only the most ridiculous examples that get attention—not just cheating or extreme incompetence of the Wansink variety, but also work with glaringly obvious flaws such as the ESP experiments, the beauty-and-sex-ratio analysis, and the himmicanes study—and this implicitly lets off the hook all sorts of run-of-the-mill bad science.
A quick rule of thumb for now is that we should be suspicious of anyone who writes books and gives Ted talks saying that everyone cheats. The trouble is that there’s tons of bad research out there being done by people who aren’t like that, they’re just people who are Greshaming our scientific discourse with low-quality work.
Just to continue on this point, another problem is what I’ve called the “research incumbency rule,” which is that, once a story is told, the burden of proof is on other people to disprove it. So, if a researcher makes some ridiculous claims and they are published, there are lots of barriers to others making the case that there’s a problem with the claims, let alone that there might be fraud. I’m not saying it’s impossible to make the case that published work is wrong—I’ve done it many times, as have Simonsohn et al.—but there’s a high burden of proof. You have to come in with really strong evidence, much stronger than the evidence for the original claims.
Well said! It’s such a shame too, when editors (including now-revealed fraudsters) of the top journals select for clickbait.
” One would think that funding agencies would be the entities that should care, and I’ve long been amazed that they are so passive — about this….”
Every incentive for fund agencies is just the opposite. :)
The more they spend, the more they can justify asking for. The agency doesn’t care what it gets for its money, it only cares how much it spends and the more the better. Outing bad work or fraud among an agency’s researchers is more likely to get their budget cut.
Until some senator gets a wild hair and decides to crack down, the funding agencies won’t do anything. Right now, no senators are jumping out of their seats to crimp the flow of cash to MyState University.
Anyway, no matter what the “stasi” say, Ted Talk, Hidden Brain, Freakonomics and similar stories all generate far more positive than negative publicity. They’re tailor-made to appeal to the junk-science audience, who loves to hear BS about how standing in front of a mirror and self-puffing increases your job prospects, or that Big Food has a massive evil plan to addict everyone to Flamin’ Hot Cheetos, or that Conservative women are hotter or whatever other snake oil gets pedaled in these quack forums.
So, don’t count on anything happening at funding agencies until the headlines on CNN are blaring about junk science.
Last but not least most scientists are reticent to expose fraud and bad work because:
1) most of them believe that it’s kind of right anyway – like the idea that the population bomb was really right, just bad timing
2) they don’t want to bring scrutiny down on science in general and especially their own work
3) they don’t want to bring scrutiny on hypothesis or claims that scientists mostly want to be widely accepted, like climate doom (many scientists don’t believe in climate doom, but would be reticent to publicly oppose it, fearing it would reduce the pressure for public policy to reduce CO2 emissions), environmentalism in general, critical race theory or the general idea that discrimination is widespread and needs to some intervention for remedy.
The big picture reality: at this point, attacking junk science is perceived as being against the interest of science in general. Until that changes, the Stasi will have to keep yelling from the sidelines.
Fantastic comment. I think it’s everyone’s job to police researcher integrity to some degree, and that the most immediately impactful solution would be wide adoption of registered reports, and organizations like Peer Community In Registered Reports are on the forefront of this. They’re a free, academic-led service that facilitates the RR process, and partnered journals will publish accepted Stage 2 RRs with no further peer review. Chris Chambers has a great slide about how the step of the scientific method scientists should have the least control over is the results, but the results are also the step most important to having a career. The incentives are perverse and I think we’ll look back on this era as another case of Arendt’s Banality of Evil.
Sandra – I think you have hit on the key solution. Many journals now require author statements regarding who did what work, and attesting to the accuracy of their work. But these statements are always statement appended after the submission. Clearly the statements should come at the beginning, before the paper. That would reduce the fraud by a large amount – a simple nudge will do it.
Dale, You are a genius! Will you come speak at my corporate event? Will pay you $100k.
Sorry, he’s too busy attending hundredth birthday parties of socialite / war criminals to deliver any talks at this time for less than $200K.
Dear Andrew, have you read the 3rd Data Colada post on Twitter? Yes, it seems like there is tampering with data. But as an econometrician/statistician, I am becoming increasingly concerned with what these simulations are trying to show. I have read Uri’s pieces in the past but now I struggle even more to understand why the null hypothesis they set out to reject is actually the null hypothesis they consider. https://twitter.com/otiliaboldea/status/1672319924260184065?s=46&t=1wJMlIZzTaI79_PrOu4Fdg
Here is the code they ran: https://datacolada.org/appendix/111/Post%20111%20-%20Falsificada%203%202023%2006%2022.R
Feel free to formalize their procedure and let us know if they ran an inappropriate testing procedure
Otilia:
I haven’t looked at that post, but just to speak generally I can say that Simonsohn and his collaborators are creative researchers and sometimes come up with their own statistical methods. As often happens when applied people make up new methods, some problems can arise: they’re beta-testing their own ideas, as it were. So I can well believe that they’re doing useful things with the data and also that there could be better ways of attacking the problems at hand. Again, I’m speaking in general terms here, thinking about this sort of thing.
Otilia, I read the post and also your twitter thread. Regarding their simulations, my impression is okay maybe there’s something here but what is it? If both samples (cheaters/non-cheaters) are drawn from the same population, then the p-value for the KS test statistic should be uniformly distributed. So what do they think a large p-value is saying?
I’d like to see them take this simulation further, i.e. 1) generate two samples from the same distribution and record p-value for KS test statistic, 2) randomly choose 13 observations to change such that the mean is increased by x, then 3) leaving those 13 observations changed, randomly choose 13 observations to change such that the mean is increased by x and record p-value. Do 1) and 2) ten thousands times and for each time you do 1) and 2) do 3) ten thousand times.
Especially when you’re relying on simulations without a clearly delineated theory, I think this is a crucial step. It’s a way of validating your method, and I think sometimes they miss this.
“There’s some top-tier irony in their allegations. Gino is considered a top expert on ethics and human behavior and yet is accused of fabricating data in a study about… dishonesty. She authored a book about the benefits of rule-breaking called Rebel Talent. And about a month ago, Gino wrote in the Harvard Business Review about making “better, more informed decisions.”
Though some professors publicly expressed horror at the scandal, others weren’t so quick to condemn. Sa-kiera Hudson, an assistant professor in Management of Organizations at the University of California Berkeley’s Haas School of Business, suggested that while fraudulent studies should be called out, the real problem is the “folks who decide to police what is real vs fake science.”
“Neighborhood watches function similarly. And while maybe they work sometimes, there’s A LOT of room for error,” Hudson said.
Later on in her sanctimonious Twitter thread, Hudson used the phrase “shut and dry.” Incredible scholarship, Professor Hudson.”
https://app.thespectator.com/2023/06/22/is-academia-rotten-to-the-core/content.html
Mike:
Hudson seems to be tapping into what I’ve called the Javert paradox, the idea that there’s something suspect about tracking down problems in science. I think part of this is an problem of “availability,” as they say in psychology. It’s easy to visualize the high-profile articles that were published, not so easy to visualize the work of all these young researchers that didn’t get published in top journals because those young researchers were not willing to make up data or p-hack or whatever.
It isn’t just the social sciences; I see a lot of the same in environmental sciences. Basically, it is a problem of incentives; you get precious little reward for blowing the whistle. Here is the last slide of a talk about a bogus study of a fishery on the Mekong River that I gave recently at a NOAA Fisheries lab:
Recognize the dual nature of scientific activity.
On the one hand, we are trying to figure out how the world works. On the other, we are trying to get ahead, to earn promotions or bonuses or better jobs, or to help our graduate students or post-docs. Only an economist could believe that some invisible hand will reliably transmute the latter activity into the former. Rather, we should recognize that both activities go on at once, and should read articles carefully and critically before we take them seriously.
De omnibus dubitandum (Doubt everything).
John:
Too bad it wasn’t a bogus study of a fishery on the Yangtze river, or they could’ve linked up with the bogus econ literature.
I think it’s *even worse* than you described in this post. Although I cannot provide empirical evidence, my experience from within b-school behavioral research is that there are FAR more active perpetrators than passive complicits. Anyone entering the field in the past 20 years was indoctrinated into “the way.” At present, the Ginos, Arielys, (and others I have witnessed but haven’t been publicly called out yet) etc. have massive power and influence as full professors and journal editors, and the younger rising stars filling in beneath them started as their grad students. Like chooses like, and when those who “succeeded” in this culture make hiring decisions, invite speakers, review papers, and make accept/reject decisions, they select atheoretical noisy clickbaity work of a particular “type”, and expect a production speed generally incompatible with high-quality science. (Not to mention that accepted work generally must “build off” bogus theories previously published by those in charge now.) I wish I had a more optimistic view of things, but it feels increasingly like there’s only one way to play, at least in certain sub-fields.
Sandra:
I haven’t heard about these other examples of outright cheating (as opposed to flamboyant promotion of work that is bad but without the data being fabricated or misrepresented), but I’m pretty much an outsider in that field so there’s no reason I would’ve heard about it.
Reminds me of the reaction to the recent exposé by Charles Piller about the apparent manipulation by Sylvain Lesné in his landmark Alzheimer’s paper about Aβ*56 oligomers (https://www.science.org/content/article/potential-fabrication-research-images-threatens-key-theory-alzheimers-disease). Amid a pile of reactions from other researchers claiming that everything was still mostly fine and they all knew the research was suspect all along, I found this comment particularly revealing. From https://www.alzforum.org/papers/blots-field.
https://twitter.com/ClicksAndHisses/status/1551844775409553408
https://twitter.com/ClicksAndHisses/status/1553605398317281281/photo/1
“What is more shocking: reading that the discovery of the elusive Aβ*56 oligomer may not have been one 16 (!) years after its publication, or reading repeatedly in the comments that this finding was doubted anyway? Who, then, were those reviewers who repeatedly asked to demonstrate Aβ*56 in Aβ-oligomer containing samples? We have never succeeded to clearly show Aβ*56.”
I know you tend to focus on social science, Andrew, but I think there’s a huge amount of this going on in biomedical research as well, in particular connected to neuroscience and Alzheimer’s disease. I wrote a bit about my experience with it here (with several nods to some of your prior blog articles):
https://ajtrev.substack.com/p/begone-good-faith
Thousands of papers and hundreds of millions spent per year but no resources for a replication. This story is more about failure of NIH and the culture of Alzheimers research.
Also, look up any disease and you will find elevated amyloids. That is the default (lowest energy) conformation of peptides, so they accumulate in all damaged/diseased tissue.
It is like someone is too bedridden to take out their trash and trying to cure them by visiting and throwing it out. Accumulating trash can cause secondary problems but removing it isn’t going to address why they were bedridden to begin with.
“This story is more about failure of NIH and the culture of Alzheimers research.”
Chicken and egg, I’d argue. The culture gets this way in part because nobody bothers to hold anyone accountable for fraudulent research. How is the NIH supposed to “succeed” if papers with 2300 citations are also ones that apparently everyone knows are fatally flawed, yet nothing is done about them?
The problem is compounded in Alzheimer’s in particular because the evaluation criteria in clinical trials (e.g. ADAS-Cog) are fungible and noisy—it’s practically a rite of passage for a disease trial to fail, then for the academic community and sponsor to say “oh well, but look at this interesting subgroup!” The subgroup literally never holds up in the next trial.
When applying for an NIH grant you need to justify the “novelty” of the project. That is why we hear about all these half-hearted attempts at replication that never got published.
The lab redirects funds from other projects towards the replication (maybe it can tie into their specialty somehow) and if it doesn’t work, they say “who knows, I’m not being paid to risk resources on this.” If it *does* happen to replicate though, that will often get published.
Every grant should include funding for another lab to do a direct replication. When this was tried on a small scale for spinal cord injury and cancer, the results were very poor. We saw only 10-30% significance in the same direction, when it would be 50% skipping the data collection and flipping a coin.
https://www.sciencedirect.com/science/article/abs/pii/S0014488611002391
But in general, the amyloid-beta idea was interesting in the 1980s but should have been abandoned before the study in the OP was even done. At least the original cholinergic hypothesis generated drugs that seemed to help symptoms, the hundreds of failed trials based on the amyloid hypothesis tells me these interventions must be actively harmful. How and why did this paradigm become dogma when the old one performed better?
Sorry, 1990s. Around here (2002) is when it became obvious that amyloids accumulate in response to underlying damage or malfunction:
https://pubmed.ncbi.nlm.nih.gov/12491403/
It’s a classic prisoner’s dilemma game. A university that installs tough criteria (to avoid scientific fraud) would be in a huge disadvantage if other’s don’t follow.
” this is an academic community that’s consistently looked away or downplayed lying, cheating, and fraud in its midst.”
An odd thing I notice in many psycholinguistics papers is that even when a p-value was larger than 0.05, the authors argued that the effect is significant. Even the published statistics don’t support the significance claim, so the authors are not doing any data fabrication per se. I suspect that the main problem, also in these behavioral sciences by Gino et al and others, is not lying and fabricating stuff, but rather near-complete ignorance of statistical theory. The problem probably comes from their advisors, who pass on their incomplete knowledge on to their students, who go on to become senior scientists themselves. “Man hands on misery to man. It deepens like a coastal shelf” (Larkin, This be the verse). (I have met linguists who were happy to remove all subjects’ data that didn’t match the theoretical predictions—they were obviously not appropriate subjects.)
I think that the reason nobody calls out this kind of bogus claim and that they even get published at all (even the reviewers of the papers didn’t notice these problems) is that the reader’s also working with incomplete undertanding. You see an F1 (by subjects) ANOVA score showing p=0.05 and an F2 (by items) ANOVA score showing p=0.05 and you say, yeah, that looks significant to me, ignoring a 1973 paper by Clark (language as fixed effect fallacy).
Another thing that puzzled me about the above case was that if Gino et al actually did fabricate their data, why did they do it so badly? I think that it should be relatively easy to fabricate data in a way that one cannot get caught? I mean, I do data simulation almost every day. When I assign projects to students, I just take a published study’s data, generate 40 posterior predictive data-sets so that each student gets a unique data set (so they can’t copy each other’s numbers), but it’s all fabricated data. Is data fabrication so hard or maybe I don’t understand the complexities of true data fabrication? All these Excel-footprints they left behind seem so 20th century. Stapel also used to edit Excel sheets by hand IIRC. Another easy thing to do is to just destroy the data after publication and say it was lost in a computer crash, or that it’s sitting on a computer in some remote and inaccessible part of Canada and cannot be recovered any more.
Andrew, maybe you should teach a course on how to fabricate data like an expert.
Shravan:
I looove fabricating data. Simulating data is a wonderful research tool. Of course when our data are simulated, we should be open about it and share the process we used to do the simulation.
Your suggestion makes me think that it could be very useful to write an article, and maybe teach a course, explaining the benefits of simulated data and demonstrating how to do it in multiple examples.
“Another thing that puzzled me about the above case was that if Gino et al actually did fabricate their data, why did they do it so badly?”
I wonder this about a lot of these deliberate frauds. Is it that dishonest people are generally incompetent, or that the small subset of dishonest and incompetent people are the ones who are caught, leaving the dishonest and competent people at large?
Raghu:
My general impression is that successful people keep doing what worked for them in the past. If someone fakes some data and it works, then I’d expect them to keep faking it. I’d think that the biggest hurdle would be the first time, because then there’d be more fear of getting caught. Once the cheater does it for awhile, he or she might feel invulnerable.
The flip side of the argument is that often we hear that rulebreakers are uncomfortable with the lifelong burden of secrecy. It’s sometimes said that cheaters subconsciously want to be caught so that they can finally relax. This would explain writing books such as “Evilicious: Why We Evolved a Taste for Being Bad,” “How We Lie to Everyone—Especially Ourselves,” “Is Everybody Cheating These Days?,” and “Why it pays to break the rules at work and in life,” and it would also explain why Frey and Gino were pushing science reform. Maybe the pressure was getting to them, and at some level they were hoping to get caught so they could start afresh.
Finally, as Sandra writes below, all this behavior reeks of laziness. It’s a lot easier to copy data (or, in Frey or Wegman’s cases, to copy writing) than to do new work. And, once you’re motivated by laziness, it makes sense that you’ll take the easy path. To put it another way, if you had the energy to do a very careful fake, why not just do the actual experiment?
The other thing is that even a careful fake can get caught. Recall Michael Lacour, the political scientist who faked an entire survey. He was pretty sophisticated with R, but he still got caught.
To get back to the laziness thing: these cheaters were lazy in some ways but energetic in others. Giving Ted talks and writing books isn’t easy, right? Some researchers put most of their effort into research; in contrast, Wansink, Hauser, etc., were lazy when it came to research but extremely energetic when it came to promotion. It could be that there are some cheaters who are extremely careful to fake their data in ways that can’t be detected . . . but then they don’t have the energy to promote their work, so you never hear about them in NPR, Freakonomics, Harvard Business Review, etc.
IIRC Lacour only got caught because he had unusually large effect sizes. I think that if he had toned down the true values he input to generate data, it would have been fine. He should have gotten some realistic estimates first. Or am I mis-remembering this case?
Incidentally, I was wondering what became of Wansink. It seems he runs a foundation of sorts. I would link to it except I don’t want people to get their weight-loss advice from there (although his advice did seem sensible, don’t expect miracles, keep the weight-loss program simple). So he is still full of energy, as Andrew discusses above.
Shravan:
I think Lacour got caught not so much because of the effect sizes but because he posted his purported raw data, and some students who were interested in the subject matter looked at the data carefully and found problems. Lacour didn’t do a bad job at faking the data, and he was a pretty smart guy, but he wasn’t as smart as those students. Which makes sense, right? Cheating is a substitute for hard work, it’s also a substitute for talent. Lacour was hard working and talented as far as cheaters go, but he wasn’t at the top level.
And, yes, there is something charming about Wansink, somehow. I’ve never met any of these people, but Hauser, for example, just seems creepy to me. Wansink is so relentlessly upbeat, kinda like Robert Preston in The Music Man. He’s a fraud, he knows he’s a fraud, that’s ok, being a fraud is his job in life—it’s all cool!
At the time of the work, there was a general expectation that no one would ever scrutinize the data. Certainly not investigate to the depths being done now. There really was no fear of getting caught. And most researchers (at least then) had no idea how Excel worked or how sleuthing might be done. There was no perceived need to be careful; laziness prevailed.
If you are also in the academic community and know the GATE KEEPERS (reviewers, editors, co-editors) to get Gino’s reviewed, accepted, and published,
it is easy to DEDUCT that we have SYSTEMIC CHEATING BEHAVIORS even in the top journal community.
In a way, we are all preys of modern academic world. WE got tenured, promoted, and rewarded for publications.
In the past, people published to share interesting findings. With the new published-and- you will be rewarded schemes (money, grants, endowed professor bonus), people are motivated to cheat (to get something). It is just human nature.
The latest on Gino: https://storage.courtlistener.com/recap/gov.uscourts.mad.259933/gov.uscourts.mad.259933.1.0.pdf.
Suing Harvard and the Data Colada investigators!
There is much to read in the suit and I’ve only started. It will be interesting to see how this plays out – some of her claims certainly should be addressed (did Data Colada never contact her about the anomalies they found?). One bit that caught my attention, however was this statement:
“Data Colada could not possibly have had any good faith basis for their purported “strong suspicions” for her published data going back to 2008, a period before Open Science Framework (“OSF”), an open-source web application, made it possible for behavioral scientists to share their data.”
I don’t understand why it was impossible to share data before the Open Science Framework.
It is a junk lawsuit and will be thrown out under SLAPP. It reads like a Trump lawsuit in a lot of ways, with the most salient being that it is a situation where the plaintiff has acted so terribly that the standard law enforcement actions are inadequate and additional actions must be taken. This is then justification for the lawyers to argue that the plaintiff was singled out for persecution.
There is also a claim of sexism because once there was a male Harvard professor who was treated differently under entirely different circumstances. This claim is what lawyers call “racehorsing” aka “throwing everything against the wall and hoping something sticks.”
But the most ridiculous statement in a lawsuit consisting entirely of them is continuously claiming that DataColada had no evidence. All they had was evidence!
Point 19 of the first section struck me:
“After the release of its defamatory blog series against Plaintiff, Data Colada admitted that it had no evidence that Plaintiff committed data fraud. This admission was made to a limited audience…”
I wonder how they’re defining “evidence” in this case. There seemed to be pretty damning evidence to me.
I read a little bit further. On page 52, point 233:
“In a July 18, 2023 webinar titled “Data Fraud and What to do Next: A conversation with Uri Simonsohn and Maurice Schweitzer,”18 Defendant Simonsohn admitted (at min 8:47) that, there was no evidence that Professor Gino manipulated data. Simonsohn stated: “Nobody has unambiguously claimed that this was Francesca Gino . . . The retractions don’t say that. They just say data she had and data she posted. Uh, my belief is that she did it, but there’s no evidence of that. But, but it doesn’t really matter.”
You can find the webinar on Youtube here: https://www.youtube.com/watch?v=OPUKyeetdt8
He’s not saying there’s no evidence of fraud, just no evidence that Gino specifically was culpable (“someone could have somehow used her computer”). I think it’s also relevant that this wasn’t said as part of a rehearsed formal speech or anything like that, he was responding to a question in what seems to me like a fairly casual seminar. Even if we accept that there’s no evidence that Gino committed fraud herself, all the data passed through her and are, at least on some level, her responsibility. Throughout the lawsuit there seems to be some subtle attempt to shift the blame of any anomalies to RAs (reinforcing how they usually collected the data and converted it to digital format and so on).
Finally, for the sake of being helpful, I want to point out that her point-by-point response to Data Colada’s blog posts starts on page 52. I imagine that’s what most readers of this blog would be truly interested in.
A couple more points:
Point 241 claims that survey data collected on paper and entered into a spreadsheet are not necessarily sorted. Of course, this does not explain why the id numbers were in fact sorted, with the exception of the observations that Data Colada focused on. In other words, the id order was clearly not random.
Point 250 correctly points out how unreliable survey data might be. Given that, it seems irresponsible to rely on it in published work.
While I find much disturbing in the case, I’m still not convinced it is “garbage” in legal terms. The essence of the case, as I see it, is whether the actual problems with the data were due to acts of a particular researcher, and then further whether those acts were intentional or not. That is a heavy burden of proof. Ariely seems to have largely escaped by acknowledging his data was bad, but not that he was responsible. Those poor research assistants keep messing up the work of these famous researchers. Perhaps. But it reminds me of the phrase used by Edward Tufte (in Beautiful Evidence) about use of the passive voice in writing: “Although often a useful writing technique, passive verbs also advance effects without causes, an immaculate conception.”
Dale:
I agree with all you wrote there, especially your second paragraph. Another big aspect of the lawsuit is that Harvard was somehow unfair in their investigation against her. That could be true, and if so it would be right for Harvard to be punished accordingly.
My fear is that if Gino wins this case (even partially) people may take that as evidence that there was never any fraud in her research at all, even if that is never refuted over the course of these legal proceedings. Her shoddy (quite probably fraudulent) research could be erroneously vindicated. And then a bunch of scientists will have to waste a ton of time and money replicating these studies to dispel these artificially bolstered myths.
I feel like regardless how this turns out, this is going to be a massive mess for the field of behavioral psychology.
Dale:
Why should Data Colada have had any responsibility to contact her about the anomalies they found? This is all public information, right?
According to her claim, they usually do contact authors about anomalies. In fact, they acknowledged that since they attached a statement explaining why they did not in this case. I’m not sure I find their explanation all that convincing, but it also seems disingenuous to simply claim that they did not without addressing the reasons they gave.
More generally, much of this lawsuit strikes me as garbage – but I’ve given up trying to predict legal cases or pretending that I understand the legal issues involved.
Wow. I wonder what her planned endgame is. Fox News commentator?
In any case, she’s at Harvard, right, so maybe she could get Dershowitz to represent her. He’s gotta have lots of time on his hands now that Jeffrey Epstein no longer needs his services. . . .
Actually, he’s quite busy. He has a podcast he puts out around twice a week where on every episode he spends close to an hour talking about himself and how many people on Martha’s Vineyard don’t like him.
Joshua:
Twice a week doesn’t sound like much work. He has lots of energy and I’m sure he could find the time to handle some lawsuits on the side.
“so maybe she could get Dershowitz”
He does seem to be missing from this story.
The court filing mentions that her lawyer prior to filing this lawsuit was Paul S. Thaler. From the filing:
“172. Dean Pisano urged Dean Datar to read the letter sent by Professor Gino’s counsel, Mr. Thaler, advising him that Mr. Thaler had handled many cases at Harvard on behalf of other faculty members and was “simply appalled” by both the investigation committee’s decision and its proposed sanctions against Professor Gino.”
But the defamation court filing did not come from Thaler or his firm, she hired a different lawyer to file this suit. Why not just retain Thaler – with so much experience with Harvard – and have him file the suit? We can guess that he declined to participate.
Another interesting aspect is that the filing is exactly 100 pages long. My guess is that she contracted with the legal firm to write at least 100 pages, and that is what she got, with the same few arguments and ridiculous allegations of malice repeated over and over again.
I called it a junk lawsuit because of the burden placed upon plaintiffs to prove actual malice in defamation suits in the US. First of all, she may be too famous to even file for defamation. Second, the idea that the Data Colada investigators somehow developed malice towards what would to them be a random researcher is just silly. Their long track record of data sleuthing should help them here.
Matt:
My favorite is the bit where they say she’s “a working mother of four young children.” I guess that’s an application of the well-known legal rule that if you have a bunch of kids, you’re allowed to sue people whenever you want.
But only as long as the kids are young. Once the kids get older, or if you’re unemployed, it’s another story.
Somehow your Nudge anecdote reminded me of the mother of all memory holes: HBS had an adoring case study of Enron, pre-collapse, which painted an adoring portrait of the company and its leadership (including a footnote explaining that really no need to worry about those strange “special purpose vehicles” engaged in peculiar cash-for-stock transactions with the company). The case was airbrushed from the HBS website, rather than serving as a lesson in financial fraud.
Ray:
That’s hilarious. Do you have a citation for that? A quick google search revealed The Fall of Enron from 2008, updated 2019, and Innovation Corrupted: The Rise and Fall of Enron from 2003, updated 2005.
Also this: Trading Truth, A Report on Harvard’s Enron Entanglements, which appears to be from around 2003 and which says:
And this:
“Herbert ‘Pug’ Winokur,” huh? Maybe I ran into him at the yacht club the other day.
And here’s more on the HBS case studies:
Also this article might be relevant: From superstars to devils: The ethical discourse on managerial figures involved in a corporate scandal.