OK, more on junk science. Sorry! But it’s been in the news lately, and people keep emailing me about it.
For those of you who want some more technical statistics content, here are some recent unpublished papers to keep you busy:
– BISG: When inferring race or ethnicity, does it matter that people often live near their relatives?
– Nested R-hat: Assessing the convergence of Markov chain Monte Carlo when running many short chains
OK, are you satisfied??? Good. Now back to today’s topic: the mysterious popularity of goofy psychology research.
Here’s the deal. We’ve been hearing a lot about glamorous scientists who go on Ted and NPR, write airport bestsellers, get six-figure speaking gigs . . . and then it turns out, first that their work does not replicate, and next that their fame and fortune were based on scientific publications that were fatally flawed. Maybe the data were fabricated, maybe the experiments never happened, maybe the data were analyzed manipulatively or incompetently, often even if everything was on the up-and-up these studies were too noisy for anything useful to be learned.
At this point we usually ask, What happened? How did this bad work get done? Or, How did it not get caught, staying aloft, Wile E. Coyote-like, for years, with no means of support? Or, What caused the eventual fall?
But today I want to ask a different question: How did this work get all the adoring publicity in the first place?
Sometimes the answer seems clear to me. Brian Wansink, for example, the guy who claimed that if you move items around on the menu or use a smaller plate or whatever, you could get people to eat 28% less for lunch. That’s a big deal. Big if true, as the saying goes. If the work was legit, it deserved all the publicity it got.
Similarly with Freakonomics, which has some strong messages regarding incentives and what you can learn from observing people’s behaviors. Some of the research they promoted was mistaken, but they really were going for important topics much of the time. And the Gladwellverse. No, I don’t believe that therapist’s claim that he can predict with 93% accuracy who’s gonna get divorced—but if it were the case, it would be worth hearing about. Again, big if true.
Or, if a researcher cures cancers in mice and then gives a Ted talk saying how he’s gonna end cancer in humans, ok, sure, that’s an exaggeration, but the relevance is clear.
Other times, though, I didn’t get it.
For example, the scandal everyone’s talking about now is a paper coauthored by two different NPR/Ted talk stars, and the key result of the paper was that if people sign an honesty pledge at the top of a form, they behave more honestly than if they sign at the bottom. Even if this was all real—even if there had not been all these data problems (for example here, and that ain’t the half of it), even if this had not been part of a whole line of research with replication problems—even without all that, why did anyone really care about this stuff in the first place?
I mean, sure, within the world of academic psychology, yes, studying small influences on behavior is important. Years ago when I read the classic collection on judgment uncertainty, edited by Kahneman, Slovic, and Tversky, I was entranced. It was super-cool and changed my understanding of human nature. But . . . research on where people should sign an honesty pledge? OK, I can see how business marketers would care about this, but how does it make you a Ted talk superstar? And one of these researchers made his name by writing books and articles saying that people are “predictably irrational” . . . this is news?
How did this happen? What got these people into the stratosphere of fame? Part of it is that they must be very engaging speakers. But, still, is it enough to just be a good performer? When I think of inspiring Ted talks, I think of biologists who are curing cancer by figuring out how proteins work. I think about people building machines that will remove pollution from the oceans. Developing new ways of teaching that will reach underpriviliged kids. Using technology to build veggie burgers. Training parrots to play the marimba. Amazing stuff. Compared to that, we have . . . running a psychology experiment using a fake paper shredder? Huh?
So I went to some of the Ted talk pages of these psychology researchers and I think I haver some sense of what was going on.
Here’s one: “Are we in control of our own decisions?”
And another: “How to change your behavior for the better”
And how about this: “The Power of Why: Unlocking a Curious Mind”
And: “Rebel Talent: Why it Pays to Break the Rules”
It’s a three-step process: First you do research demonstrating some very specific thing. Actually, the research doesn’t even need to demonstrate it, but it needs to look like it does. Second, you parlay this into a reputation for being a brilliant, innovative academic. Third, you imply that the specific thing you found (or, to be more precise, that you claim to have found) has important general implications. That’s how you get from the angels-dancing-on-the-head-of-a-pin result, “Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end,” to the big ideas such as “How to change your behavior for the better.”
I’m no saying that the researchers who do this are bad people, or that there’s anything bad about these three steps. First, it’s fine to do research on a specific topic; that’s often how we learn, building bricks that become part of the edifice of science. Second, if you do successful work, you get a good reputation; fair enough! If you have a problem with that, the problem is structural (to publish in tabloid journals it helps to make big claims); you can hardly blame researchers for doing their best. Third, if you honestly think your work can help the world, then it makes sense to get some publicity for it. Nothing wrong with that!; indeed, it would be irresponsible not to try to publicize important work.
Somewhere along the way, though, there’s a leap, unsupported by data. OK, there could be many leaps, starting with the claims in the published research in step 1. But here I’m thinking about the leap that takes you from oh-so-prosaic “Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end” to the dramatic “Rebel Talent: Why it Pays to Break the Rules.”
Here’s the problem, as I see it: the scientific credentials accrued by the journal publications give the researcher a sort of license to make large claims unsupported by evidence.
An extreme case was disgraced primatologist Mark Hauser, whose research on monkeys had no practical applications at all, but it was enough to get him to be considered an expert on morality and hang out at Jeffrey Epstein’s Edge Foundation. You might think it’s a cheap shot for me to bring up the child-molester connection here, but it’s not! Hauser didn’t just con NPR and, presumably, himself; he also managed to convince people in the Epstein orbit that he was this savant. And a key component of all this was the academic status conveyed in part by the peer-reviewed publications.
But really we see this all the time. Consider power pose, that Ted-talk favorite. The subtitle of the published article was the careful, “Brief Nonverbal Displays Affect Neuroendocrine Levels and Risk Tolerance.” OK, it turns out the research was botched and didn’t actually demonstrate effects on neuroendocrine levels or risk tolerance. But the researchers didn’t know that at the time. What they, and the journal editors, have to take some responsibility for is the title, “Power pose”—a misleading title given that there was nothing about “power” in the research—and the final sentence of the abstract, “That a person can, by assuming two simple 1-min poses, embody power and instantly become more powerful has real-world, actionable implications,” even though the actual study had nothing about anyone “instantly becoming more powerful,” nor did it have any real-world, actionable implications. I’ve brought up this sentence many times, partly because it annoys me and partly because it indicates a big blind spot of the authors and the journal editors, that they’d end the abstract with such a strong claim that had zero support from the experiment.
For another example, there was this report from Freakonomics that “sports participation causes women to be less religious, more likely to have children, and, if they do have children, more likely to be single mothers.” It doesn’t sound so impressive if you find out what they really did: “comparing women in states with greater levels of 1971 male [high school] sports participation . . . to women in states with lower levels of 1971 male sports participation.”
OK, here’s another, from a New York Times op-ed: “Knowing a person’s political leanings should not affect your assessment of how good a doctor she is — or whether she is likely to be a good accountant or a talented architect. But in practice, does it? Recently we conducted an experiment to answer that question. Our study . . . found that knowing about people’s political beliefs did interfere with the ability to assess those people’s expertise in other, unrelated domains.” Oh, interesting, so they asked people to assess the quality of doctors, accountants, and architects, and saw how these assessments were colored by the political beliefs of these professionals? I followed the link to the research article and did a quick search. The words “doctor,” “accountant,” and “architect” appear . . . exactly zero times. Actually, “Participants were required to learn through trial and error to classify shapes as ‘blaps’ or ‘not blaps’, ostensibly based on the shape’s features. Unbeknownst to the participants, whether a shape was a blap was in fact random.” And later they had to choose “who the participant wanted to hear from about blaps and how they used the information they received.”
As always . . .
I have no problem with people doing studies to address very specific research claims. I also have no problem with speculation. My problem is when these get conflated: when a demonstration of claim X gets transmuted into a statement about Y (also problems arise because often X has not really been itself demonstrated as presented). And my problem is when the conflation is done within the scientist rather than the research study: Researcher A has demonstrated very specific claim X, which now turns A into a sort of science pundit who has authority to make strong claims about Y.
I don’t know much about Jesus, but this all sounds like some sort of doctrine of transubstantiation.
Summary
The original question was, how did obscure academic research such as “Brief Nonverbal Displays Affect Neuroendocrine Levels and Risk Tolerance” and “Signing at the beginning makes ethics salient and decreases dishonest self-reports in comparison to signing at the end” get such publicity and reach a level of cultural salience to merit all this media attention?
The answer is that there has been a sort of transubstantiation, whereby the success (such as it was) of scientific studies had the effect of elevating the researchers involved into a higher state, granting them authority so that their evidence-free speculations were given the status of scientific findings.
I agree with your diagnosis, but the examples seem a bit strange. While nobody cares much about signing at the top vs the bottom, plenty of people care about whether there are simple ways to make people behave more honestly. While nobody much cares about a particular power pose, plenty of people care about whether your body language can influence the way you affect others. As you say, the key step seems to be how we get from the seemingly uninteresting research question to the more important ones.
When I think about it, it is difficult to see how to directly address the more important question. How can we induce people to become more honest? We could devise a theoretical model (expressed mathematically, of course) focused on levers that could affect the “degree of honesty” exhibited by people. There are plenty of such economic studies (not necessarily focused on honesty, but models devoid of data focused on how equilibrium behavior is influenced by changing one or more levers). On the other hand, if I look for data on these questions, there is often no obvious data source available. So, why not do an experiment? Of course, the experiment (in order to be feasible) will, of necessity, be for a narrower question, often one of limited interest – just like the examples you list.
Then the leap from the narrow question to the ones we really care about is easy to see: it enhances the status of the researcher, gives them a platform for voicing their opinions, and is interesting because the original question that motivated their research was actually of interest. I don’t see an easy way to combat this, other than the often-suggested changes such as improving peer review, improving promotion, tenure, and grant incentives, open data policies, more informed and better press coverage, etc. I do support these changes, but I’ll speculate that one reaction to the situations you are describing is the rejection of elites and academic research. While I think this is partially a healthy reaction for the general public, I lament that and wish there was a better way – a way to preserve the academic platform without continuing the trend towards misguided research and/or inappropriate conclusions drawn from the research.
The only idea I can think to contribute is that we need to start rewarding collection and curation of good data and downplaying analysis of that data. Good data means data that is designed to potentially say something about important issues, with measurement carefully designed and data made available. Rewards should be based on people using that data. For a long time, the rewards have gone to the analysis – I think that is a result of an older era when analysis was difficult. Developments in hardware and software make the analysis step far more accessible (as many on this blog lament). Anyone is capable of torturing the data long enough to say something provocative. Collection of data is also largely automated and often effortless. But careful measurement, curation, and documentation remains difficult, labor-intensive, and underappreciated in my opinion.
But, to end on a pessimistic note, I can’t see too many TED talks coming from the curation of such data, nor will it help the teaching profession to make all those required classes entertaining for students.
Dale:
Regarding your first paragraph: Sure, but my point is that in the Ted talks the researchers were making general claims, but the research addressed only very narrow outcomes. Lots of exaggeration was going on. And I don’t even know if the researchers fully realized they were exaggerating. They may have, without reflecting on it, just naively thought that their experimental measures in these narrow settings implied general findings about honesty, power, etc.
I agree with you. The chain of logic seems to be: this general question is of interest, here is a concrete experiment related to that general question, here are the results of that experiment, here is what I speculate about the general question as a result. That chain seems sensible enough to me – until you ask about the leap between the general question and the specific experiment and the leap back to a general conclusion. I don’t know if those leaps are conscious or naive, but it seems easy to make these leaps given the chain of logic that was being followed. Where the chain should have been broken is whether the specific experiment would yield anything meaningful about the general question. I think that step is not given enough critical examination and the rewards/penalties associated with it are not sufficiently large. Instead, we reward the general conclusions that are (over)reached.
Unfair to Hauser. He presented himself as an expert on morality because he was. He was expert on many things and published widely on many topics, including moral philosophy as well as moral psychology. He convinced me, for example, that Rawls’ “A theory of justice” was generally misinterpreted as implying normative status of its conclusions, while, in fact, nothing in its 600 pages implied that it was anything other than a psychological theory, like Chomsky’s theory of language. As I had made the usual misinterpretation, I was pretty astounded by this insight. His “research on monkeys” was what got him into trouble, but it was not his major effort by any means. (And, as ethical lapses go, what I know about it, from what was public, did not seem to put it very high in egregiousness.)
Hauser is/was an evolutionary biologist who held a chair in psychology. Not an expert on moral theories. And Rawls had no interest in “psychological theory.”
Kyle:
To be fair to Hauser (not something I usually have much inclination to do, but whatever), if you look him up on Google scholar, his most-cited item is a linguistics paper, his second is one of his pop-science books, and his third is a monograph, but #4 and #5 (with 2000 and 1500 citations) are at least related to moral theories:
– Damage to the prefrontal cortex increases utilitarian moral judgements
M Koenigs, L Young, R Adolphs, D Tranel, F Cushman, M Hauser, …
Nature 446 (7138), 908-911
– The role of conscious reasoning and intuition in moral judgment: Testing three principles of harm
F Cushman, L Young, M Hauser
Psychological science 17 (12), 1082-1089
And here’s his ninth-most-cited paper, with almost 1000 citations:
– A dissociation between moral judgments and justifications
M Hauser, F Cushman, L Young, R Kang‐Xing Jin, J Mikhail
Mind & language 22 (1), 1-21 992 2007
So I don’t think it’s unreasonable to label him as an expert on moral theories.
What ever happened to his “Risk Eraser” project . . . that’s another story.
I take back my last sentence, with apologies. It’s more complicated than that, but way off-thread.
I think its because you followed the link, while most people did not. Even if they did and it looked like nonsense, they would not have the self-confidence on the topic to critique it themselves. They trust in peer review, etc to filter out the ridiculous claims.
If you carefully read the “tumor growth in rats decreased by 90%” studies, I think you will conclude they are just as bad as the “blaps” type studies.
Eg, say drug slowed tumor growth by 50%. Using a standard population growth model where:
t = weeks
k = carrying capacity (size where tumor kills the organism)
p0 = initial size
r1 = control rate of growth
r2 = treated rate of growth
t = 1:20
k = 100
p0 = 1
r1 = 1
r2 = .5*r1
y1 = k/(1 + ((k – p0)/p0)*exp(-r1*t))
y2 = k/(1 + ((k – p0)/p0)*exp(-r2*t))
par(mfrow = c(2,1))
plot(t, y1, ylim = c(0, k))
lines(t, y2)
plot(t, (y1-y2)/y1)
https://i.ibb.co/cTbSjP7/growth.png
If you stop the study at 5 weeks the drug looks great (80% effective), at 20 weeks not so much (0%). There is lots of silliness like this out there.
Is that what that means? I have always interpreted a statement such as “slows tumor growth by 50%” to mean that it takes twice as long to attain a given size of tumor, not that the tumor is half as large at a given time. The two different interpretations produce vastly different values of effectiveness as a function of time.
That was sloppy writing on my part, I meant something like “cancer in rats decreased by 90%”.
The point was if you take a snapshot (of tumor size, survival, or whatever) during the exponential growth phase, you get a very misleading picture of whats going on. This is one reason why there are so many promising animal trials that end up extending the average life of cancer patients from 2 years to 2.5 years or similar. Nowadays standards are so low that is heralded as a huge success.
In fact, modelling pure division rate without any plateau looks even worse. Say the treatment reduces the rate by 5%. Then ratio of cells in control vs treatment is
n1/n2 = 2^(r*t)/2^(0.95*r*t) = 2^(0.05*r*t)
After 20 weeks the 5% growth-slowing effect has been magnified to a 50% tumor size effect and so on. Essentially, a 1 week lag gets turned into a super promising cancer drug.
To be fair, the op-ed itself describes the “blaps” study in detail. It was apparent without following the link that the study was about blaps rather than about doctors etc.
Which still leaves the question of why they presented research on blaps as if it had something to do with doctors and accountants, instead of trying to do follow-up research on doctors and accountants.
Matt:
I disagree with your statement that “it was apparent without following the link that the study was about blaps rather than about doctors etc.”
The op-ed flatly said, “Knowing a person’s political leanings should not affect your assessment of how good a doctor she is — or whether she is likely to be a good accountant or a talented architect. But in practice, does it? Recently we conducted an experiment to answer that question. Our study . . . found that knowing about people’s political beliefs did interfere with the ability to assess those people’s expertise in other, unrelated domains.” I assumed that they were telling the truth, and I was surprised that the paper had none of that doctor, accountant, or architect content. Sure, we’ll expect some hopeful exaggeration, but just making all that up . . . that’s more than I was thinking I’d see.
“Here’s the problem, as I see it: the scientific credentials accrued by the journal publications give the researcher a sort of license to make large claims unsupported by evidence.”
This isn’t new. The media have been asking actors and professional athletes for their opinions on diet and even politics for far too long. Scientists have been making broad claims – even way outside their area of expertise – meteorologists on climate change, all kinds of medical scientists on epidemiology. etc.
But it does seem to be getting worse, in part because:
1) the incentives for attention are increasing for both researchers and media.
2) decentralization of media encourages end-runs around gate-keepers, e.g. faux academic journals and TikTok as a popularizer of scientific results; and
3) so many recent failures of the research enterprise, in the form of fraud and p-hacking undermine public confidence in the scientific enterprise and its results.
It’s a vicious cycle. I’m not sure fixing failures in the research enterprise with pre-registering experiments, improving peer review will help much.
Any ideas for how to break the cycle?
And: I agree with Dale.The dishonesty research would be big if true – bigger, I’d argue, than any of Wansink’s work. If the finding in the main Gino-Ariely et regretful coauthors paper were true and extended from reporting driving mileage to reporting income (not a big leap), it would be very big. Like, an increase of tens of millions of dollars of tax receipts in a country like Guatemala, which experimented with this intervention [Fabricated data in research about honesty. You can’t make this stuff up. Or, can you? : Planet Money](https://www.npr.org/transcripts/1190568472).
Here is a comment with an open question for people who know more than me about the subject:
I was looking at the first Gino study with the experiment about cheating/lying when signing at the top vs the bottom. I wondered about the gender makeup in the study – indeed, that was collected but not reported on in the study. So, I did a few analyses and found no significant (forget p values, the data is very similar for both genders) effect of gender on the tendency to cheat or the amount of cheating. Nothing to report there. But: shouldn’t there be a difference? A little searching leads me to believe that the prior should be that there would be greater dishonesty among men than among women (the studies I saw are far from unanimous, but apart from a small number of rarely cited studies, males seem to show more dishonesty than females – perhaps someone more familiar with the literature can comment on this).
My question is: first, are my conjectures about gender differences and honesty valid? Second, if they are, then should this be relevant in the review of studies such as this Gino study? Since there is the leap from the particular experiment to reaching conclusions about the general population, shouldn’t differences in the experiment from what is expected (in this case, that we would expect generally greater dishonesty among males, but they are not found in the experiment) cast doubt on the “leap unsupported by data” that Andrew refers to above?
This may all be moot, depending on whether my conjectures are at all valid. But this is what I mean by data quality. An experiment was run, the data is available, but part of the determination of its quality should be whether it comports with what we know (or think we know) about all the relationships in the data? When we find surprising things, there should be some pertinent discussion of that – the gender effects on honesty (lack thereof) were not discussed in the original paper. I find this makes me more skeptical of claims that the results of this particular experiment can say anything reliable about the more general questions.
Dale Lehman wrote, “A little searching leads me to believe that the prior should be that there would be greater dishonesty among men than among women (the studies I saw are far from unanimous, but apart from a small number of rarely cited studies, males seem to show more dishonesty than females – perhaps someone more familiar with the literature can comment on this).”
Along this line of thought, I recommend watching the following very short video (less than a minute),
https://www.instagram.com/p/CsykXWdupge/
It was recorded just before all the accusations were made. It shows Gino, informally attired, playing hoops with her kid and depicts the sheer joy of family life, especially when she successfully puts the basketball threw the hoop–while looking the other way! Does this video alter your view of her in any way? In keeping with the “signing at the top vs. signing at the bottom”, suppose you saw this video BEFORE rather than AFTER all the accusations. Would your reaction/conclusion be different? Indeed, should your reaction/conclusion depend in any way on her physical appearance and domestic life? Do you think that others upon viewing this video would be influenced even if you are not?
Well, the video doesn’t change my view by a whole lot. But, like one of the Instagram commenters, I had to suspect the video was the product of some highly selective editing…which in turn is reminiscent of the research practices she’s been accused of. Is it possible that once you get used to manipulating data in a serious context, you’ll keep manipulating it even when you’re just clowning around?
It’s a very small point, yes. But if the video has any effect at all on my view of Gino, it isn’t a positive effect.
We performed a (Bayesian) reanalysis of many studies focusing on incentivized, anonymous dishonest behavior in heterogeneous samples. The results showed that gender had no effect (p. 365, top left): https://doi.org/10.1017/S1930297500009232
A meta-analysis found a small effect of women being more honest (Figure A.6): https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2866381
Another meta-analysis found a slight effect of men lying more than women by about 4% (Figure 7), but apparently the sample included many students: https://pure.mpg.de/rest/items/item_3037642/component/file_3047427/content
So, overall there might be some tendency, but the effect does not seem to be sufficiently robust to serve as a validation criterion for the quality of a dataset.
I am totally unfamiliar with your area of research – I had to look up what HH was. But, at a high level, I was wondering how you can separate the HH measures from the effects of sex on honesty – as you note, they are somewhat correlated. So, unless a study is able to include the HH measure, shouldn’t we expect to see an effect of sex on dishonesty?
I would also add that your study seems to share the features under discussion in this post. You identify a general problem and then conduct a fairly specific experiment(s) to examine it, ending by making more general conclusions. I’m not sure there is any other way to proceed, but my questions are about the standards that should be applied to the creation of the specific experiments to be used to cast light on the more general questions.
Thanks for spending so much time to look into these studies in detail!
You are right – if the trait Honesty-Humility (HH) is related to gender and a valid predictor of dishonest behavior, one should see a correlation of gender and dishonesty when HH is not measured. Note that this actually fits to the results of the other two meta-analyses, which did not measure or consider HH but found some tendency that women behave more honestly. The question is whether such a relatively weak association can reliably be detected in studies with small samples.
Concerning your other remark, I think conducting studies in simplified/controlled contexts or experimental paradigms is a common approach in science in general, and specifically in psychology. The core question you mention seems to be whether one prefers a deductivist or inductivist approach to knowledge generation. According to a deductivist approach, we have some theory that specifies boundary conditions under which we should observe certain regularities (e.g., causal effects). We can then design appropriate experiments to provide strong tests of such claims. Typically, such (lab or online) experiments are overly simplified while allowing us to make relatively strong claims (i.e., they have high internal validity). The question to what degree such specific results can be generalized (i.e., external validity) is generally difficult to answer. A core issue is what exactly is the target population or context one wants to generalize to. I think the best approach might be to (a) be transparent whether there are plausible reasons why the results should *not* generalize to certain context to which the theory should apply to (this can then be tested in new studies), and (b) combine the evidence from simplified (but highly controlled) experimental studies with field studies which may often have lower internal validity but are closer to realistic applications.
Anyways, this a very old question in philosophy of science, so there is probably no easy answer.
“What happened? How did this bad work get done?”
The narratives recently presented by Ariely and Gino in defending themselves against accusations of fraud give us some insights into what was happening in their labs.
Their actual words consist of various claims about what might have happened to the data that might look like fraud but not be. But if you step back from the words, the edifices they are building look really bad. Gino in particular shifts seamlessly back and forth from attacking her accusers to describing all the potential ways that the data in her lab might have gotten corrupted. She’s figuring that out now? There are five different versions of the same spreadsheet and she claims to have no idea which one has the original data (so neither does HBS!)? The entire purported effect in one paper came from what she claims was a single person who filled out multiple surveys to win gift cards? Shouldn’t that have been both anticipated and noticed? Reading her counterclaims gives a strong sense that she is looking at her own data for the first time and giving us a list of all the ways things might have gone innocently wrong. The more she tries to defuse the accusations this way, the worse she looks as a scientist. In Ariely’s private communications, he seemed to genuinely not remember taking data from one experiment and pretending that it had been collected in a different experiment that he made up in his head. The common theme seems to be zero interest in data management or even version control as standard operating procedure.
To me, all this adds up to these scientists viewing their own labs as sausage factories, with no real interest in what goes in and an emphasis on p-hacking to get something out the other end. Might as well shoot for the moon.
In Gino’s defence: She is currently preparing for her lawsuit Gino v Harvard Business School et al. I read her arguments in this light. If I were her, I would do exactly what you describe: First wave, attack the defendants. Second, in case the first wave does not suffice, and since no one will ever believe the results of the four disputed studies again, throw them under the bus and sacrifice them as “possibly riddled with honest error”.
Recently, Andrew wrote an article about (among other things) how it is common in litigation to “throw everything at them and hope something sticks” (https://statmodeling.stat.columbia.edu/2023/09/24/stupid-legal-arguments-and-moral-hazards/). Gino’s way of arguing is very much in line with this approach, in my view.
It’s pretty unlikely that Gino herself has decided on the strategy being pursued by her lawyers. Much more likely, the lawyers themselves are doing whatever they think will produce a win in court, while urgently warning her that she’d better keep quiet in public.
I am certain her lawyers will recommend silence! I completely agree with your objection. At the same time, is there anyone fighting for their life in court who would not at least try to build their defence in their head (even if they can keep quiet)? I did not mean to imply that her writings *are* her legal defence – I meant to say that her reasoning reflects legal reasoning, which is surely what she is engaged in now. I would add that there is a possibility that her public arguments may be taken from her legal defence.
Thank you for helping me to recognise my imprecise statement and correct my record!
Raphael K wrote:
“If I were her, I would do exactly what you describe: First wave, attack the defendants. Second, in case the first wave does not suffice, and since no one will ever believe the results of the four disputed studies again, throw them under the bus and sacrifice them as ‘possibly riddled with honest error’.”
You mean if you were her and were trying to cover up fraud. I would invite you to look at the counterfactual though, if she were innocent. An innocent researcher in her position would be laser-focused on what happened in the lab that caused multiple scholars to conclude that fraud was committed on multiple papers when it wasn’t. I can’t emphasize this enough: this imaginary researcher would be looking for a single root cause for the multiple misinterpretations, not a unique laundry list of potential problems for each suspect paper.
For the innocent researcher, the actions and identities of the accusers would be background noise. This researcher would have dedicated considerable time over the years to version control and data integrity, because they do serious work in a serious lab.
At this point, this is all VERY counterfactual!
Matt:
Yeah, we’re all speculating at this point. Speaking in general terms, I know what you mean. After Columbia University had that scandal with the U.S. News rankings, one thing that bothered me was that the administration didn’t seem to have any angry reaction toward whoever fabricated, manipulated, or messed up the data.
Something similar happened in the Wansink case, where he claimed that data collected from preschoolers was actually collected from 8–11 year olds. The story changed about three times until the JAMA journals lost patience. In academia, that sort of thing apparently makes you a quirky eccentric and increases your speaking fee. In the justice system, as I suspect Gino is about to find out the hard way, changing your story is treated as evidence of lying.
I think this phenomenon of weak studies (or even weaker popular airport books) being published, then leading onto TED talks and more books, is part of a broader phenomenon. Since the 2000s the number of avenues (websites, conferences, publishers) looking for easily digestible memes that can be monetized has grown. This has lead to an increase in the demand for scientific memes at exactly the same time as knowledge of substandard science has grown. These two trends are inevitably in conflict.
More than a decade ago, Evgeny Morozov wrote about this broader phenomenon of “meme laundering” where credulity is the norm : https://newrepublic.com/article/105703/the-naked-and-the-ted-khanna
Chetan:
Morozov! It’s because of him that I backed out of using the Jeffrey Epstein agents for a book I was writing.
John Brockman was a high-powered literary agent for scientist authors. He had a fairly successful strategy for linking his clients together in mutual backscratching ways to promote each other’s fame.
But he was always way too tied in with Jeffrey Epstein.
Heck, back in 1989, Epstein’s girlfriend’s dad Robert Maxwell tried to unethically chisel me, Dr. Evil-like, out of one … MILLION … dollars.
This was a couple of years before Maxwell plummeted off his yacht just before it was revealed he’d embezzled his employees’ pensions. Oh, and he’d been a spy for the Soviets and the Israelis.
These people were always bad news.
Yeah, I dodged a bullet with that one. Just my good luck that the revelations came out shortly before I signed anything. I’m sure I could’ve gotten out of any contract had I wanted to, but it was simpler to have never had any formal connection to break.
My guess is that a lot of academics saw Malcolm Gladwell become a celebrity public speaker for writing bestsellers that included references one or two of their papers, and so they got to strategizing about how they could move from mere researcher to famous pundit and motivational speaker. After all, they know more about psychology than Malcolm does, so why shouldn’t they have fabulous career like his?
Their business plan tended to be:
Step 1. Discover micro-finding X
Step 2. ???
Step 3. Become famous public speaker on macro topic Y
Unlike with the Underwear Gnomes, however, Step 2 is pretty obvious: Do a lot of self-promotion exaggerating the importance of your micro-finding X as the skeleton key that unlocks macro topic Y.
Citing South Park references (underwear gnomes to explain the labor theory of value) warmed my heart!
“This is big if true” has become a kind of flag for me.
For me it signals someone pushing an agenda and trying to use information to advance that agenda even if they have no idea and/or haven’t take the time to assess whether that information is reliable.
Small typo here: “Third, if you honestly your work can help the world,” missing a “think” after “honestly”
Another typo: The coyote’s first name is Wile, not Wiley. That’s why he also has an “E.” Perhaps Andrew was primed by thoughts of a large academic publisher. ;-)
Oof! Pretty embarrassing to be corrected on that one by a foreigner.
I agree with your larger point about leaps in logic from narrow studies to large claims and the power of reputation. But I think you are missing a big player here in who cares about honesty pledges – governments! It has huge implications (big if true!) for public financial management and systems of self-reported taxes, for example. It also magnifies the harm of this kind of fraud – governments spent time and money trying to replicate and test this in the real world – waste of taxpayer money and wasted opportunity to pursue something else that could have been more effective.
The latest Ted Talk controversy was over a Black speaker who advocated color-blindness. This raised some objections, and a social scientist was asked to give an assessment.
https://www.thefp.com/p/adam-grant-chris-anderson-respond-coleman-hughes
Roger:
Fun link. Watch that B.S. flying! My favorite is when the Ted talk guy writes, “He’s off-the-charts smart.” I guess he’d already used “whip-smart” that day so he needed to go for a different cliche.
(Trying not to sound like Jerry Seinfeld or Andy Rooney.)
Have you noticed a rise in the term “off-the-charts” lately? I have seen many examples in baseball commentaries where some player’s statistic went from X to Y and, accompanied by a *bar chart* the guy describing it would call it an “off-the-chart” result, which seems particularly humorous while showing a chart which includes all the data.
I didn’t notice that, but the word “legendary” seems to be used everywhere. Part of the hype and exaggeration cycle, I believe.
The answer to this question has to come from outside of the content of science itself. I find recent work in the Sociology of Science, SSS/STS literature most informative.
Just one example, directly on the point of the rise of the impact of market forces on scientists:
https://journals.sagepub.com/doi/10.1177/0306312718772086
Kevin:
Agreed. The sort of introspection and reflection that I do in the above post is limited as a research method!
Your comment about why anyone should have cared about these studies in the first place fits with Adam Mastroianni’s informal survey of how many academic psychologists he knew who could name specific papers that failed replication in the wake of the replication crisis (as highlighted today at Marginal Revolution). Nobody cares which studies were wrong because there was never any reason to care about said studies in the first place.
Yes. The big questions are, of course, what’s intelligence, how does it work, and how much of it do any of the other animals around have. We’re doing rather badly on these questions, so the questions actually asked end up being inane, trivial, silly. And forgettable.
A funny thing here is that attempts to figure out how we’re different from the other animals is failing so miserably. As I love to point out, not only are we the only animal that’s figured out the standard model, we’re the only animal on the planet that can understand that sex causes pregnancy and pregnancy leads to childbirth. (Note the “can”; “can” for humans is big: it tells us what we are capable of when we don’t mess up. Messing up doesn’t mean “can’t”.)
However, in the surprise of the year, someone’s actually found something.
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0290546
It seems we humans are the only animals that are good at recognizing sequences of events. And all the other animals are really really bad at it.
IMHO, this is actually really big. I see too many articles showing birds can count to three and arguing that therefore doing mathematics isn’t uniquely human. Give me a break city on steroids.
Just wanted to share this paper that uses hierarchical modeling (and references you) and portfolio optimization in a novel way from Deepmind. I thought it was brilliant
https://arxiv.org/abs/2309.07893v1?j%3DeyJ1IjoiMmV1ZHUifQ.fs2VjqM3aPdldmJz73mQF0JBoyNUfLbvLl3t-9RyvxI=&source=gmail&ust=1696311058954000&usg=AOvVaw00mT7itEl-SPrwANGPyxW9&utm_source=substack&utm_medium=email
i’ll say the answer is that people who buy these airport books are morons with no scientific training…the (supportive!) comments on Gino’s LinkedIn posts about the suit are something else
Dl:
The people who buy these books are not all morons. My mom read Freakonomics when it came out, and she loved it. And my mom is no dummy.
Ok, she didn’t actually buy the book, she checked it out of the library, but still.
I don’t fly very much any more, but when I do I make sure to check out the airport bookstore, because airport bookstores tend to stock a lot of pretty good nonfiction books aimed at c. 115 IQ frequent flyers that I might find either informative or debunkable.
Steve:
I’ve become frustrated at the way in which airport-style nonfiction books have become so standardized, just pushing one big idea in chapter after chapter, and with the whole book padded using big print, wide margins, and thick pages.
OK, the big print I understand—we’re all getting older, and our vision is getting worse (see recent post on this blog regarding glasses).
But the rest of it . . . I feel like the authors have so little to say, and they’re hoarding their material. Even when there’s good stuff, it’s padded with barely-relevant stories.
I was rereading Earl Weaver’s autobiography the other day—it came out in 1981—and I was struck by just how dense it was, full of good stories, speculations, character sketches, all sorts of things. Airport-style books—even good ones—don’t seem to do this anymore.
Andrew, did you just notice this about pop nonfiction books? I’ve noticed for at least 20 years that they are articles padded into books.
Anon:
Back in 2008, one frustration I had with the reception of our Red State Blue State book is that lots of people said there was no reason to read the book because the material was all in our Red State Blue State journal article. That wasn’t true at all! The book had tons and tons of material that was not in the article. Entire chapters of material on economic inequality, religion and voting, political polarization, income and voting in other countries, and lots more. But people just assumed the book had nothing more, because we designed the book to be friendly and accessible. I was soooo annoyed! We were harmed by the general perception of airport-style nonfiction. In retrospect I wish we’d abandoned the cute, memorable title and given it a more hardcore title such as American Voting: A Multidimensional Perspective. Something that would look less airport-like. Grrrrr.
Andrew, I wouldn’t say that it was just fluffy bizbooks that dud you wrong, but your own misleading title. You decided to package it in that mode.
Anon:
Sure, that’s what I just said! I gave it a bad title. In my defense, I was thinking not of the sort of empty bizbook that has one simple message repeated in chapter after chapter. I was loosely modeling it after Freakonomics, which had something new in every chapter. There were some ways that I wanted to do better than Freakonomics, in particular by graphing all of our findings so that the reader could actively participate in the process of discovery rather than just take our claims on faith, but I was aiming for that same fun-to-read treatment of important ideas in social science.
The trouble was that readers weren’t expecting something special like Freakonomics; they were primed to expect a generic bizbook, hence their annoying assumption that the entire book would be nothing more than an expansion of a 30-page research article. As I said, had I realized that this would be their expectation, I would’ve chosen a more hardcore title.
I read Freakonomics during my lunch break at a retail job with a small book section. Some things haven’t held up (I remember the abortion-cut-crime paper was found to have mistakes), but it wasn’t really a One Big Theory book and instead a collection of surprising findings. I think the fact that Levitt himself was writing it alongside Dubner distinguished it from Gladwell, who is just reporting back his version of things from experts in fields he’s far from an expert in (hence “Igon Value”).
A lot of the popularit of this work at the end of the day is about finding an easy way around a hard problem.
Controlling one’s appetite at the dinner table? Try this one weird trick.
People cheating on their taxes? One weird trick.
People seeing women as of less stature? One weird trick.
People saw these tactics as little cheat codes to improve the world without really changing anything, dealing at all with distributive justice, or otherwise making any group sacrifice anything.
I think this explains a lot of the demand side of the research. The supply side, I guess, follows a similar logic. Perform this one weird trick and you can get a 1 million dollar salary from Harvard, a named professorship, and 100k speaking gigs.
Well said. The cognitive bias here (laziness) is the culprit.
Jesse Singal wrote a book about that, “The Quick Fix: Why Fad Psychology Can’t Cure Our Social Ills”. I admittedly haven’t read it.
Absolutely yes; a lot of the appeal was that it purports to offer change without much work and without changing anything important.
If you really believed in the whole subtle-influence research program, surely you’d also have to believe that advertising and political campaigning should be regulated about as strictly as nuclear reactors, and probably with a view to eventually getting rid of them, like cigarettes? But one thing they fought very shy of was proposing anything big business, or for that matter the political establishment, wouldn’t like.
I came back to this post and the comments, because I remembered some people have been talking about the Francesca Gino case here, and more specifically the options of staying quiet or go on the attack or defend herself.
I just read a bit on her website “Francesca V Harvard”, and I wonder whether she is doing herself a favor writing on there. In the recent post about the New Yorker article she mentions that there were a number of studies that did not produce results for example, which reminded me of possible file drawer issues and overall problems with publishing psychology studies which she might now put a spotlight on.
There also seem to be several possible mistakes in that post according to my interpretation, which seem to hinge on her interpretation of certain sentences and words used in the New Yorker article which to me however seem to be depicted correctly. The entire case is a trainwreck unfolding live in a way. It’s perhaps representative of many things that might be wrong in parts of social science, one thing being a pet peeve of mine: scientists talking to journalists.