There’s been a lot of news coverage lately on the replication crisis in psychology and related fields. Simine Vazire and I wrote something a couple years ago, Why did it take so many decades for the behavioral sciences to develop a sense of crisis around methodology and replication?, exploring more formally the timeline I’d discussed in a much-discussed post from 2016. Why this has been motivating so much discussion just now in the news media is another story, I guess related to their being a big lawsuit in the air. Lawsuit = news, I guess? This article by Gideon Lewis-Kraus seems like a good summary.
Just one thing, though. The subtitle of that article is, “Dan Ariely and Francesca Gino became famous for their research into why we bend the truth. Now they’ve both been accused of fabricating data.” We see two things that should help the reader engage: (a) a focus on individual personalities and life stories, and (b) the accusation of fabricating data. Both these things are worth discussing—people’s choices matter, and fraud is always a concern (as would be mistaken accusations of fraud). Indeed, we had a long discussion just a couple months ago on cheating in science, sports, journalism, business, and art, riffing on a book by financier Dan Davies.
So, sure, but . . . I continue to think that the big problem of non-replicability is not fraud so much as the misguided expectations: the idea that science is supposed to be some endless stream of discoveries and a refusal to admit error. Here are a few relevant posts:
– Honesty and transparency are not enough
– Psychology needs to get tired of winning
– Clarke’s Law: Any sufficiently crappy research is indistinguishable from fraud
– Here’s why I don’t trust the Nudgelords . . .
– The real problem of that nudge meta-analysis is not that it includes 12 papers by noted fraudsters; it’s the GIGO of it all
One refreshing difference of the current headlines compared to what came in the past is that, for whatever reasons, some of the researchers involved in the latest scandals appear open to admitting that much of their past work is just wrong, that its nonreplication is not just some technical problem but rather is a reflection that in the past they were basically doing the experimental equivalent of generating random numbers and using them to tell stories. I still have unpleasant memories of political scientists insisting, in the face of all evidence, that subliminal smiley faces have large effects on attitudes toward immigration; of a sociologist avoiding looking at careful explanations of how his much-publicized claims were nothing more than noise mining; of the himmicanes and air rage people never giving up; the Freakonomists not coming to terms with their promotion of climate change denial; the nudgelords memory-holing their former adoration of a now-discredited food behavior researcher; etc etc etc.
I guess what I’m saying is an important step forward in the current discussion of replication problems, both in science and in the news media, is to recognize that so much of this research is just no good, at best ridiculously overestimating effect sizes and setting up a false sense of certainty in a world that is highly variable. All this is separate from any questions of fraud and blame.
Economics stands alone. I visited UW milwaukee econ dept yesterday and was impressed. They have a 5 year BA and MS combined degree. Only requires intro calc. That is ok. Not everyone needs to be deep in stats, probability, advanced calc to earn a meaningful degree. A grad student told me about recent trends in hot areas like environmental and labor economics using game theory to model firm power etc.
Most econ people have no interaction with the sociology clownshows. The enmity between those fields goes way back.
Speaking of sociology. It is rock bottom of the social sciences and i with Andrew wouldn’t lump them all together. Below is a recent email from a distinguished active sociologist and a so called progressive
“The field only started to have signs of wokeness in the 1970s (maybe the late-1960s, but they were fringe elements at the time), and the field didn’t become heavily woke until probably the 1990s. Even then, those at the top of the field were still holding their ground, although wokeness was welling up in less distinguished parts of the discipline. By the 2000s and more recently, however, wokeness has completely taken over.”
Joey:
There’s bad work in econ too, some of which is published by top researchers in top journals. Here are some examples:
https://statmodeling.stat.columbia.edu/2013/01/10/that-controversial-claim-that-high-genetic-diversity-or-low-genetic-diversity-is-bad-for-the-economy/
https://statmodeling.stat.columbia.edu/2013/08/05/evidence-on-the-impact/
https://statmodeling.stat.columbia.edu/2017/07/20/nobel-prize-winning-economist-become-victim-bog-standard-selection-bias/
Joey,
I think your correspondent is talking about postmodernists or critical theorists (or whatever they are called these days), who infect other departments besides sociology, especially humanities departments and anthropology. Anyway, let’s call them out by a better name than woke.
But, why do you call your correspondent a “so called progressive?” I call myself a progressive, was so before the postmodernists came along, and seem to agree with your correspondent.
Sorry for semantics confusion. I should say liberals rather than progressives.
The precepts of woke for my purposes are here under 4th heading
https://www.ihatesociology.com/todays-sociology
So this may be only quarter-knowledge on my part, but I suspect one reason so much “woke” scholarship comes from sociology departments is that sociology uniquely straddles two traditions: positivism and interpretivism. Most other social sciences—economics and psychology, for example—adhere to a positivist paradigm, treating social phenomena like natural phenomena and studying them through observation, measurement, and statistical models. In contrast, interpretivism argues that understanding human action requires grasping the subjective meanings individuals attach to their experiences. It favors qualitative, context-rich methods over falsifiable laws and aims to interpret social reality and offer new perspectives rather than produce universal theories.
Critics may label this approach “unscientific,” but that may be besides the point: just as a novel can reveal truths about our lives or a therapy session can generate insights without being scientific, interpretive sociology produces its own form of knowledge.
so to summarize, this interpretivist lineage explains both the prevalence of “woke” work in sociology and why it resists critique by positivist standards.
But I admit I didn’t read any woke study—if any explicitly claim positivist rigor and fall short empirically, they should of course be criticized.
I don’t like the “woke” label (I haven’t seen a good definition – it appears to represent anything disliked by MAGA types) and I don’t find the blanket criticisms of sociology insightful. I am not a sociologist and personally find reading sociology painful (a few equations would help!). But I accept that sociologists generally have “liberal” views and don’t seem to represent “conservative” views very often. Just like the fact that a large percentage of academics vote Democratic rather than Republican – this is not evidence of a lack of diversity, it is evidence of their preferred positions. I’m tired of the unstated belief that somehow every viewpoint should be equally represented, otherwise it is evidence of “wokeism” or censorship. I am willing to believe that sociologists have come to their views as a result of their subject matter expertise, training, and research. It may not be the research I would do, but I’m not willing to label the field as inferior or fundamentally flawed. We are too quick to dismiss fields that are not our own (I’m an economist and my fellow economists are among the worst in this respect). I think you will also find that female academics research somewhat different topics than male academics (this has been documented in econ) – is that evidence of “wokeism,” “lack of diversity,” or is it a result of different life experiences? Perhaps these generalities are not useful, and instead of blanket criticisms, we should engage with the actual content of these subjects. I think I am agreeing with huan here.
“woke” has become basically just a code word for leftist. I admit there are a number of grifts on the left (one that was pointed out to me was that the national level “Black Lives Matter” organization’s finances look a lot like a scam, though grassroots local organizations seem largely independent). But of course right wing Trump politics is almost nothing BUT scam with Trump himself running crypto coin rug pulls to launder money and seemingly take bribes, promoting unsupported lies about 2020 election outcomes, and of course exporting essentially all of the US govts sensitive data on individuals to unelected tech oligarchs through the whole DOGE fiasco.
So, american politics is full of crap at the moment, but the history of the use of “Woke” is by black americans describing white people who were aware of systemic injustices for minorities and fought against them. Systemic injustices, like redlining, like asymmetric law enforcement, like creating welfare systems designed primarily to control and lock people into poverty, like the CIA supporting overseas organizations without regard to whether they traffic drugs in minority US communities, like race targeted vote suppression laws esp. in the south, like failure to respond to natural disasters in minority communities, or even harmful responses (Katrina comes to mind), are a huge problem in the US. And many people refuse to acknowledge them, close their eyes and ears to them, or deny them or attribute the outcomes to some “inevitable, inherent” failing of the various groups.
The existence of these systemic injustices however becomes a fertile ground for creating academic grift, where by writing junk academic articles you can promote your career on the backs of the oppressed minority groups you claim to support. I wouldn’t claim to have any kind of quantitative view of how much of that is going on, but you’d have to argue very strongly to make me believe it was near zero.
We need a new subfield of economics, we’ve got micro and macro, now we need grifto.
Dale, I think I mainly agree with you. In my view, woke sociology is criticized in three ways—one of them clearly invalid, one valid, and one I’m unsure about.
1. “Oh look, these woke sociologists are researching minorities again, so their work must be garbage.”
This is not a valid criticism. The topic of research alone doesn’t determine the quality of the work. I believe this is what you push back against when you point out that a research focus often reflects different life experiences rather than any inherent bias.
2. “Oh look, these woke sociologists are researching minorities and openly describe themselves as activists. That must mean their research is biased.”
While not logically airtight—activists can produce rigorous research—I think there’s some weight to this concern. In today’s polarized climate, someone passionately advocating for, say, trans rights may hold strong moral convictions. If such a person then researches these topics, there’s a real risk that objectivity is compromised, or that objectivity isn’t even considered a goal.
3. “Oh look, these woke sociologists are researching minorities, but their hypotheses aren’t even falsifiable.”
This seems to point to the longstanding divide between positivism and interpretivism. I’m not sure whether this criticism succeeds. It’s a bit like saying, “This novel is bad because it’s fictional.” I assume many so-called woke sociologists are working within critical theory, which operates under a very different paradigm than what we know and do. Perhaps there are multiple valid ways to generate knowledge—not all of them strictly scientific.
That said, reasonable scholars should still be able to agree on some shared standards for studying society. Even arguments from interpretivist or critical theory traditions can benefit from coherence, empirical grounding, and intellectual transparency.
So yeah, I’m not sure.
Woke means “looking at everything through the lens of race.” A better word is “race reductionist”. This is a right wing view (see Susan Nieman’s “Left is Not Woke” for the argument; tl;dr it is essentialist and distracts from class issues by focusing on race), but liberals love wokeism because it allows them to feel like they are morally superior.
Daniel makes some good points, as well.
That description of econ sounds pretty woke to me.
They should first figure out where money comes from and how it trickles through the economy before worrying about “environmental economics”.
Ie, bank loans and Cantillon effect. If that’s not it, what is it? Econ is silent, distracted by political junk while wealth inequality skyrockets along with government budget/debts.
A new low for you – what in the world do you mean that they should ” figure out where money comes from and how it trickles through the economy before worrying about “environmental economics”?” Are you claiming a hierarchy of importance here or are you saying there is a hidden connection between monetary theory and environmental issues? If so, please educate us. Some of us (me) care deeply about environmental economics and not so much about banking. I know these are interconnected and following the money is important to understand how anything works in the economy, including environmental impacts. But you seem to be making some kind of criticism of “environmental economics” without specifying anything. I actually disagree with much of the “environmental economics” methodologies, but I have reasons and they don’t come from ignoring the Cantillon effect.
You can’t shoot a rocket into space without first figuring out some basics of chemistry and physics.
Just like econ theories that ignore the dynamics of money. Doing certain things out of order can only be wasting (sic) money, sorry.
In the end, I can easily be made to change my mind. Can you share a single finding from environmental economics that stood up to independent replication or made an otherwise surprising (yet accurate) prediction about the future?
Disclaimer: Of course, *anything* can yield interesting wild speculations (mushroom trips, divination).
The New Yorker article has some amazing sentences in it.
> The data manipulation and fraud allegations were hard to reconcile (for people who worked with him) with the fact that “none of them ever saw him get anywhere close to data”.
> “Dan was way too famous to worry about the publication process. There were papers I could barely get him to read.”
Interesting argument, that Dan could never have manipulated or faked data because he never goes anywhere near the data anyway.
Shravan:
I guess the idea is that he’s the idea man.
Part of the debate here is what we mean by “fraud.” The accusations against Gino etc refer to something extreme.
What I see in my field: papers where the authors have worked to get the data to agree with some hypothesis that they hold for other reasons. We see specifications that apparently support the author’s view but that have no justification in the statistics or in the substance of the issue. We also see efforts to claim either evidence is not what it seems or to claim there is no evidence on something when there absolutely is.
There are two “smoking guns” for this kind of work. Sometimes the replication code shows that the author mis-represented a variable or model in the text. This is not part of the “garden of forking paths.” This is a simple misrepresentation. The tip-off comes when, using the definition the text provides, the result does not favor the author’s case. In other cases, the variables used in statistical tests have been highly “processed” from the source: think of someone constructing an index as a weighted average of 4 things, instead of just controlling for the four things, and never showing that some alternative index construction yields similar results. Most of us would realize that we need to investigate the consequences of arbitrary modeling decisions; does it matter if we construct the index differently, for example.
Unfortunately, long online appendices with lots of “robustness checks” are just part of the problem. Most robustness checks have little to do with any serious concern. The long appendix makes the paper look serious and careful when it is not.
This is not “fraud” in the sense of just making up the data. The data are real and “replication” studies can reproduce the results using the public data and code journals provide.
The outcome is the same as fraud: research results one cannot trust because of the author’s motivations in doing the research. And the motivations are similar to fraud: to produce novel, clean results that will make the paper stand out in a field crowded with lots of research that does something similar. We would be deeply disappointed if our grad students did this stuff in papers they write for us, yet we reward people with tenure and promotion for doing it in the main journals.