There was this Freakonomics podcast, “Why Is There So Much Fraud in Academia?” Several people emailed me about it, pointing out the irony that the Freakonomics franchise, which has promoted academic work of such varying quality (some excellent, some dubious, some that’s out-and-out horrible), had a feature on this topic without mentioning all the times that they’ve themselves fallen for bad science.
As Sean Manning puts it, “That sounds like an episode of the Suburban Housecat Podcast called ‘Why are bird populations declining?'”
And Nick Brown writes:
Consider the first study on the first page of the first chapter of the first Freakonomics book (Gneezy & Rustichini, 2000, “A Fine is a Price”, 10.1086/468061), in which, when daycare centres in Israel started “fining” parents for arriving late to pick up their children, the amount of lateness actually went up. I have difficulty in believing that this study took place exactly as described; for example, the number of children in each centre appears to remain exactly the same throughout the 20 weeks of the study, with no mention of any new arrivals, dropouts, or days off due to illness or any other reason. Since noticing this, I have discovered that an Israeli economist named Ariel Rubinstein had similar concerns (https://arielrubinstein.tau.ac.il/papers/76.pdf. pp. 249–251). He contacted the authors, who promised to put him in touch with the staff of the daycare centres, but then sadly lost the list of their names. The paper has over 3,200 citations on Google Scholar.
I replied: Indeed, the Freakonomics team has never backed down on many ridiculous causes they have promoted, including the innumerate claim that beautiful parents are 36% more likely to have girls and some climate change denial. But I’m not criticizing the researchers who participated in this latest Freakonomics show; we have to work with the news media we have, flawed as they are.
And, as I’ve said many times before, Freakonomics has so much good stuff. That’s why I’m disappointed, first when they lower their standards and second when they don’t acknowledge or wrestle with their past mistakes. It’s not too late! They could still do a few shows—or even write a book!—on various erroneous claims they’ve promoted over the years. It would be interesting, it would fit their brand, it could be educational and also lots of fun.
This is similar to something that occurs in the behavioral economics literature: there’s so much research on how people make mistakes, how we’re wired to get the wrong answer, etc., but then not so much about the systematic errors made in the behavioral economics literature itself. As they’d say in Freakonomics, behavior can be driven by incentives.
P.S. Some interesting discussion in comments regarding the Gneezy and Rustichini paper. I’ve not looked into this one in detail, and my concerns with Freakonomics don’t come from that example but from various other cases over the years where they’ve promoted obviously bad science; see above links.
the israeli daycare paper was replicated by the original authors in a second experiment that specifically addressed measurement issues highlighted by ariel rubinstein:
https://arielrubinstein.tau.ac.il/papers/WC05/GR1.pdf
looks like it was never published aside from this rough draft posted on rubinstein’s website
I was not aware of this paper (on the day care center fines) but found this interesting. I’m not sure why Nick Brown said the number of children was constant over the time period – I did not see that assumption in the paper. All I saw was a single value for the number of children and then the average number of parents late over time. The single figure for the number of children could be the result of the same number every week, or sloppy reporting of the total number registered in each center and no details on how many showed up each week (if that is the case, then I’d certainly like to see the data on how many no shows there were and perhaps the % of parents late would be a better measure than the absolute number). The link for the Rubinstein critique did not work for me but the replication did.
I’m inclined to believe the results and I found their “cognitive dissonance” explanation somewhat sensible. I’d put it in these terms: there was an incomplete contract in that parents were not told what would happen if they came late. It seems reasonable that they might view teachers having to stay late as altruistic behavior. Once the fine (small one, at that) is introduced, they have monetized that part of the transaction. Parents might then see this as a feature of the contract: am I willing to pay the small fine in order to pick up my child late – gone is the altruistic behavior of teachers that was modifying parents’ behavior.
There is a similar literature about paying for blood donations. When payment occurs, there may be two distinct groups of donors – those that do it for altruistic reasons and those that do it for money. The payment is a positive incentive for the second group, but may be a negative incentive for the first group: “if they are going to pay people, then I am no longer interested in donating.” I’ve seen some philosophical literature along these lines that addresses what is different about market transactions and non-market behavior.
So, I’m not sure I find this example of low Freakonomics standards compelling, although I think the point about their widely varying quality to be true.
Indeed, in the paper A Fine is a Price (open access at https://www.jstor.org/stable/10.1086/468061) we see large variations between day-care centres. The number of late comers moves around quite a lot in some day-care centres, and the four weeks of measurement before the treatment may not give us enough information about the individual baseline variance of the individual day-care centre. The same applies to the post-treatment period. My overall takeaway from the study is that it looks like a bad idea to fine the parents of children from Israeli nurseries.
Inspired by the comment above, I also found a ‘replication’ called ‘Is a fine still a price? Replication as robustness in empirical legal studies’ (available for free on SSRN). I put quotation marks around ‘replication’ because they used MTurk for online experiments – it tries to replicate the principle, not the setup. In the abstract, they summarise their findings as
‘Our survey results do not replicate the original findings. In both our daycare and tax studies, the introduction of fines causes respondents to reduce non-compliant behaviour. Respondents expect others to behave similarly. Fines do not cause respondents to adjust their concerns about an incomplete contract consistently with Gneezy & Rustichini’s theory. They also show very little evidence of fines crowding out social motivations, despite being responsive to our social treatments. The effects of fines on outcome behaviours and respondents’ reasons are transitory. Once the fines are removed, our respondents return to their baseline behaviours. The survey results are consistent with our intuitions (and standard rational-choice theory) that fines deter.’
Something that took me a while to realise is that ‘A Fine is a Price’ is a misleading title – they offer several possible explanations for their findings: ‘a model of differential information and incomplete contracts’ (parents avoid arriving late for fear of severe consequences, and the fine signals mild consequences, reducing their fear of punishment), multiple equilibria models (the introduction of the fine shifted outcomes from an equilibrium with few late arrivals to a new equilibrium with more late arrivals), and of course social norms (parents’ perception of arriving late changed, the fine reduced their perception of inconvenience to the nursery school teacher). All of these explanations fit the data, and I am still puzzled as to why the authors chose only one explanation in the title (except that it is really catchy!).
On a completely different note, I have fond memories of studying this study. In a behavioural economics class, I was part of a group that had to make a video summarising the paper. It counted towards our final grade. I came up with most of the content, and one of the MBA students in my group came up with lots of nice animations and effects. He even found us an incredibly talented voice actor. I didn’t learn a lot of useful things from that course, but it was a lot of fun and motivated me to dig deep into the paper.
The idea of trying to replicate “a fine is a price” with an MTurk survey feels strange … I mean, generally, the idea that you can replace the act of running an experiment with just giving people a survey that says, “Imagine you’re a different person and we did this experiment to you. What do you think you would have done?” … and get comparable results to the actual experiment? On looking at the paper ( https://scholarship.law.ua.edu/fac_working_papers/730/ ) I do see their conclusions section mentions this concern, saying:
> “While we believe the survey tells us something important about how respondents think they would behave, and perhaps how they want to behave, differences between this ex ante decision environment and real life may lead them to behave differently.”
and
> “anecdotal evidence like this suggests the effect may still exist in the field. While our results suggest most people would not intend to behave this way, and that it is unlikely they do so because the simple existence of the fine displaces social controls, a many studies approach is likely needed to fully explain the behaviour.”
Also, intuitively it seems like to use MTurk workers is to specifically select for price sensitive subjects (after all, they responded to the monetary incentive to do the survey), which is an interesting choice for replicating a study about price/fine insensitivity …
Dale: This URL worked for me just now for the Rubinstein commentary:
https : / / arielrubinstein.tau.ac.il / papers / 76.pdf
I have put a bunch of spaces (8 in total) in there because this blog is set up to send comments to moderation if they contain links.
Regarding whether the results seem plausible or not, I don’t know if that tells us anything about their veracity. As long as social scientists don’t tell us something that is either hugely counterintuitive (which goes way beyond the “quirkiness” of Freakonomics) or socially unacceptable, I think that our acceptance of it is to some extent dependent on how much we like the story and want to agree with it.
Consider Diederik Stapel’s most famous paper, in which he demonstrated (in a very clever and plausible experiment that did not actually take place) that people express more racist attitudes and behaviours in a messy environment. We all hate dirt and we all hate racism, so that was an easy sell. But it’s not at all hard to come up with a theory of why a messy environment might reduce racism (“collective recognition of our common humanity in the face of adversity”, etc), for which Stapel would merely have to invert his fake results. But this would have caused huge controversy and been the subject of great skepticism from day one (and might well not even have been accepted in Science). It would imply, for example, that the mayor of a city would have to choose between the sanitation budget and the anti-racism budget. Stapel himself said in an interview with the NYT that he avoided making claims that might have been remotely controversial for exactly this reason.
Quote from above: “Why Is There So Much Fraud in Academia?”
I have been wondering this for a while, and also concerning other forms or versions of “bad” science and “bad” scientists in academia. My current reasoning and wondering concerning this is that I think there might have been 1) a “natural selection” of bad psychological scientists, among which 2) a subgroup of “bad” scientists which psychopathic personalities and/or characteristics might be present. This latter group might be especially related to unethical behavior, fraud, and research misconduct.
Should anyone be interested, the general ideas about this can be read in a manuscript I posted on SSRN titled “The Natural Selection Of Bad Psychological Scientists”, and a more detailed depiction of the possibility of the presence, and even possible prosperousness, of psychopathy in psychological science can be read in a manuscript I posted on SSRN titled “Things I Have Wondered (So Far)”.
Side Note: both titles of these manuscripts, and the content, are influenced by two papers titled “The Natural Selection Of Bad Science” by Smaldino & McElreath (2016) and “Things I Have Learned (So Far)” by Cohen (1990). Points for those that thought about both of these original papers!!
Oh yes, it’s all about psychopathology! If one wants to make a grabby claim, be sure to throw in some psychopaths.
How big could the incentive of being featured in Freakonomics possibly be? Of all the academics using QRPs or committing outright fraud, what percentage of them could possibly be doing so in an effort to be featured in Freakonomics? I have a hard time imagining this could account for a meaningful amount of fraudulent science.
Noah:
I agree that the incentive for being featured in Freakonomics itself is not so large. My concern with incentives here is that the same sorts of things that will get you featured in Freakonomics can also get you featured in NPR, Ted, Gladwell, Nudge, and other prestige media sources, which then provide a sort of validation for everyone else to just assume your work is correct and important.
If Freakonomics were to disappear today (which I wouldn’t want to happen; it does a lot of great things!), the incentives to appear in NPR, Ted, etc., would still be there. I’m certainly not claiming that Freakonomics has caused the junk science problem or that it is, by itself, a major contributor to it. But it is part of the problem! And it could do better, by forthrightly acknowledging past errors instead of acting as if they never happened. From the journalism standpoint, I understand their approach of moving on and not examining the past. But as a scientist and a scholar, I think we can learn from our past mistakes. And, if they’re gonna go to the trouble of covering the topic of “Why is there so much fraud in academia,” I think it would make sense for them to cover the junk science they’ve featured. And, yes, they’ve featured the work of Brian Wansink and Dan Ariely. Beyond that, though, I’m just as concerned about the junk science they’ve promoted that isn’t fraud but just the frothy mixture of statistical significance and bad data that is sometimes the byproduct of noisy measurements, weak theory, and overclaiming of results.
Andrew:
Do media really “incentivize” junk science?
Isn’t the reality is that all of this garbage sciece exists not because of NPR or Ted Talks, but because it’s accepted, encouraged and treated as legit in the sub-community of “scientists” that produces it? All of this “research” is published in science journals and the funding for it is approved by other “scientists.” Seems to me like it’s more the other way around: popular media rely on the authortative endorsement of junk science to sell it. Without that endorsement, it’s all National Enquirer crap. With that endorsement – like Biden’s dimentia – it’s “news.”
Chipmunk:
I think it’s a complicated system. PNAS and Psychological Science are entirely within academia, and they publish and promote crap. That said, I think the fact that this crap gets media attention is part of the appeal. Both players in the game win: the media gets science-endorsed studies to write about, well-connected scientists get some amount of fame and some amount of money. Remember that the American Sociological Association gave an award to Malcolm Gladwell!
I’m the first to claim that journalists are pretty much the most maleable and incompetent “profession” in western culture. Just the same, in my comment above I wrote:
‘All of this “research” is published in science journals and the funding for it is approved by other “scientists.” ‘
I failed to mention that “this research” is carried out by cerified and approved PhD professors. Journalists are not supposed to question other claims made by the Infallible Truth Tellers of Academia [e.g., climate disaster, peak oil, populatation bomb] – in fact such questioning has been directly attacked by the Federal Government. So what makes quack social science claims special, that journalists are supposed to understand that, here, in this one particular context, they are are supposed to question academics and relieve the SFBrtds in academia of their responsibilities?
I guess it’s complicated because no one in academia wants to risk their funding by attacking quacks, they’d rather just blame journalists?
The point isn’t Freakonomics per se has a giant impact, but that it’s a clear example of a broad class of low-standards media / public relations platforms that credulously promote shoddy science. Since publicity is valued, e.g. by administrators, this class of platforms incentivizes mediocre or over-hyped work, and even fraud. (See also yesterday’s post, on the American Heart Association’s press releases. This annoyed me so much that I wrote to them.) Freakonomics complaining about rampant fraud is a bit like a drug dealer complaining that too many people use drugs — it’s not that a single dealer has a giant impact on the drug market, but it shows rather a lack of self-awareness.
“… I have a hard time imagining this could account for a meaningful amount of fraudulent science.”
Freakonomics promoting questionable work is just an example of the bad incentives in place not the sole source of them.
In an exchange with Chipmunk, Andrew just wrote,
“Remember that the American Sociological Association gave an award to Malcolm Gladwell!”
This criticism surprised me because I thought that Andrew has had a recent (reluctant) rapprochement with Gladwell’s writings. Unfortunately, I am unable to back up my assertion.
Paul:
Yeah, I liked this one. But that’s not what they were giving him an award for!
Well, I did find this
https://www.youtube.com/watch?v=HSFmBf3Yl00
“Getting out of Dodge”
Malcolm Gladwell explores how classic westerns may have influenced SCOTUS rulings on gun laws
where Gladwell does a fantastic explanation regarding the wackiness of the U.S. gun laws.
“Gladwell does a fantastic explanation regarding the wackiness of the U.S. gun laws.”
If by ” does a fantastic explanation,” you mean make shit up, then yes, Gladwell does a fantastic explanation of that.
“Malcolm Gladwell explores how classic westerns may have influenced SCOTUS rulings on gun laws”.
If by “explore,” you mean make shit up, then yes, Gladwell does explore that.
A couple of minor points.
The “fine” is not immediate. It shows up on the monthly bill. IIRC, in Operant Conditioning the consequence should be as close to the behaviour as possible. This model is more like getting your credit-card bill at the end of the month and discovering you have have been binging on lattes.
The NIS 10 is called “relatively small but not insignificant.” The authors then state that “the average gross salary per month in Israel at the time of the study was NIS 5,595”. This is not the population of interest. The population of interest is the parents using the day cares.
There is a distinction between the books and the podcast franchise. Steven Dubner seems to have spun off the NPR show into a separate business, with Steven Levitt as one of the hosts of a podcast, not his main partner. Dubner dabbles in different social topics–some of which he takes deep dives. I found the podcast about Dallas very informative as I knew little about the fastest growing American city. The series on academic fraud pulled together different strands I knew from this blog and other places.
“I’m disappointed, first when they lower their standards and second when they don’t acknowledge or wrestle with their past mistakes”
If Freakonomics did not lower its standards or did acknowledge or wrestle with past mistakes, it could no longer make the grabby claims at the heart of its business.
P.S Grabby claims is all Malekan is about.