Here:
Michael Sanders based his World Bank-funded study in Guatemala on a trick supposed to increase tax compliance, but now says Francesca Gino’s work misled him
At the time Michael Sanders remembers being embarrassed, even ashamed. He had convinced the World Bank to fund a study on an entire country to show that one simple trick could drastically increase tax compliance. If you made people pledge their honesty beforehand, research showed, they were more honest. But the one simple trick had proven tricksy. Despite strong prior evidence it should have had dramatic effects, it had done nothing at all. . . .
“I’m furious,” he said. The former member of the British government-created Behavioural Insights Team believes he has good reason to feel misled about that evidence. . . . Now Sanders is left wondering how they didn’t spot it earlier. Not least because Gino, who was paid $1 million a year for her media-friendly research, went on to write a book titled Rebel Talent: Why it Pays to Break the Rules at Work and in Life. “If only there was a clue,” Sanders said, “that she thought you could get ahead by breaking the rules.”
Here’s the full story:
Back in 2015 Sanders, now a professor at King’s College London, was working with the group better known as the Nudge Unit. Their job was to use behavioural science to help governments work better.
They had, they thought, found one such method. To keep people truthful when filling in official documents, make them sign the declaration of honesty before, rather than after. Then, when they put in their details, they will remember they had promised to tell the truth.
A trial by some of the world’s leading researchers, including from Harvard, had shown stunning success. It was Sanders’ job to validate it at scale. They went to Guatemala, a country with very low tax compliance. They conducted a study on all the tax returns, comparing those with honesty declarations at the start with those at the end.
He was confident. “It was an enormous amount of data. If there was an effect, we would see it. With samples that size, you can detect the sound of a gnat’s wing.” And the effect? “Nothing.”
They had promised the Guatemalans big impacts, and with justification. The authors of the original research were, said Sanders, “a who’s who of behavioural science.” Yet they couldn’t replicate the work. It was inexplicable.
The news article continues:
It is, today, more explicable. There is compelling evidence in the raw data to suggest Gino manipulated her results. She denies it, but Harvard apparently concurs. Which means that hers is just the latest Ted-talk-friendly research paper to tumble. . . . Many studies involved small numbers of participants and questionable statistics. . . .
He said they really should have thought more about her book. “It’s like if she wrote a book, My Adventures as a Pirate by Long John Silver, and then we still get surprised when it turns out Long John Silver is a bit of a wrong ‘un.”
This sort of news story is important. It’s easy to focus on the sloppy researchers and the careerists and the ideologues and the people whose names keep appearing on papers with fake data and the copiers-without-attribution and the cartoonishly overconfident hypesters and the megalomaniacs and the people who won’t acknowledge their mistakes and the people who never back down (see page 51 here) and the plagiarists and the torment executioners and the would-be manipulators of the system and the people who copy without attribution or shame, even from wikipedia, or . . . ok, you get the idea. It’s easy indeed to focus on the wrongdoers, or even to come up with elaborate justifications for their behavior or analyses of their character. It’s the fascination of moral and intellectual train wreck.
I have this image in my mind, that scene from The Fugitive where they’re hijacking the bus and then it crashes into the moving train. It’s that volatile combination of intellectual failure (bad research, statistical errors, lack of imagination, the one-way-street fallacy, etc.) with moral failure (not letting go, defensiveness, careerism, etc.).
Anyway, yeah, let’s look away from the villains for a moment and spare some thought for the victims, who in this case include some individual Guatemalans whose time was wasted, the Guatemalan government which isn’t getting the revenues they deserve and for which this failed intervention may have served as a distraction from a more direct approach to tax collection, the researchers who wasted some chunks of their careers on this (it’s ok to spend time on bad ideas, that happens to all of us; the problem here is that the wasted effort was based on falsified evidence, so it was really an unnecessary blind alley down which they were led), the British taxpayers who were funding that “nudge unit,” and . . . all of us. I just spent 20 minutes writing a blog post instead of making a valuable research contribution to the world, you just spent 5 minutes reading this, etc.
Now, you might say that this last inefficiency was unnecessary, as nobody was forcing me or paying me or otherwise strongly motivating me to write this post. But I think it’s important! Research incompetence, research fraud, and the promotion of fraudulent or incompetent work (recall Clarke’s law) . . . these are not victimless crimes.
Quote from the blog post: “Research incompetence, research fraud, and the promotion of fraudulent or incompetent work (recall Clarke’s law) . . . these are not victimless crimes.”
Yes! I wrote some stuff at some point in time which might be related to (the gist of) this post. I am sharing it here because I reason it underlines what I think might be part of the main point of this blog post:
“The problematic issues mentioned in the introduction made me wonder whether psychological scientists wonder (enough) that they are doing. If psychological scientists have engaged in, and contributed to, the problematic issues mentioned in the introduction, do they ever wonder whether they have actually been a “scientist”? I wonder whether psychological scientists who have engaged in questionable research practices ever think about how this might have affected, and might still affect, the literature and other research(-ers). I wonder whether psychological scientists who have engaged in questionable research practices ever think about how much resources may have been, and might be, wasted because of these kinds of actions (cf. Ioannidis, 2005). If academic tenure can be thought of as receiving a permanent position which may better allow for academic freedom in service of the common good, do psychology professors think about what that could mean regarding their options, choices, and actions? I wonder whether psychological scientists ever think about all the reasoning, hypotheses, theories, study designs, research findings, and papers that could be produced and published if other people would be in their position. I
wonder whether psychological scientists ever think about what the literature in their field could look like if other people would have designed, performed, and published research and papers. I wonder whether it would have been, and be, better and cheaper for Psychological Science and society at large if certain psychological scientists were, and are, paid to not do research and to not publish papers.”
Anon:
One way to say this is that, if psychological science is worth supporting, it’s because it has some value in itself, not just as the production of beautiful careers. And I do think that psychological science has both intellectual and social value. But, to the extent that’s true, bad or misguided research psychological science can have negative intellectual and social value. As for example what seems to have happened in Guatemala.
Quote from above: “But, to the extent that’s true, bad or misguided research psychological science can have negative intellectual and social value. As for example what seems to have happened in Guatemala.”
Yes! Yes! It’s just that I hardly read about anything beyond step 1 in recent years wich seems to be that “there might be some problematic issues” in psychological science. Well if there are, perhaps we should also think about step 2, 3, 4, 5 etc. One way to do this perhaps is merely ask some questions and wonder about things, which is what I did above, and will do again now here by quoting another section of what I wrote at one point in time. Hopefully I can now do it without any spelling mistakes (“that” was supposed to be “what” in the first sentence I quoted above), and without any space left open (that sometimes happens when I copy and paste here):
“I wonder whether there are psychological scientists who think they have the knowledge, wisdom, and/or right to (directly or indirectly) “nudge” or “steer” people and/or societies in a certain direction (cf. Neuhaus, 2022). Do psychological scientist ever wonder whether they might make things “worse” (in the long term) when trying to make things “better” (in the short term)? Do psychological scientists ever wonder about the potential negative consequences of certain knowledge they might help accumulate (cf. Popper, 1971, p. 283). I wonder whether psychological scientists that (directly or indirectly) help design and implement “behavioural interventions” are
aware of the possibility that these “behavioural interventions” can have unexpected effects or even opposite effects to those proposed and/or intended (cf. Stibe & Cugelman, 2016). If you are aware of the possibility of unexpected effects or even opposite effects to those proposed and/or intended of “behavioural interventions”, could and/or should you view the people that are affected by these “behavioural interventions” as participants in an experiment? Is there an “informed consent form” that members of society have to sign before “behavioural interventions” that might affect them are implemented? I wonder whether it is currently the case that “(…) most scientists, at least most creative scientists, value independent and critical thinking very highly. Most of them hate the very idea of a society manipulated by the technologists and by mass-communication.” (Popper, 1971, p.
283). I wonder whether psychological scientists think about the possibility that psychological treatments can cause harm, such as symptom worsening, the appearance of new symptoms, and excessive dependency on therapists (cf. Lilienfeld, 2007). I wonder whether psychological scientists think about the possibility of lasting negative effects of psychological interventions (cf. McQuaid et al., 2021). I wonder whether psychological scientists think about potential negative effects of psychotherapy in relation to informed consent (cf. Rozental et al., 2018, p. 309). I wonder whether the principle and/or perspective “first, do no harm” (primum non nocere) should receive much more attention in Psychological Science.”
I don’t think the question is whether well-done psychological science has value. That’s doing a p-test of value > 0. The harder but more useful question is estimating *how* much value is it produces. And in turn, are we as a society getting a reasonable return on publicly funded social science research.
Should this sentence by Michael Sanders,
“If only there was a clue,” Sanders said, “that she thought you could get ahead by breaking the rules.”
be amended to
“If only there were a clue,” Sanders said, “that she thought you could get ahead by breaking the rules.”
because he is being ironic/sarcastic, as in “wink, wink”?
I tried to find more details of the subjunctive mood in English
https://www.grammarly.com/blog/grammar/subjunctive/
but I gave up because it started to get too complicated.
The non-grammatical issue is that Gino, I assume unintentionally, was supposedly tipping her hand.
I have pondered whether Gino (and Stapel some 10 years back) performed some giant hoax. In light of this, I am also reminded of a paper by James (1995) who wrote about fraud and hoaxes in science (copy paste doesn’t work, so I’m typing this out, so please check and verify):
“Scientists seem to lack curiosity about what drives fraudsters. There clearly are several motivations, some of which are, admittedly, reprehensible. The more squalid fraudsters simply want tenure, or more research funds, or easy money from drug companies. They merit contempt. But others (for whom one may have some sympathy) go to immense trouble to pull an eloborate hoax. I doubt whether the harm done by scientific hoaxers is commensurate with the fuss made about them. Whoever planted the Piltdown skull provided an interesting problem for others to unravel, a stimulus for further research and some entertainment.” (p. 474)
When it comes to fraud, it is hard to top Cyril Burt who not only created phony data, he also created/credited the assistants who did the collecting and analyzing, Miss Howard and Miss Conway.
https://en.wikipedia.org/wiki/Cyril_Burt
From the above, here is an interesting defense of Burt:
“[n]o one with any statistical sophistication, and Burt had plenty, would report exactly the same correlation, 0.77, three times in succession [as sample size of separated identical twins increased] if he were trying to fake the data.” This, in a way, is similar to a defense of Gino: how could she not have known that the spreadsheet would indicate her alterations?
That reminds me of a recent comment here concerning the Gino case, where if I remember correctly someone pointed out a series of p < 0.001 or something like that in her first-authored earlier papers (?).
It also reminds me of the Levelt et al. (2012) report concerning the Stapel fraud case where the following is written:
"The statisticians found countless flaws while studying and re-analysing all the survey material. This is
inevitable to some extent; social psychology researchers cannot be expected to be aware of the latest,
specialized techniques in the statistics field. But it is disturbing that statistical flaws frequently revealed a
lack of familiarity with elementary statistics. For instance, one article had an entire series of t-test results
on the difference between pairs of means, of the type: t = 0.90, p< .05 and therefore ‘significant’. The co-
author was astounded to be told by the Levelt Committee that a co-author should have seen this absurdity." (p. 52)
The subjunctive in counterfactual conditionals is commonly replaced by the past indicative form in many dialects of English, especially in informal speech. In this case, “if only” makes it pretty clear that it’s a counterfactual anyway, so I don’t think changing the verb form would change the meaning.
You know another not-victimless crime? Economists saying that Guatemala, a country that issues its own currency, needs tax compliance because it would help them “raise revenue”.
The purpose of taxes is to create demand for the currency, and destroy excess money so that inflation isn’t excessive, and also to some extent to redistribute economic power from those who are taxed more to those who are taxed less.
Guatemala’s GDP/capita is about $8000, Gini coefficient is about 0.45, and inflation rate is about 3% (all based on world bank or our world in data charts). According to this:
https://taxsummaries.pwc.com/guatemala/individual/taxes-on-personal-income they have tax of 5% on the first 300k GTQ (about $39k USD) and 7% on amounts over that.
If they made their tax 0% on amounts under 300k GTQ and say 12% on amounts over that they’d get huge compliance for free, reduce their tax collection complexity (only a tiny fraction of people would need to file or pay taxes at all), reduce their Gini coefficient, and after a brief adjustment, by adjusting the 300k cutoff and the percentage for overage retain their low inflation rate.
Instead we get people pushing the idea that what’s really needed is more tax compliance and that poor people should be made to accept state control and surveillance over their economic lives and that the state “deserves” pointless income from poor people so poor people should be “nudged” to “comply” with state economic surveillance and state mandated reduction in their standard of living.
Same in the US, the original income tax was only “on the rich”:
https://en.wikipedia.org/wiki/Revenue_Act_of_1913
There is no debate that tax compliance increases when fewer individuals are liable for taxes. Lowering taxes for the poor would increase the tax burden on more affluent people, leading to greater tax evasion and tax avoidance on their part though.
However, I am not sure how many people would support such a reform. I think a significant proportion of people consider it important that *everyone* contributes to funding the government. This is not intended as a rebuttal to your comment, but rather to add nuance to the discussion. Designing an income tax schedule is not entirely easy!
But taxes **don’t** fund a government that issues its own currency. that’s fundamentally the thing I’m pointing out. If there were a pit into which money would inevitably just fall on its own (or, say, money was made of leaves or something that rotted away on its own), the government could just issue money to buy whatever it wanted to do, and then not do any taxation at all. Forever.
Since there isn’t such a place where money goes to die, instead the money supply would inflate and money would accumulate into the hands of those who spend it least (the rich) and so the government needs taxes to create that place where money can go to die, thereby keeping its quantity from getting too big and the concentration of power from getting too concentrated in the rich’s hands.
Saying that “everyone should contribute to funding the government” fundamentally misunderstands what taxes do, and how governments are “funded”. They issue currency, the *spending* simply *creates* money, and is in essence an order to submit your resources to the government’s control.
Let’s go a little further. Just for illustration, let’s create groups of people A,B,C,D etc.
Let’s say A is the people who directly sell services to the government: civil servants, companies that provide asphalt and steel, government contractors, etc. Let’s say B is the people that A primarily buy things from, say industrial suppliers, luxury vehicle makers, restaurant owners, grocery stores, and C is the workers at those firms, and D is the lowest level day laborers and agricultural workers who provide all the base materials needed by the others.
Now, the flow of material goods goes like this (arrows would be good but they bork the blog so I’ll use –} to indicate flow direction)
D –} C –} B –} A –} Govt
So what we see is that everyone **already** economically supplies the government even when there’s just money printing. And in fact most of what’s given up is given up by the people at the “bottom” while people higher up “close” to the government funding get resources by virtue of being “close to the money spigot” and therefore being able to simply purchase what’s needed from people farther from the spigot.
Adding in taxes on groups C and D just adds insult to injury as they then are forced to give up things like free time, quality shelter and food in excess of what they already gave up because they were in reality “funding” the government in the beginning (in terms of real resources given up).
When governments spend they create money, when they create money it shifts resources from people “far” from the money source, to people “close” to the money source. This tends to make people who are “close” very rich. When we tax we take money from various people, reducing their claim on resources. When we tax the people “far” from the money, they pay “twice” once, when they gave up their resources to people who get them due to money printing, and once when we take the money they got in exchange for those resources to make sure they don’t get uppity.
Creating a “tax compliance” system which taxes broadly across all of society is about establishing and perpetuating and growing inequality that causes poverty for the base level producers in society and wealth for those who suck up to the guys with guns.
@Daniel Lakeland:
I do not say that I want everyone to contribute to government funding. I said that many people want everyone to contribute. This is reflected in different attitudes, such as cutting social security to avoid ‘free riders’, calling for a general military draft, and introducing requirements to vote or participate in judicial proceedings.
When I say that the government needs to tax people, I do not necessarily mean monetary transfers to the government. Here, I am using the word ‘tax’ in a broader sense to signify the opportunity cost that people incur when they do something for the government. If you do something for the government, it gives you an IOU (usually in the form of money) to compensate you for the opportunity you have lost by spending your time in a different way. This IOU can then be used to make other people miss out on their opportunities.
My point is that the government receives real services, and these need to come from somewhere. Money shifts the timing and location, but ultimately, opportunities are taken from people. The same would hold true for a cashless society. The buck needs to stop somewhere. Wherever it stops, there is redistribution from individuals to the government. Where the buck stops must be carefully designed — that was my point.
Raphael,
Yes I fully agree with you that opportunity and real physical assets are the important consideration. But when we talk about taxation we talk about money unless you are carefully making a distinction as you did in your most recent post.
When we talk about getting tax compliance in Guatemala we talk about them forcing at gunpoint a large number of poor people to give up 5% of the money they earned, and thereby to give up 5% of the opportunity they were afforded as a compensation for doing something the government wanted done (indirectly). So first they gave up time and physical resources to provide something to the rich people close to the government, without those rich people giving up much if anything… And THEN they had 5% of what little opportunity for economic control they earned taken away. A 5% which must needs be coming out of very basic needs they have because they’re impoverished, by the system that creates money and points the guns that collect the taxes.
In other words, I argue it is a noble thing for the poor of Guatemala to avoid taxes and its a good thing they dont give a shit what Francesca Gino thinks.
Remember that it isn’t just income tax. Figuring in sales taxes, etc., the ITEP figures that the total tax rate for the top 1% is only about twice that for the bottom 20%: https://itep.org/who-pays-taxes-in-america-in-2024/
In the US, what do we even accomplish by taxing the lowest say 50% of earners at all?
Suppose you didn’t tax them and they instead spent the money and didn’t save any of it. Then the tax would have had no effect on the money supply because you’d have eventually taxed it out of the pockets of the wealthier people who accumulate it instead (if you had a sane tax system), accomplishing the goal of stabilizing the money supply. All you’ve accomplished is the reduction of demand for goods needed by the relatively poorer people, and therefore the reduction of *production* of goods for those poorer people, and therefore the systemic entrenchment of their poorness.
So then the only other reason you might legitimately tax the poorer groups is to reduce inflation of the goods they demand, which would be legitimate if there were some kind of out of control inflation. But in normal times, they demand very basic things, so by artificially suppressing the price signal and keeping the price of goods they want lower, you ensure fewer people want to produce those goods, and a generally lower supply of the basic goods they need, again systemically entrenching their poverty.
The main illegitimate thing that taxes on the lower 3/4 of society does is it suppresses the economic power of the bottom 3/4, it allows you to stabilize the money supply (sort of) *without* taxing away accumulated wealth and therefore power for the richest group, and ensures the stability of the inequality that the richest people like and want.
That all probably goes even more so for Guatemala.
People in the poorest say 1/3 of society already physically produce the bulk of all the actual *material goods* utilized by everyone, they already ship those goods to people above them in the hierarchy in exchange for a fraction of their market value, then they consume primarily goods necessary for basic maintenance of their bodies… shelter from hot/cold, food, transportation to and from work, healthcare, childcare, those put together make up a giant fraction of the income of people making less than say 50%tile of income in the US (median household income is $84k/yr, median household size is like 2.5 people so that’s like $34k/yr/person)
Taxing the bottom half of earners, or even bottom 75%tile of earner probably has no function whatsoever in terms of stabilizing the money supply, long term stability of prices, or the demand for currency. What it does, is it structurally ensures cheap labor and lower demand for consumer material goods… thereby ensuring the richer groups can accumulate money and power and control.
The one thing it really doesn’t do is FUND THE GOVERNMENT which is of course the excuse used to justify it. “We need to fund the roads somewhow” or “we need to pay for the military somehow” or whatever. None of that is true even in the slightest.
Just to clarify, when I’m talking here about funding the govt i’m talking about the **federal** govt, which issues currency. State governments in the US **ARE** funded by tax receipts.
Pretty sure banks issue currency, fed gov issues treasuries that are used as collateral for banks to issue the currency (and lots of it right back to the gov for new treasuries).
But the general idea of government “money printing” increasing wealth inequality, then income tax doubling down on oppressing the lower classes seems right.
Like if you are against wealth inequality, you should be against increasing the size of the government (at least to the point its funded by debt).
Anon.
The treasury issues actual paper currency and coins. But thats a tiny fraction of the money supply. Banks handle accounting deposits into checking and savings accounts and thats essentially all of the money supply. They increase it with two operations:
1) deposits of treasury checks / fund transfers (ie. Depositing social security payments, salaries for govt employees and payments to govt contractor companies)
2) Making bank loans. Where they take a promissory note as an asset and create a new bank liability (ie. They increase your checking balance by the amount you borrowed)
So yeah banks are critical to the money supply as they are the organizations that keep track of money.
Broad taxation of the population of Guatemala is clearly a bad idea, as this population has an enormous number of impoverished people. Taxing them takes food out of their mouth and has no social purpose.
1. Researchers (myself included a bit) were convinced it was worth moving mountains to run fairly small-sample studies only powered to detect these very large and dramatic ‘nudge effects’. Then we failed to detect any substantial effect, we could only statistically rule out “very large effects” … which was deemed uninteresting/unsurprising.
2. When I suggested that data coming from social science labs and other trials needed to be witnessed and publicly verified. See [here](https://docs.google.com/presentation/d/1qk8UNpaHPXWWCGikzErb7XpvrDazMfuWTY_jZUBMBxA/edit?slide=id.p19#slide=id.p19) the response was typically “do you really think researchers would falsify this stuff?”. My response: yes, some will, because it’s so easy, and the incentives are large.
I wrote about this in my paper “Is Psychological Science Self-Correcting?” (https://pmc.ncbi.nlm.nih.gov/articles/PMC10460510/pdf/nihms-1913977.pdf):
“New investigators need more than sound research practices. They also rely on the published literature to judge what is known and unknown, what warrants further investigation and what can be taken for granted. If a large fraction of published findings are not trustworthy, then new scientists may be something like explorers setting off to survey the Kong Mountains or the coast of New South Greenland—discoveries claimed by early explorers and copied for decades from one map to another until later explorers, after wasting time on futile expeditions, convinced cartographers that there was nothing there.”
Harriet Hall, MD called this tooth-fairy science. “You can get all kinds of good data that is reproducible and statistically significant. Yes, you have learned something. But you haven’t learned what you think you’ve learned, because you haven’t bothered to establish whether the Tooth Fairy really exists.”
Is this similar, or the same as, or related to, the difference between reliabiltity and validity?
Anyway, I was again reminded of something I have been wondering about in light of certain large-scale proposals of “collaborative” research where “redefining statistical significance” proposals might be adhered to. The results of such studies might result in communicating some p values < 0.005, and the findings might even have been replicated amongst the "collaborating" researchers. However, it might not really be clear what has exactly been found, and how valid the conclusions are. They may reliably replicate, but are they valid with regard to the conclusions and possible implications or recommendations that might follow?
This might all be especially pertinent as I reason that the nature of the format described above may also result in the findings being pretty hard to independently replicate in the same manner. This might be because such large studies are relatively hard to replicate by critical or interested or whatever other researchers. And, a smaller replication might be easier to ignore or brush aside for reasons X, Y, or Z (think about all the discussions concerning "conceptual replications" in this light). Or, a new perhaps more theoretically based attempt, possibly involving a different measure or construct to try and figure out what really is crucial, might go unnoticed due to this study being smaller and getting less attention. So, the thing that has been "found" or concluded in the large-scale study may stay around for a while, or is relatively hard from a certain perspective to counter.
In general, I have wondered whether it is possible to design studies that include certain measures and exclude others, that only involve certain constructs and not others, and that only look at the p-value and not the size of the effect, and (deliberately or accidentally) paint a certain picture that way that may not be valid or true.
I then thought about what this might mean or imply regarding recent proposals for "improvement".
I then got a bit sad, and thought I'd done what I think is best to try and point to this.
And then I thought that I'd better listen to some music.