From A to Z
A couple years ago we discussed a post by Mark Zimbelman expressing skepticism regarding a claim by psychologist (also business school professor, Ted talk star, NPR favorite, Edge Foundation associate, retired Wall Street Journal columnist, Founding Partner of Irrational Capital, insurance agent, bestselling author, etc etc) Dan Ariely. Zimbelman wrote:
I have had some suspicions about some experiments that Professor Ariely ran using a shredder that was modified so it looked like it was shredding but really wasn’t . . . He claimed it was “quite simple” to convert a shredder by breaking the teeth in the middle with a screwdriver . . . We were unable to break any teeth on the shredders we purchased but ended up finding a way to remove some of the teeth in the center by taking the shredder apart. Unfortunately, when we did this the papers would no longer go through the shredder without getting turned to one side or another and they inevitably got stuck because the shredder no longer had enough teeth to pull them through. We concluded that it was impossible to modify any of the shredders we bought . . .
A couple weeks after his first post, Zimbelman followed up with further investigation:
I [Zimbelman] did an extensive literature search (involving several others who helped out) looking for the research that he claims was done with the modified shredder. The end result is that I can’t find any published paper that discusses using a modified shredder. I even called one of his co-authors and asked him if the experiment that they ran together used a modified shredder. He said the shredder in their study was not modified.
I did find a few papers that used a regular shredder but did not mention any modifications. I also found several statements (including this one and the one linked above) where he claims to use this mysterious modified shredder. Overall then, here’s where we are with the shredder:
1. Dr. Ariely has made numerous claims to use a modified shredder in his matrix experiments.
2. I am unable to find any published papers by Dr. Ariely that use a modified shredder.
3. Modifying a shredder to do what he has claimed appears to be very unlikely.
Zimbelman’s posts were from late 2021, and I reported on them in early 2022. That’s where things stood for me until an anonymous tipster pointed us to this video that Ariely posted to Youtube in September 2023, along with this note:
Over the years I ran many different versions of honesty/ dishonesty experiments. In one of them, I used a shredder. Here is a short piece from the Dishonesty movie in which this shredder is starring and can be seen in action.
Here are some screenshots:
The Dishonesty movie is from 2015. Here’s a press release from 2016 promoting the movie. The press release says that the experiment that used the shredder was performed in 2002.
I assume the scene in the movie showing the experiment was a reconstruction. It just seems doubtful that in 2002 they would’ve taken these videos of the participants in that way. There’s nothing dishonest about reconstructing a past scene—documentaries do that all the time!—I’m just trying to figure out exactly what happened here. Just to be sure, I watched the clip carefully for any clues about when it was shot . . . and I noticed this:
Let’s zoom in on that mobile phone:
I don’t know anything about mobile phones, so I asked people who did, and they assured me that the phones in 2002 didn’t look like that, which again suggests the scene in the documentary was a reconstruction. Which is still fine, no problems yet.
Home Depot . . . or Staples?
Zimbelman’s post links to a voicemail message from Ariely saying they bought the shredder from Home Depot. But in the video, the shredder is labeled Staples:
Wassup with that? In the label on the Youtube video, Ariely says that the shredder in the video is the same as the one they used. (“Over the years I ran many different versions of honesty/ dishonesty experiments. In one of them, I used a shredder. Here is a short piece from the Dishonesty movie in which this shredder is starring and can be seen in action.”) Which makes sense. It’s not like they’d have a whole bunch of modified shredders kicking around. But then where did they buy it? Back in 2002, was Home Depot carrying Staples brand products? I guess a real shredder-head could tell just by looking whether the shredder in the video is vintage 2002 or 2015. The shredder in the video doesn’t look 13 years old to me, but who am I to say? Unfortunately, the image of the label on the back of the machine isn’t readable:
You can make out the Staples brand name but that’s about it.
So, what happened?
There are several possibilities here:
1. The 2002 experiment happened as was described, they really modified that shredder as claimed, they kept the shredder around, it was still operating in 2015, they demonstrated it in the video, and it worked as advertised. Also, that shredder in 2002 had actually been bought at Staples, and Ariely just had a lapse in memory when he said they’d bought it at Home Depot.
2. The 2002 experiment happened as was described, they really bought that shredder at Home Depot and modified it as claimed, but then the shredder was lost, or discarded, or broke. When they made the movie, they went to Staples and bought a new shredder, modified it (using that approach which Ariely said is simple but which Zimbelman in the above-linked post said is actually difficult or impossible), and it worked just as planned. Under this scenario, Ariely was not telling the truth when he wrote that the shredder is the same as in the original experiment, but, hey, it’s just a movie, right? The Coen brothers had that title card saying that Fargo was based on a true story, and it wasn’t—that doesn’t mean they were “lying,” exactly!
3. The 2002 experiment happened as was described, they really bought that shredder at Home Depot and modified it as claimed, but then the shredder was lost, or discarded, or broke. When they made the movie, they went to Staples and bought a new shredder, but at this point they didn’t bother to try to modify it. It was just a dramatization, after all! Why ruin a perfectly good shredder from Staples? Instead they just filmed things just as they literally look in the video: they put paper in the shredder, then off screen they mangle the edges of some other sheets of paper, put them in the bin in the bottom of the shredder, and then turn on the video again and take the partially-mangled papers out. The way I wrote this, it seems kinda complicated, but if you think about it from the standpoint of someone making a video, this is much easier than getting some modified shredder to work! Indeed, if you look at the video, the sheets of paper that went into the shredder at the beginning are not the same as the ones that they took out at the end.
4. The 2002 experiment never happened as described. Ariely or one of his collaborators had that cool idea of modifying the shredder, but it wasn’t so easy to do so they gave up and just faked the study. Then when making the 2015 movie they just said they lost the shredder, and the moviemakers just did the reenactment as described in option 3 above.
5. I’m sure there are other possibilities I didn’t think of!
Is the video “fake”? I doubt it! I assume the video is either a clip from the movie or was filmed at the same time as the movie, and in either case it’s a reenactment of Ariely’s description of what happened in the experiment. In a reenactment if you show some sheets of paper fed into a shredder and then later you show some sheets of paper removed from the shredder, there’s no reason that they have to be the same sheets of paper. Similarly, if a movie reenacts a flight from New York to Chicago, and it shows a shot of a plane taking off from LaGuardia, followed by a shot of a plane landing in O’Hare, they don’t have to be the same plane. It’s just a reenactment! The fact that someone made a video reenacting a scene with a shredder does not imply that this shredding actually happened in the video—there’s really no reason to for them to have gone to the trouble to have done this shredding at all.
So, there are several possibilities consistent with the information that is currently available to us. You can use your own judgment to decide what you think might have happened in 2002 and 2015.
“Dishonesty can permeate through a system and show up not because of selfish interest but because of a desire to help.”
In the above-linked press release—the one that said, “we modified the shredder! We only shredded the sides of the page, whereas the body of the page remained intact”—Ariely also said:
There are pressures for funding. Imagine you run a big lab with 20 people and you’re about to run out of funding – what are the pressures of taking care of the people who work with you? And what kind of shortcuts would you be willing to take? Dishonesty can permeate through a system and show up not because of selfish interest but because of a desire to help.
With regards to research, I don’t think that most people think long-term and think to themselves that somebody would try to replicate their results and find that they don’t work. People often tend to “tweak” data and convince themselves that they are simply helping the data show its true nature. There are lots of things in academic publications that are manifestations of our abilities to rationalize. . . .
I think that the pressures of publication, funding, helping the group, and reputation are very much present in academic publications.
Which is interesting given the fraudulent projects he was involved in. Interesting if he was doing fraud and interesting if he was the unfortunate victim of fraud. He’s an admirably tolerant person. Fraud makes me angry; I would not be so quick to refer to it as “not because of selfish interest but because of a desire to help.”
Wassup, NPR? You raised concerns in 2010 but then promoted
One of the above sources links to this NPR article from 2010:
Should You Be Suspicious Of Your Dentist Or NPR’s Source? . . .
Last month, Dan Ariely, a behavioral economics professor, talked with All Things Considered host Robert Siegel, about how incredibly loyal, almost irrationally so, people are to their dentists – more so than with other medical professions. . . . when he appeared on NPR’s air, there was every reason to trust him.
Ariely offered information certain to unnerve listeners and anger dentists ¬– information based on a fact that he cannot back up.
If two dentists were asked to identify cavities from the same X-ray of the same tooth, Ariely said they would agree only 50 percent of the time.
Ariely cited Delta Dental insurance as his source. However, Delta spokesman Chris Pyle said there is no data that could lead to that conclusion. . . .
Here is what Ariely said:
Prof. ARIELY: And we asked both dentists to find cavities. And the question is, what would be the match? How many cavities will they find, both people would find in the same teeth? . . . It turns out what Delta Dental tells us is that the probability of this happening is about 50 percent. . . .
It’s really, really low. It’s amazingly low. Now, these are not cavities that the dentist finds by poking in and kind of actually measuring one. It’s from X-rays. Now, why is it so low? It’s not that one dentist find cavities and one doesn’t. They both find cavities, just find them in different teeth. . . .
“According to Dr. Ariely, he was basing his statement on a conversation he said he had with someone at Delta Dental,” said Pyle. “But he cannot cite Delta Dental in making that claim because we don’t collect any data like that which would come to such a conclusion.”
So what happened?
Ariely said he got that 50 percent figure from a Delta source who told him about “some internal analysis they have done and they told me the results. But they didn’t give me the raw data. It’s just something they told me.”
Ariely did not provide the name of the Delta medical officer, whom Ariely said was not interested in talking with me. . . .
Ariely told me he happened upon that figure when he was conducting research analyzing 20 years of raw data on Delta claims. . . . But Ariely did not see or analyze any data that would lead to a conclusion that dentists would agree only 50 percent of the time based on studying an X-ray.
Wow. The NPR report continues:
But what is NPR’s responsibility? . . . NPR can’t re-report and check out every thing that an on-air guest says. . . . In this case, the interview with Ariely was taped ahead of time and edited for air – but no one thought it necessary to challenge his undocumented statement.
ATC executive director Christopher Turpin said NPR had no reason to question Ariely, given his credentials as a tenured professor and an expert on how irrational human beings are. . . .
ATC has other pre-taped segments with Ariely, and those should be double-checked before they are aired. There’s no doubt that Ariely is both entertaining and informative about how irrational we humans are — but he also must be right.
It’s funny that, after all that, they write that there’s “no doubt” that Ariely is informative. I have some doubt on that one!
But here’s the interesting thing. The above warning was from 2010. Not 2022, not 2020. 2010. Fourteen years ago. But if you google *NPR Ariely*, you get a bunch of items since then:
2011: Is Marriage Rational? : Planet Money
2011: For Creative People, Cheating Comes More Easily
2012: TED Radio Hour: Dan Ariely: Why Do We Cheat?
2012: ‘The Honest Truth’ About Why We Lie, Cheat And Steal
2014: Dan Ariely: Where’s The Line Between Cheating A Little and Cheating A Lot?
2014: Rethinking Economic Theory: The Evolutionary Roots Of Irrationality : 13.7: Cosmos And Culture
2015: Dan Ariely: What Pushes Us To Work Hard — Even When We Don’t Have To?
2017: Dan Ariely: When Are Our Decisions Made For Us?
2018: Everybody Lies, And That’s Not Always A Bad Thing
2020: Why Some People Lie More Than Others
And then in the past year or so there have been some skeptical stories, such as this from 2023: Did an honesty researcher fabricate data?
But until recently NPR was running lots and lots of stories quoting Ariely completely without question—for years and years after they’d been explicitly warned back in 2010.
The scientist-as-hero narrative is just so strong that NPR kept going back to that well even after their own warning.
Why post on this?
God is in every leaf of every tree. It’s interesting how the more you look into this shredder story the more you can find. But there’s always some residual uncertainty, because there’s always some elaborate explanation that we haven’t thought of.
In the meantime, I’d recommend following the advice of that 2010 NPR report and asking people for their evidence when they make claims of scientific breakthroughs. There’s nothing you can do to stop people from flat-out lying, but if you can get purported experts to specify their sources, that should help.
If scientists know ahead of time that they’re expected to produce the shredder, as it were, maybe they’d be less likely to make things up in the first place.
I personally have “no doubt” that Ariely is informative … just not in the way that NPR would like. Yes, he’s informative as an example of a liar, fabricator, or unreliable historian. Similar to how a failure is still informative in a study, uncritical media’s use of Ariely’s opinions is informative of those media’s processes and credibility.
That being said, it would be simple for a large organization such as NPR (or any major news center), which likely employs many reporters, to have a diversity of opinion on whether one “expert” is credible or not (and for many reporters to not understand the concerns). I would hope that they’d have a central database of potential “sources” which would note that he’s not reliable (augmented by linking to all of the previous instances where he’s provided evidence). But given that I have a hard time searching this very blog for something I *know for certain* exists (and usually give up), it could be challenging for a radio show to thoroughly re-vet each source when they’re considered.
Leaving aside Ariely and his possibly fictitious shredder, Wikipedia has an extensive article about paper shredders:
https://en.wikipedia.org/wiki/Paper_shredder
“The first paper shredder is credited to inventor Abbot Augustus Low, whose patent was filed on February 2, 1909.[1] His invention was never manufactured because he died prematurely soon after filing the patent.[2]”
Pasta making and anti-Nazi leaflets of 1935 are mentioned immediately after the above quotation. And immediately after that, regarding your rights and your garbage,
“Paper shredders became more popular among U.S. citizens with privacy concerns after the 1988 Supreme Court decision in California v. Greenwood; in which the Supreme Court of the United States held that the Fourth Amendment does not prohibit the warrant-less search and seizure of garbage left for collection outside of a home.”
And for those who are not shredder cognoscenti:
“Large organizations or shredding services sometimes use “mobile shredding trucks”, typically constructed as a box truck with an industrial-size paper shredder mounted inside with storage space for shredded materials.”
Other than the fact that Zimbelman could not disable a shredder in the same way as Ariely claimed, and noting that the video clip shows a reconstruction of the experiment that may or may not use the original shredder, is there any specific concern about the so-called “matrix” experiments done with the shredder? I don’t mean the generalized concern with Ariely’s work, but something specific to these particular experiments, like the auto insurance company “data” case had.
Gregory:
Saying that “Zimbelman could not disable a shredder in the same way as Ariely claimed, and noting that the video clip shows a reconstruction of the experiment that may or may not use the original shredder” is understating what’s going on here! Beyond Ariely’s has a track record of misrepresenting his psychology experiments (not just “generalized concern” about his work, but concern that he will confidently say things that are not true), the clip from the movie is not consistent with his earlier claims about the shredder . . . that’s a concern that’s specific to this particular experiments. If there was no shredder, what was happening? It’s anyone’s guess.
“…The Coen brothers had that title card saying that Fargo was based on a true story, and it wasn’t—that doesn’t mean they were “lying,” exactly!”
What does this mean exactly? I expect it is possible to find true stories with some resemblance to plot points in Fargo.
I think he meant that Fargo is a fictitious story and you’re not intended to take that claim seriously in the way you are supposed to take a news story published in say the LATimes seriously. On the other hand, science journal articles are 100% supposed to be things we can take seriously…
The film also has a standard disclaimer to the effect that “all resemblance to real people is coincidental.”
Matrices and a shredder are used in Mazar, Amir, and Ariely 2008, “The dishonesty of honest people: A theory of self-concept maintenance.” In this paper the “recycle condition” is supposedly one where the student is told to recycle their test sheet – which is where they actually answer the questions – but not their answer sheet, which is where they record how many they got right. The students presumably think that there is no way for the researchers to verify whether they lied in the “recycle” condition. Here is where the shredder is mentioned:
“The recycle condition (second cheating condition) was similar to the no recycle condition, with the difference that participants were instructed to transfer their answers to the premarked bubble sheet and then walk to a shredder, shred their original test sheet, and take only the bubble sheet to the experimenter, at
which point they would be paid accordingly. Because of the shredding, this condition offered a lower probability of being caught cheating than the no-recycle condition.”
Throughout the paper, the authors are somehow able to tell if someone cheated in the recycle condition. In Experiment 1, the students are simply told to keep the test sheet (that would show how they really did) and take it home for recycling. How did the researchers know whether cheating occurred when the students went home thinking they had all the evidence in their bags? There is no mention of a modified shredder. But since they apparently did know in Experiment 1, they wouldn’t need a modified shredder in Experiment 6.
Ariely is a fabulist. We already knew that from the USC debacle, where the person who ran his experiment said that his described methodology could not possibly be right.
Ariely is trustworthy. He wears a lab coat after all! And here, neither me or any economist I know have ever worn one when running experiments. If we’d only known that hack, maybe we’d all have ted talks…
Psychologist, doing a study on honesty. Maybe that should have been a red flag.
Francesca Gino might disagree.
Aaa:
I think she might agree. After all, setting aside whatever Gino did or did not do, it’s undeniable that several high-profile psychology studies on honesty were fraudulent. At the very least, Gino has had the unfortunate experience of having collaborators who have violated scientific ethics.
Quote from above: “At the very least, Gino has had the unfortunate experience of having collaborators who have violated scientific ethics.”
Maybe she mistook them for what she might view as “rebel talents”, which she wrote a book about if I am not mistaken. From some website about that all:
“We think of them as troublemakers, outcasts, contrarians: those colleagues, friends, and family members who complicate seemingly straightforward decisions, create chaos, and disagree when everyone else is in agreement. But in truth, rebels are also those among us who change the world for the better with their unconventional outlooks. Instead of clinging to what is safe and familiar, and falling back on routines and tradition, rebels defy the status quo. They are masters of innovation and reinvention, and they have a lot to teach us.”
Ahh, yes, good point. What we would consider fraud or violations of scientific ethics, Ariely or Gino might consider creative unconventional behavior by troublemaking rebels. Proving your theories by making up or altering data is a way of going outside the box. And they could hold that view, whether or not they’d done the fraud themselves or just had ended up in repeated collaboration with fraudsters.
“So, there are several possibilities consistent with the information that is currently available to us. You can use your own judgment to decide what you think might have happened in 2002 and 2015.”
I think I am ready to at least push this further. The shredder paper was indeed Mazar et al 2008. There is a bunch of information on a weird blog named The Deception of Deception that seems to be anonymous, here:
https://deceptionofdeception.wordpress.com/the-simple-model-of-rational-crime-smorc-2/
The Eureka! moment came when I realized that the experiment design used for these matrix tests was not set up to use any information from what went into the shredder. In the control condition, they validated the self-reported performance of the students, and so determined how many matrices they could really solve in the allotted time. In the cheat condition, they didn’t care anymore how many were actually solved because they had a statistically valid baseline for the human population from the control experiments. So they gave a different sample of students a chance to cheat and shred the evidence, and then used the self-reported performance to compare to the actual performance in the control. When the students who shredded claimed on average six puzzles in the allotted time versus the four puzzles actually solved in the control, they assumed the difference was lying.
In these experiments, what would be the point of modifying the shredder? The only thing you could test is whether the control condition and the cheat condition pulled samples from the same population or entirely different populations, fast math-puzzle-solvers and slow ones. This test would be nice if you are dotting every i, etc., but their sample sizes seem sufficient to support the randomness assumption they made. So not only was there no modified shredder in Mazar 2008, there was no good reason to have one.
I suspect that most likely Ariely was trying to remember this experiment during some later talk but couldn’t really remember how they did it so he just winged it. (That is very close to what must have happened in the USC debacle as well.) Later, once the fraud accusations started popping up, and questions arose about the shredder, he was no longer in a position to go back and reread his own paper and say “oops I remembered that wrong.” With his reputation already on the ropes, all he could do was double down.
Matt:
I kinda know what you mean with your last sentence there, but . . . telling the truth is always an option!
Matt seems to be surmising that the entire modified shredder story was Ariely’s mis-recollection; that there never was a modified shredder; that it was not an integral part of any experiment and no claims in a research paper were dependent on it’s existence. Maybe Ariely had a dream or something that was so vivid that he recollected it as reality.
This seems like a possibility – until we come to the video that purports to show the experiment, where *the* infamous modified shredder is purported to be shown in operation.
So I guess I don’t see the out. it’s one thing to double down on a mis-recollection in a conversation with friends. It’s something completely different to gin up a video purportedly showing events that never happened with a shredder that never existed and release that video pulbically in an effort to refute allegations of fraud. He had no co-authors or grad students who could help clarify the events and/or verify the existancer of the equipment in question? Are all his former grad students shaking their heads about this shredder, or are they outraged at the attacks on their trusted mentor? Seems like one of them could have some relevant thoughts on the situation.
Incidentally, it struck me as odd that someone would go to Home Depot to buy a paper shredder. Drywall tape, OK, but a paper shredder? Would you go there for printers and ink too? So I checked and at least at the moment Home Depot does sell paper shredders – though none that are Staples brand. But it’s hardly surprising that the model in the video shows no sign of having been purchased at Home Depot, where psychology research teams probably don’t typically have an account.
It’s conceivable that the shredder was bought at “Office Depot”, which Ariely later misremembered as “Home Depot”. That would be an easy mixup to make, especially if he’s in the habit of personally buying stuff from the latter.
Also, the claim of 50% difference about identifing cavities from the same X-ray, might be a garbled version of something about differences in recommended treatment for those cavities. Some treatments are much more expensive than others. I can see an insurance company trying to figure out a standard, and finding there’s a lot of variation in recommendations.
Re: Ariely doubling down. Here is something he wrote back in April 2024:
“I have also spent the better part of the past three years reliving a set of decisions I made and actions I took over a decade ago, around a now-infamous study of human behavior using car insurance data. Working in partnership with trusted colleagues, I did what was expected and right at the time. When the findings of the original study were called into question years later, we re-tested our assumptions, publishing results that put doubt on the original findings. When issues with the original data set were uncovered, we retracted the paper. Later we re-examined and replicated the findings, and in early 2024 published two papers that show that the effects of signing first matter (links to the papers are at the bottom of this post), but also that they matter in a more nuanced and complex way. Such is the scientific process. (See link to my official statement about this event).
As I was trying to understand what went wrong with the 2012 paper, people were very quick to criticize and question my professional judgment, casting a shadow over my life’s work. “Where did it all go wrong?” I have asked myself many times a day for the last few years. I have looked long and hard, but given that so much time has passed, I have come up empty-handed, and I simply don’t know for sure what went wrong. “Should I have done things differently?” Evidently. But as I examine all my decisions, I feel that I did the best I could at the time. And I’m okay with that.”
[For those who have not been following along, the car insurance data was extensively reworked to support his hypothesis, in such a clumsy manner that it was obvious to later investigators. After years of investigation, the motive and all the evidence point back to Ariely as having committed the fraud. But in the end Duke U. swallowed his obfuscations.]
Later in the same post he wrote:
“For my part, I am planning on many more years of mistakes. I don’t intend to let fear and regret drive my decisions. I want to keep learning and growing. I intend to continue to make my own mistakes, and I even hope to normalize making more of them.”
> Later we re-examined and replicated the findings, and in early 2024 published two papers that show that the effects of signing first matter (links to the papers are at the bottom of this post), but also that they matter in a more nuanced and complex way.
This reminds me of the red is the sexiest color papers, in which the authors failed to replicate the original claim but then argued that their results hold only during a particular season or something like that. Nuanced and complex indeed. More generally, it is so gratifying that almost every single researcher eventually finds out that they had been right after all.
Shravan:
Yup, as the saying goes, “the mistake does not affect the paper’s conclusions.”
It kinda makes you wonder why they bother collecting data at all, given that the conclusions never seem to change.
Lao don‘t forget that replication rates are statistically indistinguishable from 100%. I see that as a compelling reason not to run any experiments at all. It‘s gonna replicate for sure, so just write down what you think will happen. Reminds me of the good old days of intuition based research in linguistics. The experimenter and the subject are the one and same person.
Shravan wrote:
“…the authors failed to replicate the original claim but then argued that their results hold only during a particular season or something like that. Nuanced and complex indeed.”
The math puzzles experiment underwent a formal replication in 2018:
https://journals.sagepub.com/doi/full/10.1177/2515245918781032
The replication found a tiny effect going in the opposite direction. The authors issued a response:
https://journals.sagepub.com/doi/pdf/10.1177/2515245918769062
The response had this:
“The goal of the current replication effort was to directly replicate one of the six experiments in our original article (Mazar, Amir, & Ariely, 2008). When we began working on the protocol with the Editor and proposing authors, though, we realized that there were
more challenges than one would think. Our original experiment was run with students, not in a lab but in a large classroom, in one session, as part of a course requirement, in a particular location and at a particular time, as part of a packet that included other studies that we do not have a record of, and in a particular culture
and country. Because we do not know whether and how such factors contributed to our results, we tend to favor conceptual, rather than direct, replications of the experiment. In addition, this realization suggests that it may be useful more generally for authors to report many more details about their procedure than is
required for publication purposes, so that subsequent replication attempts could come closer to reproducing meaningful contextual factors.”
For blog readers, this quote might as well be coming from The Onion.
LOL. In my replication attempts I now include Heraclitus‘ observation: you can never step into the same river twice. Every dataset’s a one time event.
Actually this reminds me of psycholinguistics papers in top journals. It is almost inevitable that actually p is larger than 0.05, but authors simply claim that it is lower than 0.05 and reviewers go, oh ok, accept. I have even seen impossible, i mean Huberman level impossible, p values and nobody notices. So much for perr review.
One common trick used is to do a by subjects and by items analysis (by aggregating data, thereby concealing a source of variance) getting by subjects only to come out barely significant, and using that to claim they found an effect. I learned some years ago that US psycholinguists are taught this is ok for claiming significance.
Hmm. I’m offering some skeptical views—skeptical of the skeptics in this case.
I don’t think that the device is a cell phone. It looks more like a neurostimulation device for controlling pain. Recall that Ariely was severely injured when in the military. Such devices were built by Advanced Neuromodulation Systems and later by St. Jude Medical. See https://www.dotmed.com/listing/disposables-general/st-jude/7883/proclaim-drg/3512419 for an image of model from St. Jude.
If the shredding was faked, it was an impressively good job of fakery. Notice at 1:21 in the video that many pages are stacked and have parallel rough edges. Compare the yellow sheet at 1:18 with the blue sheet at 1:19. The sheets being reviewed at 1:22 and 1:26 look similar but actually differ in the top row of answers. The chad in the shredder box is consistent with the ragged edges. So, if they faked the shredding, how did they do it?
If I wanted to generate such faked pieces of paper, I’d get a real shredder and would remove the teeth in the center. I’d probably use a tool a little less subtle than a screwdriver. Think angle grinder, Dremel tool, hammer, a double-cut bastard file, end nipper, etc. I clearly recall a project in which I built stuff after a trip to the hardware store and cannibalizing some suppliers in the kitchen. The guy running the project and supervising me liked the rather professional-looking result, but I doubt if he could properly describe how it was built. Just as Ariely was mistaken (or just making it up) about the supplier of the shredder, he might have spoken in a similar fashion regarding the method of modifying the shredder.
Bob:
It could be that Ariely hired someone else to buy and alter the shredder and then just made up where it was bought and made up how the shredder was altered. It’s so hard to know! Once trust has gone, communication is so much more difficult.
This relates to the point that Dan Davies made in his book about the deadweight costs of lying and fraud.
When people publicly lie, we pay three times: first for the consequences of believing the lie, second for the lack of trust we then end up applying to honest people, third for the blurry zone where the possible presence of different sorts of lies makes it hard for us to untangle what is going on.
Andrew wrote:
It’s so hard to know! Once trust has gone, communication is so much more difficult.
I agree completely. Indeed, in some jurisdictions it is a legal principle that if a witness fibs on one point, the jury can ignore everything the witness says. See https://en.wikipedia.org/wiki/Falsus_in_uno,_falsus_in_omnibus
Good catch on the neurotransmitter Bob, I wonder if that model was available in 2002.
It does look like those papers went through a modified shredder. But as Andrew pointed out, what is shown in the video is very likely a re-enactment. Here’s the thing: In 2015 Ariely had an incentive to modify a shredder. He was trying to rescue his reputation. In 2002, he had no incentive to modify a shredder because the experiment design (cleverly!) eliminated any need to know what went into the shredder.
The thing in the video isn’t a pain-management neurotransmitter; it’s an iPhone or iPod Touch in a protective case with a cutout for the Apple logo and the camera. You can look at the link to the listing Bob gave and see that the second photo clearly shows the same thing: https://www.dotmed.com/listing/disposables-general/st-jude/7883/proclaim-drg/3512419. You can even make out the Apple logo in the case cutout. You can also see that the USB power brick has an iPod Touch product card underneath it. This is because the St Jude device consists of several pieces, one of which is an iPod Touch.
The actual pain management device is an additional piece. It’s called the Implantable Pulse Generator or IPG. The way they work is that an app running on an iPhone, iPod, or iPad is used to remotely control the IPG device. See for example Fig 1 (page 17) of the manual for the St Jude device: https://www.northernpaincentre.com.au/wp-content/uploads/2017/04/Proclaim-DRG.pdf
Yes, and the massive loss of time and resources that could’ve been used to do good science … (these scientists are not “learning” by mistakes or that you and I are “teachers” providing them valuable guardrails! They’re actively showing future fraudsters on how to “succeed” by “winging” it … 😢)
Regarding making up details, as Pooh-Bah said in The Mikado, such are “Merely corroborative detail, intended to give artistic verisimilitude to an otherwise bald and unconvincing narrative.”
Home Depot, Office Depot, Staples—just a detail.
Bob:
Yeah, it’s deniable because he can just say he misremembered the name of the store. The claim that the shredder from the 2015 video reconstruction is the same as the supposed shredder from 2002 experiment, that’s a bit iffier, but, again, he could always claimed to have misremembered, saying something like, “Oh, yeah, now that you mention it, I guess it wasn’t the same shredder after all.” Once someone is committed to brazening it out, there’s always a way to keep altering the story.
Hmm, lots of stuff, e.g.:
https://pubmed.ncbi.nlm.nih.gov/22482245/
Variations in caries diagnoses and treatment recommendations and their impacts on the costs of oral health care
Results: Inter-examiner agreement for caries detection was moderate (kappa = 0.42) while for treatment decisions it was fair (kappa = 0.29). The sensitivity and specificity were 0.69 and 0.65 for caries detection and 0.56 and 0.65 for treatment decision respectively. Dentists overestimated the presence and depth of carious lesions and there was a tendency to treat enamel lesions using invasive therapeutic procedures. Mean treatment cost across the two cases was 32US$ (range 9-65) while the histologically validated cost was 23US$.
Conclusion: The variability in caries detection and treatment decision negatively affected the cost of the dental treatment.
At the risk of sounding like the guy in the conspiracy-theory meme, in the video one can see the top of the shredder very clearly, and the “15” on it in a paper icon means it’s a “15 sheet” shredder. That narrows down the possible models significantly, and makes it feasible to search: Staples “15 sheet” shredder
This looks like the correct one:
https://www.allsold.ca/staples-15-sheet-paper-shredder-spl-txc15a-19160
Model SPL-TXC15A
https://www.youtube.com/watch?v=gLpxUbsGZAY
There seem to be a few available secondhand.
Here’s one for sale, CA 100 https://www.facebook.com/marketplace/item/3802913156639629/
(anyone want to buy it For Science?)
Another: https://www.ebay.com/itm/256510845673
And here’s a reference to that model existing in at least 2010:
(now I really feel like the meme guy!)
https://procure.ohio.gov/pdf/SR900810%20GPC005%20Staples%20Net%20Prices%20Eff%2004_01_10.pdf
OSINT at work. We need our own Bellingcat for science.
Maybe this has already been covered in past posts, but I clicked through to the Elsevier Connect coverage of the experiment from 2016, where we read this:
>What did we find?
>On average, people solved four problems but reported solving six.
>Nearly 70% cheated.
>Only 20 out of the 40,000 were “big cheaters”, people who claimed to have solved all 20 problems. They cost the experiment $400.
>We also found more than 28,000 “little cheaters” who cost the experiment $50,000.
>So although there are some big cheaters out there, they are very rare and their overall economic impact is relatively low. On the other hand, there are a lot more “little cheaters” out there and their economic impact is incredibly high.
Leave aside the easy jab that it is indeed “incredibly” (i.e., not credibly) high, there’s a claim that a lot of money was dollars was paid out. Consider:
>Over 40,000 people, form all walks of life, participated in The Matrix Experiments.
And
>They received $1 for each problem they claimed to have solved correctly.
And
>On average, people solved four problems but reported solving six.
This means that the Matrix experiments paid out almost a quarter of a million dollar in cash. Has that money been accounted for?
Thomas:
The thing abut the large number of people reported in that experiment rings a bell . . . I think it came up in one of our posts, but I can’t find it.
Regarding your final question: As I wrote in an earlier post, “OK, it’s 2024, and maybe this all feels like shooting a rabbit with a cannon. A paper by Dan Ariely on the topic of dishonesty, published in an Elsevier journal, purporting to provide “guidance to managers and policymakers” based on the results of an online math-puzzle game? Whaddya expect? This is who-cares research at best, in a subfield that is notorious for unreplicable research.”