Christopher Chabris reviewed the new book by Malcolm Gladwell:
One thing “David and Goliath” shows is that Mr. Gladwell has not changed his own strategy, despite serious criticism of his prior work. What he presents are mostly just intriguing possibilities and musings about human behavior, but what his publisher sells them as, and what his readers may incorrectly take them for, are lawful, causal rules that explain how the world really works. Mr. Gladwell should acknowledge when he is speculating or working with thin evidentiary soup. Yet far from abandoning his hand or even standing pat, Mr. Gladwell has doubled down. This will surely bring more success to a Goliath of nonfiction writing, but not to his readers.
Afterward he blogged some further thoughts about the popular popular science writer. Good stuff. Chabris has a thoughtful explanation of why the “Gladwell is just an entertainer” alibi doesn’t work for him (Chabris). Some of his discussion reminds me of my article with Kaiser Fung on Freakonomics. The situations are different—Levitt does his own research whereas Gladwell describes the research of others, and overall I’m a lot more positive about Freakonomics than Chabris is about Gladwell—but in both cases there is the tension of applying academic standards to popular entertainment, of taking wonderful and complex stories and flattening them into cliches (see discussion here).
All of Chabris’s post is worth reading, but here’s a part that I noticed because it relates to some of our recent blog discussions regarding small-N counterintuitive findings:
This leads to my [Chabris’s] last topic, the psychology experiment Gladwell deploys in David and Goliath to explain what he means by “desirable difficulties.” The difficulties he talks about are serious challenges, like dyslexia or the death of a parent during one’s childhood. But the experiment is a 40-person study on Princeton students who solved three mathematical reasoning problems presented in either a normal typeface or a difficult-to-read typeface. Counterintuitively, the group that read in a difficult typeface scored higher on the reasoning problems than the group that read in a normal typeface.
In my review, I criticized Gladwell for describing this experiment at length without also mentioning that a replication attempt with a much larger and more representative sample of subjects did not find an advantage for difficult typefaces. One of the original study’s authors wrote to me to argue that his effect is robust when the test questions are at an appropriate level of difficulty for the participants in the experiment, and that his effect has in fact been replicated “conceptually” by other researchers. However, I cannot find any successful direct replications—repetitions of the experiment that use the same methods and get the same results—and direct replication is the evidence that I believe is most relevant.
This would be even better, blog-style, with links for the study and the failed replication.
P.S. As a bonus, here’s a post on disgraced science writer Jonah Lehrer, where Chabris writes, “Jonah Lehrer was never a very good science writer. He seemed not to fully understand the science he was trying to explain; his explanations were inaccurate, overblown, and often just plain wrong, usually in the direction of giving his readers counterintuitive thrills and challenging their settled beliefs.”
Particularly resonant to me was Chabris’s linking of Lehrer’s ethical lapses with his scientific lapses:
The fabrications and the scientific misunderstanding are actually closely related. The fabrications tended to follow a pattern of perfecting the stories and anecdotes that Lehrer — like almost all successful science writers nowadays — used to illustrate his arguments. Had he used only words Bob Dylan actually said, and only the true facts about Dylan’s 1960s songwriting travails, the story wouldn’t have been as smooth. . . .
After the Dylan episode, others found more examples of how Lehrer did this. I think one of the clearest was Seth Mnookin’s analysis of Lehrer’s retelling of psychologist Leon Festinger’s famous original story of “cognitive dissonance,” based on Festinger’s experience of infiltrating a doomsday cult in 1954. Of the moments after an expected civilization-destroying cataclysm failed to start, Festinger wrote, “Midnight had passed and nothing had happened … But there was little to see in the reactions of the people in that room. There was no talking, no sound. People sat stock still, their faces seemingly frozen and expressionless.” Lehrer narrated the same event as follows: “When the clock read 12:01 and there were still no aliens, the cultists began to worry. A few began to cry. The aliens had let them down.” Do you see the difference? Lehrer’s version is more dramatic: people worry, they cry, they feel let down. It’s more human. Each one of these little errors or fabrications makes the story work a little bit better, makes it match our expectations more closely, and thus gives it greater influence on our beliefs.
This seems closely related to the idea that Thomas Basbøll and I had, that plagiarism (or, more generally, obscuring of the provenance of data) is a statistical crime in that it reduces our ability to learn from reality.
Simplification is necessary for storytelling, but when you smooth away the parts of the story that don’t fit your template (whether you’re Levitt, Gladwell, Lehrer, or anyone else) you close off a possibility for learning.
Another example I like is Mike Daisey’s story about Apple factories. Man, that guy is shameless.
I tend to think in terms of personality because we fixate, naturally it seems, on the deliverer of the message and we become infatuated. I also think some people feed that demand. Aimee Semple McPherson and so on. I see Gladwell as pushing himself; his agenda is to convince others of his value, of his contribution. He writes for the renown.
I’ve long thought Andy Warhol’s 15 minute comment was misunderstood. It isn’t that the world will pick you and then you and then you for a short burst of fame but that you and you and you will seek it and the modern age will allow you to do that much more. We see lots of it, from youtube dance videos to faked claims about government ineptness: you create renown for yourself or for your idea. “You” are the one who got this rolling, etc.
My very personal take about Gladwell and the dyslexia stuff is that he is absolutely wrong. I don’t extend that conclusion to everything he’s written. I have a form of dyslexia. I can read well but I can’t retain and can barely process symbols. But that is much of what I do and it often involves painful transformations from words and images into symbols – or worse, having to read a bunch of symbols and figure them out for meaning AND having to do that nearly every time I see them. It’s a real drag.
Reading music at any speed is impossible – so I play by ear. Days of the week confuse me, which is a family joke. (Think about that: it involves matching symbolic identities so Tuesday is some number. I can’t keep both straight, which exposes the processing failure. It’s genetic. One of my kids has the same issue.) One could say, “Oh, you’ve benefitted from having to overcome” but that’s nonsense. I’m drawn to my work despite my issues. Anyone who thinks this kind of thing makes for closer reading misses the point entirely: if you can read the symbols, then a device or trick that makes you pay closer attention can be worthwhile but if you can’t read the symbols then you’re … look at a child who can’t read and tell me that kid is doing better at understanding as the kid works to decode words. Some people with disabilities do very well but, dear Lord, if you cherry pick success stories you get the wrong conclusions. It’s like the ex post reasoning I hear on idiot radio (meaning sports talk) when they bring up how terrific Julio Jones has been for Atlanta and argue your team should have made such a trade, which ignores how many top wide receivers have been total busts, etc.
If there is any truth in Gladwell’s generality it would be that some people, a relative handful, are drawn to work with or against their own difficulties. We see this with body issues: the handicapped runner or other athlete who has taken on a challenge directly in opposition to the handicap. But there are limits. I would never try to make myself into a “professional” musician, though I can play that well – or could when I was younger – because I know I couldn’t overcome my processing issues to be able to transpose on the fly or even remember complex sections at speed. Heck, I have to remind myself what notes are in which scale! I work in areas that require a lot of symbols only because the rest of my mind sees the answers. My other skills work just fine. I associate broadly and think very well in a straight line. If the problems I work on could only be approached through my symbolic processing issues, I wouldn’t touch them. I simply could not do the work. That would be like trying to be a sight-reading professional musician. And those handicapped runners are able to run with their handicaps – and we don’t see the ones who can’t.
“If there is any truth in Gladwell’s generality it would be that some people, a relative handful, are drawn to work with or against their own difficulties.”
Well, that’s pretty much exactly what he actually wrote in his book. He didn’t claim that 100% of people faced with what he termed “desirable difficulty,” or most of them, or even a large minority of them would benefit in some way from a particular difficulty, only that there exist some people who do. He made no claim about any measure of central tendency, his only claim is that the number of people who in some way benefit from having dyslexia is not zero. One person’s assertion of having been afflicted with X and not benefited from it does not contradict an argument that people who benefit from X exist. Whether David Boies or Gary Cohn actually benefited from having dyslexia or whether the were mistaken in their own belief that they benefited from it is probably not knowable; however, your assertion that you did not benefit from dyslexia contributes virtually no information about whether anybody does.
I do believe that “desirable difficulty” is a bit of a misnomer, as for some unknown but probably large proportion of people, the difficulty in question is not desirable at all and likely yields no benefits. However, Gladwell does not argue that all, most, or even a sizable minority of people benefit from these “desirable difficulties,” only that some (i.e., more than zero) do.
> only that there exist some people who do. He made no claim about any measure of central tendency, his only claim is that the number of people who in some way benefit from having dyslexia is not zero
You do realize that’s obvious, right? Really: his “only claim” is that at least one person benefits, in _some_ way?
That’s not remotely interesting and doesn’t need a book or specific anecdotes to demonstrate it. Critics of Gladwell are doing him an honor by assuming he has something of interest to say. I would hope you’ve mischaracterized his work since if his “only claim” is an intellectual vacuity like this, since if true it would make his work worthless even beyond the
depths of even what the most severe critic I have read has said.
I’ll freely admit that I enjoy reading his books. I don’t read them for intellectual enlightenment; I read them for the cool stories. You can decide for yourself whether or not a pretty obvious claim backed up by a well-told story is worth reading. I, personally, thought reading about David Boies and Gary Cohn was pretty interesting, even though they clearly do not apply to most people or even to the average person, but that’s not my point.
My point is that many of these criticisms are based on Gladwell’s failure to present rigorous statistical evidence of arguments that are not statistical in nature in the first place.
That you find his writing dull, boring, and not worthwhile is a perfectly fair criticism. I don’t happen to share your opinion on that front, but it is a perfectly valid opinion to have.
However, the argument that Gladwell is “wrong about dyslexia” because his stories of admittedly-exceptional people, whose experience he makes no attempt to generalize to any population, differ from the experiences of the average dyslexic has all the merit of claiming that comparing the height of my wife and the height of myself does not constitute sufficient evidence that my wife is taller than I am. A man need not procure random samples of men and a random sample of women and compute the average and standard deviation of their heights and perform a statistical test to determine whether or not his wife is taller than he is.
I imagine you know the saying that goes something like “If your only tool is a hammer, every problem looks like a nail.” I just stumbled across this post while goofing off after reading a really helpful article about fixed-effects and random-effects models. However, when (if) reading Gladwell, it’s time to put down the statistical hammer, because virtually nothing that he talks about has anything to do with phenomena that are statistical in nature.
On an : Outliers, on the whole, is one of my favorite books. That said, Gladwell totally screwed up his discussion of a study that he cited in support of a “10,000 hour rule.” He must have either not read, misunderstood, or mis-stated what that study did and what it found. I found that very disappointing, and I’ll stipulate that Gladwell blew that one bigtime. That is another fair criticism of Gladwell. (If you’re keeping a file of stuff to throw at him, add that one.) That issue hasn’t even come up in this discussion here, and I think most of the criticisms here stem from hammers mistaking various other things as nails.
Late to this party but I can’t not respond to this problematic argument. It is one thing to say that an author like Gladwell is putting forth a theory for debate – and if this were so, one would expect a more constructive response to critics; it is another thing to disclaim any intention to generalize. First of all, from the title of his books to the language used throughout the pages, it is clear that Gladwell is interested in sociology. The fact that he uses the term “theory of desirable difficulties” gives away his intention.
>> virtually nothing that he talks about has anything to do with phenomena that are statistical in nature.
Is there anything in nature that is not statistical? I don’t even know how to interpret this accusation.
What I’ll say is let’s assume that Gladwell is only interested in a singular subject. Paradoxically, this turns a difficult project into something impossible. Having a sample size of 1 creates a hopeless scenario.
Let’s examine the theory that Boies is a great lawyer because he is dyslexic. How would you prove/disprove such a statement? All we observed are two separate things: Boies is a great lawyer; and Boies has dyslexia. Well, maybe a third thing, which is that Boies said he became a great lawyer because he is dyslexic. A correlation does not a causation make. First-person speculation is not science either.
If Gladwell is attempting a generalization, he could expand his data set to multiple Boies, and hope to find stronger evidence for the theory. If he is forced to use N=1 samples, his task is more formidable, not less.
Of course it isn’t. That is my point.
“What proportion of people in the world are named Haile?” is a statistical question.
“What proportion of visitors to this blog are named Haile?” is a statistical question.
“Is my name Haile or is it not?” is not a statistical question. Given that you do not know the answer, you can go get random samples of people and find out their names (somehow) and compute the proportion of them whose name is Haile; and if that proportion is very small, or zero, you might conclude that the probability that my name is Haile is small; however, at the end of the day, either my name is Haile or my name is not Haile and your statistical analysis of the proportion of people named Haile contributes virtually nothing to the matter.
Neither Gladwell, nor his readers, nor David Boies himself can truly know whether or not Boies’s dyslexia contributed to his success as a lawyer, but whether it did or did not is not a statistical question any more than whether or not my name is Haile is a statistical question.
As you described it, it is an unanswerable question, not worthy of discussion. Any claim should be labeled as purely speculative, and unfalsifiable, and uninteresting.
To make it answerable, you have to make it statistical. If there is high likelihood that people with dyslexia became great lawyers, then the theory that David Boies is a great lawyer because of dyslexia has support.
It also makes no sense to compare the statement “David Boies is a great lawyer because of dyslexia” equivalently “David Boies would not have been a great lawyer if he was not dyslexic” with a purely descriptive statement like “David Boies was born in 19XX”. The Gladwell critics have never complained that he does not fact-check.
It’s an unanswerable question regardless of how I described it and does not depend on the strength of the relationship between dyslexia and success as a lawyer in the population unless the relationship is several orders of magnitude stronger than basically anything ever observed in the real world.
Even if hypothetically a rigorous study found an odds ratio of 10.0 (95% CI: 9.7 – 10.4) in favor of dyslexics becoming successful lawyers vs. non-dyslexics becoming successful lawyers, that would still tell us very little about whether David would have become a successful lawyer in the absence of his dyslexia. The base rate of people becoming successful lawyers is really low, and while an odds ratio of 10 would be mind-bogglingly large effect, it would still make for a small probability, in absolute terms, that any given dyslexic person would become a successful lawyer, and we would still know hardly anything about whether David or any other given individual would have become a successful lawyer had he not had dyslexia.
The strength of the relationship between X and Y in the population is a statistical question. Whether X caused Y in a single person observed to have both X and Y is not a statistical question. Knowing the statistical relationship between X and Y in the population can given you a tiny little bit of insight into the likelihood that X caused Y in a single case where both are observed, but it doesn’t even come close to answering the question.
So if I tell my 12 year old cousin that the mass of an object is a constant no matter how fast it is moving or that the sum of the angles of a triangle is always 180 am I also guilty of “smoothing away the parts of the story that don’t fit your template”?
To close off “a possibility for learning” temporarily might sometimes be the cost one pays for pedagogic efficiency.
I’m not condoning what Levitt, Gladwell or Lehrer did or didn’t do; but all I’m saying is it’s not all black and white. Teaching is not always entirely consistent and perfectly aligned with the exact, literal truth.
Rahul:
Here’s what I wrote:
I didn’t say you shouldn’t simplify, I just said that there’s a tradeoff. The tradeoff can be worth it. That is, I think we agree here.
Personally, I think Levitt made a good tradeoff whereas Gladwell’s wasn’t worth it. Of course, that’s subjective.
Rahul:
I think your example is counterintuitive, int he following sense: Feynman described the case you propose as expecting things to always work in the same way regardless of the domain. You cannot expect laws in the macro Universe to behave exactly as laws in the micro Universe (Gravity is one of them).
So in your well stated example, you put 2 examples one of physics and another from math, where indeed, if we switch to a different domain (that where the speed of the object is near relativistic terms, and that where the space is non euclidean) the basic simple rules do not apply any more.
The thing is, Gladwell and Levitt (to some extent) tend to generalize these punctual results as laws that govern the human condition in every situation. What they are doing, to put it in the same term as your example, is taking all the equilateral triangles and telling your son: “Every triangle is equilateral”, furthermore “every angle is always 60 degrees”, you simplified the triangle so much, you are giving a misconception of it.
> the sum of the angles of a triangle is always 180
But that’s math, so there is no “smoothing away the parts of the story that don’t fit your template”
Representations in math are self-referential so they can’t be wrong – they are exactly true.
If you draw a triangle as your re-representation of it for some purpose, the imperfections don’t count as they don’t matter mathematically.
(Given statistics can be mis-construed as math, thus sort of distinction concerns me)
Imperfections wasn’t what I had in mind when I made that comment.
It’s a really good idea to read Malcolm Gladwell’s work at the same time you are reading Freakonmics. This drives home the point that Gladwell is making it up as he goes along. Direct peer reviewed evidence is cited in Freakonomics that contradicts a lot of Gladwell’s claims.
Freakonomics was a two man team, consisting of a fine professional magazine writer (Dubner) and an award winning economist (Levitt). The Gladwell operation is more like a Dubner trying to play both roles at once.
Gladwell could afford to hire first-rate research assistants who could show him when he’s wrong, but instead he just lets his current RA pick his next one.
Slate posted Gladwell’s response. Pretty much the same he said on WNYCs Brian Lehr show last week. Paraphrasing…..”.just a couple of disgruntled academics don’t appreciate what I do.”
Didn’t we go through all this 4 years ago with Pinker’s review of What the Dog Saw? Sounds like Chabris leveled a very similar charge. I haven’t read the new book yet, or the reviews, but I did see in the Slate rebuttal that Gladwell urges Chabris to remember “that the world is not improved when those who create knowledge condescend to those who try to popularize it”. Isn’t that the point though? If so many scientists are frustrated that Gladwell’s storytelling is actually misrepresenting science, aren’t they allowed to critique him? And is he threatening to stop writing about science if academics don’t let up on him? Sounds like he’s saying he’ll pick up his laptop and go home.
I don’t want him to feel like he’s being condescended to. He doesn’t deserve that. I do wish, however, that he’d be more thoughtful and reflective about the reviews he’s getting. Instead, he appears to have given Chris the same shrug off that he gave Pinker, kind of a passive aggressive “it’s not me it’s you”, more intent it seems on preserving his brand than improving his craft. Ah well.
These problems seldom if ever occur when popular science is written by scientists. One can think of lots of examples like Feynman, Kahneman, Ariely to name but three.
Kevin:
I dunno about this. Ariely had some issues (see also here). Kahneman had the advantage of decades of research to write about, and Feynman wrote about physics in general, not just his own research, so he had an even wider range of things to choose from. But if you’re a Levitt or an Ariely, then you can run into problems when you run out of material. Sort of like what happened to David Sedaris.
Pingback: Half-Assed Book Review of David and Goliath by Malcolm Gladwell | Stupid Opinions Written Poorly
Malcolm Gladwell’s rejection letter (http://bit.ly/1cXaoP1)