A couple weeks ago I blogged on John Gottman, a psychologist whose headline-grabbing research on marriages (he got himself featured in Blink with a claim that he could predict with 83 percent accuracy whether a couple would be divorced–after meeting with them for 15 minutes!) was recently debunked in a book by Laurie Abraham. Discussion on the blog revealed that Laurie Abraham had tried to contact Gottman but he had not replied to the request for an interview.
After this, Seth wrote to me:
A few months ago one of my Tsinghua students asked me to recommend a psychology book about marriage. And I recommended Gottman’s book. Now I feel bad. It’s awful that Gottman didn’t take the time to defend his work. For all I know he has something to say in answer to Abraham’s criticisms but I guess I’ll never know what it is.
I replied:
Yeah, who knows? My guess is that Gottman believes he is fundamentally correct (based on the same sort of “clinicial intuition” that many experienced practitioners have), that he doesn’t understand the principles of modern psychological research methods, and that he enjoys the publicity he’s gotten. Were he put face to face with some criticism, my guess is that he’d just say that, sure, he’s no expert on statistics but he really understands marriages very well. Which may be true!
Seth:
Gottman is very good at math for a psych prof — that’s his background (math major). Not a clinician at all. This is one reason that his failure to defend his methodology shocks me.
Me:
Being a math major perhaps gave him a sense of overconfidence–thinking he was sooo much smarter than those doofuses who taught research methods, he decided he never needed to learn trivial things such as ROC.
The underlying question is: how could such a big-shot make such a basic statistical mistake? I don’t know, but once you have some success, I guess there’s not much of a motivation to change your ways.
Also, I could well believe that, for all its flaws, Gottman’s work is better than much of the other research out there on marriages.
I also wonder how Gladwell could’ve gotten fooled so easily. When someone claims a 94% predictive accuracy rate, wouldn’t you want to check his numbers?
And, of course, it’s possible that Laurie Abraham (and I) are completely wrong and actually Gottman has really done what he claims. As I put it earlier:
I eagerly await the Abraham vs. Gladwell showdown on Colbert. Could someone please tape that for me when it happens? You can record this on the same tape that already has the Bartels/Frank WWF bout, and the one where they challenge my namesake to see if he can read two full pages from his oh-so-well-reviewed opus of some years ago without the entire studio audience falling asleep. Oh, and if there’s room, you could throw in that clip of Johnny Carson and Zsa Zsa Gabor’s cat. . . .
Gladwell's job is really to entertain people by telling them compelling stories. Those stories occasionally touch upon truths, much as a stone skipping across the water or a stopped clock telling the right time. A psychologist who claims uncanny insight into marriages makes a much better character in a Gladwell story than one who doesn't understand elementary methods in statistics. That's about all it comes down to.
It's not just that he was an undergrad math major. Early in his career he wrote a book on time series analysis. So I'd think he ought to be on top of statistics and methodology, not just good at math.
Sanjay: statistics and methodology (for empirical science) can actually have almost no overlap at all.
Perhaps a good example was those _misguided_ comments that randomization plays no role in Bayesian statistics or that multiple comparisons are something that do not even need to be _thought about_ in Bayesian statistics…
Technical mathematical statistics can be done just as mathematics – and we should be happy that many chose to do that because they contribute there.
But also I have found that some of my worst clients had the best mathematical training, for instance a medical biophysicist who once told me in response to my possible role as consultant in his group – we don't need any help with formulas!
(at the time they were being sold microarray analysis methods that assumed independence of the gene expressions)
K
83% accuracy doesn't sound that impressive given 50% of all marriages in America end in divorce. He's only 33% more accurate than a coin flip.
And Uri Geller bent spoons with his mind. I saw him do it. His method was quite similar to that Gottman used – as described in your prior post: he used misdirection to distract you while he bent the spoon with his hands. Ex post facto, he bent it with his mind because he conceived of the trick and executed it in his head.
I drink tea with my mind. It tells my hand to lift the cup and bring it to my mouth.
I'm also extraordinarily good at telling whether marriages will last. If someone tells me they're divorced, I know the marriage didn't last. I am 100% accurate.
Does anybody know who has done the best research in this area? I'm curious to learn about it.
I found this — http://www.ncbi.nlm.nih.gov/pmc/articles/PMC16229… which suggests that researchers in this field have appreciated the defect in Gottman's methods for 9 yrs, or within 3 of Gottman's first of many studies. The issue isn't just whether Gottman can do math but how peer review works (or more so often doesn't)
Raymond: I think a 33 percent unit (do you call them percent units? English is my second language) increase in accuracy is pretty great, instead of flipping a coin when guessing, you could compare it to rolling a (six-sided) die and hoping it doesn't come up a "1". In fact, it's a 66% increase in accuracy (83/50)=1.66 compared to just assuming that the couple is like the average american couple.
I guess what i'm trying to say is that it's easy to see the appeal: if the guy actually was right, he would have had a method that was potentially cheap and accurate (given the research subject). Which of course is all the more reason to check his research carefully.