I’ll be on Radio 4 at 8.40am, on the BBC show “Today,” talking about The Honest Rainmaker. I have no idea how the interview went (it was about 5 minutes), but I’m kicking myself because I was planning to tell the hookah story, but I forgot. Here it is:
I was at a panel for the National Institutes of Health evaluating grants. One of the proposals had to do with the study of the effect of water-pipe smoking, the hookah. There was a discussion around the table. The NIH is a United States government organisation; not many people in the US really smoke hookahs; so should we fund it? Someone said, ‘Well actually it’s becoming more popular among the young.’ And if younger people smoke it, they have a longer lifetime exposure, and apparently there is some evidence that the dose you get of carcinogens from hookah smoking might be 20 times the dose of smoking a cigarette. I don’t know the details of the math, but it was a lot. So even if not many people do it, if you multiply the risk, you get a lot of lung cancer.
Then someone at the table – and I couldn’t believe this – said, ‘My uncle smoked a hookah pipe all his life, and he lived until he was 90 years old.’ And I had a sudden flash of insight, which was this. Suppose you have something that actually kills half the people. Even if you’re a heavy smoker, your chance of dying of lung cancer is not 50 per cent, so therefore, even with something as extreme as smoking and lung cancer, you still have lots of cases where people don’t die of the disease. The evidence is certainly all around you pointing in the wrong direction – if you’re willing to accept anecdotal evidence – there’s always going to be an unlimited amount of evidence which won’t tell you anything. That’s why the psychology is so fascinating, because even well-trained people make mistakes. It makes you realise that we need institutions that protect us from ourselves.
I like that last bit—“if you’re willing to accept anecdotal evidence, there’s always going to be an unlimited amount of evidence which won’t tell you anything.” Of course, what makes this story work so well is that it’s backed up by a personal anecdote!
Damn. I was planning to tell his story but I forgot. Next time I do radio, I’m gonna bring an index card with my key point. Not my 5 key points, not my 3 key points, but my 1 key point. Actually, I’m gonna be on the radio (in Seattle) next Monday afternoon, so I’ll have a chance to try this plan then.
Can't see this item listed in their running order. Did it get bounced to another day, perhaps by the more immediately topical item on technology in cricket (0843) ?
Sorry please ignore / delete my previous comment – the item did indeed go out in the 0840 'Business news with Adam Shaw' slot, n fact broadcast at around 0844-0846.
Actually, I think anecdotal evidence is quite useful if it's used the right way. The key is to use an anecdote that is objectively well-known, but is well-known not for the topic you are interested in.
Steve:
I agree that anecdotal evidence can be useful–I use it all the time! I didn't say it was useless, only that if you're willing to accept anecdotal evidence, there's always going to be an unlimited amount of evidence which won't tell you anything.
On the contrary, every piece of anecdotal evidence is telling you something, it's just not always telling you what you happen to be looking for.
"My grandpa smoked hookah til he was 90" doesn't mean smoking is safe, and "My brother, a hookah smoker, died at 30" doesn't mean it isn't. Using either one alone gives you an excessively rosy/dismal view. The key is to listen and collect anecdotes, then you can do actual extrapolation and figure out that tobacco sometimes kills.
Here's today's Today program(me) – the Rainmaker part starts at 2:43.45
Your story got me thinking about different ways we humans understand information.
If the data is presented sequentially over a long period of time then we may tend to view it as a sequence of individual cases (anecdotes), and might not be able to integrate it to identify general trends. Focusing on individual cases (i.e., anecdotes) might be a reasonable strategy when answering complicated questions using small amounts of data (a kind of nearest neighbour inference).
But our visual system is very good at extracting general trends, as well as picking out outliers. This may be part of the explanation of why graphical displays are so much more effective than tables — graphs exploit the power of our visual system to communicate what might be a very complicated structure.
If this general line of thought is correct, it might be worth trying to use insights from psychology when designing graphs. For example, there are interesting results on subitizing that may be useful.
I heard someone say, "Look at George Burns: he smoked cigars constantly but he lived to be 100." And then someone else says, "Yes, but cigars aren't as bad for you as cigarettes," as if that offers an explanation. These people are observing variance and confusing it with expectation.
The psychological aspect is interesting: I've flown airlines that consistently arrive 15-30 minutes late and I don't complain, but I'll get mad if my flight arrives 3-4 hours late, even if the airline is generally on-time.