What follows is a “meta” sort of discussion, so I’ll put it below the fold and most of you can skip it.
After reading Steven Levitt’s “drunk driving is safer than drunk walking” argument, I was pleased to see that many of the commenters on his Freakonomics blog pointed out various flaws in his argument. Levitt and Dubner are lucky to have a blog with such alert commenters, and I think they deserve much of the credit for setting up this space where commenters can engage in intellectual discussions.
As a sort of natural experiment, we can look at how some other blogs’ comment sections handled this same topic. The commenters here were mostly sympathetic to my argument, which is no surprise given that youall are used to my modes of reasoning. (One commenter did accuse me of “ignoratio elenchi,” which I’d never heard of before, but I think i dealt with the substance of the comment later on in the thread.)
MIchael Bishop pointed me to the discussion on Jeff Ely’s blog. In my post, I pointed out the problems with Levitt’s general line of reasoning (treating different modes of transportation as substitutable on a per-mile basis); Ely goes to the trouble of crunching Levitt’s numbers more carefully. Like me, Ely is puzzled by Levitt’s apparent certainty about his claims. And, like me, Ely has commenters who agree with and elaborate on his points.
But here’s one more case. Tyler Cowen linked to both my entry and Ely’s. Only a few of Tyler’s commenters remarked on the drunk driving thing, but the ones who did were pretty much supportive of Levitt, basically characterizing Ely and me as typical know-nothing academics. That’s fine–I certainly don’t expect commenters on some other blog to have a sense of what I do. What surprised me was that the commenters at Tyler’s blog were more supportive of Levitt than were the commenters on Levitt’s own blog (who are generally on his side, I’d say, based a quick scan of some other entries). Interesting.
"no-nothing academics"?
I'm very interested in what you think of Hanson's interpretation of the smoking-cancer link:
http://www.overcomingbias.com/2009/12/what-anti-s…
Having read the chapter, Levitt's blog post, the comments on Levitt's post, your earlier blog post, etc., I agree with the commenter who said that he can argue for distance as the proper metric but he hasn't made the point, haven't shown that it's better than other metrics, etc. As many people have noted, this seems related to the fact that this book is not Levitt's work. Or as you note in your linked post, this stuff isn't easy and thus when it's not your work you really don't get into it in sufficient depth.
I thought that the comment on the Levitt blog #99:
http://freakonomics.blogs.nytimes.com/2009/12/14/…
was one of the best; It points out that the population of those that walk drunk might be overpopulated by those that live under bridges, compared to those that drive drunk. The assumption of the Levitt-Dubner hypothesis is that the underlying populations are identical (enough). But there's no reason to assume that. I would say that this fact undermines their hypothesis so much as to render their argument moot.
Noah: Typo fixed.
Retired:
Hansen questions whether smoking is actually bad for your health, citing the lack of any strong evidence from clinical trials of interventions that attempted to reduce smoking. The story I've always heard along those lines is that smoking kills people from lung cancer but one could imagine it saving lives in other ways, for example by relaxing people and allowing them to live less stressful lives and not get heart attacks or crash their cars or whatever.
That seems like a potentially plausible story to me, but something seems funny about Robin's examples because in one of those studies, reducing smoking doesn't even seem to reduce deaths from lung cancer. This seems suspicious to me, and I think it suggests these studies can't quite be interpreted so cleanly.
Robin's larger point is that there's a lot still unknown about the total effects of smoking (or, presumably, of any other public health intervention, since smoking has bigger effects than just about anything else out there).
A related point, which was touched on by Robin, is that at the individual level, it can be hard for people to quit, and anti-smoking interventions are often not so successful. (This makes sense, given that many of the people who could easily quit, have already done so.) On the other hand, there are large differences in smoking rates between countries and between social groups.
are we reading the same MR comments? I see exactly one post there that's somewhat defensive of L&D – and that one just says that they don't advocate drunk driving.
Then there's the same ignoratio elenchi guy who showed up on your blog.
I thought Jeff Ely's post was really outstanding and has one of the best comments I've read in a while:
http://cheeptalk.wordpress.com/2009/12/16/drunk-c…
Good grief, to use your jargon Andrew
– lack of statistical significance is seldom significant
Assuming that one study Hansen refered to was – "lung cancer (deaths+registrations) was 11% lower."
but the sample size was just too small to label it as statistically significant
A more careful look at the study may of course identify other propblems, but on what was provided the hypothesis was "replicated" in a weak sense.
Keith
Sebastian: I agree that Ely's post was excellent. It was also interesting to see that he and I offered completely different arguments, both of which were relevant. Ely was arguing about the statistical inference, and I was arguing about the decision analysis.
Keith: I thought that in one of Robin's studies, lung cancer was lower in the smoking group. In any case, I'm certainly not recommending that anyone go out and start smoking; my point was that if there appear to be near-zero total effects in some settings, this might be better interpreted as cancellation of various positive and negative effects, rather than some sort of overall null effect.
The "good grief" was due to my developing concern that the location, extraction, contrast and rendering of all the relevant evidence on a topic requires too much effort and care – for blogging and blog commenting to be very helpful.
Except maybe for the pointing out of new perspectives such your cancellation one – cancellation of various positive and negative effects, rather than some sort of overall null effect.
Keith