The “all else equal fallacy,” one more time (this time using econ jargon), and also a discussion of the perils of “crossover” arguments

I was sorry to see Steven Levitt repeating the claim about driving a car being good for the environment. I wrote about this last week when it appeared in the other New York Times column of John Tierney, but perhaps it’s worth repeating:

These guys are making a classic statistical error, I think, which is to assume that all else is held constant. This is the error that also leads people to misinterpret regression coefficients causally. (See chapters 9 and 10 of our book for discussion of this point.) In this case, the error is to assume that the walker and the driver will be making the same trip. In general, the driver will take longer trips–that’s one of the reasons for having a car, that you can easily take longer trips. Anyway, my point is not to get into a long discussion of transportation pricing, just to point out that this seemingly natural calculation is inappropriate because of its mistaken assumption that you can realistically change one predictor, leaving all the others constant.

Unintended consequences of an economist forgetting about unintended consequences

I’m surprised that Levitt didn’t notice this, given that the distinction between “exogenous” and “endogenous” variables is such a big deal in economics. In fact, an important contribution that economists often make to public policy debates is to emphasize that you can’t simply assume “all else held equal” in an analysis. In fact, Levitt himself made this point is his column a couple months ago, in discussing unintended consequences. One of the consequences of switching from driving to walking is that you take shorter trips. Maybe this is a good thing, maybe it’s a bad thing, but I don’t think it makes a lot of sense to say, “Be Green: Drive” without realizing that distance traveled is affected by the choice.

P.S. Levitt buttresses his argument with the statement, “Chris Goodall [the person who made the walking/driving comparison] is no right-wing nut; he is an environmentalist and author of the book How to Live a Low-Carbon Life.” How relevant is this? Even a “right-wing nut” could make a good point, right? More to the point, I think we have to be careful about automatically trusting “crossover” arguments. Do we have to believe something, just because it comes from somebody who we wouldn’t expect to say it? I worry that this sort of crossover appeal is so appealing that otherwise-skeptical commentators (such as Levitt) forget their usual skepticism.

P.P.S. Yes, I realize that Levitt might just be trying to be amusing and thought-provoking rather than making a claim about public policy. From the standpoint of economics and statistics, though, I think this really a great opportunity to explain why the “all else equal” assumption can cause problems. A great example for a course in linear regression or econometrics.

10 thoughts on “The “all else equal fallacy,” one more time (this time using econ jargon), and also a discussion of the perils of “crossover” arguments

  1. Your basal metabolism rate is not that much less than your rate of using up calories while walking. The calculation would only be valid if you didn't burn calories while driving to the store (or for that matter just lying around on the couch doing nothing). This was pointed out by one of the comments to the Tierney article; one that Levitt appears to have missed.

    This is just an embarrassment to anyone pushing the argument.

  2. Bill,

    Exactly. My question is, how did this slip by Levitt's economics-trained skepticism? Is it just the "Even an environmentalist says …" factor?

  3. In this case, the error is to assume that the walker and the driver will be making the same trip.

    While it's true that people with cars will tend to take longer trips, isn't the challenge in getting people to take a given trip by walking or biking rather than driving as per usual, in which case the trip is the same? For example, I have to decide whether to take a five minute drive or a twenty minute bike ride to the grocery store. This might be the situation they're envisioning when they assume equal distance trips.

  4. '"…Chris Goodall [the person who made the walking/driving comparison] is no right-wing nut; he is an environmentalist and author of the book How to Live a Low-Carbon Life." How relevant is this?'

    It's only relevant to the extent that someone would respond with an ad hominem about the argument coming from a right-wing nut.

    Plus, I suppose a lot of people misunderstand the gem "only Nixon could go to China."

  5. Andrew,

    I am a bit surprised that you bring this up again given that in my assessment you lost the original argument fair and square.

    But wait, I opposed you in the original argument, so shouldn't neutral readers take this fact into account when deciding whether to trust this assessment? Absolutely, I'm an interested party!

    Likewise, one should be more skeptical of a pro-car finding when it comes from the car industry (or a "right-wing nut") than when it comes from an environmentalist. Was it just chance that the cigarette industry was the last party that denied a link between lung cancer and smoking?

    You ask:

    'Even a "right-wing nut" could make a good point, right? (…) Do we have to believe something, just because it comes from somebody who we wouldn't expect to say it?'

    Put like this, the answers are obviously "yes" and "no", and one could add that an environmentalist could unwittingly submit an analysis that makes cars look too good. But I think that one should think about this in probabilistic terms. This is especially so because in statistics there are always decisions to be made which could go this way or that. Do you leave outliers out? Which method of factor or cluster analysis is chosen? Which control variables go in, which don't? How are these operationalized?

    I've read quite a bit on the deterrence of crime. In some cases, I can predict the result of an empirical paper by looking at the name of the author. Joanna Shepherd comes to mind. So I'll discount "deterrence works" results from that author even if I personally can't find anything wrong with the analysis.

    Bayesians talk about how one should update one's beliefs if one finds others disagreeing with one's priors. But the extend to which they are updated should depend on who says it.

  6. Cmon, the guy is a nut himself, please dont judge economists by him. Hope youve seen freakonomics critics by dinardo, its all there.

  7. Lemmus,

    I agree that the source of information is relevant, but here it seemed that Levitt was using an analysis of the source _instead_ of thinking seriously about the argument.

    The larger issue is selection bias. Calculation of energy expenditure are not difficult for a someone with physics training; see, for example, Phil's calculations in his comments to the earlier entry. But, Phil's commonsensical conclusions wouldn't get picked up the international news media. Instead, you get someone who makes an attention-getting claim–that walking is worse for the environment than driving–and, hey, surprise, that's what gets the attention! The fact that the guy is an environmentalist is just part of the bigger story: if the study had been published by General Motors, nobody would've given it a second look. The "probabilistic analysis" you describe does not take into account the process by which research becomes publicized.

    So, from a "systemic" perspective, I think there's very little information to be gained from the fact that the author of this study is not a "right-wing nut." I think that all this did was allow Levitt to let his guard down and forget some standard lessons of causal inference.

    P.S. I seem to recall Joanna Shepherd saying that she opposed the death penalty, or something to that effect, but I don't recall exactly. (You can search this blog for "death penalty" for my own work in this area.)

    P.P.S. I think this "right-wing nut" thing is a distraction. If you want to say that the work is not being supported by the auto industry, fine, but I'm really opposed to this implicit policy of giving people credentials based on their political affiliations. The physics here is straightforward enough that I don't think right-wing or left-wing comes into it at all.

  8. Andrew,

    I agree that Levitt doesn't seem to have thought hard about the issue – but then, he doesn't pass judgment; all he says is that the commmon sense view has been "called into question", which is certainly accurate. (As for the title of the post, I'll give him the benefit of the doubt and call it "tongue-in-cheek", a mode of communication Levitt seems to love.)

    My interpretation of Phil's comment that you mention differs from yours – in my view it makes clear that these calculations are anything but straightforward and that you always have to make crucial assumptions – something that was also mentioned in the Tierney piece (how many miles per gallon, what does the driver eat and drink).

    The most straightforward message I got from this whole thing is that it is good for the environment to eat little or no meat – something I already knew, of course.

    Your point about the press (and scientific journals, one might add) being more likely to report "man bites dog" stories is a very good one. That would definitely have to be taken into account when updating one's beliefs, something I overlooked in my first comment.

  9. According to the EPA, there are about 28,000 kcal in a gallon of gas. Walking a mile, for a 150-lb person, consumes about 60 kcal above basal metabolism. Driving a mile at 28 mpg consumes about 1,000 kcal in gasoline. Tough to fit another 940 kcal into 2/3 of a cup of milk (metaphorically speaking.) In short, I have a hard time believing it.

    I also note Tierney's quote talked about the cost of producing and shipping the milk, environmentally speaking, but not the cost of producing and shipping the gas. Hmmmm… I wonder which has the bigger environmental impact, producing and transporting 2/3 cup of milk or 1/2 cup of gas? And milk is not an environmentally friendly product. What if I ate a slice of bread instead?

  10. Why is there any expectation the question of walking vs driving can be settled on any quantitative basis other than an isolated physical one when we don't understand the couplings to other economic and resource sinks and sources any better than we do those of sea surface temperatures?

Comments are closed.