Story time meets the all-else-equal fallacy and the fallacy of measurement

Alex Tabarrok with a good catch:

In Why Don’t Women Patent?, a recent NBER paper, Jennifer Hunt et al. [Jean-Philippe Garant, Hannah Herman, and David Munroe] present a stark fact: Only 5.5% of the holders of commercialized patents are women. One might think that this is explained by the relative lack of women with science and engineering degrees but Hunt et al. find that “women with such a degree are scarcely more likely to patent than women without.” Instead, most of the difference is “accounted for by differences among those with a science or engineering degree” especially the fact that women are underrepresented in patent-intensive fields such as electrical and mechanical engineering and in development and design.

Predictably, the authors do not ask why women might self-selection into non patent-intensive fields, perhaps because this would require at least a discussion of politically incorrect questions. The failure to investigate these questions leads to some dubious conclusions, notably:

Closing the [gender] gap among S&E degree holders would increase commercialized patents by 24% and GDP per capita by 2.7%.

Right; and since only 10% of construction workers are women, closing the gender gap would result in many more houses. In the case of construction, my suspicion is that gender equality would reduce not increase the amount of construction. In the case of patents, I am not sure what would happen, indeed the point is that without a much better understanding of what causes differences in patent proclivities one shouldn’t jump to conclusions.

This one hits three of my tropes! (Recall the lexicon.)

First, the all-else-equal fallacy: Assuming that everything else is held constant, even when it’s not gonna be.

As a person who’s written many times about unintended consequences, Alex is well aware of the problems arising from extrapolating a pattern in observational data to make claims about the effects of future intervention.

Second, there’s story time: When the numbers are put to bed, the stories come out. It’s that all-too-quick moment when the authors pivot from the causal estimates they’ve proved, to their speculations, which, as Kaiser Fung has written, are “no more credible than anybody else’s story.” Maybe less credible, in fact, because researchers can fool themselves into thinking they’ve proved something when they haven’t.

The third problem with Hunt et. al conclusions is the measurement fallacy of taking something that can be easily measured and identifying it with something we care about. Statisticians and quantitative social scientists always have to watch out for this: progress can be made via quantitative analysis, but there is always the difficulty of translation.

In this case, the problem is looking at patents and making claims about technical innovation. As Alex writes,

The quick jump from patents to innovation is also unwarranted—there is very little evidence that patents increase innovation. Moreover, most innovations are not patented.

Indeed. As anecdotal evidence, I will submit this in favor of Alex’s first point, and my entire career in support of his second.

Just to be clear . . .

This is not to say that the work of Hunt et al. is valueless. I’m a big fan of looking at patterns in data. My colleagues and I wrote a whole book about income and voting, even though (a) the incomes being analyzed were imperfect survey responses and (b) we’d probably be interested in wealth more than income, if we only had good measurements of wealth. And then we yammered on about “rich” and “poor.” So it’s all a matter of degree. I have not read the Hunt et al. paper (clicking through took me to a page with the abstract and a note that I’d have to pay $5 for the whole article), but based on the abstract, it looks like they’ve gone a bit further in their conclusions than their data warrant.

7 thoughts on “Story time meets the all-else-equal fallacy and the fallacy of measurement

  1. “problems arising from extrapolating a pattern in observational data to make claims about the effects of future intervention.”

    This reminds of the claims made in the recent nutritional scare out of Harvard regarding red meat and how replacing red meat with nuts or whatever would provide some beneficial effect. http://www.nytimes.com/2012/03/13/health/research/red-meat-linked-to-cancer-and-heart-disease.html?_r=1&scp=1&sq=red+meat&st=cse

    Besides the many other problems with this analysis, that “substitution association” (aka blatant causal claims) analysis represents some serious unwarranted extrapolating…

  2. “the measurement fallacy of taking something that can be easily measured and identifying it with something we care about.”

    Okay, but having some data is better than having no data. So, can you think of a better or comparable measure that would show that women are more equal to men in technological innovativeness? I can think of a number of ways to approach this, but none of them seem all that likely to come up with a much more politically correct result.

  3. Andrew – have you read Banerjee and Duflo’s “Poor Economics”? If so, do you think it contains many examples of your “story time” pet-peeve? I’d think you would, but curious to hear further thoughts on that.

    • Brett:

      I’ll delegate this one to Kaiser, who’s already taken a look at that book. I will say, thought, that I think “story time” is ok if it is labeled as such. My problem is when people assume that an extreme level of rigor at one part of an analysis gives license for inappropriate certainty later on. As if the extra rigor can be spread into the other conclusions the way you might even out the peanut butter in a sandwich. I have no comment on Banerjee and Duflo on this issue, having not seen their book.

Comments are closed.