In a post entitled, “Why the New Pollution Literature is Credible,” Alex Tabarrok writes:
My recent post, Air Pollution Reduces Health and Wealth drew some pushback in the comments, some justified, some not, on whether the results of these studies are not subject to p-hacking, forking gardens and the replication crisis. Sure, of course, some of them are. . . . Nevertheless, I don’t think that skepticism about the general thrust of the results is justified. Why not?
First . . . my rule is trust literatures not papers and the new pollution literature is showing consistent and significant negative effects of pollution on health and wealth. . . . It’s not just that the literature is large, however, it’s that the literature is consistent in a way that many studies in say social psychology were not. In social psychology, for example, there were many tests of entirely different hypotheses—power posing, priming, stereotype threat—and most of these failed to replicate. But in the pollution literature we have many tests of the same hypotheses. We have, for example, studies showing that pollution reduces the quality of chess moves in high-stakes matches, that it reduces worker productivity in Chinese call-centers, and that it reduces test scores in American and in British schools. . . . from different researchers studying different times and places using different methods but they are all testing the same hypothesis, namely that pollution reduces cognitive ability. . . .
Another feature in favor of the air pollution literature is that the hypothesis that pollution can have negative effects on health and cognition wasn’t invented yesterday . . . The Romans, for example, noted the negative effect of air pollution on health. There’s a reason why people with lung disease move to the countryside and always have.
I also noted in Why Most Published Research Findings are False that multiple sources and types of evidence are desirable. The pollution literature satisfies this desideratum. Aside from multiple empirical studies, the pollution hypothesis is also consistent with plausible mechanisms . . .
Moreover, there is a clear dose-response effect–so much so that when it comes to “extreme” pollution few people doubt the hypothesis. Does anyone doubt, for example, that an infant born in Delhi, India–one of the most polluted cities in the world–is more likely to die young than if the same infant grew up (all else equal) in Wellington, New Zealand–one of the least polluted cities in the world? . . .
What is new about the new pollution literature is more credible methods and bigger data and what the literature shows is that the effects of pollution are larger than we thought at lower levels than we thought. But we should expect to find smaller effects with better methods and bigger data. . . . this isn’t guaranteed, there could be positive effects of pollution at lower levels, but it isn’t surprising that what we are seeing so far is negative effects at levels previously considered acceptable.
Thus, while I have no doubt that some of the papers in the new pollution literature are in error, I also think that the large number of high quality papers from different times and places which are broadly consistent with one another and also consistent with what we know about human physiology and particulate matter and also consistent with the literature on the effects of pollution on animals and plants and also consistent with a dose-response relationship suggest that we take this literature and its conclusion that air pollution has significant negative effects on health and wealth very seriously.
This all makes a lot of sense—enough so that I quoted large chunks of Tabarrok’s post.
Still, I think actual effects will be quite a bit lower than claimed in the literature. Yes, it’s appropriate to look at the literature, not just individual studies. But if each individual study is biased, that will bias the literature. You can think of Alex’s two posts on the effects of air pollution as a sort of informal meta-analysis, and it’s a meta-analysis that does not correct for selection bias within each published study. Again, his general points (both methodologically and with regard to air pollution in particular) make sense; I just think there’s a bias he’s not correcting for. When we talk about forking paths etc. it’s typically in settings where there’s essentially zero signal (more precisely, settings where any signal is overwhelmed by noise) and people are finding patterns out of nothing—but the same general errors can lead to real effects being greatly overestimated.
P.S. The comments to Tabarrok’s post are pretty wack. Some reasonable points but lots and lots of people just overwhelmed by political ideology.
I’m not familiar with the literature. Is he right that things like high level chess moves being worse with pollution has been replicated? How do you cleanly isolate the effect, anyway?
Either way I think he undersells the apparent strength of priming literature. There were replications (just with heavy selection bias or through adding silly interactions). And the literature was incredibly consistent: people’s decisions can be affected by small primes. And everyone agrees that context can change behavior, so isn’t it reasonable to find effects from a small dose of context in the form of a prime?
The problem is with the size of the intervention vs. The size of the effect and the noisiness of the outcome measured. I haven’t read the new pollution literature, but I can definitely understand people’s skepticism.
I’ll also add that the pollution effects we strongly believe in are from the accumulation of pollution in the body over time, no? I don’t see a clear extension to the claimed instantaneous effects of pollution on cognition (I’m assuming the studies look at panel data over some time frame rather than a cross-section across different people from different environments)
It makes me hope some psychologist tries to test it by having subjects perform random tasks while a couple cows are farting in the room.
Immediate impacts of high pollution have been recognized for a long time, see here for example: https://en.m.wikipedia.org/wiki/Great_Smog_of_London
Sure, and people die from smoke inhalation as well. I still don’t feel that really informs us of call center or chess performance from a moderate, temporary increase in pollution. That feels akin to using the effects of drowning to predict the impact of drinking an extra glass of water.
If even mild to moderate pollution is dulling our brains, we’d see the most scientific progress by shipping our best scientists out of the polluted cities and to some rural part of New Mexico.
Kj:
I don’t know that it would be a good idea for us to ship our best scientists to some rural part of New Mexico. They might then go and develop a superweapon that could destroy the human race. Better to keep them in the big cities where, with dulled brains, they’ll devote their careers to doing various bits of incremental research. Or, put them in a big and polluted enough city and they’ll just spend their time responding to blog comments.
kj, you said “I’ll also add that the pollution effects we strongly believe in are from the accumulation of pollution in the body over time, no?”
I thought you were trying to imply that we do _not_ strongly believe in acute effects that take place almost immediately.
If what you meant is that we strongly believe in effects from both accumulation over time AND from immediate exposure then sure, OK, yes, “we” strongly believe in both of those things.
This is so interesting because if you cast about for the impacts of global warming you’ll “learn” that it’s already shaving gazillions of collective years off people’s lives – yet sans COVID and other known direct causes of death, life expectancy is rising. Right?
What gives? How can we be collectively dying of global warming and pollution while life expectancy is rising?
The problem with this approach is that, while pollution may indeed create particles that have negative health effects, the pollution (and global warming) are created by economic activity that dramatically improves health and well being. What if we pass “zero pollution” laws? Then how will we have ambulances to get us to hospitals and how will we have hospitals to address our critical care needs?
This way of thinking about both pollution and global warming is analogous to Ehrlich’s thinking about the “Population Bomb” – it fails to recognize that technology has already and/or will outrun the danger. Pollution and global warming result from technological progress. Technological progress improves life expectancy faster than pollution and global warming undermine it. While both pollution and global warming can be addressed in some practical sense within some limit, eliminating both altogether by fiat (rather than through further technological progress) would cause more harm than good because it would destroy the technological progress that’s actually causing life expectancy to rise.
Whether Alex’ point is technically right (I agree with Andrew, it’s probably overstated) or not is moot. In the larger picture, it’s wrong.
The trend is your friend, until…
What data you are referring to? Life expectancy in the US hasn’t been rising since about 2014. Of course there may be other estimates and countries out there.
For the US though, the CDC wonder data showed this was mainly due to an increase in “poisonings”. Ie, the “opioid crisis” and middle aged people over/under-dosing blood pressure medications.
Also, I don’t know why you would leave out covid as if that is an exception rather than part of the system. It reminds me of when people try to “control for” el Ninos when looking at temperature data as if that isn’t part of the climate.
Is Alex Tabarrok a pen name of John Ioannidis? If not, why does he write “I also noted in Why Most Published Research Findings are False that…”? (I am not a native speaker, so I might be getting something wrong here.)
“Why Most Published Research Findings are False” is the title of Ioannidis’ paper, but it is also the title of Tabarrok’s blog post about that paper. In the full text, but not the excerpt quoted above, Tabarrok writes “First, go back to my post Why Most Published Research Findings are False …” which makes clear that he is referring to his blog post and not Ioannidis’ paper.
The never-ending search for simple answers to complex questions. The health of our population is shaped by all kinds of inputs that are hard to tease out. I think that the reduction in cigarette use in the last generation has been a positive, and I’m glad that the government has put some effort into this. When I was a young resident, I saw a man with berylliosis that he got working on the Manhattan Project in Oak Ridge. I think that the Manhattan Project saved many American lives, but I favor great restrictions on aerosolizing beryllium. I have seen quite a few patients with asbestos related cancers. Asbestos is a good fire retardant; retarding fire is good, but I urge caution around asbestos. A few years ago, there was literature on leaded gasoline and crime; the evidence is worth considering. Nineteenth century coal mining led to real economic development, but they were careless about the health of the miners.
The roots of the word “analysis” mean to cut apart. In a lot of social and economic comments I see very little analysis, cutting apart, and a lot of affinity conglomeration.
I am always puzzled by why economists are trying to estimate the negative health effects of pollution. As Messieurs Tabarrok and Gelman (and any other mildly aware person would) note, the face validity of the hypothesis that pollution is bad for health is unquestionable. The real issue is the gradient, or the dose-response effect of pollution.
I assume that by referring to a “new” pollution literature, Mr. Tabarrok is referring to studies with hi internal validity. These are often very narrowly constructed studies that have little external validity. Consider:
Deryugina, Tatyana, Garth Heutel, Nolan H. Miller, David Molitor, and Julian Reif. 2019. “The Mortality and Medical Costs of Air Pollution: Evidence from Changes in Wind Direction.” American Economic Review, 109 (12): 4178-4219.
First, note the result:
“We estimate that a 1 microgram per cubic meter (μg/m3) (about 10 percent of the mean) increase in PM 2.5 exposure for one day causes 0.69 additional deaths per million elderly individuals over the three-day window that spans the day of the
increase and the following two days.”
Given 50 million elderly, this is about 35 deaths (95% CI, 29 to 41)! So, a 10% increase in pollution, as the authors measured it, is worth about $245 million at $7 million per life saved. How much would it cost to reduce pollution by 10%? My guess much more than $245 million.
This assessment illustrates the problem. It is incontrovertible that pollution adversely affects health. We need to study the issue to get an estimate of the dose-response relationship to use in a cost-benefit analysis to devise and implement policies to control pollution that may save lives.
The “new” pollution literature with its limited external validity does this poorly. For example, in the above cited study, the effect of pollution (PM2.5) was constrained to be linear–same effect whether the change in PM2.5 was from a low or high base. Whether this is a valid approach is a question, but it is reasonable and likely biologically appropriate to specify a non-linear relationship between pollution and health–there is likely (but maybe not) a threshold level at which the body has an ability to respond without adverse effect. In addition, people may adapt and the behavioral response will genrate a non-linear relationship.
In addition, the change in pollution was form a change in wind direction–no policy can change wind direction.
Third, the effect was measured off of a temporary increase or decrease in pollution–not a change in the sustained level of pollution, which is the dominant variation in pollution that affects health. We don’t want to be like Beijing or Moomba.
So, what did we learn from this “new” pollution study? I suggest the answer is not much.
The article was published in the flagship journal of the American Economic Association because economists love cleverly designed internally consistent studies no matter how useless they may be. Mr. Tabarrok seems to be one of those economists.
Some of these critiques don’t really make sense. If the change in the wind improves air quality in line with the expected improvement from a proposed policy, do we really care that we got the estimate from a study where the improvement was caused by the wind and not the policy? Why would we? Moreover re: the idea that sustained air quality is “the dominant variation in pollution that affects health”, while this is true, it somewhat ignores that that there appear to be some non-trivial health effects from temporary exposure to poor air quality as well.
I highly recommend you actually read an entire regulatory analysis including CBA of a proposed air quality rule from US EPA, because this comment reads to me like you haven’t done so. (Or you have, but are throwing that information out without explanation.)
Did the responder miss the fact that the results suggest that a 10% change in pollution, for example, an increase is associated with 35 deaths among those 65 and older. This is NOT a substantial effect in a nation of 50 million elderly people.
Also, from a recent EPA RIA (EPA-452/R-18-006, August 2018):
“For adult PM-related mortality, we use the effect coefficients from the most recent epidemiology studies examining two large population cohorts: the American Cancer Society cohort (Krewski et al. 2009) and the Harvard Six Cities cohort (Lepeule et al. 2012). The Integrated Science Assessment for Particulate Matter (PM ISA) (U.S. EPA 2009) concluded that the ACS and Six Cities cohorts produce the strongest evidence of the association between long-term PM2.5 exposure and premature mortality with support from additional cohort studies.”
Note the use LONG-TERM PM2.5 EXPOSURE in the analysis. Temporary wind changes and pollution are just that–temporary and EPA doesn’t think they are important.
And the EPA is well aware of the possibility of a threshold effect for PM2.5, although they use a log-linear assumption in main analyses.
“In general, we are more confident in the magnitude of the risks we estimate from simulated PM2.5 concentrations that coincide with the bulk of the observed PM concentrations in the epidemiological studies that are used to estimate the benefits. Likewise, we are less confident in the risk we estimate from simulated PM2.5 concentrations that fall below the bulk of the observed data in these studies. There are uncertainties inherent in identifying any particular point at which our confidence in reported associations decreases appreciably, and the scientific evidence provides no clear dividing line.”
I stand by all my remarks, although I would like to correct spelling of Mumbai (lol).
The cost of reducing pollution is not constant. Fracking made natural gas much cheaper in the US, and that pollutes much less than coal. Furthermore, those 35 deaths are just over 3 days. For an upfront cost of some pollution reducing technology you could get reduced pollution over a much larger number of days.
I’ve been reading Marginal Revolution since about 2005. After some fits and starts, they eventually decided on hands-off editing stance to comments that has made the blog dramatically less readable. Cowen and Tabarrok are still worth reading, but wading into the comments without cumbersome protective gear is more hazardous that indoor charcoal fumes. And with coherent on-point comments so rare, you lose the give-and-take that makes good blogs like this one and Scott Alexander even more valuable.
Which leads to the question: are the comments so much better here because of something that’s been done here, or is it just that this blog has not attracted the same trolls?
Jonathan:
I’ve wondered about that. One thing that in particular baffles me about that blog is that Cowen seems to be writing his posts to an audience of liberals, but his commenters mostly seem to range from the right to the far right, politically.
Cowen likes to be contrarian, which can be fine as long as he has some actual evidence to back up his claims. However, it attacts all of the nutty contrarians who things its smart to be against the consensus regardless of evidence. Taleb attracts those types too, but he blocks them.
“think”
> Which leads to the question: are the comments so much better here because of something that’s been done here, or is it just that this blog has not attracted the same trolls?
I can only assume the more technical posts make their brains explode, so Andrew should make more of them!
“my rule is trust literatures not papers”
Sometimes you can’t even trust literatures! Quoting from my replication of the literature on meritocratic promotion in China:
“The original study of meritocratic promotion for provincial leaders, Li and Zhou (2005), has been cited over 2500 times. But follow-up work has repeatedly failed to confirm its finding of a positive correlation between provincial GDP growth and promotion. And as I have shown in this post, attempts to extend the meritocracy story down to prefecture leaders have also failed.
How did this happen? How could a whole literature get this wrong?
Here’s my guess: researchers set a strong prior based on the provincial result in Li and Zhou (2005), combined with the elegance of the theoretical model of a promotion tournament. Since the idea of a promotion tournament is generic, researchers naturally expected it to apply to prefecture and county politicians as well. In short, researchers doing follow-up work knew that they had to confirm the original results.
However, when they studied prefecture leaders and didn’t find a positive correlation between growth and promotion, the researchers had to fiddle around with their models and data until they got a result that matched the original. And given the multiplicity of design choices, it wasn’t that difficult to find a specification that yielded statistical significance.”
https://michaelwiebe.com/blog/2021/02/replications
Michael:
Indeed. And let’s not forget the literature on priming in social psychology. The discredited elderly-walking paper from 1996 has over 6000 citations on google scholar—that’s an entirely untrustworthy literature all in itself!
Is it too gadfly-like to quote the Kalama Sutra, when Buddha was asked how to distinguish true experts from others?
《”It is proper for you, Kalamas, to doubt, to be uncertain; uncertainty has arisen in you about what is doubtful. Come, Kalamas. Do not go upon what has been acquired by repeated hearing; nor upon tradition; nor upon rumor; nor upon what is in a scripture; nor upon surmise; nor upon an axiom; nor upon specious reasoning; nor upon a bias towards a notion that has been pondered over; nor upon another’s seeming ability; nor upon the consideration, ‘The monk is our teacher.’ Kalamas, when you yourselves know: ‘These things are bad; these things are blamable; these things are censured by the wise; undertaken and observed, these things lead to harm and ill,’ abandon them.》
Can you grade me into having the same gut feelings as the experts?
Did Socrates need bans or grades, according to the Euthydemus?
《Do you then be reasonable, Crito, and do not mind whether the teachers of philosophy are good or bad, but think only of philosophy herself. Try and examine her well and truly, and if she be evil seek to turn away all men from her, and not your sons only; but if she be what I believe that she is, then follow her and serve her, you and your house, as the saying is, and be of good cheer.》