Last year I wrote a post, “Why the New Pollution Literature is Credible” . . . but I’m still guessing that the effects are being overestimated:.
Since then, Vincent Bagilet and Léo Zabrocki-Hallak wrote an article, Why Some Acute Health Effects of Air Pollution Could Be Inflated, that begins:
Hundreds of studies show that air pollution affects health in the immediate short-run, and play a key role in setting air quality standards. Yet, estimated effect sizes vary widely across studies. Analyzing the results published in epidemiology and economics, we first find that a substantial share of estimates are likely to be inflated due publication bias and a lack of statistical power. Second, we run real data simulations to identify the design parameters causing these issues. We show that this exaggeration may be driven by the small number of exogenous shocks leveraged, by the limited strength of the instruments used or by sparse outcomes. These concerns likely extend to studies in other fields relying on comparable research designs. Our paper provides a principled workflow to evaluate and avoid the risk of exaggeration when conducting an observational study.
Their article also includes the above graph. It’s good to see this work being done and to see these type M results applied to different scientific fields.
P.S. I’m putting this in the Multilevel Modeling category because that’s what’s going on; they’re in essence partially pooling information across multiple studies, and individual researchers could do better by partially pooling within their studies, rather than selecting the biggest results.
The caption for that figure:
Why are there so many inverses involved in the making of this figure?
It looks very different from the histogram of the t-statistics in the online supplement, but maybe it’s not the same thing?
In addition to the obvious peak at p=0.05, the histogram also has a general periodicity pattern due to the rounding effect of p values. It will be useful to further separate the rounding effect vs the potential p hacking/censoring effect.
I looked into the reliability of some estimates of nitrogen deposition on species richness in grasslands (using some data from a couple of influential papers in ecology) and also came up with smaller effect estimates then originally claimed. See https://peerj.com/articles/9070/
I’m sure readers of this blog will recognise the impacts of Andrew’s work on my thinking in that paper.
Our paper has been barely cited by the ecological air pollution literature, except by the authors of the original studies re-analysed, who prefer a scenario where the coarse estimates of nitrogen deposition confound the effect signal (we noted this in passing in our discussion fwiw).
Criticisms of what we did welcome, we don’t claim that it’s perfect.