This post is from Witold.
I thought some of you may find this pre-print (that I am a co-author of) interesting. It’s a meta-analysis of improving water quality in low and middle income countries. We estimated this reduced odds of child mortality by 30% based on 15 RCT. That’s obviously a lot! If true, this would have very large real-world implications, but there are of course statistical considerations of power, publication bias etc. So I thought that maybe some of the readers will have methodological comments while others may be interested in the public health aspect of it. It also ties to a couple of follow-up posts I’d like to write here on effective altruism and finding cost-effective interventions.
First, a word on why this is an important topic. Globally, for each thousand births, 37 children will die before the age of 5. Thankfully, this is already half of what it was in 2000. But it’s still about 5 million deaths per year. One of the leading causes for death in children is diarrhea, caused by waterborne diseases. While chlorinating [1, scroll down for footnotes] water is easy, inexpensive, and proven to remove pathogens from water, there are many countries where most people still don’t have access to clean water (the oft-cited statistic is that 2 billion people don’t have access to safe drinking water).
What is the magnitude of impact of clean water on mortality? There is a lot of experimental evidence for reductions in diarrhea, but making a link between clean water and mortality requires either an additional, “indirect”, model connecting disease to deaths, which is hard [2], or directly measuring deaths, which are rare (hence also hard) [3].
In our pre-print [4], together with my colleagues Michael Kremer, Steve Luby, Ricardo Maertens, and Brandon Tan we identify 53 RCTs of water quality treatments. Contacting the authors of each study resulted in 15 estimates that could be meta-analysed, with about 25,000 children. (Why only 15 out of 53? Apparently because the studies were not powered for mortality, with each one of them contributing just a handful of deaths, in some cases the authors decided to not collect, retain or report deaths.) As far as we are aware, this is the first attempt to meta-analyse experimental evidence on mortality and water quality.
We conduct a Bayesian meta-analysis of these 15 studies using a logit model and find a 30% reduction in odds of all-cause mortality (OR = 0.70, with a 95% interval 0.49 to 0.93), albeit with high (and uncertain) heterogeneity across studies, which means the predictive distribution for a new study has a much wider interval and slightly higher mean (OR=0.75, 95% interval 0.29 to 1.50). This heterogeneity is to be expected because we compare different types of interventions in different populations, across a few decades.[5] (Typically we would want to address this with a meta-regression, but that is hard due to a small sample.)
The whole analysis is implemented in baggr, an R package that provides meta-analysis interface for Stan. There are some interesting methodological questions related to modeling of rare events, but repeating this analysis using frequentist methods (random-effects model on Peto’s OR’s has a mean OR of 0.72) as well as various sensitivity analyses we could think of all lead to similar results. We also think that publication bias is unlikely. Still, perhaps there are things we missed.
Based on this we calculate about $3,000 cost per child death averted, or under $40 per DALY. It’s hard to convey how extremely cost-effective this is (a typical cost effectiveness threshold is equivalent of one years GDP per DALY; this is reached at 0.6% reduction in mortality), but basically it is on par with the most cost-effective child health interventions such as vaccinations.
Since the cost-effectiveness is potentially so high, there are obviously big real-world implications. Some funders have been reacting to the new evidence already. For example, some months ago GiveWell, an effective altruism non-profit that many readers will already be familiar with, conducted their own analysis of water quality interventions and in a “major update” of their assessment recommended a grant of $65 million toward a particular chlorination implementation [6]. (GiveWell’s assessment is an interesting topic for a blog post of its own, so I hope to write about it separately in the next few days.)
Of course in the longer term more RCTs will contribute to precision of this estimate (several are being worked on already), but generating evidence is a slow and costly process. In the short term the funding decisions will be driven by the existing evidence (and our paper is still a pre-print), so it would be fantastic to see if readers have comments on methods and its real-world implications.
Footnotes:
[1] For simplicity I simply say “chlorination” but this may refer to chlorinating at home, at the point from which water is drawn, or even using a device in the pipe, if households have piped water which may be contaminated. Each of these will have different effectiveness (primarily due to how convenient it is to use) and costs. So differentiating between them is very important for a policy maker. But in this post I group all of this to keep things simple. There are also other methods of improving quality, e.g. filtration. If you’re interested, this is covered in more detail in the meta-analyses that I link to.
[2] Why is extrapolating from evidence on diarrhea into mortality hard? First, it is possible that reduction in severe disease is higher (in the same way that vaccine may not protect you from infection, but it will almost definitely protect you from dying). Second, clean water also has lots of other benefits, e.g. it likely makes children less susceptible to other infections, nutritional deficiencies, and also makes their mothers healthier (which could in turn lead to fewer deaths during birth). So while these are just hypotheses, it’s hard a priori to say how a reduction in diarrhea would translate to a reduction in mortality.
[3] If you’re aiming for 80% power to detect 10% reduction in mortality you will need RCT data on tens of thousands of children. Exact number of course depends on how high baseline mortality rate is in the studies.
[4] Or, to be precise, an update to a version of this pre-print which we released in February 2022. If you happened to read the previous version of the paper, both main methods and results are unchanged, but we added extra publication bias checks, characterization of the sample and rewrote most of the paper for clarity.
[5] That last aspect of heterogeneity seems important, because some have argued that the impact of clean water may diminish with time. There is a trace of that in our data (see supplement), but with 15 studies the power to test for this time trend is very low (which I show using a simulation approach).
[6] GiveWell’s analysis included their own meta-analysis and led to more conservative estimates of mortality reductions. As I mention at the end of this post, this is something I will try to blog about separately. Their grant will fund Dispensers for Safe Water, an intervention which gives people access to chlorine at the water source. GW’s analysis also suggested a much larger funding gap in water qulity interventions, of about $350 million per year.
Witold,
Thanks for sharing. This is interesting and potentially important work. I was struck by the first two sentences of the abstract of your paper:
The idea that this intervention wasn’t considered because it wasn’t on some earlier list . . . that’s interesting, and I wonder how often this occurs in other settings too, that effective treatments aren’t being considered because they were thought of as being part of a different category.
We expand on this point in the paper. From what I understand (having seen some conversations with policy makers), this issue is uniquely bad in the case of water, because you can think of provision of clean water is a health issue or an infrastructure/sanitation issue, which means there is never a clear funder. There isn’t such an ambiguity with, say, vaccines. That’s why being on these lists (we talk particularly about two WHO-endorsed lists) is important. But to do that you need cost effectiveness estimates.
What’s the posterior expected increase in life years per $1000 spent?
Good question. Per our calculation you get about 25-30 healthy life years per $1000 spend. Providing 5 years of chlorine (including distribution costs) for the two specific examples we considered is $25-50. This is sensitive to take-up rate, which in turn depends a lot on program design (e.g. giving out chlorine bottles vs distributing through health clinics vs maintaining dispensers at water sources).
By the way, in my experience it’s hard to capture the uncertainty in cost when measuring health impacts, because we end up with a ratio distribution X/Y, where X, cost, is known with good precision, Y, health impact, has less precision and may be (as it is in our case) heavy-tailed (predictive distribution of OR is log-normal and uncertain). So the mean of X/Y is not defined. Hence the DALY calculation uses cost per person divided by expected reduction in DALYs per person, as we explain in the paper. Perhaps there is a better way of doing this?
I hope there is some kind of chlorine monitoring system involved in this plan. Adding chemicals to the water without making sure they are diluted to safe levels does not make water “clean”, even if the parasites/etc all died. If not considered, how much cost would that add?
As mentioned below, you could cause a lot of harm (via miscarriages, and also response to birth defects) while thinking you are helping. Miscarriages/abortions confound your childhood mortality outcome. This isn’t something to shrug away or laugh at.
@Anoneuoid: giving out chlorine bottles is in fact not the preferred approach anyway, I used it as a basic example, but the current thinking is to distribute via chlorine dispensers (dosing the appropriate amount per container at the point where water is collected) or providing water treatment solution via health clinics (e.g. during routine pregnancy visits).
I don’t know enough about chlorine toxicity to comment on pre-term mortality, but I’d think if there is risk increase, it would result from long-term exposure, same as with exposure to most pollutants. But I will ask my colleagues.
As for monitoring for chlorine, studies of these interventions check water samples for presence of chlorine (with households permission: but the refusal rate is very low) so that researchers don’t rely on self-reported data on whether people chlorinate. (But, just to be clear, our estimates are intent-to-treat.)
Are RCTs for water quality really needed at XX or XXI century?
That was my thought too. This seems like a problem that could be very well addressed with a BACI (before-after-control-impact) design.
For some in public health still the absence of evidence is evidence of absence, but the bigger question is of the magnitude of the effect. While all reasonable people agree that water quality will impact diarrhea, hence impact mortality, I believe (although I am not a “water person” myself) that experts’ priors were in single digit reductions in mortality, probably closer to 0 than 10%. Moreover, without an estimate (whether it’s 3% or 30% reduction) it’s hard to make a case for including it on the lists of recommended child health interventions (see Andrew’s comment above).
I agree that water quality is important and should be brought up to modern standards. But with a limited budget and many other problems to tackle, you need to prioritize.
Of course, this is “evidence-based medicine”. Every time a slight variation comes up there is “no evidence”. Like when the WHO claimed in 2020 there was “no evidence” antibodies towards covid conferred immunity, ignoring literally everything known about the topic.
One issue with these mortality numbers though, is they ignore miscarriages and abortions. An intervention can decrease child mortality by increasing either of those rates in the riskiest population. I wouldn’t think clean water would do this, but perhaps chlorine toxicity?
https://www.ewg.org/news-insights/news/chlorinated-tap-water-called-risk-pregnant-women
Is it really a benefit if the child dies in the womb rather than a few days to years after birth? That needs to be considered and accounted for.
> Like when the WHO claimed in 2020 there was “no evidence” antibodies towards covid conferred immunity,
“At this point in the pandemic, there is not enough evidence about the effectiveness of antibody-mediated immunity to guarantee the accuracy of an ‘immunity passport’ or ‘risk-free certificate,’ ” WHO said.
Dr. Maria Van Kerkhove from WHO has previously said it’s not known whether people who have been exposed to the virus become completely immune. The new WHO brief underscores that stance, and jibes with other scientific statements about the idea of developing immunity.
During a Friday briefing, the Infectious Diseases Society of America warned that not enough is known about antibody testing to assume immunity.
Dr. Mary Hayden, spokesperson for IDSA and chief of the Division of Infectious Diseases at Rush University Medical Center, said, “We do not know whether or not patients who have these antibodies are still at risk of reinfection with Covid-19. At this point, I think we have to assume that they could be at risk of reinfection.”
“We don’t know even if the antibodies are protective, what degree of protection they provide, so it could be complete, it could be partial, or how long the antibodies last,” Hayden added, “We know that antibody responses wane over time.”
Right in line with your bad faith characterization of what Wolensky said about vaccine-induced immunity. I guess you just don’t give a shit.
It sounds like the same thing is happening regarding water quality, so here is the “no evidence” claim:
https://washingtontimes.com/news/2020/apr/27/who-walks-back-no-evidence-claim-coronavirus-immun/
From there you are linked to a press release on the WHO site that *still says*:
This is totally false. There was tons of evidence at the time that antibodies would be protective until they waned and the virus mutated.* That is what always happened for coronaviruses.
Some think the WHO and CDC actually knew that protective antibodies followed by mutations and waning was by far the most likely outcome, but kept it from the public. If I hadn’t done medical research myself, I might also think so.
But it really is a severely confused philosophy. They busted out the same “no evidence” line about human-to-human transmission, then nosocomial transmission, aspiration during injections, and apparently are doing the same regarding clean water.
If someone wants to claim clean water would *not* reduce child mortality, they should offer some argument why. Due to the mountains of evidence, the default is that clean water would be beneficial.
Thanks for an interesting post with focus on discussing the uncertainties. Science as it should be done.
From my non-expert perspective it looks like another public policy situation where given the uncertainties along with a potential huge effect, interventions are merited. The effective altruism folks have taken some big hits lately, but maybe they got this one right.
Thanks so much! I mentioned that I will do follow-up posts on EA, cost-effectiveness and uncertainties, this is something I definitely want to cover in a bit more detail.
Witold –
I look forward to reading that. I think it’s a very interesting (and important) topic. It gets to the heart of so much related to risk, decision-making, uncertainty in science and decision-making, etc.
Did you think about ways to incorporate non-RCT evidence?
Yes, thanks, we have been looking into it for a follow-up paper. We haven’t done this yet, but I guess the question then becomes how to weigh RCT vs observational data. Somewhat similar issue to how to weigh direct (reduction in mortality) vs indirect (reduction in waterborne disease, leading to reduction in mortality) evidence. I will do a follow-up post on that.