A major update in our assessment of water quality interventions

By GiveWell @ 2022-04-26T00:09 (+82)

This is a linkpost to https://blog.givewell.org/2022/04/06/water-quality-overview/

As we continue to grow, GiveWell seeks to maximize both the cost-effectiveness of the funding we direct and the likely room for more funding of the programs we support. We think we’ve identified a category of interventions that rates really well on both: water treatment, such as chlorination.

This is a major update for us. Before 2020, based on the available evidence, we didn’t believe that water quality interventions had a large enough effect on mortality to make them a competitive target for funding. We’ve since seen new evidence that has led us to significantly increase our estimate of the mortality reduction in young children that’s attributable to these interventions: a 14% reduction in mortality from any cause,[1] up from around 3%.

Though we have remaining uncertainties about these numbers, we’ve substantially updated our view of the promisingness of water treatment. Where we previously found that Evidence Action’s Dispensers for Safe Water program was about as cost-effective as unconditional cash transfers, we now believe it’s about four to eight times as cost-effective, depending on the location. That was a primary factor in our decision to recommend a grant of up to $64.7 million to Dispensers for Safe Water in January 2022.

We’re sharing this news in brief form before we’ve published a grant page, because we’re excited about the potential of this grant and what it represents. It’s an area of work we haven’t supported to a significant degree in the past, but one that we now think could absorb hundreds of millions of dollars in funding for cost-effective programming.

The problem and the intervention

In low-income settings, contaminated water is a major cause of diarrhea, a leading cause of death in children under five years old.[2] Several interventions exist to either purify water or protect it from contamination in the first place, but chlorination has a number of features that made it an attractive intervention for us to explore: it is inexpensive and widely used, several charities are already set up to implement it, and the technology behind it is well established.[3]

Chlorine is well-known as a disinfectant; it reacts with disease-causing microorganisms in water, inactivating viruses and bacteria.[4] There is evidence that chlorination programs, such as distributing chlorine to households, reduce diarrhea in children, but there had been scant evidence that such interventions reduce mortality, which is the single largest driver of cost-effectiveness in our models of water quality interventions.[5] Before 2020, we estimated the effect chlorination had on the prevalence of diarrhea, and then extrapolated from that effect to arrive at an indirect mortality reduction estimate of roughly 3%.[6] This translated to a low cost-effectiveness estimate relative to the programs GiveWell typically funds.

What led us to update

In mid-2020, Michael Kremer, one of the researchers who conducted the ​​initial trial studying dispensers for safe water, and now a researcher at the University of Chicago who has extensively studied global health interventions, shared with us preliminary results from his team’s new meta-analysis of mortality data from randomized controlled trials (RCTs) of water quality interventions.[7] (The working paper has since been published, here.) Pooling studies allowed Kremer’s team to obtain statistically significant results from studies that weren’t individually large enough to produce such results on their own. These results suggest that water treatments have a significant effect on mortality from any cause (“all-cause mortality”) in children under five—Kremer et al. currently estimate a roughly 25% reduction.[8]

Their finding is especially noteworthy because it comes from an analysis of direct evidence on mortality, and it is eight times larger than an indirect estimate that extrapolates from diarrhea morbidity. This implies that water treatment yields a mortality benefit that cannot be fully explained by reduction in diarrhea alone.

This led us to take a deeper look at the evidence and revisit our cost-effectiveness analysis. Using the new data as a foundation, we completed our own meta-analysis of RCTs of chlorination interventions, arriving at a mortality reduction estimate in children under five of about 14%.[9] After adjustments, which were informed by (among other things) local diarrhea mortality rates and differences between the interventions used in the trials versus in the charity context, we estimate that two types of chlorination interventions—Dispensers for Safe Water and in-line chlorination (both described below)—reduce all-cause mortality in under-five children by 6 to 11%,[10] depending on the program and location. This increase in mortality reduction, combined with the additional benefits modeled in our current analysis,[11] corresponded with an increase in cost-effectiveness, which made these programs look much more promising.

In a future blog post, we’ll explain more about what led us to complete our own analysis of the mortality data. More on the process we undertook to arrive at our estimate is in our water quality intervention report.

The program

Dispensers for Safe Water installs, maintains, and promotes use of chlorine dispensers at rural or remote water collection sites in Uganda, Malawi, and Kenya. Our 2018 report on Dispensers for Safe Water concluded that, due to our uncertainty over mortality effects at the time and a model that found Dispensers for Safe Water to be about as cost-effective as unconditional cash transfers, it didn’t meet our bar for directing funding.[12]

When we revisited Dispensers for Safe Water after our re-analysis of mortality effects, it began to look much more cost-effective—between four and eight times as cost-effective as unconditional cash transfers, depending on location.[13] That new estimate, combined with our confidence in Evidence Action as an implementer[14] and our understanding from conversations with Evidence Action and others that water treatment is a particularly neglected focus area, led us to recommend a grant to Dispensers for Safe Water of up to $64.7 million[15] over a period of seven years.

Longer-term, we think there’s a true funding gap for this work that’s unlikely to be filled by other funders. We speculatively estimate that Dispensers for Safe Water, or programs targeting similar populations at a comparable level of cost-effectiveness, could cost-effectively use $350 million per year globally.[16] We’ll publish more details in our forthcoming write-up of the grant.

Other water quality interventions

Our current view of water treatment’s effectiveness at reducing mortality and its relative neglectedness has led us to begin exploring water quality interventions besides Dispensers for Safe Water. We’ve reached a fairly advanced stage in our investigation of one such intervention, Evidence Action’s in-line chlorination program, a technology that automatically disinfects the water in communal water tanks with chlorine rather than requiring an individual user to add it.[17] We’re in much earlier stages of researching other programs, such as distributing vouchers for at-home chlorination supplies, distributing water filters, and protecting natural springs from contamination.

In conclusion

We’re excited about the potential of this grant to Dispensers for Safe Water to maintain and expand on a much-needed, life-saving service, and we’re hopeful that water treatment generally will provide even more opportunities for us to do good with our growing resources. Our research is far from done—due to our remaining uncertainties, we want to fund additional studies on the effects of water treatment to help us refine our mortality reduction estimate and pressure-test our current findings. This is an integral part of GiveWell’s commitment to truth-seeking—seeking out new data and being open to revising our views.

For more on our investigation, see our intervention report on water quality. More details on the Dispensers for Safe Water grant will be available in a forthcoming grant page.

  1. Our estimate suggests that chlorination reduces all-cause mortality in children under five by about 14%. After adjustments, including those related to local diarrhea mortality rates and differences between the interventions used in the trials versus in the charity context, we estimate that two types of chlorination interventions we’re currently assessing—Dispensers for Safe Water and in-line chlorination—reduce all-cause mortality in under-five children by 6 to 11%, as described in the "What led us to update" section. ↩︎

  2. "Diarrhoeal disease is a leading cause of child mortality and morbidity in the world, and mostly results from contaminated food and water sources." WHO, "Diarrhoeal disease," 2017.
    "Diarrhoeal disease is the second leading cause of death in children under five years old." WHO, "Diarrhoeal disease," 2017.
    The Institute for Health Metrics and Evaluation's Global Burden of Disease (GBD) tool estimates that in 2019, enteric infections caused 15.3% of all deaths in under-five children in sub-Saharan Africa (see here). ↩︎

  3. Water chlorination has been used in major cities in the United States since the early twentieth century: "Following the demonstration of chlorine's use for disinfection in 1908, most major cities began water chlorination within the next decade." Cutler and Miller 2005, p. 3.
    See, for example, Evidence Action's Dispensers for Safe Water as one charity set up to implement chlorination.
    In our cost-effectiveness analysis we estimate that in-line chlorination in Kenya costs about $1.68 per person served and that Dispensers for Safe Water costs between $1.22 and $1.87 per person served, depending on the country. ↩︎

  4. See the "Mechanism of action" section of our intervention report on water quality. ↩︎

  5. See this 2016 blog post for more on our views of the evidence at the time. ↩︎

  6. This assumes a 1:1 relationship between reductions in diarrhea morbidity and reductions in diarrhea mortality. We describe further and provide sources in our water quality intervention report: "The most recent Cochrane meta-analysis of water quality interventions reports that chlorination interventions reduce the risk of diarrheal illness by 23% in low-income settings. GBD estimates that 14.5% of all-cause mortality in children under age five in countries with low socio-demographic index is caused by enteric infections. Together, these estimates imply that chlorination might reduce all-cause mortality by 3.3%." GiveWell, "Water quality interventions," 2022. ↩︎

  7. Michael Kremer co-founded Deworm the World, which later became part of Evidence Action. Additionally, Stephen Luby, a co-author of this paper, is on the Board of Advisors for Evidence Action (note: this is separate from Evidence Action’s Board of Directors). ↩︎

  8. Kremer and his team's initial estimate was slightly higher, 28 to 30%. These are the estimates we currently cite in our water quality report: "Depending on the method used to pool results, the analysis reports that water quality interventions reduce the odds of all-cause mortality in children under five by 28% (95% confidence interval, 8% to 45%) or 30% (95% credible interval, 8% to 51%)." Since we wrote the intervention report, Kremer et al. (2022, p. 1) have adjusted their estimate to account for the greater uncertainty of implementation in a new context: "Taking into account heterogeneity across studies, the expected reduction in the odds of all-cause child mortality in a new implementation is 25%." In other words, the new estimate is not simply a statistical description of previous RCTs, but a prediction of what one would expect from conducting a new trial. ↩︎

  9. See our calculations here and an explanation of our approach here. ↩︎

  10. See here and here in GiveWell, Water quality CEA (ILC and DSW). ↩︎

  11. Beyond reductions in under-five mortality, the current analysis also incorporates benefits from reduced mortality in people five and over, as well as development benefits (increases in consumption resulting from reduced illness). See our analysis here. The 2017 version of our Dispensers for Safe Water CEA modeled only reductions in under-five mortality. ↩︎

  12. In 2018 we named Dispensers for Safe Water a “standout charity,” a designation it maintained until GiveWell discontinued the use of that designation in 2021. ↩︎

  13. See here in GiveWell, Water quality CEA (ILC and DSW). An updated CEA will be published with the upcoming Dispensers for Safe Water grant page.

    We use GiveDirectly's unconditional cash transfers as a benchmark for comparing the cost-effectiveness of different funding gaps, which we describe in multiples of "cash." Thus, if we estimate that a funding gap is "10x cash," this means we estimate it to be ten times as cost-effective as unconditional cash transfers. As of this writing, we have typically funded opportunities that meet or exceed a relatively high bar: 8x cash. We also consider funding opportunities that are between 5 and 8x cash.

    Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes to other grants we have made or considered making, and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible. ↩︎

  14. Evidence Action is the parent organization of one of our top charities, Deworm the World Initiative. We've also directed funding to its Accelerator program and the since-discontinued No Lean Season. ↩︎

  15. We describe this grant, which will be funded by Open Philanthropy, as "up to $64.7 million" because $15.9 million is exit funding that will be released only upon our decision to withdraw support from the program. More details on this grant will be available in a forthcoming grant page. ↩︎

  16. Based on our rough estimates of long-term room for more funding in countries Dispensers for Safe Water might expand to where we estimate the program would be at least eight times as cost-effective as unconditional cash transfers. See here in our room for more funding analysis. ↩︎

  17. See GiveWell, "Water quality interventions," 2022: "In-line chlorination is a technology for automatically disinfecting water at shared water collection points in low-income settings with unsafe water." See Pickering et al. 2019 for a more detailed description of an in-line chlorination system. ↩︎

Joel AH @ 2022-04-28T08:09 (+4)

Thanks for all your hard work! This is very exciting. Do you feel that the mortality gap relative diarrhea is something in need of explanation? It increase my confidence in the results of the meta-analysis if there was some indication where it was coming from. Or would something like waterborne parasites, malarial mosquito larvae in the water tanks be enough to cover it?

GiveWell @ 2022-05-03T11:25 (+4)

Hi, Joel,

This is a big question, and one we'd very much like to have a better explanation for! It's important to note that our estimates are uncertain, so the gap between the mortality benefit of water quality interventions and mortality attributed to diarrhea is not necessarily as large as it appears. But we do find it plausible that a larger-than-expected mortality reduction would exist, for a few reasons:

  • Global Burden of Disease (GBD) estimates of diarrhea deaths in children under five assume that each death has just one cause. However, deaths are often the result of several contributing causes, and in some situations removing any single one of those causes would mean averting a death. So we think it's likely that just extrapolating diarrhea mortality from diarrhea morbidity underestimates the total mortality benefit of avoiding diarrhea.
  • On a related note, there is observational evidence that repeated bouts of diarrhea may be a significant cause of malnutrition, and evidence that malnutrition increases risk of dying from infectious diseases, such as malaria and measles. Frequent diarrhea generally seems to lead to undernourishment and worse health, leaving people, especially children, vulnerable to other illnesses such as respiratory infections.
  • There is evidence from studies of historical water quality improvement projects that improving a population's drinking water leads to mortality benefits beyond what would be expected from a reduction in waterborne diseases alone, such as reductions in death from respiratory infections; this is known as the Mills-Reincke phenomenon. The effect on child mortality observed in these studies is consistent with the estimate we arrived at in our own meta-analysis. More recent observational evidence points to a link between diarrheal infection and increased risk of respiratory illness.

For more details and citations for the above, see this section of our water quality intervention report (particularly beginning with "We are uncertain about our central estimate of the mortality impact of chlorination…").

In the course of our investigation, we also spoke to several researchers who found it plausible that water quality improvements would have a larger-than-expected mortality benefit. These researchers offered their own insight into why water treatment might also reduce non–diarrheal illness, or why the reduction in diarrhea mortality might be larger than what one would calculate indirectly from diarrhea morbidity. See our notes from these conversations here, here, and here.

In sum, we still have uncertainty about what explains the gap between mortality reduction estimates, but the impact of averting waterborne diseases on death from non-waterborne diseases probably plays a role. We think that further empirical evidence of the size of the mortality effect, or a better understanding of the mechanisms that link water quality to mortality, could help to substantially reduce our uncertainty.

jah @ 2022-05-14T16:11 (+1)

Thanks you very much for taking the time to reply!

DevonFritz @ 2022-04-28T06:26 (+2)

Love it! Keep up the excellent work.

BenStewart @ 2022-04-26T05:34 (+2)

Fantastic news!