Research I'd like to see

By Alex Cohen @ 2024-12-17T14:08 (+51)

I’m Alex Cohen, Principal Researcher at GiveWell. From time to time, we get questions from people outside of GiveWell about ways they could help our research. I decided to write down a few concrete research questions that I think would be informative to us and seem doable primarily with desk-based research, rather than original data collection or field visits (though those would probably help).

These are listed below. They’re not in any particular order and are roughly similar levels of priority. While our team may work on some of these questions over the next year, even in those cases, extra input is still likely to be useful.

If you decide to take these on, I’d be interested in what you find, so please post your answers to the forum. Also, if you find these types of questions interesting, please consider applying for our Senior Researcher role.

“Red teaming” some newer grantmaking areas

We recently published our findings from “red teaming” our top charities and previously did a Change Our Mind contest to find errors in our top charities and other programs where we’ve directed a lot of funding.

However, we’ve solicited less external scrutiny beyond these large grantmaking areas. Can you find errors in our published material on newer programs?

A few programs where we’ve made grants already and may consider providing more funding in the future are below:

• Chlorination^[1]

• Malnutrition^[2]

• Syphilis screen and treat^[3]

• Kangaroo Mother Care (KMC)^[4]

• Tuberculosis contact management^[5]

• Participatory Learning and Action (PLA)^[6]

You can see the type of questions that we ask during red teaming in this rough document.

Moral weights questions

We’ve spent a decent amount of time trying to understand the trade-offs between increases in consumption, averting deaths, and reducing morbidity via our moral weights, but there’s some desk-based work we haven’t done.

Two examples:

• How do our estimates compare to recent estimates from value of a statistical life (VSL) studies in low- and middle-income countries specifically? If they’re different, how plausible do we think these differences are? We haven’t done a comprehensive review of this literature recently to understand how they compare to our moral weights, and it’s possible more work could lead to changes on how we trade off averting deaths for increased income, for example.^[7]

• Is there any evidence we could use to check our assumptions on averting morbidity vs. consumption? For example, we assume that averting a year of clubfoot is as valuable as increasing consumption by ~50% and that averting a year of severe anemia is as valuable as increasing income by ~60%.^[8] Is there any evidence that exists that would help us discipline these estimates (either for these morbidities or others like fistula or cataracts)? How much are individuals in LMICs willing to trade off to avert morbidity?

Burden of disease questions

Our cost-effectiveness estimates rely heavily on data on burden of disease from the Institute for Health Metrics and Evaluation (IHME). However, we have some outstanding questions about these data.

We’ve noticed some cases where these data are out of sync with other sources. Our current approach is to consider putting weight on both. For example, in our report on red teaming our top charities, we note a couple examples:

• All-cause mortality across Nigerian states. Our estimates of under-5 all-cause mortality rates in Nigerian states, which are a key driver of our decisions about where to fund New Incentives' conditional cash transfer program, rely heavily on data from the Institute for Health Metrics and Evaluation (IHME). However, we found that IHME's estimates are negatively correlated with estimates from the United Nations Interagency Group for Child-Mortality Estimation (UN IGME) across the states where New Incentives operates. If UN IGME estimates are more accurate, relying solely on IHME could lead us to fund New Incentives’ program in less cost-effective states.
  Additionally, the UN IGME’s estimates are, on average, 35-40% higher than IHME's estimates in states where New Incentives operated as of January 2024. If we come to believe that UN IGME’s results are more reliable, or it is best to take an average of the two, then relying on IHME alone is causing us to underestimate overall disease burden and, consequently, underestimate the potential impact of New Incentives' program.
• Malaria mortality in Chad. Our estimates of the malaria burden in Chad are significantly lower than in other countries where we fund malaria interventions, but we have not thoroughly investigated the reasons for this discrepancy. A comparison of under-5 malaria mortality estimates from the UN IGME and IHME across several African countries reveals substantial variation. For example, the UN IGME estimate for under-5 malaria mortality in Chad is ~2.5 times higher than IHME's estimate, while in Guinea, the estimates are virtually the same, and in Uganda, the UN IGME estimate is only ~45% as high as IHME's. These inconsistencies raise questions about the reliability of the data we use to estimate the malaria burden and, consequently, the cost-effectiveness of malaria interventions in different countries.

We include some additional examples in our report on the optimizer’s curse and uncertainty in our grantmaking.

The broad question here is: How do IHME’s estimates of mortality and burden of disease compare to other sources?  Which source(s) should we put more stock in?

• For all-cause mortality, how does IHME compare to the UN IGME and UN World Population Prospects?
• For cause-specific child mortality, how does IHME compare to estimates from the CA CODE project (UN IGME)?
• For cause-specific adult mortality, how does IHME compare to estimates from the WHO’s Global Health Observatory?
• For maternal mortality, how does IHME compare to the Maternal Mortality Estimation Inter-agency Group (MMEIG)

Some more specific questions:

• The example above from Chad suggests we may have been undervaluing malaria programs in Chad. (We’ve since begun accounting for this in our grantmaking decisions.) Can you find other examples of our work where incorporating estimates from non-IHME sources might change our decisions?

• IHME says the under 5 mortality rate in Uganda in 2021 was 6.5%, slightly higher than neighboring DRC (around 5.8%) and nearly twice as high as Kenya (3.7%).^[9] IGME says it was 4.2% in both Uganda and Kenya, and nearly twice as high (7.8%) in DRC.^[10] What estimate should we use?

• Across 5 countries in Africa where we have supported or considered supporting campaigns to distribute insecticide-treated nets (DRC, Uganda, Chad, South Sudan, and Zambia), GBD’s estimates of the share of 1-59-month-old deaths due to malaria vary from 8% to 44%, while IGME’s estimates for all 5 countries are between 15% and 30%.^[11] How much variation should we expect across countries? This is important for us. If the share of deaths due to malaria is similar across countries, then we mostly care about prioritizing areas with high all-cause mortality. If it’s more spread out, then we’d want to prioritize finding the most malarious regions.

• IHME’s 2021 maternal mortality rate estimate in Nigeria is 299 per 100,000 live births. The WHO/UNICEF ("MMEIG") estimate is 1,047 per 100,000.^[12] That is a huge difference and could materially affect whether programs focused on reducing maternal mortality would or wouldn't look cost-effective to us. Which source should we believe? What should our best guess be?

• How many people over the age of 5 are dying of malaria, and how much does this vary across areas? Across states in Nigeria, the IHME estimates anywhere from 0.3 to 2.5 over-5 deaths per under-5 death.^[13] Should we believe this variation reflects true differences across states?

Indirect deaths

In our recent report on red teaming our top charities, we write:

“Indirect deaths” are deaths that wouldn’t have occurred without the incidence of a disease, but aren’t directly attributable to that disease in disease burden data. For example, someone who had contracted malaria might die from an unrelated disease because malaria weakened their immune system. We account for the indirect deaths averted by a program by estimating how many indirect deaths are averted for every direct death averted. Our assumptions about indirect deaths vary significantly across programs, ranging from ~5 indirect deaths averted for each direct death averted for vitamin A supplementation (VAS), 0.75 for malaria and vaccines, and ~2 for water chlorination. We have not thoroughly assessed whether these magnitudes, or their relative sizes across top charities, are plausible.^[14]

Is there research available that could help us pick the right values for these indirect deaths for different interventions? How does this research accord with our current work? Are there heuristics we should use for estimating indirect effects across different interventions or countries?

Some more specific questions:

• We based this comparison partially on a comparison of all-cause mortality effects to cause-specific effects in randomized trials, but we haven’t done anything systematic.^[15] For example, looking at the biggest meta-analyses, what's the typical relationship between cause-specific effects and all-cause effects? What should our best guess be based on those studies? How should we think about internal validity (e.g., publication bias) or external validity (e.g., whether indirect deaths should be lower today as new vaccines or other programs have come online)?

• Is there anything we can learn from the literature on non-specific effects on vaccines specifically? Our impression is that there's a large literature here, but we’re not sure how reliable it is and how much guidance it could give us on this question.

1. ^{^}

    See our intervention page on water quality interventions, that focuses on chlorination programs, and overall water quality CEA here. We’ve made grants to several types of chlorination interventions including Evidence Action’s Dispensers for Safe Water (CEA) and in-line chlorination (CEA).

2. ^{^}

    See our intervention report on community-based management of acute malnutrition (CMAM) and CEA. We’ve made recent grants in malnutrition here, here, and here.

3. ^{^}

    See our interim intervention report on Syphilis Screening and Treatment During Pregnancy (CEA). We’ve also made several grants to Evidence Action’s syphilis program including this most recent grant, as of the time of this post.

4. ^{^}

    See our recent intervention report, preliminary CEA, and grant page.

5. ^{^}

    See our intervention report on Community-Based Household Contact Management (CEA). We have also made a recent grand to Clinton Health Access Initiative (CHAI)’s  Community-Based Tuberculosis Household Contact Management

6. ^{^}

    See our recent intervention report, preliminary CEA, and previous grant pages here and here.

7. ^{^}

    In our current page, we say:

   A further challenge is that there is little revealed preference or stated preference research conducted in LMICs; most VSL and similar analyses estimate how much people value life in LMICs by extrapolating from high-income country research.11 A key issue with extrapolation is that one needs to make an assumption about how much the relative value of income versus health changes when a population is much poorer (often referred to as "the elasticity of demand for health"). Perhaps someone who barely has enough money to survive would greatly prefer any increase in income more than an additional year of life. Different assumptions about how to extrapolate can lead to estimates of the value of a DALY that vary by at least an order of magnitude.12

   Though the literature on VSL in LMIC contexts is limited, we are aware of a few potentially relevant empirical papers on the topic, which are briefly summarized in León and Miguel 2016, itself an estimate of VSL in an LMIC context (see following footnote).13 These papers generally appear to find substantially lower values of health relative to income than are estimated in high-income countries.14 We have not yet carefully vetted these papers and expect to review them more closely in the future, but our impression is that estimates of the value of life from these papers have not yet been used by major decision makers and are based on different methodologies than typical VSL estimates, so they should not yet be interpreted as "standard" assumptions.15

   Because of limitations in the existing literature, we do not see current "best guess" estimates of the relative value of income versus health in LMICs as robust.

   Our current moral weights from here:

   • Value of doubling consumption for one person for one year: 1
   • Value of preventing one under-5 death from malaria: 116.9
   • Value of preventing one 5-and-over death from malaria: 83.1
   • Value of preventing one under-5 death from vitamin A deficiency: 118.4
   • Value of averting one stillbirth (1 month before birth): 33.4
   • Value of averting one neonatal death from syphilis: 84
   • Value of averting one year of life lived with disease/disability (YLD): 2.3

    

8. ^{^}

    See calculations here.

9. ^{^}

    Source here.

10. ^{^}

     Source here.

11. ^{^}

     These figures are based on this lightly vetted spreadsheet comparing IHME and IGME malaria mortality estimates and how incorporating IGME figures would affect our malaria mortality estimates.

12. ^{^}

     Source here. Filters need to be set to [World - Nigeria - 2020 - Maternal deaths]

13. ^{^}

     Source: Institute for Health Metrics and Evaluation (IHME). Used with permission. All rights reserved

14. ^{^}

     For more details, see the “Indirect malaria mortality” section of our report on ITNs, the

    “Non-malaria deaths indirectly averted” section of our report on SMC, and the “Adjustment for all-cause mortality effect” section of our report on New Incentives.

15. ^{^}

     From our ITNs intervention report:

    Our estimate of 0.75 is based on triangulating three different sources of information:

    Our analysis of the relationship between all-cause mortality and malaria incidence in Pryce et al. implies a value of up to 1.5 indirect deaths for every direct malaria death.221 Our best guess is that this is an overestimate, because:

    The studies underlying the Pryce et al. meta-analysis took place in the 1980s, 1990s, and 2000s.222 It is plausible that the ratio of direct to indirect deaths has fallen since that time in malaria-endemic countries as overall health has improved and under-five mortality has decreased.223

    We use national-level estimates from the Global Burden of Disease project from the countries where the studies in Pryce et al. were conducted as inputs to our analysis (see here). Intuitively, we might expect the Pryce et al. RCTs to be conducted in areas where malaria mortality was higher than the national average. If that assumption is correct, our analysis would be likely to overestimate the share of indirect deaths and underestimate the share of direct malaria deaths.

    We have also spoken with malaria experts who told us that it is widely accepted there are roughly 0.5 to 1 indirect malaria deaths for every direct malaria death.224

    Our analysis of water chlorination programs, another intervention that reduces child mortality by averting infectious diseases, suggests a ratio of 2.7 deaths indirectly averted for every death directly averted from enteric infection.225 This leads us to believe that a high ratio of indirect to direct deaths is plausible.

    Our analysis of the relationship between all-cause mortality and malaria incidence in Pryce et al. implies a value of up to 1.5 indirect deaths for every direct malaria death.221 Our best guess is that this is an overestimate, because:

    The studies underlying the Pryce et al. meta-analysis took place in the 1980s, 1990s, and 2000s.222 It is plausible that the ratio of direct to indirect deaths has fallen since that time in malaria-endemic countries as overall health has improved and under-five mortality has decreased.223

    We use national-level estimates from the Global Burden of Disease project from the countries where the studies in Pryce et al. were conducted as inputs to our analysis (see here). Intuitively, we might expect the Pryce et al. RCTs to be conducted in areas where malaria mortality was higher than the national average. If that assumption is correct, our analysis would be likely to overestimate the share of indirect deaths and underestimate the share of direct malaria deaths.

    We have also spoken with malaria experts who told us that it is widely accepted there are roughly 0.5 to 1 indirect malaria deaths for every direct malaria death.224

    Our analysis of water chlorination programs, another intervention that reduces child mortality by averting infectious diseases, suggests a ratio of 2.7 deaths indirectly averted for every death directly averted from enteric infection.225 This leads us to believe that a high ratio of indirect to direct deaths is plausible.

    Our analysis of the relationship between all-cause mortality and malaria incidence in Pryce et al. implies a value of up to 1.5 indirect deaths for every direct malaria death.221 Our best guess is that this is an overestimate, because:

    The studies underlying the Pryce et al. meta-analysis took place in the 1980s, 1990s, and 2000s.222 It is plausible that the ratio of direct to indirect deaths has fallen since that time in malaria-endemic countries as overall health has improved and under-five mortality has decreased.223

    We use national-level estimates from the Global Burden of Disease project from the countries where the studies in Pryce et al. were conducted as inputs to our analysis (see here). Intuitively, we might expect the Pryce et al. RCTs to be conducted in areas where malaria mortality was higher than the national average. If that assumption is correct, our analysis would be likely to overestimate the share of indirect deaths and underestimate the share of direct malaria deaths.

    We have also spoken with malaria experts who told us that it is widely accepted there are roughly 0.5 to 1 indirect malaria deaths for every direct malaria death.224

    Our analysis of water chlorination programs, another intervention that reduces child mortality by averting infectious diseases, suggests a ratio of 2.7 deaths indirectly averted for every death directly averted from enteric infection.225 This leads us to believe that a high ratio of indirect to direct deaths is plausible.