Why is it so hard to know if you're helping?

By Vasco Grilo🔸 @ 2024-10-16T16:08 (+58)

This is a linkpost to https://www.theintrinsicperspective.com/p/why-is-it-so-hard-to-know-if-youre

This is a crosspost for Why is it so hard to know if you're helping? by Erik Hoel, which was published on 25 January 2024.

[Subtitle:] Even sending mosquito bed nets to Africa carries uncertainty

“We are willing to help, Lord, but what, if anything, is needed? ... Either we don't know what part of ourselves to give or, more often than not, the part we have to give is not wanted.” - A River Runs Through It, Norman MacLean

A long time ago I was sitting on an airplane next to a sociable teenager. We ended up chatting, and she told me about life on her reservation, since she was Native American. As a teenager her thoughts were on her soon-to-be disclosed inheritance, because members of her tribe, due to the collective ownership of casinos, received a lump sum of the profit when they came of age. The lump sum was sometimes enormous, calculated in some obscure way concerning bloodlines, and ranged from $80,000 to $250,000. Right when they turned 18.

“That must be amazing for the community!” I said.

“Sometimes,” she replied, considering. It turned out she knew one guy who had used the money to go to college and then get advanced degrees, and had a highly lucrative and far-away career (concerning what, I forget), and a few others she didn’t know personally had done something similar. Most people, she said, since they’re teenagers, just went on a massive spending spree. But she emphasized, to avoid giving the wrong impression, that the spending spree wasn’t necessarily some selfish one. It was culturally expected that the money get doled out appropriately. People bought their families cars and presents. Shining white BMWs rolled up over dusty roads. You bought your friends drinks, and then you bought the entire bar drinks. Of course you spent a lot on yourself, but it wasn’t just a materialistic orgy of consumption and greed, it sounded more like, for several months, people hosted a massive party.

I tentatively tried to suggest, given she was about to come into this inheritance, that the money might be spent in objectively better ways. She laughed. She’d already gotten this lecture. Everyone she knew had gotten this lecture. She knew exactly my type, and kindly allowed me to pontificate uselessly about how much such an inheritance would be worth in a longterm index fund. And then she said something like how she was going to buy her parents a new car. She didn’t need my advice, and what I was suggesting didn’t make sense for her life, and she loved her parents.

I had thought I was helping with my hastily-arrived-at advice. But it seems just as likely to me now that I was an outsider moralizing, blindly tackling the rules of a world I had no familiarity with, one that perhaps cannot be obviously optimized. After all, sometimes it is rational to go along with the traditions of your community, and everyone you know and love, instead of trying to build your own personal life raft. I just don’t know the answer. What should she have done? I’m still thrown off balance by the tale she told, and unsure, in the end, what the best thing to do with the money would have been.

A long while later, I would think about the teenager’s story when I read about the uncertainty around this question: Do direct cash transfers work to alleviate poverty? While there are some positive findings, a series of studies in 2022, like an analysis of GiveDirectly’s $1,000 donations in 2020, found surprisingly few.

Maybe it’s just that the money involved in such studies ($500, $1,000, $2,000, etc.) is not life changing? Perhaps. But then it’s worth considering that 70% of all lottery winners go broke within three years, leading Time Magazine to once pessimistically title a piece “Here’s How Winning the Lottery Makes You Miserable.” And I actually knew someone who won the lottery, shortly out of high school. He doesn’t have any of the money left. But you know what he did do? He bought his mother and brother cars.

I thought about the teenager’s tale yet again when reading the recent scuffle online about whether or not the insecticide-treated (and therefore mosquito-repelling) bed nets being distributed by charities to Africa for years have actually been repurposed for fishing, which could—theoretically, it should be stressed—lead to fishery problems. Some, like venture capitalist and “techno-optimist” Marc Andreessen, seized on the 2021 study to argue that bed nets are somehow a “triple threat,” while others who donate to bed-net-focused charities like the effective altruist organization GiveWell, such as Matthew Yglesias, were quick to reply that saving lives is good actually. Versus, you know, fish.

The scuffle didn’t have much new to say on the topic. The New York Times already wrote “Meant to Keep Malaria Out, Mosquito Nets Are Used to Haul Fish In” back in 2015. On the other side, an academic review in 2011 called the issue of bed net misusage “not evidence-based.” But the concern has never fully gone away; The Guardian returned to the issue after a 2018 study on inappropriate bed net use for fishing.

Personally, I think the fishing thing is (excuse the pun) a red herring. But the perennial debates around misuse (or use, from the user’s perspective?) of nets is an illustration of how hard it is to find an effective scientifically-proven intervention that is, at any given moment, for sure saving lives in the manner intended. In other words, the difficulty of actually helping. For the overall take-away story of bed nets is more mixed than it’s commonly portrayed as. From 2004 to 2019, there were two billion bed nets distributed to Africa. Yet, according to the World Health Organization, the prevalence of malaria deaths in Africa stopped declining around 2015, having “stalled.” Another Times article from last year read “The mosquitoes are winning.”

This stalling out of progress might be surprising news; insecticide-treated bed nets have been a staple of organizations like the Bill and Melinda Gates Foundation for decades. And there are many cogent arguments that money toward bed nets is the most effective existent form of charity. How can both the stalling out, but also the claimed maximal cost-effectiveness, be true?

The easiest-to-analyze argument for bed net effectiveness comes from GiveWell, in large part via donations to the Against Malaria Foundation, which is calculated to have saved almost 200,000 lives. Yet GiveWell’s fundraising only took off after 2015, which is also after malaria death decline in Africa had already stalled out. The overwhelming majority of the money that has flowed to bed net purchasing via GiveWell has occurred after malaria deaths plateaued (in 2014 GiveWell was 17 times smaller than it is now).

Of course, despite how its giving hasn’t obviously reduced the absolute numbers of malaria deaths, it’s possible that GiveWell’s funding has still causally saved as many lives by preventing a reversal of the trend, fighting hard to keep malaria deaths merely steady since 2015… but this hypothesis is less than ideal for clarity.

To make an argument for causal responsibility for lives being saved we’d have to figure out total bed net distribution and usage, which indeed experienced the greatest leaps in coverage in the lead-up to 2015, at the time tracking well how deaths were decreasing; but since then bed net usage and distribution have trended upward nation-wide since 2015 without much shift in total malaria deaths. Additionally, we’d also need to factor in everything else, not just bed nets, that’s going on when it comes to combating malaria (e.g., maybe bed nets are increasingly ineffective but other interventions, like the rollout of malaria vaccines, are filling the gap, or just improvements in basic medical tech and access, etc). And, oh god, then you have to factor in that the price for life-saving interventions has been decreasing! So more lives should be being saved for the same amount of funding, but somehow deaths still stalled. As Science reports:

… some of the most commonly procured malaria commodities now cost less than half of what they did in 2005, even as the donor funding available to procure them has more than tripled from <$500 million to >$1.6 billion.

This is the classic problem of causal attribution in the sciences—causation is damn hard to show, and you need to constantly adjust away all sorts of confounds, and charity in the real world has a million of them. So then how can we be highly confident what a contribution to GiveWell brings? Luckily, unlike so many charities, GiveWell offers a public model which is, well, basically a big Excel spreadsheet, but it’s quite a good one, with all sorts of variables you can conceivably tinker with.

Despite the complexity, there is arguably a core of GiveWell’s model: the finding that providing bet nets in randomized controlled trials (RCTs)—experiments where some individuals or villages received bed nets and others didn’t—reduced the prevalence of malaria and specifically overall child mortality, indicating causality. Studies of excess mortality peg the protective effect around saving something like ~40 per 1,000 children per year.

Why focus on child mortality? Because, despite a popular conception of malaria as a death sentence, malaria is usually not a leading factor for adult mortality in regions where it’s prevalent, since people acquire immunity if they survive a critical vulnerable period. Studies examining decreases in mortality from bed nets therefore focus on young children (and usually right after the rainy season). Malaria’s mortality rate—high enough to be a plague on humanity, but low enough (dozens among the thousands) to be difficult to track—is one of the things that necessitates using RCTs to make the case for mortality reduction.

The last time there was a controversy over bed net misuse for fishing in 2015, GiveWell’s official response cited precisely the older RCTs in their defense. Overall, the RCTs are the gold standard and a meta-analysis of them conducted in 2018 (Pryce et al. 2018) is cited by GiveWell for key variables in their model.

Yet this central meta-analysis of RCTs consists of only five studies that examine child mortality. What’s more, the meta-analysis (where the data is, essentially, pooled) is itself dominated by just a single study, one conducted in 2003. As GiveWell writes:

Phillips-Howard et al. 2003 is the bednet mortality trial that makes up the majority of the child mortality effect estimate in the Cochrane meta-analysis.

In theory that could be fine. But if you read the five RCTs on child mortality themselves, a more complicated picture emerges. One of the studies showed basically no effect at all. And the other studies which do show a positive life-saving effect also often observe that bed nets quickly stall out in their effectiveness after a short time, like less than a year, and then have either smaller, or in some cases, zero effect for reducing child mortality after that. Consider the earliest of the five studies: Do insecticide-treated curtains reduce all-cause child mortality in Burkina Faso?

In the first year, post-intervention mortality was substantially lower in the clusters receiving curtains compared with the control clusters (rate ratio = 0.74; ~95%) but in the second year, there was no difference between mortality in the two groups (rate ratio = 0.99)

This is despite how the researchers would re-treat with insecticide every six months, refreshing the effect (even 9 months post re-treatment, the nets killed 97% of mosquitoes). While villager usage did dip from 79% the first year to 57% the second, the total vanishing of the positive effect remained mysterious. In Phillips-Howard et al. 2003, the large study that forms the bulk of the data for mortality reduction, this negative finding in the literature is acknowledged, because they see the same thing:

Several of the previous studies on insecticide-treated bed nets and curtains have shown a higher efficacy in the first compared with the second study year. In Burkina Faso, treated curtains were highly effective in the first year, but no effect was observed in the second study year. With the exception of the Ghana bed net trial, other studies show similar but less marked trends.

And here’s the actual data from the central Phillips-Howard et al. 2003 so you can judge for yourself:

There’s a clear result after the first rainy season, that initial spike of deaths in the control villages, but the difference between who receives bed nets and who doesn’t quickly attenuates. Critically for the fate of future assumptions by charities, the researchers argue this attenuation isn’t real. They do that, in turn, by controlling for if insecticide-treatment was delayed, and blaming that for the vanishing effect.

We also observed a significant reduction in ITN [insecticide-treated nets] efficacy in the second year… further stratified analysis that took into account whether bed nets had been re-treated according to study protocol or not (every six months), suggested that almost all of the reduced efficacy in the second study year could be explained by late re-treatment of nets.

Wait—but why, or how, did researchers even have samples to create this comparison condition where there was a late re-treatment of nets, and so use that to explain away the attenuation? Because:

Both insecticide and nets were imported duty-free via the U.S. Embassy in Nairobi. The terrorist bombing of the Embassy in August 1998 resulted in delays in net re-treatment in some villages in 1998 and 1999.

Essentially, the researchers created a new model that compared the villages that were delayed in insecticide re-treatment due to the bombing to those that weren’t, and found that this correlated with the attenuation in mortality decline, and so the study reported an overall strong effect that didn’t decay to nothing, and that’s why, in turn, the 2018 meta-analysis cited by GiveWell is so sure that bed nets are effective beyond a suspiciously short burst. Yes, the world is actually that historically contingent.

The problem is that the authors’ analysis doesn’t prove that delayed re-treatment was the cause of the drop-off in their experiment. It might be! But there could also be alternative explanations, and the scope of these alternative hypotheses depend on many details not included in the 22-year-old paper. Did the researchers lose contact with the villages? Did they continue to check and promote the usage of bed nets, leaving everything the same except re-treatments of the nets? Keep in mind, the bed nets in the earlier RCTs were often accompanied by a bevy of outreach, which in the 2003 study

included public meetings (baraza), Luo and English language leaflets, field staff workshops, participatory educational theatre, school competitions, message calendars, and songs produced by traditional birth attendants.

In an ideal world, once limited re-treatment abilities were forced by external events, villages should have been randomly assigned to be either delayed or non-delayed; nothing like that process is mentioned, and that makes the design susceptible to common causes and so on. In other words, the mini-study within the gold-standard RCT on child mortality reduction from bed nets, the one that contributes the most to the key meta-analysis, might not have really been a RCT.

I stress again—their results might still be true! It could have been the re-treatment delay, and not some other factor, that mattered. This would be good news, because the newer nets don’t have to be re-treated, they just slowly release insecticide, and so then we shouldn’t worry about the rapid attenuation in the earlier RCTs.

But a few things give me pause. Like how the insecticides used should theoretically last a year, not six months. Or how delayed re-treatment doesn’t explain the attenuation seen in other studies, since in those re-treatments weren’t delayed. Or how the overall data for malaria deaths in Africa also shows attenuation and so broadly fits the same pattern. Or how, even if you skirt around the RCTs on child mortality and calculate expected lives saved solely off of malaria prevalence, you’ll run into the same issue; concerns go back to 1996 about the benefit from preventing malaria via bed nets at one stage of life being washed out by preventing immunity at other stages, and even modern studies continue to stumble across attenuation when tracking prevalence alone, like a study published in 2022 identifying that:

a fast and steep decline of malaria incidence was noted during the first year after mass LLIN distribution (p<0.0001). In years 2 and 3 after distribution, malaria cases started to rise again to levels higher than before the control intervention.

What’s critical to note is that the potential ineffectiveness of bed nets beyond a brief initial effect may go against the consensus of charities, but the actual academic literature is more conflicted. One example is the idea that the bed nets themselves are creating a “Red Queen” dynamic where they force changes in the mosquito population that obviates the initial intervention. The literature on this is also mixed, with some authors arguing that it matters (along with other factors), but other authors argue it’s overrated as a cause of concern.

While GiveWell attempts to incorporate insecticide-resistance, estimating that resistance only decreases effectiveness of bed nets by 1/4, by their own admittance:

There are no high-quality studies that estimate the impact of insecticide resistance on ITNs’ effectiveness in the field…

and they also admit their conclusion that bed nets are still highly effective in the face of resistance-race dynamics

seems to be based on low-quality information.

In a 2022 paper, “Stalled malaria control – root causes and possible remedies,” the authors write:

In Kenya, there was no significant decline in malaria incidence during the last decade, despite 50-80% of the human population owning and using insecticide-treated bed-nets (ITN). Similar situations exist in Tanzania and Uganda… we suggest that this is largely due to a change in the species composition and behaviour of the vector populations, consequent on the widespread use of ITN.

Ultimately, no one knows why the effectiveness decline of bed nets keeps popping up. Maybe the insecticide stops working, or maybe there’s rapid Red Queen dynamics, or maybe community involvement by the researchers plays a big role in the effectiveness of the studies, and a substantial portion of recipients stop bothering unless the mosquitoes are really bad that night, and they just leave the nets hanging up but ill-fitted and ill-used (it’s worth mentioning the Against Malaria Foundation only has thirteen employees, likely making RTC-level outreach impossible nation-wide).

All this uncertainty means any claim that goes: “given X dollars donated you are saving Y lives in expectation” possesses intrinsically large error bars; this is true no matter how complicated the models behind it are, since the data that underlies the models, the primary literature itself, is often in dispute, or taken too uncritically, and there’s enough it can be cherry-picked either way to highlight either the successes or the failures. The real answer is likely in the middle—but even the clearest results on lives saved, the older RCTs, come bearing unexamined assumptions and quirks that make the effects more questionable than anyone would like.

It doesn’t look like we’ll be getting many new RCTs. In a 2023 modeling study that showed a positive effect for bed nets, the authors admit that:

It would be unethical now to test these distinctions empirically because it would require leaving a cohort of people without nets and therefore exposed to potential transmission risks.

After finishing this essay, I donated to GiveWell. Specifically their bed net program (they fund more than that). Why? Perhaps because, even if the error bars hidden behind the sale’s pitch are massive, lives are probably still saved via bed nets alone. Or perhaps selfishly, as a shield so I can’t be accused of being overly cynical. Or perhaps because I’m trying to be comfortable not having stamped approval that my donation is “saving X number of lives.” My uncertainty is vast. I’d like to think a donation does something, but, given the mysterious stalling out of effectiveness both overall on malaria progress, along with mysteries seen in the original gold-standard RCTs, and given the potential Red Queen dynamics of immunity, there’s a non-negligible chance that I could have set the money on fire and gotten the same result at this point. People might not want to hear that life is so uncertain. What I do think is that there are moments of clarity, like the big international push for bed nets and other aid in the 1990s and early 2000s that coincided with malaria deaths dropping rapidly across Africa, and in those moments the best way to help people becomes easy to identify, like a beam of sun breaking through the clouds. Just like that beautiful initial spike in the graph of the key RCT. But then the clouds close, and the clarity ends. The clarity for the effectiveness of bed nets, at least compared to other causes (like new vaccines), looks to me like it closed sometime around 2015, and it’s been clouds ever since.

Should a lack of sunlight matter? Maybe in an ideal world, we humans wouldn’t mind doing things that are pointless, as long as they are good.

Håkon Harnes 🔸 @ 2024-10-17T10:17 (+21)

Funding to malaria prevention has stalled since 2010.^[1] It's misleading to point to GiveWell funding alone. I don't think it's particularly surprising that given ~constant funding the progress has slowed. As noted the nets last only a couple of years, and presumably you get diminishing marginal returns when scaling up, as always. I'm not sure exactly how they are counting here, as "other funders" is suspiciously small, but the general point still stands, that the largest donors have stalled over the last decade. WHO estimates for the counterfactual^[2] show that just keeping the deaths constant is an achievement in itself (although the counterfactual is of course more uncertain). I think this comes down to population growth and a little bit of climate change, but I haven't looked into it deeply.

I think there is an argument to be made here, that as the world shifted priorities with the SDGs and funding for the "old" efforts stalled, we unfortunately got a unusually good opportunity in malaria prevention in terms of marginal impact. While just keeping up the pressure doesn't yield equally spectacular sounding results as the initial ramp up, it's still likely saving thousands of lives.

1. ^{^}

   World Health Organization. (2023). Funding for malaria control and elimination, 2000–2022, by channel (constant 2022 US$) [Figure 6.4]. In World Malaria Report 2023 (p. 49). WHO.

2. ^{^}

   World Health Organization. (2023). WHO methods for estimating cases and deaths averted. In World Malaria Report 2023 (p. 123), Annex 1. WHO.

Vasco Grilo🔸 @ 2024-10-17T11:37 (+3)

Thanks for the comment, Håkon! I strongly upvoted it.

lexande @ 2024-10-16T23:44 (+5)

Note that those graphs of malaria cases and malaria deaths by year effectively have pretty wide error bars, with diferent sources disagreeing by a lot:

(source)

Presumably measurement methodology has improved some since 2010 but the above still suggests that the underlying reality is difficult enough to measure that one should not be too confident in a "malaria deaths have flatlined since 2015" narrative. But of course this supports your overall point regarding how much uncertainty there is about everything in this sort of context.

Vasco Grilo🔸 @ 2024-10-17T09:46 (+3)

Great context, lexande! Relatedly, below is the disease burden per capita from malaria for a age-standardised population, and respective uncertainty^[1]. There seems to be a stabilisation from 2017 to 2019, but this period is only 2 years. I assume the higher values in 2020 and 2021 were caused by COVID-19.

1. ^{^}

   I guess the shaded area respects the 90 % or 95 % confidence interval. I did not quickly find information about this.

Håkon Harnes 🔸 @ 2024-10-17T10:25 (+3)

WHO claims the bump is due to covid-19 disruptions in world malaria report 2023.

From page 18:
"Between 2000 and 2019, case incidence in the WHO African Region decreased from 370 to 226 per 1000 population at risk, but increased to 232 per 1000 population at risk in 2020, mainly because of disruptions to services during the COVID-19 pandemic. In 2022, case incidence declined to 223 per 1000 population at risk."

This is an argument for the effectiveness of existing interventions.