GiveWell Misuses Discount Rates

By OscarD🔸 @ 2022-10-27T06:57 (+73)

Summary

• GiveWell currently uses a time discount rate of 4% for all their cost-effectiveness analyses (CEAs).
• I argue that it is a mathematical mistake to pick any single best guess value to use for the CEAs.
  • Instead, GiveWell should use a probability distribution over possible discount rates.
• This is not just an aesthetic judgement for mathematical puritans; it materially changes the CEAs, notably by making all the deworming interventions more attractive relative to other interventions.
  • This is because deworming interventions rely on multi-decadal effects, and so a lower discount rate would make them much more valuable.

Epistemic Status

• On the object level, I cannot think of any reasons to justify GiveWell's current modelling choice over my proposal.
• However, I still doubt my conclusion because on the meta level it seems like an obvious thing that would be surprising if no one at GiveWell had ever thought of doing, which is evidence I am missing something important.

Main

GiveWell’s CEAs are an impressive attempt to model many different factors in assessing the near-term impacts of various interventions.^[1] I will ignore all of this complexity. For my purposes, it is sufficient to note that the CEA for most interventions is well characterised by decomposing impact into several constituents, and multiplying these numbers together. Consider Helen Keller International’s Vitamin A Supplementation program:

$V=M\times R\times \frac{1}{C}$ ^[2] where:

$V$ is cost-effectiveness [deaths/dollar],

$M$ is baseline mortality [deaths/year/child],

$R$ is mortality reduction [%], and

$C$ is treatment cost [dollars/child/year]

Obviously, all of these terms are uncertain. Treatment costs we can estimate quite accurately, but there may be fluctuations in the price of labour or materials needed in the distribution. Mortality data is generally good, but some deaths may not be reported, and mortality rates will change over time. The mortality reduction is based on a solid-seeming meta-analysis of RCTs, but things change over time, and circumstances differ between the trial and intervention locations.

GiveWell’s model makes a subtle mathematical assumption, namely that the expectation of the product of these three random variables is equal to the product of their expectations:

$$E\left[V\right]=E[M\times R\times \frac{1}{C}]=E\left[M\right]\times E\left[R\right]\times E\left[\frac{1}{C}\right]$$

This is not, in general, true.^[3] However, if the three random variables are independent, it is true. I cannot think of any plausible ways in which these three random variables correlate. Surely learning that the price of vitamin A tablets just doubled ($C$) does not affect how effective they are ($R$) or change the baseline of how many kids die ($M$). Thus, while GiveWell’s method is mathematically unsound, it gives the correct answer in this case. It could well be that GiveWell has considered this, and decided not to explain this in their CEAs because it doesn’t change the answer. I think this would be a mistake in communication, but otherwise benign.

The one place where I believe this mathematical mistake translates into an incorrect answer is in the use of discount rates. From GiveWell’s explanatory document:

“The discount rate's primary effect in the cost-effectiveness analyses of our top charities is to represent how much we discount increases in consumption resulting from the long run effects of improved child health for our malaria, deworming and vitamin A charities (which we call "developmental effects"). It also affects the longer-run benefits from cash transfers. We don't discount mortality benefits in our cost-effectiveness analyses.”

This figure shows the cost-effectiveness of all the charities in the CEA spreadsheet, when varying the discount rate.^[4]

Deworming interventions, shown in dashed lines, vary considerably in the discount rate because their path to impact relies on multi-decadal earnings increases for the recipients. The other interventions are less sensitive to the discount rate because more of their impact comes via direct mortality reduction, which is not subject to a discount rate in GiveWell’s analysis. However, all CEAs include an analysis of long-term income changes, so display some sensitivity to the discount rate.

The GiveWell approach of making their best point estimate of a random variable and using it for subsequent calculations breaks down here because the cost-effectiveness is not linearly proportional to the discount rate. The precise function relating discount rate to cost-effectiveness for the deworming interventions is a bit complex.^[5] We can just note that the cost-effectiveness $V$ is some function $f$ of, among other things, the discount rate $d$:

$$V=f\left(d\right)$$

We care about the expected value of the cost-effectiveness:

$$E\left[V\right]=E\left[f\right(d\left)\right]$$

GiveWell makes the further implicit assumption that

$$E\left[f\right(d\left)\right]=f\left(E\right[d\left]\right)$$

But this assumption is false!

Because $f$ is nonlinear, our best guess for the discount rate being wrong in one direction will mean our estimate for cost-effectiveness will be off by more than if we are wrong about discount rates in the other direction. In this case, if the appropriate^[6] discount rate is significantly lower than GiveWell’s current value of 4%, then deworming looks very good indeed, while if we choose a higher discount rate, deworming is only moderately worse.

This means that simply plugging in our best guess for the discount rate into the CEA is a mathematically invalid move that gives the wrong answer.^[7] The correct approach would involve deciding on our probability distribution in credences over different discount rates, and integrating the product of f and the probability distribution. Others have argued GiveWell should do uncertainty analysis to keep better track of the risk their recommendations are badly off. My concern differs from this, as even if we just care about maximising expected value without being risk averse, we should avoid using point estimates of the discount rate.

I make no claim to have a good way of choosing our probability distribution, just that the current implicit one - 100% confidence in 4% - is clearly wrong. For illustrative purposes, here are three slightly less wrong discrete distributions we could have over discount rates: equal credence between 2% and 6%, a binomial distribution, and uniform credence on 0%-8%.

The more credence we place on lower discount rates, the more deworming charities look better compared to the baseline methodology, even in relation to other charities:

Thus, GiveWell improving their modelling of discount rates could increase the cost-effectiveness of deworming charities by upwards of 10%. The takeaway I am hoping for is in pointing towards a better process, rather than defending my new very dodgy numbers.

My accompanying post argues the large chance of TAI within the coming decades should cause GiveWell to use a higher discount rate.

Notes