In Cost-Effectiveness Analyses, when should we take the Expected Value?

By EdoArad @ 2024-03-13T14:49 (+15)

Say we have a choice between two types of bednets, each offers the same malaria protection but their production and distribution process is different.

1. One producer, NetCo^[1], is making bednets using a highly variable process. Each bednet costs differently, but NetCo guarantees an average cost of $1 per bednet for the foreseeable future, which we easily verify by looking at their past data. 
2. Another producer, SleepWell^[2], has a solid factory with highly regular production. All bednets will therefore cost the same. However, they have yet to finalize the whole process and cannot yet commit to the cost. They also, annoyingly, estimate that their bednet cost will be centered at $1.

Say that we need to choose only one of the two producers, and we can't wait for any more information. Should we go with NetCo or with SleepWell? Maybe it doesn't matter?

You may reason:

Well, let's first imagine that I'm buying only one bednet. In this case, buying from NetCo will cost me some amount of money $$c$, which is $1 on average.

Hmm, but isn't it the same for SleepWell? There we don't know the cost, but the average estimate is also $1. I mean, while Epistemic and Statistical Uncertainty differ, can't I just also average out over my own world-views? 

This thought process is basically correct^[3]. It's also correct if we want to buy a thousand bednets. However, we generally ask "where should I spend my money for the highest returns?" rather than "what would be the cheapest I would need to pay for some amount of bednets?". So, say AMF expects to fundraise $1 million, which producer should they choose?

You, again, reason:

I want AMF to buy as many bednets as possible with the same amount of money. If I choose NetCo, I know that sometimes I'll buy cheap nets and sometimes expensive nets. All in all, the more money I put into it the more nets I'd get and the closer the cost distribution of these nets to the actual cost probability. In this case, we can average everything out (say, by grouping together cheap+expensive nets for the average price of $2 per two nets), and we expect to buy about a million bednets. 

Shouldn't it again be basically the same for SleepWell? No? Okay, I'll do this as accurately as I can.

Going with SleepWell, we would spend $1 million on either cheap or expensive nets, or anything in between. That's complicated to calculate so maybe I should first try a silly edge-case. Say the cost is either $0.01 or $1.99 with equal probabilities. The average cost is indeed $1. We have two cases:

1. If the cost turns out to be $1.99, then with $1 million we'd get roughly 500 thousand bednets. 
2. If the cost is $0.01, we can buy 100 million bednets.

Oh shit, that's a lot! 

Wait, even if the cost could have been $199 or $199,999 instead of $1.99 that would still mean that in expectation we'd get more than 50 million bednets, much higher than NetCo. What happened here? Is this because I selected this silly edge case?

The point is that when we ask "how much good can I do with my money" we intuitively guess that the answer is 

Because $\text{Amount of Good}=\frac{\text{Spent Money}}{\text{Cost per Unit of Good}}.$

That's true if all these variables are constant. They may still be random variables! ... TODO: explain the proper formula in the statistical uncertainty case. 

1. ^{^}

   Mnemonic: net cost

2. ^{^}

   Mnemonic: In this case, buyers can not worry about prices changing and they can sleep well

3. ^{^}

   Except, maybe, that it doesn't account for different types of possible risk aversion. It may be the case that EVM is the best way to deal with statistical uncertainty but that it's rational to be averse to epistemic uncertainty. [Actually, I have a hunch that epistemic uncertainty should generally be rewarded rather than penalized due to option-value / value-of-information, but that should ideally be already modeled in].