What is the state of the art in EA cost-effectiveness modelling?

By Froolow @ 2022-06-04T12:08 (+26)

Hi all,

A friend linked me to the 'red team' contest, and since I undertake cost-effectiveness modelling professionally I thought that would be a useful place I could contribute, potentially.

I'm not an active member of the EA community, so I'd like to ensure I don't straw man the state of the art here; would it be fair to say GiveWell's evaluation of some of its most cost-effective charities represents what the EA community would consider to be a high-quality cost-effectiveness model? If not, what would the EA community consider to be a high-quality cost-effectiveness model in an EA context?

https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness/cost-effectiveness-models

(To be clear, the model is great but I think there are a number of areas where it could be improved upon!)

Thanks so much,

Froolow

Lorenzo @ 2022-06-04T17:33 (+9)

I would say that GiveWell's cost-effectiveness analyses are considered excellent (here is a guide from 2019), but they should be taken in context.
From https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness/cost-effectiveness-models "we consider our cost-effectiveness numbers to be extremely rough."
"There are many limitations to cost-effectiveness estimates, and we do not assess charities only—or primarily—based on their estimated cost-effectiveness."

And this old blog post: https://blog.givewell.org/2011/08/18/why-we-cant-take-expected-value-estimates-literally-even-when-theyre-unbiased/

They definitely have a much smaller weight than what I first assumed: I initially thought all GiveWell did was to make cost-effectiveness analyses of dozens of charities and recommend the most cost-effective ones. It seems that's completely wrong, and they rely a lot on other criteria and less quantitative or public information.

You might also be interested in the series: Concerns about AMF from GiveWell reading (especially Part 3 and Part 2)

And this FAQ from 2017 from GiveWell

Froolow @ 2022-06-04T21:07 (+5)

Thank you - really helpful additional information and very useful to have it confirmed that GiveWell are considered high quality models by the EA community. Really appreciate it.

Vasco Grilo @ 2023-05-20T08:25 (+2)

Hi Lorenzo,

They definitely have a much smaller weight than what I first assumed: I initially thought all GiveWell did was to make cost-effectiveness analyses of dozens of charities and recommend the most cost-effective ones. It seems that's completely wrong, and they rely a lot on other criteria and less quantitative or public information.

I think GiveWell actually puts a major weight on their cost-effectiveness analyses. Elie Hassenfeld (co-founder and CEO of GiveWell) mentioned in the Clearer Thinking podcast that (emphasis mine):

ELIE: You know, there's a lot of attention paid to the numbers, and certainly plenty of high profile institutions were behind the report. I think that, in my experience, GiveWell is one of the few institutions that's, I don’t know, trying to make decisions based on cost-effectiveness analysis in doing that in a sort of, consistent and principled way. GiveWell cost- effectiveness estimates are not the only input into our decisions to fund malaria programs and deworming programs, there are some other factors, but they're certainly 80% plus of the case. I think we're relatively unique in that way. I don't think there are other groups, certainly I can't think of any ones as I'm sitting here now, that are using numbers in that same way. In some ways, I think that is why we have real value added in the world, because I don't think that explicit cost-effectiveness estimates is the only way to give effectively, but it's certainly a strategy that I think should be employed significantly. I'm glad that we can be the ones to come in and play it.

Gavin @ 2022-06-04T12:58 (+4)

I'd say Michael Dickens and Sam Nolan are probably peak performance.

This is a classic, devastating methodological piece too.

Froolow @ 2022-06-04T13:57 (+4)

Thank you - these are really helpful to help me understand. On the Sam Nolan piece especially quantifying uncertainty was one of the biggest critiques I had of the GiveWell model so I'm glad this has already been considered!

NunoSempere @ 2022-06-05T16:51 (+2)

this has already been considered!

This has been considered by Sam, but it's still a valid criticism of GiveWell's models as they are now.

Stefan_Schubert @ 2022-06-04T14:44 (+2)

I would guess that other orgs besides GiveWell also have cost-effectiveness models/analyses.

Gavin @ 2022-06-04T14:54 (+4)

Couldn't find any public OP analyses on a cursory look

Stefan_Schubert @ 2022-06-04T14:59 (+4)

I guess that if one wants to red team effective altruist cost-effectiveness analyses that inform, e.g. giving decisions, non-public analyses may be relevant.

Gavin @ 2022-06-07T10:43 (+2)