We don't have evidence that the best charities are over 1000x more cost effective than the average

At 80,000 Hours, we published an article on this topic in 2023 by Benjamin Todd. It's a follow-up to Toby Ord's original work, and looks at other datasets and cause areas.

Benjamin concluded:

Overall, I think it’s defensible to say that the best of all interventions in an area are about 10 times more effective than the mean, and perhaps as much as 100 times.

And also:

People in effective altruism sometimes say things like “the best charities achieve 10,000 times more than the worst” — suggesting it might be possible to have 10,000 times as much impact if we only focus on the best interventions — often citing the DCP2 data as evidence for that.

This is true in the sense that the differences across all cause areas can be that large. But it would be misleading if someone was talking about a specific cause area in two important ways.

There's a ton more detail in the article.

In our chapter for the World Happiness Report we think we provided the most direct evidence answering this question. But the sample of charities isn't necessarily representative, but more of a convenience sample. More work needs to be done estimating the effect of the "typical charity" which is tough since a large amount of total expenditure is done by large, practically unevaluatable charities. The differences are still massive though. Especially when we compare them to expectations. 

I agree with point 2 to some extent but not point one. We have direct evidence from multiple randomized controlled trials that show us without much doubt that the best interventions are 10-100x more cost effective at saving lives than many others. Like @Cody_Fenwick  says maybe not 1,000

Just because an intervention is complex doesn't necessarily mean the outcome is complex as well. Many complex interventions are more measurable than we think.

I discuss this a littler more here https://forum.effectivealtruism.org/posts/w44oxwXpRkzyEEEHr/the-best-health-systems-strengthening-interventions-barely

Here's some good reflections as well from Kevin Starr from Mulago foundation along similar lines
https://www.mulagofoundation.org/articles/in-numbers-we-trust

Yes there are interventions which are hard to measure, but not as often as we might think. 

I agree with other comments that the 80k article is the place to go.

But I also want to specifically praise and thank the original poster for (1) noticing an important seeming empirical claim being bandied around (2) noticing that the evidence being used seemed insufficient (3) sharing that potentially important discovery.

(For what it's worth, before the 80k article, I also worried that people in the EA community were excessively confident in similar claims.)

Also, even if charities differ significantly on a specific, narrow metric, they may differ less substantially in terms of various indirect and knock on effects (which also matter). See https://reducing-suffering.org/why-charities-dont-differ-astronomically-in-cost-effectiveness/

Since we're on this topic, I recently saw that the Happier Lives Institute estimated that the best charities (based on WELLBYs per dollar) are about 100x more cost-effective than the average charity https://www.happierlivesinstitute.org/world-happiness-report/

I have given a shallow recap of their medium-depth report, so take with a grain of salt.

The data in this paper comes from the 2006 paper "Disease Control Priorities in Developing Countries".

I don't understand. Does this paper not support the claim? 

I've actually never heard this claim before, personally. Instead, people like Toby Ord talked about how the cost of curing someone's blindness through the Fred Hollows Foundation was 1,000x cheaper than training a seeing eye dog. 

What is the average charity? I don't have a good intuition for what it looks like, is, how big it is, what it works on etc.^[1] I think pinning this down will help make the comparison clearer. Will, how do you think about this? 

1. ^{^}

   Sidenote: At least in the US, I would be open to the argument that the average charity -- defined as being the midpoint of some multidimensional array of size, cause area, staffing, location, etc. -- produces literally zero charitable benefit on net, and might even be doing harm. You might not share this intuition, but we have a long list of mostly null effects for pro-social interventions once they're evaluated rigorously (enterprise zones in California, medicaid enrollment in oregon, head start, etc. -- any of which you might take issue with but I think the broader point is defensible that on average, interventions don't work.) If the average social utility gain of a given nonprofit America is zero, then I don't know how we're going to say some other cause is X or Y times "better" than that. The seeing eye dog vs curing blindness comparison is a lot more coherent, I think.

    

I don't have anything to add about the intra-cause effectiveness multiplier debate. But much of the multiplier over the average charity is simply due to very poor cause selection. So while I applaud OP for wanting rigorous empirical evidence, some comparisons simply don't require peer-reviewed studies. We can still reason well in the absence of easy quantification

Dogs and cats vs farmed animal causes is a great example. But animal shelters vs GHD is just as tenable.

This isn't an esoteric point; a substantial amount of donations are simply to bad causes. Poverty alleviation in rich countries (not political or policy directed), most mutual aid campaigns, feeding or clothing the poor in the rich world, most rich-world DEI related activism lacking political aims (movement building or policy is at least more plausible), most ecological efforts, undirected scholarship funds, the arts. 

I'm comfortable suggesting that any of these are at least 1000x less cost effective.

You may find this 80K article useful, both for their analysis and for all the data they collected: How much do solutions to social problems differ in their effectiveness? A collection of all the studies we could find. Bottomline is 3–10x not >1,000x for measurable interventions, and stack on a 2–10x spread for harder-to-measure interventions: 

Overall, I roughly estimate that the most effective measurable interventions in an area are usually around 3–10 times more cost effective than the mean of measurable interventions (where the mean is the expected effectiveness you’d get from picking randomly). If you also include interventions whose effectiveness can’t be measured in advance, then I’d expect the spread to be larger by another factor of 2–10, though it’s hard to say how the results would generalise to areas without data.

Also this section: 

3. How much can we gain from being data-driven?

People in effective altruism sometimes say things like “the best charities achieve 10,000 times more than the worst” — suggesting it might be possible to have 10,000 times as much impact if we only focus on the best interventions — often citing the DCP2 data as evidence for that.

This is true in the sense that the differences across all cause areas can be that large. But it would be misleading if someone was talking about a specific cause area in two important ways.

First, as we’ve just seen, the data most likely overstates the true, forward-looking differences between the best and worst interventions.

Second, it often seems fairer to compare the best with the mean intervention, rather than the worst intervention. ...

Overall, my guess is that, in an at least somewhat data-rich area, using data to identify the best interventions can perhaps boost your impact in the area by 3–10 times compared to picking randomly, depending on the quality of your data.

This is still a big boost, and hugely underappreciated by the world at large. However, it’s far less than I’ve heard some people in the effective altruism community claim.

In addition, there are downsides to being data-driven in this way — by insisting on a data-driven approach, you might be ruling out many of the interventions in the tail (which are often hard to measure, and so will be missing).

This is why we advocate for first aiming to take a ‘hits-based’ approach, rather than a data-driven one.

("Hits-based rather than data-driven" is quite counterintuitive, especially to someone like me who's worked most of my career in data-for-decision-guidance roles, but a useful corrective to the streetlight effect.) 

Edit: whoops just saw Cody's comment above pointing to the same article.