How frequently do ACE and Open Phil agree about animal charities?

By Ben_West🔸 @ 2019-12-17T23:56 (+66)

I'm curious what the concordance between Animal Charity Evaluators (ACE) and The Open Philanthropy Project (Open Phil) is. Specifically: of the charities that ACE recommends, what fraction get grants from Open Phil? And conversely, of the charities that ACE considers but does not recommend, what fraction of those get grants from Open Phil?

It seems like there is pretty high concordance:

1. All of the charities which have ever been recommended as a "Top Charity” have received a grant from Open Phil.
2. About half of the charities ever described as a "Standout Charity" have received a grant from Open Phil. (Three of the four "general interest standout" charities have received a grant, although ACE only started separating general and special interest recently.)
3. None of the charities ACE comprehensively reviewed but did not recommend have received a grant from Open Phil. [Edit: there actually is one discordant charity – see Kieran's comment below.]
4. 5% of the charities ACE did an "exploratory" review of received a grant, as did 3% of the ones they "considered" but did not review.

Note the y-axis is logarithmic. This particularly affects the “Standout” column, which is about half concordant and half discordant. For standout and Top Charities, “concordance” means they got grants from Open Phil; for other recommendation types, “concordance” means that they did not get grants.

The biggest source of discordance seems to be with standout charities. This could be because some of their early standout charities were based on evidence that Open Phil disagreed with (possibly true of Vegan Outreach, maybe FARM) or because the criteria they use are different from Open Phil’s (maybe true of Faunalytics?).

Open Phil started their farmed animal welfare focus area about two years after ACE published their first recommendations, and Open Phil pretty quickly granted to all of ACE's Top Charities after starting. Since then, it seems about 50-50 whether Open Phil grants to something before or after ACE recommends it. So neither organization seems to be simply copying the other.

Should ACE and Open Phil agree?

My guess is that the criteria that ACE uses for their Top Charities is similar enough to what Open Phil looks at that all top charities “should” get grants.

I’m less clear about standout charities. For example, if a charity is highly effective but has limited room for more funding, ACE may choose it as a Standout and also Open Phil may choose not to grant to it. A more careful analysis of this would look at the reasons why charities were or were not selected for recommendations, and see how they line up with Open Phil’s criteria.

Raw data

Copy of the spreadsheet is here.

Data notes

1. If an organization had multiple recommendation/review status levels at different times in history, I chose the “most recommended” level, using this ranking of recommendations from most to least: Top Charity, Standout Charity, comprehensive, exploratory, considered.
2. I tried to match up organizations which have different names in ACE and Open Phil’s data set, but I probably missed some.
3. I wrote this before ACE’s 2019 recommendations were published, but I think the results will be similar: the one new Top Charity (Anima International) received a grant from Open Phil, as did the new Standout Charity (FIAPO).

Animal Welfare Fund

Three charities which were named “Standout Charities” by ACE but did not receive Open Phil grants did receive grants from the Centre for Effective Altruism’s Animal Welfare Fund (Animal Ethics, Faunalytics, and Sociedade Vegetariana Brasileira). As Lewis Bollard is both the Chair of the Animal Welfare Fund and the Program Officer for Open Phil’s Farmed Animal Welfare focus area, it could be interesting to consider these as additional sources of concordance.

For example, maybe Lewis thought a given charity was high-quality, but that it would be better funded via the Fund instead of through Open Phil. None of the payout reports explicitly stated this (and the other fund managers could have been the ones who recommended these charities, instead of Lewis), but it seems reasonable to assume that Lewis at least did not think they were terrible grants.

Here is what concordance looks like if you include these three charities as concordant:

Credits

I would like to give a shout out to ACE for publishing such an easy to use list of their recommendations, as well as to Open Phil for providing a download of their grants. This post was inspired by conversations with JP Addison. Aaron Gertler and Gina Stuessy provided helpful comments.