Should EA grantmaking be subject to independent audit?

By Milan_Griffes @ 2019-04-17T17:18 (+19)

Inspired by this awesome comment by Evan Gaensbauer.

Tangentially related: comment thread about compensating Long Term Future Fund grantmakers

Evan did a great, independent audit of the recent round of Long Term Future Fund grants.

This seems like a clear value-add. It'd be good to have similar audits happen for each round of EA grantmaking.

Currently, we're relying on folks taking it upon themselves to do audits like this. There's no formal structure, and if such audits stopped, there wouldn't be any institutional consequences (though there'd likely be indirect consequences, such as lower grantmaking quality over time).

Given that audits like this seem like a clear value-add to the EA grantmaking process, building formal structure to support grantmaking audits seems leveraged.

Many firms do something analogous by maintaining "Internal Audit" business units, though it's not clear that those units are well-designed. See also Open Phil's recent announcement of their intention to build out an internal "Impact Evaluation" function.

Should the EA community find a way to financially support independent audits of its grantmaking?

Khorton @ 2019-04-17T19:55 (+7)

What level of audit are we talking about? Are we just talking about someone who ensures people aren't giving money to (undeserving) friends and family? Or are we talking about scrutinizing the decisions to some extent?

Milan_Griffes @ 2019-04-17T20:35 (+9)

I don't have a particular specification in mind. Evan's comment seems like a good starting point for what audits could look like.

John_Maxwell_IV @ 2019-04-17T19:30 (+5)

It feels inefficient to second-guess a decision which has already been finalized. I think you could argue that something like a grant decisions thread should get posted before money gets disbursed, in case commenters surface important considerations overlooked by the grantmakers. There might also be value in auditing a while after money gets disbursed, to understand what the money actually did. Auditing right after money gets disbursed seems like the worst of both worlds.

Milan_Griffes @ 2019-04-17T19:36 (+11)

I think a lot of the value of an internal audit business unit flows from other stakeholders being aware that internal audit exists, during their decision-making process.

i.e. the common knowledge that internal audit is a thing / one's decisions will be later subject to independent scrutiny generates a lot of the value.

JoshYou @ 2019-04-18T02:47 (+13)

Open Phil would be a good candidate for this, though that's a difficult proposition due to its sheer size. It is a somewhat odd situation that Open Phil throws huge amounts of money around, much of which happens without any comment from the EA community.

Evan_Gaensbauer @ 2019-04-19T02:28 (+15)

A lot of this is the private sensitivity many community members feel about publicly criticizing the Open Philanthropy Project. I'd chalk it up to the relative power Open Phil wields having complicated impacts on all our thinking on this subject, since with how little the EA community comments on it, the lack of public feedback Open Phil receives seems out of sync with the idea they are the sort of organization that would welcome it. Another thing is the quality of criticism and defense of grantmaking decisions on both sides is quite low. It seems to me EA has overgeneralized its conflict avoidance to exclude scenarios when adversarial debate or communication is fruitful for a community overall, and so when adversarial debate is instrumental, EA is poor at it to the point it doesn't recognize good debate.

A pattern I've seen is for critics of something in EA will parse disagreement with some aspect(s) of their criticism as a wholesale political rejection of everything they're saying, or taking it as a personal attack on them on retaliation for attacking a shibboleth of EA. These reactions are usually patently false, but this hasn't stopped EA from garnering a reputation of being hypocritically closed to criticism, and impossible to affect change in.

While I wouldn't say I generally agree with all of Open Phil's grants, and simply by chance most EAs or other people wouldn't because they're are so many, the impression I've gotten is that the EA community and Good Ventures don't have identical priorities. EA is primarily concerned with global poverty alleviation, AI alignment, and animal welfare. An example of something Open Phil or Good Ventures prioritizes more than EA is criminal justice reform. EA agrees criminal justice reform is one of the more promising areas in public policy to do good, it's not literally one of EA's top priorities. So, criminal justice reform is a top priority more particular to Dustin Moskowitz and Cari Tuna.

My impression is that as long as motivations in Open Phil's grantmaking don't pull away from effectiveness and other EA values in the cause areas the community cares most about, they don't mind as much what Open Phil does. A good example of when the EA community is willing to strongly criticize Open Phil when ineffective grantmaking infringes on a cause area EA is more passionate about is the criticism Open Phil received from multiple points over how they made their grant to OpenAI.

Milan_Griffes @ 2019-04-19T16:22 (+3)

Tangentially relevant: https://www.openphilanthropy.org/blog/update-how-were-thinking-about-openness-and-information-sharing

vaidehi_agarwalla @ 2019-04-21T01:41 (+9)

I was going to mention this post as well, to summarize it: Open Phil has deliberately lowered their transparency levels on the grants they make because unlike GiveWell they aren't making direct recommendations to the public and because the projects are too complex/would take too many resources to justify/sharing information would have 'programmatic risks', and they wouldn't be able to be completely transparent.

So Open Phil has actively moved away from public discussion. Given the feedback process for the Long Term Funds recently, I can understand this decision, although if it wasn't so research intensive I'd like to see more write-ups, especially for US policy grants. I think international opportunities (not just global health but on a whole plethora of issues) are very neglected and tractable, but currently there is an overwhelming focus on NTD's (mainly GiveWell's research). I'm assuming Open Phil doesn't want to crowd out/double up on GiveWell's research, or because of Good Ventures' interests. I predict these international opportunities will become less neglected as EA reaches developing countries and is able to incubate local EA's who have comparative advantage in those areas.

The flipside of less transparency is that Open Phil has been doing the most active recruiting over the last two years within EA, and so it's position as such is relatively strong in the community, not just in terms of money but also from a careers perspective.

Evan_Gaensbauer @ 2019-04-19T02:33 (+7)

I just wanted to channel Aaron's comment in clarifying the following:

  • While I don't mind the characterization, I didn't originally intend my comment as a kind of audit.
  • I was not under the impression the money had been disbursed yet, and it wasn't ever my intention to criticize grantmaking decisions after disbursement, or to evaluate individual grant recommendations from this round in particular, only a general trend in the LTF Fund.
aarongertler @ 2019-04-18T18:25 (+6)

You probably know this, but to clarify for other readers: no money has yet been disbursed for the LTF grant round on which Evan commented.