Should EA grantmakers make p(success) public?

By Kaleem @ 2023-05-01T13:42 (+26)

Summary

This article asks whether EA grantmakers should publicly disclose the probability of success (p(success)) for their funded projects, and discusses the potential benefits such as improved community norms and accountability, as well as potential drawbacks and implementation considerations.

Introduction: 

Recently I’ve been thinking about the possibility of EA grantmakers publicly sharing the p(success) of their funded projects. This article is intended to start a discussion by exploring the potential benefits and drawbacks of this approach, and is by no means exhaustive or super detailed. 

This is an idea I had this week and initial conversations with people in Trajan House about the idea were interesting and positive enough that I thought it’d be worth opening up the conversation. I have no experience as a grantmaker, only as a grant applicant, and so I’m sure that I have a very poor understanding of how grantmaking actually works. Therefore I’m sure there are reasons I haven’t heard or thought about as to why this suggestion might have already been rejected, or wouldn’t be a good idea - which I’d like to read.  I tried to get some grantmakers to take a look/comment on a pre-post draft of this, but didn’t have much luck. In any case, thanks to the two people that did give brief feedback.

Suggested Implementation: 

I’d suggest that grantmakers publish a grantee-independant p(success) alongside the public grant disclosures that some of them already publish,^[1]. ^[2]That is to say that I assume grantmakers are able to look at a project proposal independent of the grant applicant, and assign a p(success). Once they’ve done that, they then might factor in their subjective belief in the aptitude/competence/track record of the applicant and adjust the p(success|grantee) up or down, and keep that private. I think to avoid worries about how this disclosure might affect grantee’s mentality regarding their proposed project pre-execution, it could be that p(success) only be shared publicly after the grant period is over.

Why this might be a good idea:

• Improved community norms around failure: Although the EA community uses expected value for decision-making, success and failure still play a significant role in shaping people's reputation in the community.^[3] Being transparent about the p(success) could help us better appreciate and acknowledge both ambitious projects with lower probabilities of success and those who work on them. It might also make it easier for people to be more open about their failures, and discuss ways to avoid similar pitfalls, if they’re able to point to the failed project not having a high p(success) in the first place.
• Career stability: Sharing p(success) can also assist community members in obtaining funding or employment even after experiencing multiple project failures, as it helps clarify that the actual probability of success may have been lower than perceived. In a future scenario with multiple independent EA funders, if a grant proposal is rejected by one grantmaker, another grantmaker can fairly evaluate it with more information to properly assess the applicant's track record, without being influenced by the previous grant assessor's decision.
• Greater grantmaker accountability and grant benchmarking: Making p(success) public could hold grantmakers accountable for their track records, particularly if they consistently overestimate or underestimate the success of certain classes of projects or grantees.
  • Publicizing p(success) can facilitate learning and benchmarking among grantmakers, allowing them to compare their success rates and identify areas for improvement. I can see that this could already be happening/ could happen in private channels, but I think this happening in the open would strengthen trust that the community has in grantmakers’ independence.  
  • Being transparent about p(success) might strengthen trust between grantmakers, grantees, and the wider EA community, by demonstrating their commitment to honesty and accountability, which can enhance their credibility and reputation.
• Enhanced EA integrity and public reputation: Demonstrating transparency in how we use expected value when making individual grant decisions could strengthen the credibility of the EA community and convince other grantmakers to adopt similar decision-making frameworks.^[4]
  • This could also lead to the EA community being more appealing to some external funders and potential partners. By demonstrating a data-driven and transparent approach to decision-making, we may attract additional resources and support from organizations and individuals who share similar values and goals.

Potential reasons for caution:

• Discouraging ambitious projects: Publicizing p(success) could deter people from applying for funding for ambitious projects with lower probabilities of success.
  • Because reputation is still built on success/failure in the community
  • People might not want to ‘waste their time’ (given that time is often our most valuable resource) on projects that the think won’t help them further their careers or have an impact
• Lower morale: Applicants might feel demotivated or less excited if the disclosed p(success) is lower than they initially anticipated, potentially affecting their commitment to the project.
• Distorted incentives: Public p(success) information might lead to strange social dynamics, where individuals may prefer to work on projects with very low or very high probabilities of success for the sake of prestige.
• Complexity in separating grantee-independent p(success) from p(success|grantee): It might be challenging, a waste of time,  or even counterproductive for grantmakers to separate the two probabilities as the grantee's capabilities could be critical in determining the project's overall success.

What do you think? 

1. ^{^}

    This used to be on FTXFF’s website but it no longer exists.

2. ^{^}

    Here are Givewell’s - although I don’t know if I think my p(success) suggestion would be as important for them.

3. ^{^}

    e.i. People in the community who seem to have a good reputation/high status are people who seem to have a track record of success, and people with a track record of failure don’t get much fame or recognition, even though we don’t know anything about how ambitious/low-p either person's projects were. 

4. ^{^}

    i.e. it shows that we put our money where our mouth is when it comes to how we make individual grant decisions, not just which causes we choose or how we choose causes.