EA needs to understand its “failures” better

By mariushobbhahn @ 2022-05-24T14:24 (+67)

Introduction

I recently posted “How many EAs have failed in high-risk, high-reward projects?” and got some insightful answers. Shortly afterward, Michael Dickens posted “What are some high-EV but failed EA projects?” and also got insightful answers. 

My main realization from the responses to both questions is that the EA community knows too little about its “failures” (at least what’s publicly available). However, I think it would be very important to understand our failed attempts better. Perceived failure has implications on people’s mental health, it has implications on which projects get started and which ones don’t and it has implications on who feels included and who doesn’t. 

Therefore, I want to lay out why we should know more about our “failures” and what could be done to address the problems we are currently facing. 

What does “failure” mean?

Definition: the fact of someone or something not succeeding (Cambridge Dictionary). 

In an EA context, I would guess that most people mean something like “An individual or a group of people attempted something and did not achieve their goals” when they think about failure.

The word failure has a very negative social connotation. Even when you are self-confident and have good mental health, failure is something scary. It means you haven’t achieved what you set out to do, that you “didn’t deliver” or “could have done better”. Part of the reason why failure feels bad is that it is often attached to your identity and not your actions--it’s very easy to go from “I failed a project” to “I am a failure”. 

I think this negative connotation is not only bad for mental health reasons but also for instrumental reasons (see below). Therefore, I think the EA community should think and communicate differently about “failure”. To remove the link to personal identity, I will call them failed attempts for the rest of the post. 

Why do failed attempts matter?

Expected values

A lot of reasoning in EA is based on expected values (EV). To calculate an EV, we need an estimate of the impact and an estimate of the probability of success. The probability of success is often hard to estimate but (failed) previous attempts can give us information in multiple ways. 

1. We could naively approximate the probability of success as the number of successful previous attempts divided by the number of total attempts.
2. We can use projects of similar difficulty as a reference class for our own probability.
3. We can use comparable people’s success rates as a reference class for our own.

In all of these cases, our estimates are much more informative if we know about the failed attempts and not only the successful ones. Otherwise, we might systematically overestimate our chances of success. 

A culture of ambition

Many of the projects that EAs are attempting are increasingly difficult to pull off. For example, there is a need for more megaprojects, many of which have a low probability of success. 

If we treat failed attempts as failures, we reduce the number of people trying them. If even under the best circumstances, your chances of success are less than 10%, it just makes less sense to try if you “fail” 9 times out of 10 unless people explicitly embrace your failed attempts as well. 

Alternatively, we should want to nurture a culture that embraces ambitious projects. This means not only talking to or about the people who succeeded but also those that attempted and didn’t succeed. Trying something with high upsides and high risk should be praised, not ignored, independent of the outcome. I think the EA community is on a good trajectory (see, e.g. this post on getting hired by EA orgs) but could do better (myself included). 

Combatting imposter syndrome

Imposter syndrome is quite common among EAs. I would assume that part of this comes from skewed images of failed attempts. We mostly see the people who stayed in the EA scene or successfully achieved something. We rarely see the people who silently left the bubble because they felt like a failure. 

Rationally, I know that most people have failed at something, but it’s not tangible to me because I don’t know about the many people who think they failed, e.g., because they don’t come to EAG. At most EA events, people rarely speak about their failed attempts (in public). If we had a culture that more openly talked about failed attempts and perceived failures, we might be able to reduce the degree to which people feel like an imposter.

Learning from failed attempts

Failed attempts provide so much information value that is just lost if we don’t talk about them. 

1. We can understand why a project failed. Are there fundamental barriers that others will face too or were there mistakes in execution that could be avoided by others?
2. Someone else can start where you left off. Even if a project failed, the information is still relevant. Which paths looked promising but failed? Which people did you talk to and what did you learn from these conversations?
3. Which things went well? Often projects fail not because everything is a disaster but because some key elements go wrong. What were the things that went well and should be replicated in a new attempt?

There is just so much information that just gets lost if we don’t know that other people attempted the same project already. We can start the new project with more information, save lots of time and get to impact faster. In case our attempts fail, someone else can start with our knowledge. 

What could be done?

There is a previous post detailing the risks and benefits of sharing your "failures" with more nuanced explanations of when and how one can talk about them. I think the post was pretty good. 

Embrace failed attempts with high EV

If someone fails an attempt at a promising project, this person should be praised. I think we are already on the right path but there are definitely things we could improve. 

For example, I would love to hear an 80K podcast episode like “Hi, I’m one of the 10 invisible Sam Bankman-Frieds; yeah, the one that didn’t become a Billionaire”. Think about how much we could learn from that person about what went wrong and what went right. I guess it might be hard to find a person willing to talk about their failed attempts but I would love to get to a state where people feel comfortable doing that. 

Another thing that I think is interesting would be “failed attempt workshops” on EAGs. There, different people could present their attempts at a project and analyze why they didn’t achieve their desired outcome. They could share their experience and others could learn from them. For example, I didn’t know that the current version of Rethink Priorities is their second attempt. Their story could help and motivate others.  

More Postmortems

There already is a postmortems category on the forum. However, there are probably still lots of stories untold that deserve to be known. 

Maybe one could do something like a postmortem month or failed attempts month where people are nudged to write about their failed projects and what we can learn from them. It could be similar to the red-teaming contest, but I want to prevent a competitive vibe. If someone has an idea how to facilitate that, feel free to run it. 

Keeping track of failed attempts

Somebody should vaguely keep track of project attempts and document if they succeeded or failed. If they failed, we might want to understand at what point they failed and for what reasons. There could be a public and a private version of such a spreadsheet such that people who failed an attempt but don’t want it to be public have an option to submit. 

I’m not sure who would be the best to keep track of that; OpenPhil, 80K, RP or CEA spontaneously come to mind. 

Potentially, such a list already exists but isn’t public for privacy reasons. If it does exist, I wouldn’t know who to talk to if I wanted to submit a failed attempt right now. 

Conclusions

I think the EA community is on a good trajectory. I feel like people don’t judge me for failed projects and I feel like I’m able to talk about them. However, I don’t think we are at the optimum yet. In a perfect world, people should be equally willing to talk or write about a promising failed project as a promising successful project and I suspect that is not the case yet. 

If you want to get informed about new posts, you can follow me on Twitter.