Some benefits and risks of failure transparency

By Vaidehi Agarwalla 🔸 @ 2022-03-27T02:09 (+52)

Key takeaways

• The goal of this post is to discuss some heuristics on when it makes sense to discuss our own and other’s failures and mistakes in the EA community. I also recap the state of failure discourse in EA, including the recent interest and activity around red-teaming. (Read More)
• Benefits of failure transparency
  • Building a more accurate map of the world so that resources are better allocated, we can identify and highlight trends in mistakes, and also help the broader world outside of EA improve. (Read More)
  • Creating a strong and trustworthy signal of our values (Read More)
  • Creating a stronger community (Read More)
• Risks and Costs
  • Opportunity costs (Read More)
  • Reputational costs (Read More)
  • If we engage in discourse less responsibly, there is a risk of harming existing discourse norms, which could result in less failure discourse in the long run. (Read More)
  • Transparency being used against you or unfairly advantaging others (Read More)
• When to engage in discourse
  • When considering our own mistakes, keep in mind the relevance of the mistake, the value of information you could gain from reflecting further on it or getting external opinions, whether others would find it valuable, and if you’re ready to hear your mistakes. If you’re higher status in the community, it may make sense to have a lower bar for sharing mistakes. If you’re in a competitive field outside of EA, it may be worth having a higher bar. (Read More)
  • When considering others’ mistakes, consider what you are hoping to achieve with the feedback, and how likely the other actor is to be open to feedback. Also consider whether the appropriate forum of engagement is private of public. It may also be worth factoring in the status of the actor you’re criticising. If you choose to communicating publicly, be careful when expressing your level of confidence and extrapolating. (Read More)
• A few links that might be more concretely helpful when you’re actually trying to talk about failure (Read More)

Linguistic note: I refer to organisations and individuals collectively as “actors”. I also use the terms “failure” and “mistakes” somewhat interchangeably throughout this post. But I try to stick to “failure” for the most part.

Thanks to Arjun Khandelwal for extensive copyediting and review, and Adam Gleave for brainstorming on whiteboards. Thanks to Nathan Young, Abi Olvera, Ben Millwood, Arjun & Adam for many (many) helpful suggestions and comments. I’ve tried to cite in the footnotes where possible but I’m sure I’ve forgotten a bunch!

Introduction

The goal of this post is to discuss some heuristics on when it makes sense to discuss our own and other’s failures and mistakes in the EA community. I hope it provides a balanced and fair view of the considerations.

Failures are pretty complex and can be caused by a lot of different things. Some are caused by mistakes that were knowable in advance, others by things that could not have been predicted or were out of someone’s control. Some failures are related to not living up to one’s values and causing harm to other people, while other mistakes negatively impact one’s own productivity or growth without causing direct harm to anyone else.^[1] It seems like all of the above are valuable, depending on who is reading. People external to EA might particularly care about value-related failures, while those working at EA organisations might benefit from knowing how other organisations have made mistakes that negatively harmed their productivity.

Failure discourse in EA currently

This section might be most useful to those new to the community or interested in a summary of failure discourse in the EA community.

The EA community can be keen to learn from mistakes and not punish people overly for making them in general, which can be rare.^[2] A highlight is the presence of several public “Mistakes” pages where organisations discuss substantive ways they’ve made mistakes. This is partly through the presence and influence of GiveWell, which has been a champion of transparency in general, including failure transparency, in the nonprofit world and has maintained a prominent and substantive Mistakes page since its inception. Some other prominent EA orgs like CEA, 80,000 Hours, Giving What We Can, and Animal Charities Evaluators also maintain substantive Mistakes pages ^[3], which is likely very unusual in the broader nonprofit world. 

These pages are a costly signal of an intention to learn from mistakes and, to some extent, be publicly accountable for them. While some of the mistakes have long been common knowledge among parts of the EA community, putting them out in the open for anyone to see allows outsiders to form a much more accurate understanding of EA organisations without having to undergo a potentially costly process of becoming socially connected within the community. Having public documentation instead of relying on social connections and word of mouth is more robust for community insiders as well. 

Discussing mistakes at public events (e.g. EA conferences) is much less common. The only instance I’m aware of is the Celebrating Failed Projects Panel at EAG SF 2017.

In the past 6 months or so, there has been a lot of interest in having more critiques, and specifically red-teaming. Cremer and Kemp’s Democratising Risk post sparked some conversation about how open EA is to critiques, and many EA funders and leaders commented in support of more criticism. A notable recent post from Jan Kulveit and Gavin Leech discusses the EA and rationalist communities’ failures in response to COVID-19. ^[4] Training for Good is running a red-teaming workshop, and there are multiple contests which support critiques - the Effective Ideas’ blog contest and just this week a contest for critiques and red-teaming was pre-announced.^[5]

It seems like now might be a good time to reflect on some of the benefits and risks associated with failure discourse.

Benefits and risks of failure discourse

What are the potential benefits?

Building a more accurate map of the world

With more information we can help the community build a more accurate map of the world together and update our decision-making accordingly. EA is young and new, it stands to reason that our map of the world has a lot of room for improvement.

Building a more accurate map may result in resources being better allocated, which seems generally net good for the world even though it can often be bad for individual actors, i.e. directed away from actors who are consistently underperforming or making mistakes. In some cases, it may even be a good sign to actors to try to find other ways to contribute, resulting in a better allocation of talent. ^[6]

A more accurate understanding of the world would enable us to identify and highlight trends in mistakes. Especially if done well/robustly, this should help us improve as a community. ^[7]

Sharing our mistakes can help the broader world outside of EA that is doing good improve as well (not just from a community building perspective). ^[8]

A strong and trustworthy signal of our values

When we talk about our mistakes publicly (which is not always advisable, see below), we are creating a very strong signal of our values. This benefits the community internally by encouraging and codifying our norms of transparency. Hopefully, this could create an environment that is more welcoming of criticism (both internally and externally) and makes us less susceptible to things like motivated reasoning. It also sends a strong signal to the outside world about our credibility. The higher the status of the person who talks about their own mistakes, the stronger the signal. Finally , it could nudge the outside world towards normalising this kind of transparency and discourse more. 

A stronger community

Demonstrating vulnerability and honesty can help communities grow stronger and closer together. Here are two examples from within the EA community:

• Howie’s 80,000 Hours podcast on mental health goes into a lot of detail on mistakes that he made during his time at Open Philanthropy and his experiences with mental illness. This episode has become 80K’s most “popular episode ever (both in terms of feedback and listening time).”

The EA student group PISE has had some success with being open and vulnerable in their community. 

What are the potential risks and/or downsides?

Opportunity Cost

If an actor chooses to engage (by sharing their own or others’ mistakes, or engaging in the discourse that follows), they are effectively taking time away from “direct” work they could be doing. ^[6]

It can also cost others resources to consume the information in a way that's useful to them, though this is more a reduction in benefit than a real cost since people can just not read it. ^[9]

Transparency in general can be very costly. Open Philanthropy changed their stance on transparency in 2016 after reflecting that it wasn’t directly aiding their goals (unlike GiveWell, whose goal is to make recommendations to the public). Charity Entrepreneurship decided to continue to publish their work in 2020 and accept the increased cost of spending more time polishing reports, but said they would evaluate in 2021. As of writing, their 2021 annual report has not yet been published.

Reputational Costs

Sometimes, writing about your mistakes could result in an actual loss of status or reputation. How much it could affect your reputation could be the function of how long ago it was, how understandable it was, how much you’ve changed since then and other factors, whether other people know about them.

If you demonstrate that you’ve learnt from your mistake, and have taken steps to improve, then sharing your mistakes could have a neutral or even positive effect. There are also instances where not sharing can be harmful to your reputation. If the mistake you are not sharing is something that might come up - say, if a potential employer was to speak to a reference of yours, then not owning your mistake may be perceived as either not being self-aware or reflective, or at worst, like you are trying to hide it.

Harming discourse

The way we react to actors sharing their own mistakes, or when we raise others’ actors’ mistakes can negatively harm discourse norms going forward. Jeff Kaufman writes, “If people react critically and harshly to … failure, it makes organizations much less likely to be willing to be so transparent in the future. And not just this organization [or actor]: others will also see that sharing negative information doesn't go well.” This is less true “if the norm is very strong, then the pressure is not going to keep people from sharing similar things in the future, and it also means that seeing a failure from this organization but not from others is informative. On the other hand, if the norm is weaker we need to be careful to nourish it, not pushing it harder than it can stand.” It seems that EA does have somewhat strong norms around sharing failure, but it’s not clear to me how strong.

Another way that discourse norms can be harmed is that having very high standards for criticism, extending to things like tone can make it less likely that criticisms are raised. It’s possible that some people wouldn’t make the criticism rather than investing resources into trying to say it perfectly.^[10] Some people I discussed Democratising Risk with thought that it was tonally inappropriate - that the authors should have expressed their negative experience separately from announcing their paper and/or expressed the negative experience more objectively. I can understand where this is coming from, but personally I’d rather live in a world where something is said imperfectly than a world where it isn’t said at all. 

Career risks

If you’re in a competitive or reputation-heavy field such as politics, transparency could be used against you and affect your future prospects. 

When there is imbalance in transparency, organisations that are more transparent could be evaluated unfairly. If some actors in a space are much more transparent than others, and they seem to be doing less well than other, less transparent orgs, they would be evaluate against different standards. This could set up unhelpful incentive structures, which could encourage less transparency. ^[9]

When might we engage in discourse?

Although I personally think it’s usually quite valuable, and would want the community to discuss our failures less than we could, it may not always be worth it for individual actors to engage in failure discourse. The following are some decision-making heuristics that might help you make this decision. The examples are somewhat focused on whether you’d want to share your thoughts publicly, but I think they could also be helpful for deciding whether to engage in more private reflection too (e.g. sharing your thoughts with a small group of people).

Our own failures

Are you in a position where you can accept your mistakes?

• Opening oneself up to criticism and reflecting on mistakes can be (really) difficult. It’s okay to not be ready to do that immediately. Sometimes a little bit of distance can help with this.

How is it relevant to your ability to have impact or your work? How important is the mistake?

• A small accounting error which underestimates your organisation’s spending by 1% is probably not very important, but a mistake which understates it by 10% could be.

How much information do you stand to gain from investing resources into reflecting on this?

• Sharing our mistakes can help us to reflect in a more rigorous and focused manner. It’s possible that making something publishable (even if you don’t publish it) could encourage this more so than doing it independently. ^[9]

Do you think others could help you learn? Are you worried about motivated reasoning?

• It could be that others have useful insights on your mistakes and you can learn from sharing them. Maybe putting too much faith in internal / self criticism leads to neglect of external criticism.

How many other people could find your insights valuable to learn about? 

• You might have observed that others are thinking in similar ways to you, or working in the same field. If your failure was particularly surprising or unusual to you, it could be valuable to spend time reflecting - perhaps you could identify a community blindspot. If you’re a first actor in a field or space, then your insights seem especially important.

What is your status within the EA community?

• If you are a high-status individual, it seems valuable to have a lower bar for sharing your mistakes and failures because of the positive impact it can have on establishing and codifying community norms.

How competitive is your industry or profession - could this mistake be used against you?

• This mostly applies to people working in non-EA paths such as politics - a handful of policy EAs have insights on the Forum anonymously to avoid issues.

Others’ failures

What is the appropriate fora of engagement?

• Not all feedback and discussion of mistakes needs to be public to achieve it’s goal, and sometimes providing private critiques and feedback can be more effective. However, there does seem to be a lot of value in being able to have difficult public discussions as a community, even if it does sometimes cost us. Often it’s very valuable to provide feedback privately to an organisation first, and let them respond.
• Sometimes, there are good reasons to share criticisms publicly. It can be useful to know if others share your views and there is broader support for the feedback you’re sharing. If the org is unresponsive to your feedback and you still feel it’s worth raising, this may be another reason.
• If you do think there is a good reason to make criticism public, it’s good practice to share a draft with the organisation before publicly posting to give them a chance to respond or make corrections. This doesn’t mean you need to make all the changes they suggest - but it could lead to a more productive conversation (e.g. this critique of Giving Green).

What is the positive impact you think your feedback will have? What is your theory of impact or change?

• Try to think about whether your feedback is actually something the organisation could take action on. Suggesting a small or capacity-strained org focus on 10 different projects might not cause any change - but suggesting they clearly communicate what they aren’t doing, and how the plan to coordinate with other actors could be really helpful.
• A sub-question: How open are they to feedback? How likely are they to change their actions? Some actors are more resistant to feedback than others. If you’ve already seen others make the same critiques as you and haven’t seen much positive, it may not be as valuable to invest a lot of time in the critique (it may be more resource efficient to just signal boost / engage with existing critiques)

What is the status or position of the actor you are criticising?

• When an actor has outsized influence in the EA community (e.g. a major grantmaker or public intellectual) then the benefit of scrutinising their work may be high and the cost to the actor is low as they are already in an established position.
• In general it seems good to be cautious about directly criticising the work of individuals in a public space because it is hard to have sufficient context on an individual and this risks creating a hostile environment and harming discourse (see above).

Do you have time to be careful when expressing your level of confidence and extrapolating?

• Because some discourse could reduce transparency, it might be worth spending some extra time thinking about how to express your thoughts. If you’re talking about anecdotal data, it might be worth asking in the post about whether others resonate with what you are saying. (This shouldn’t prevent you from sharing your own experiences - quite the contrary - many personal experiences have sparked valuable community-wide discussions)

Further reading

If you’re interested in actually doing this, some relevant reading might be a post I wrote on suggestions for online discourse norms. There are also lots of interesting posts in the discussion norms tag. I also found Giving and receiving feedback by Max Daniel very actionable - he suggests many specific questions and gives lots of examples, and other resources are linked in the comments of that post.

1. ^{^}

   See GiveWell’s current and past mistakes pages for examples of both.

2. ^{^}

   Here are some mostly non-EA examples. The EA-adjacent development charity Evidence Action shut down one of its programs a few years ago after publicly saying it wasn’t effective enough. In academia, there is a strong norm against transparency around failure - exemplified by economist Johannes Haushofer’s CV of failures which went viral in 2016 (and this earlier 2010 call in Nature for A CV of failures). One place where failures are discussed a lot, and sometimes embraced is in the startup world. Startup culture talking about failures - the leading thinkers in this space such as Eric Ries (The Lean Startup) and Paul Graham talk a lot about failures. VCs sometimes look favourably on founders who’ve got one failed startup behind them, if they can demonstrate they’ve learnt things. Ramit Sethi, a popular blogger who writes about careers, has a failure file where he aims to have 4 failures a month. 

3. ^{^}

   For those curious, the following organisations do not (to the best of my knowledge) have mistakes pages: FTX Future Fund, Open Philanthropy, One for the World, Founders’ Pledge, Rethink Priorities, Charity Entrepreneurship, Global Priorities Institute, Future of Humanity Institute. Some of these organisations do write about their mistakes in their annual reports. Other orgs that have mistakes pages: Sentience Institute. 

4. ^{^}

   Some other posts are the Good Technology Projects' postmortem, a postmortem of a mental health app by Michael Plant, organisations discuss their learnings in retrospectives like Fish Welfare Initiative or in posts announcing decisions to shut down like Students for High Impact Charities. In the Rationalist community, there was the Arbital Postmortem. You can see more examples on the Forum postmortems and retrospectives tag, and examples from the LessWrong community in their analogous tag.

5. ^{^}

   H/T Arjun Khandelwal for most of this section! 

6. ^{^}

   H/T Aaron Gertler & Adam Gleave

7. ^{^}

   H/T Abi Olvera

8. ^{^}

   H/T Arjun Khandelwal

9. ^{^}

   H/T Ben Millwood

10. ^{^}

    Chris Leong raised this concern on the pre-announcement for the contest for critiques and red teaming