Ways in which EA could fail

By michel @ 2022-10-24T23:46 (+112)

This is the first out of two posts attempting to make EA strategy discussions more productive. The second post examines EA movement course corrections and where you might disagree. 

Summary

• Following an influx of funding, media attention, and influence, the EA movement is speeding along an exciting, yet perilous, trajectory.
• A lot of the EA community’s future impact rests on this uncertain growth going well (and thereby avoiding movement collapse scenarios). 
• Yet, discussions or critiques of EA’s trajectory rarely feel action-guiding. Even when critiques propose course corrections that are tempting to agree with (e.g., EA should be bigger!), proposed course corrections to make EA more like X often don’t rigorously engage with the downsides of being more like X, or the opportunity cost of not being like Y. Proposals to make EA more like X also often leave me with only a vague understanding of what X looks like and how we get from here to X.
• In hopes of making discussions of the EA community’s trajectory more productive (and to clarify my own thinking on the matter), I will lay out a series of posts that provide an overview of: 
  • (1) Important ways in which the EA movement could fail
    (2) “Domains” in which EA could make course corrections (e.g., more cause-area tailored outreach, new professional networks and events, etc.)
    (3) Key considerations that inform course corrections (i.e., places to disagree about course corrections)
    (4) Next steps to help guide the EA movement through exciting and perilous times
• This is the first post in this series: Ways in which EA could fail. Consider it an attempt at bounding the later discussions of strategy updates. 

Ways in which EA could fail

The EA movement could collapse. Many movements before us have, and we’re not that special. But other movements, like abolitionism, have left lasting social change. In hopes of being the type of movement that does so well it doesn’t need to exist anymore, this section outlines many of the ways EA could fail. 

In this post, I’ll define “failure” as the EA/EA-adjacent ecosystem achieving substantially less impact (let's say, 30%) than it could have along some other trajectory. Note that this is a pretty broad definition. Depending on your worldview and person-affecting ethical views, failure could look more like millions or billions of people alive in the near-future suffering in ways that could have been prevented – or failure could look more like an existential catastrophe that permanently stops sentient life from achieving its full potential. 

Implicit in this definition of failure is a statement about the influence EA already has, or could grow to have: I think our ideas and resources are powerful enough to seriously influence how many beings suffer today and how many beings live beautiful lives in the future, such that there’s a massive difference between the most good a flourishing EA ecosystem could achieve (i.e., upper bound) and the good, or possibly even harm,^[1] a collapsed EA ecosystem leaves us with (i.e., lower bound). 

In order to land closer to the heights of positive impact, let’s think concretely about the worst cases that we must collectively avoid. In what ways might the EA movement fail?

I identify four clusters of failures:

1. Reputation failures: failures that result in EA losing a substantial amount of possible impact because the EA community develops a negative reputation.
2. Resource failures: failures that result in EA losing a substantial amount of possible impact because the community becomes preventably constrained by financial, human, or infrastructure resources.
3. Rigor failures: failures that result in EA losing a substantial amount of possible impact because EA enthusiasts aren’t thinking clearly about prioritizing problems and their solutions.
4. Reservation failures: failures that result in EA missing a substantial amount of possible impact because we are too risk averse, or just not ambitious enough

But note a few caveats before I elaborate on these different failure clusters:

• This taxonomy is imperfect. Some failures could fit into multiple categories, or don’t fit cleanly into any. I discuss other ways to group failures in the footnotes.^[2] 
• Causes of failures are likely to chain into each other across different clusters of failures. For example, diluted epistemic norms (which I categorize as rigor failure) could lead to risky unilateral moves that result in a PR scandal and media firestorm (which I categorize as a reputation failure). In turn, this could lead to a loss of people and funding support (which I categorize as a resource failure). Failures are a messy business – don’t let a nice 4R alliteration taxonomy fool you.   
• There are subtleties within failure modes. Some of the failures I discuss below like, for example, EA becoming politicized, could further be broken down into different consequences that plausibly have a different impact (e.g., the consequences of EA becoming disliked by left or right US political parties are probably very different). Similarly, internal disenchantment and negative press are already happening at some level, so these causes of ‘failure’ are clearly a matter of degree.
• Not all failure modes below are equally bad or equally likely. Considering both the badness and likelihood of different failures is an important next step in prioritizing which failures to course-correct away from. 
• Forgive me for vaguely using “EA.” For the sake of simplicity – and because I really am gesturing at the whole movement – I often invoke the mysterious, definitely-not-an-agent “EA” in reference to some action. But try to avoid doing this!
• EA could still be on its optimal trajectory despite some evidence of ‘failure’. For example, I expect at least some parts of the optimal version of the EA movement just aren’t everyone's vibe and some people will feel disenchanted by them. While I classify disenchantment as a failure mode below, the fact that we might still expect to see disenchantment in the optimal version of EA means evidence of the beginning of a failure doesn’t necessarily mean EA needs to course-correct away from that failure.^[3]

Reputation failures

Reputation failures result in EA losing a substantial amount of possible impact because the community develops a negative reputation, either internally, externally, or both. 

Resource failures

Resource failures result in EA losing a substantial amount of possible impact because the community becomes constrained by financial and/or human resources.

Rigor failures

Rigor failures result in EA losing a substantial amount of possible impact because EA enthusiasts aren’t thinking clearly about prioritizing problems and their solutions.

Reservation failures

Reservation failures result in EA missing a substantial amount of possible impact because we are too risk-averse, or just not ambitious enough. While reservation failures are less likely to lead to movement collapse, they are arguably more likely to increase the likelihood of an existential catastrophe – or just the preventable suffering of (m?b??t???)illions.  

Reservation failures are closely related to invisible impact loss.^[10] 

Next steps re: failure modes

• Analyze relative badness and the likelihood of failure modes to decide which to prioritize.
• Analyze how different failure modes could play out. I picture many “failures” of EA still leaving behind a core group of people committed to maximal impartial altruism. But the damage to branding, coordination, and resources might well vary depending on what caused the failures, and I expect this to have a big influence on how much any group could accomplish. 
• Identify what warning signs we might expect (or are already seeing) for each failure mode and some action threshold. I expect this action threshold to vary depending on people’s idealized vision for EAs trajectory, which will be the focus of the next posts.
• Identify more EA failure modes or debate the ones I listed: What did I miss? 
• Improve the taxonomy: Would refactoring the failure modes make them more action-guiding?
• Proposals to make cause-areas more robust to EA brand collapse. Depending on how concerned one is about the reputation of the EA brand, there may be arguments to preemptively make work in different areas (e.g., AI safety) not too coupled to EA.

Coming soon 

UPDATE: The second post on EA movement course corrections and where you might disagree is live.

The next post, or posts, will identify:

1.  “Domains” in which EA could make course corrections (e.g., more cause-area tailored outreach, new professional networks and events, etc.).
2. Key considerations that inform course corrections (i.e., places to disagree about course corrections).
3. More next steps proposals to help guide the EA movement through this exciting and perilous period of growth.

I hope to have these next sections out within the next two weeks.

1. ^{^}

   Examples of ways EA could have a net-negative impact is by directing finite altruistic talent in the wrong direction, permanently tainting ideas like doing good with an emphasis on rationality, or by discouraging others to tackle X important problem because we give the impression we're on it when we're really not. 

2. ^{^}

   Other failure mode taxonomies include:

   (1) Sequestration, Attrition, Dilution, and Distraction
   (2) Failures from pushing too hard, not pushing hard enough, or pushing in the wrong direction. 
   (3) Failures where the recognizable EA collapses before an existential catastrophe vs. failures where EA is still intact but fails to prevent an existential catastrophe 
   (4) Failures of Commission vs. omission: causing harm vs. squandering an opportunity.

3. ^{^}

    You can imagine that this makes it really difficult for CEA and other EA thought leaders to know when to course-correct upon criticism, especially when you factor in an uncertain switching cost one incurs trying to coordinate and implement a trajectory change.

4. ^{^}

    See Leaning into EA Disillusionment for more discussion 

5. ^{^}

    For more discussion, see EA considerations regarding increasing political polarization. 

6. ^{^}

    Linch accurately notes in comments that the consequences of infighting are more complicated than a mere reputation failure.

   See also: Will MacAskill’s writing on resentment and the Book How Change Happens by Leslie Crutchfield, which notes effective management of infighting as a consistent attribute of the most effective movements in recent US history (Chapter 5).

7. ^{^}

    See also: Will MacAskill’s writing on the current funding situation harming quality of thought

8. ^{^}

    See the epistemic deference forum tag for more discussion 

9. ^{^}

    Idea borrowed from this post on movement collapse scenarios

10. ^{^}

     For what it’s worth, I think reservation failures are the most overlooked cluster of failures. I think it’s just much easier to criticize ostensibly bad things EA does than all the invisible impact EA loses by being overly cautious or doing some things that turn people off. However, reservation failures are a delicate matter, because they are often – although not always – associated with downside risks that require serious consideration.

11. ^{^}

     See Will MacAskill’s writing on “Risks of Omission” for more discussion of scalability: 

    It seems to me to be more likely that we’ll fail by not being ambitious enough; by failing to take advantage of the situation we’re in, and simply not being able to use the resources we have for good ends.

12. ^{^}

     Credit to Nick Beckstead for "Bureaucrat’s curse"