8 possible high-level goals for work on nuclear risk

By MichaelA🔸 @ 2022-03-29T06:30 (+46)

Summary

For people aiming to do the most good they can, what are the possible high-level goals for working on risks posed by nuclear weapons? Answers to this question could inform how much to invest in the nuclear risk space, what cruxes^[1] we should investigate to determine how much and in what ways to work in this space, and what specific work we should do in this space.

I see eight main candidate high-level goals, in three categories:

1. Longtermist & nuclear-focused: Reducing nuclear risk’s contribution to long-term future harms
   1. Direct: Reducing relatively direct, foreseeable paths from nuclear risk to long-term harms
   2. Indirect: Reducing more indirect/vague/hard-to-foresee paths from nuclear risk to long-term harms
2. Longtermist & not nuclear-focused: Gaining indirect benefits for other EA/longtermist goals
   1. Career capital: Individuals building their career capital (knowledge, skills, credibility, and connections) to help them later work on other topics
   2. Movement strengthening: Building movement-level knowledge, credibility, connections, etc. that pay off for work on other topics
   3. Translatable knowledge: Developing research outputs and knowledge that are directly useful for other topics
   4. Movement growth: Improving the EA movement’s recruitment and retention (either narrowly - i.e. among expert communities - or broadly) by being seen to care about nuclear risks and/or by not being seen as dismissive of nuclear risks
   5. Epistemic hygiene: Improving EAs’^[2] “epistemic hygiene” by correcting/supplanting flawed EA work/views
3. Neartermist & nuclear-focused: Reducing neartermist harms from nuclear weapons

I expect we should put nontrivial weight on each of those high-level goals. But my current, pretty unstable view is that I’d prioritize them in the following rank order: 

1. longtermist & nuclear-focused, both direct and indirect
2. career capital and movement strengthening
3. translatable knowledge and movement growth (especially the “narrow version” of the movement growth goal)
4. neartermist & nuclear-focused
5. epistemic hygiene

(Note: Each of the sections below should make sense by itself, so feel free to read only those that are of interest.)

Why did I write this?

With respect to the nuclear risk space, I think people in the effective altruism community are currently unsure about even what very high-level goals / theories of change / rationales we should focus on (let alone what intermediate goals, strategies, and policies to pursue). More specifically, I think that: 

• EAs who’ve done some research, grantmaking, or other work in this area often disagree about or notice themselves feeling confused about what our high-level goals should be, or seem to me to be overlooking some plausible candidate goals.
• EAs who haven’t done work in this area are often overly focused on just one or a few plausible candidate goals without having considered others, and/or often don’t clearly recognise distinctions and differing implications between the various plausible goals.

I felt that it would be useful to collect, distinguish between, and flesh out some possible high-level goals, as a step toward gaining more clarity on:^[3]

1. How much to invest in the nuclear risk space
   • E.g., if we’re mostly focused on using nuclear risk as a “training ground” for governance of other risky technologies, perhaps we should also or instead focus on cybersecurity or other international relations topics?
2. What cruxes we should investigate to determine how much and in what ways to work in this space
   • E.g., is the crux how likely it is that nuclear winter could cause existential catastrophe and how best to prevent such extreme scenarios, or whether and how we can substantially and visibly reduce the chance of nuclear war in general?
3. What specific work we should do in this space

Epistemic status

I drafted this post in ~3 hours in late 2021. In early 2022, Will Aldred (a collaborator) and I spent a few hours editing it.^[4] I intend this as basically just a starting point; perhaps other goals could be added, and definitely more could be said about the implications of, arguments for, and against focusing on each of these goals. 

I expect most of what this post says will be relatively obvious to some readers, but not all of it to all readers, and having it actually written down seems useful. 

Please let me know if you’re aware of existing writings on roughly this topic!

1. Reducing nuclear risk’s contribution to long-term future harms

(Meaning both existential catastrophes and other negative trajectory changes.)

1a. Reducing relatively direct, foreseeable paths from nuclear risk to long-term harms

• Relevant paths include:
  • A nuclear winter that kills almost literally everyone, with existential catastrophe following shortly afterwards (e.g. because something else finishes the population off or we get locked in to a worse system of values and political systems)
  • Perhaps the use of very large numbers of very large salted bomb type weapons / radiological weapons
    • Caveat: I haven’t looked into the plausibility of this as a relatively direct path to long-term harms.
    • See some notes here.
• For an attempt to zoom in on lower-level goals/actions that have this high-level goal, see Shallow review of approaches to reducing risks from nuclear weapons.
• To the extent we’re focused on this, we might be especially concerned to reduce the chance of scenarios that are very high on some function involving number of warheads, median yield, and mean population density of targets, or perhaps scenarios involving things like many huge radiological weapons.

An unusual subtype of this goal: Reducing the risk nuclear weapons detonations/threats being used as a tool that helps enable AI takeover

• This is less commonly discussed, I’ve hardly thought about it, and I’m unsure whether it warrants attention.
• Is power-seeking AI an existential risk? [draft] notes that one (of many) possible mechanisms for that existential risk scenario is “Destructive capacity. Ultimately, one salient route to disempowering humans would be widespread destruction, coercion, and even extinction; and the threats in this vein could play a key role in a [power-seeking]-misaligned AI system’s pursuit of other ends. Possible mechanisms here include: biological/chemical/nuclear weapons; [...]” (emphasis added).
• I expect that what’s best for reducing this sort of risk would be quite different from what’s best for reducing other pathways from nuclear risk to long-term harms.
  • E.g., for this sort of risk, it seems (approximately?) useless to change the likelihood that political and military leaders would choose to detonate many nuclear weapons and to do so against cities. (Conversely, it seems like it would potentially be useful to change force sizes and structures such that it’s impossible or harder to detonate many nuclear weapons against cities.)
• If we are worried about this sort of risk, then…
  • …it seems plausible/likely that we should focus on interventions more related to the AI than to nuclear weapons (e.g., technical AI safety work to prevent the relevant forms of misalignment)
  • …it also seems plausible/likely that we should focus on one of the many other tools a misaligned AI could use for takeover (e.g., bioweapons, hacking)
  • …but it seems plausible that some nuclear-risk-like work would be warranted, as part of a defense-in-depth/portfolio approach, at least once our resource pool grows sufficiently large and we’ve plucked the lower hanging fruit
• (I have some additional rough notes on this that I could potentially share on request.)

1b. Reducing more indirect/vague/hard-to-foresee paths from nuclear risk to long-term harms

• Relevant paths include:
  • The kind of paths discussed in The long-term significance of reducing global catastrophic risks
  • Nuclear weapons/risk/war serving as an existential risk factor
• To the extent we’re focused on this, we might spread our resources and attention across a relatively wide set of scenarios.
  • E.g., for these risks, it’s not like a scenario involving 10 detonations against cities or 400 against silos are basically not a concern at all, since those things could still have effects like substantially disrupting geopolitics.
  • So, given that I do think we should focus somewhat on these broader risk pathways, I think I would push back a bit against arguments that we should focus almost entirely on questions like “How many warheads would it take to kill basically everyone?” and on interventions that prevent the chance of seemingly highly unlikely extreme scenarios.
  • ...but only a bit. And I think it’d be useful to flesh out this type of risk pathway some more.
    • E.g., maybe the concern is mostly about massively changing great power relations and geopolitics? If so, perhaps strikes against powerful nations’ capital cities would be especially problematic and hence especially important to prevent?
    • E.g., maybe the concern is mostly about shrinking or disrupting the EA movement and its work, since that in turn presumably raises existential risk and other issues? If so, perhaps strikes against cities with large numbers of EA orgs or people would be especially problematic and hence especially important to prevent?
    • E.g., maybe the concern is mostly about how nuclear conflict could trigger riskier development and deployment of bioweapons or AI? That might then have implications for what to prevent - though I haven’t thought about what the implications would be.

2. Gaining indirect benefits for other EA/longtermist goals

General thoughts on this category

• I’d also include in this category “preventing indirect harms to other EA/longtermist goals”.
• The distinction between this category and the above category is analogous to the EA community or specific members gaining career capital vs having a direct impact already.
• Two people told me that their EA-aligned orgs saw this goal as a major argument (though not the only one) for them doing nuclear risk work.^[5]
• I currently think this category of ToCs is important, but I also worry we might overestimate its importance due to motivated reasoning / privileging the hypothesis, since we were already interested in nuclear risk for other reasons.
  • (See also Beware surprising and suspicious convergence.)
  • Specifically, I’m worried we might (a) overestimate the benefits of nuclear risk work for this purpose and/or (b) fail to consider alternative options for getting these benefits.
    • I’m somewhat more concerned about the second of those possibilities.
    • For example, I worry that we might get excited about nuclear risk as a “training ground” without sufficiently considering alternative training grounds like cybersecurity, near-term/low-stakes AI issues, emerging tech policy, international relations / national security more broadly, and maybe climate change (e.g., because that involves difficult coordination problems and major externalities).
    • (But I know some people have also thought about those alternatives or are pursuing them. This is just a tentative, vague concern, and it’d be better if someone looked into this and made a more quantitative claim.)

2a. Individuals building their career capital to help them later work on other topics 

• The “other topics” in question might typically be AI, but sometimes also bio or other things.
• It seems to me useful to separate this into (a) building knowledge and skills and (b) building credibility^[6] and connections.
• In both cases, we can ask:
  • What specific career capital do we want these individuals to get?
  • Is working on nuclear risk better for getting that specific capital than working on other topics (either the topic the individual will ultimately focus on or some other “training ground”)?

2b. Building movement-level knowledge, credibility, connections, etc., that pay off for work on other topics

• Here’s one way this could work: If some EAs / EA-funded people have done or are doing nuclear risk work, then other EAs could draw on their:
  • expertise (e.g., discussing ideas about AI risk pathways where knowledge of nuclear weapons history or policymaking is relevant),
  • connections (e.g., ask them for an intro to a senior national security policy advisor), or
  • credibility (e.g., having them signal-boost a policy proposal)
• Here’s another way this could work: If people know that EAs / EA-funded people have done or are doing nuclear risk work, this could itself help other EA actors (e.g., whoever funded or employed those EAs) or EA as a whole be perceived as credible and have people accept requests to meet (or whatever).
  • I’ve heard a few EAs discuss the potential benefits of being perceived as “the adults in the room”, and I believe this is what they have in mind with that
  • This benefit may be especially pronounced if (a) EA actors have a major influence on the nuclear risk field as a whole, or on a large and distinguishable chunk of it, and (b) people who we’d like to see us as credible are aware of that and appreciate that.
    • E.g., if policymakers notice a batch of think tanks whose work seems less alarmist, more strategically aware, and more useful than the work of many other think tanks, and they notice that those think tanks were all funded by an EA funder.
• We could also frame this as an attempt to counterbalance the possibility that work on nuclear risk or other topics that’s by or funded by EAs could harm other EA actors’ credibility and connections.
  • E.g., perhaps messaging such as the Future of Life Institute’s “Slaughterbots” video, and/or some EAs’ statements that nuclear war would be extremely catastrophic and nuclear arsenals need to be massively shrunk, have lost or could lose EA(s) credibility with important national security people?^[7] And if so, perhaps we need to add some other high-visibility, high-credibility work to distract from that or repair our reputation?

2c. Developing research outputs and knowledge that are directly useful for other topics 

• E.g., historical research on the Baruch plan, nuclear weapons treaties, and nuclear arms races may provide useful insights regarding potential governance and race dynamics with respect to AI risk and biorisk issues.
• E.g., international relations research on present-day nuclear risk strategy and policymaking may provide useful insights for AI and bio issues.
• If this is the primary goal of some nuclear risk work, we would probably want the outputs to be created and written with this goal explicitly in mind. And we might think of funding that as AI or bio grantmaking rather than nuclear risk grantmaking.
• But we could plausibly have this instead as a secondary reason to fund some project that also serves one of the other ToCs discussed in this doc. And if we expect this sort of benefit to crop up often in a particular area, that could push in favor of investing somewhat more in that area than we otherwise would.

2d. Improving the EA movement’s recruitment and retention

General thoughts on this goal

• This would be via EAs working on or discussing nuclear risk and that leading to us (a) being seen to care about nuclear risk and/or (b) not being seen as dismissive of nuclear risk.
• I think we could split this into a narrow version - involving improving recruitment/retention of a relatively some number of people with some nuclear risk expertise - and a broad version - involving a fairly large number of fairly junior or not-nuclear-specialized people.
• It might be fruitful to try to figure out whether any EAs who seem to be on track for careers with high expected impact got into EA in part because of something at the intersection of EA and nuclear risk, or think they would’ve got in earlier if there was more EA nuclear risk stuff, or think they nearly “bounced off” EA due to EA not having enough nuclear risk stuff.^[8]
• If this (especially the broad version) is our focus, we might want to prioritize relatively visible work that looks clearly helpful and impressive?
  • Or maybe not - maybe the people it’d be most valuable to attract to the EA community are best captured by basically doing what’s actually best for other reasons anyway?
• Personally, I’m wary of people doing nuclear risk things primarily for the broad version of this goal, for reasons including the risk that it could end up being perceived as - or perhaps genuinely being - somewhat underhanded or bait-and-switch-like. But it seems ok if it’s just an additional goal behind some effort, and if we’d feel comfortable publicly telling people that (rather than it being something we’re hiding).

Narrow version

• E.g., maybe the more EA is involved in and cares about nuclear risk issues, or the more that that is known, the more likely we are to end up recruiting into EA researchers, grantmakers, policymakers, or policy advisors who have specialized to some extent for nuclear risk issues?
  • And then maybe they could do more impactful work than they would’ve otherwise, either on nuclear risk or on other topics, due to now being part of the EA movement?
  • And/or maybe they can help other EAs do more impactful work, as per possible high-level goal 2b (“Building movement-level knowledge, credibility, connections, etc., that pay off for work on other topics”)
• I guess here it might not just be about being “seen to care”, but rather also actually funding things and/or collaborating with people such that those people start interacting with the EA community?
  • A question that might be interested in this context: Have there been many cases of grantees coming to interact more with the EA community as a result of receiving grants from EA funders, even when this was not required by their grants?

Broad version

• The argument for this goal would be analogous to arguments sometimes made that (a) the EA community can seem dismissive of climate change concerns and (b) that might lead to some people bouncing off EA or just not noticing and getting excited about joining EA.
• It’d also be analogous to arguments sometimes made that EA’s global health and development work helps bring people into EA as a whole, and then some of those people end up in longtermism, such that ultimately we have more or similarly many longtermists as we would if all outreach focused on longtermism.
• But I’m somewhat skeptical of such arguments, because:
  • I haven’t tried to evaluate those arguments on their own terms.
  • I haven’t tried to think about how strong the analogy is.
    • One disanalogy is that I think far more people care a lot about climate than about nuclear risk, which presumably reduces the movement building benefits from doing nuclear risk work relative to those from doing climate change work.
  • I haven’t tried to think about the marginal returns curve for additional signals (of various types) of EA caring about nuclear risk, and where we already are on that curve.
    • E.g., maybe it’s just really important that (as is already the case) nuclear risk topics are discussed in an 80,000 Hours problem profile and occasional 80,000 interviews, EA Global and SERI conference talks, and EA Forum posts, and anything beyond that adds little?
  • I haven’t tried to think about things that the above arguments might be overlooking.
    • In particular, how much the value of these additional community members compares to community members we could gain in other ways^[9], and possible backfire effects from doing nuclear risk work for the movement growth benefits.
    • E.g., maybe doing nuclear risk stuff with this as a substantial part of the motivation will itself turn some particularly useful people off (e.g., due to seeming unprincipled, wasteful, or a distraction from more directly useful work)?
    • E.g., maybe that’ll just actually cause us to build bad habits of focusing on convoluted power-seeking strategies rather than saying precisely what we really think and trying to more directly improve the world?

2e. Improving EAs’ “epistemic hygiene” by correcting/supplanting flawed EA work/views

• EA and EA-adjacent communities have made various mistakes in their thinking and communications about nuclear risk. Correcting these things, or superseding them with better work and statements, may be useful not only for benefits like credibility (see goal 2b), but also for our community’s actual epistemics. E.g.:
  • This could usefully highlight the general point that EAs (especially generalists producing things quickly) can get things wrong, especially in well-established, complex, “strategic” domains.
  • It could be useful to point out specific mistakes and what sort of patterns may have driven them, since those patterns may also cause mistaken thinking in other areas. Examples of possible drivers include:
    • Our community somewhat selecting for people who believe risks are high
    • Our community often being idealistic and internationalist and so maybe overlooking or underweighting some key aspects of statecraft and strategy
• An example of an output that was intended to be somewhat optimized for this goal is 9 mistakes to avoid when thinking about nuclear risk.
• (But note that I haven’t spent much time thinking about what specific mistakes EAs have made, how often, what they suggest, how useful correcting them is, etc.)

3. Reducing neartermist harms from nuclear weapons

• I place a small but nontrivial weight on a neartermist, human-focused worldview, and I think I should pay a bit of attention to that when making decisions. And I think it’s plausible that nuclear risk should be one of the top priorities from a neartermist, human-focused worldview. And I think those two things together deserve some attention when thinking about whether to do nuclear risk work.
• In his 80,000 Hours interview, Carl Shulman indicated a similar view (though he seemed  to me to put more weight on that view than I do).
• But:
  • I place only a fairly small amount of weight on a neartermist, human-focused worldview.
  • I haven’t really tried to analyze whether nuclear risk should be one of the top priorities from a neartermist perspective, and it doesn’t seem to me high-priority to do that analysis.
    • But see a research proposal along those lines here.
• The more weight we place on this goal, probably the less we’d focus on very unlikely but very extreme scenarios (since badness scales roughly linearly in fatality numbers for neartermists, whereas for longtermists I think there’s a larger gap in badness between smaller- and medium-scale and extremely-large-scale nuclear scenarios).

Conclusion

I try to keep my bottom lines up front, so please just see the Summary and “Why did I write this?”!

Acknowledgements 

My work on this post was supported by Rethink Priorities. However, I ended up pivoting away from nuclear risk research before properly finishing the various posts I was writing, so I ended up publishing this in a personal capacity and without having time to ensure it reached Rethink Priorities’ usual quality standards. 

I’m very grateful to Will Aldred for a heroic bout of editing work to ensure this and other rough drafts finally made it to publication. I’m also grateful to Avital Balwit, Damon Binder, Fin Moorhouse, Lukas Finnveden, and Spencer Becker-Kahn for feedback on an earlier draft. Mistakes are my own.

1. ^{^}

    I.e., crucial questions or key points of disagreement between people. See also Double-Crux. 

2. ^{^}

   In this post, I use “EAs” as a shorthand for “members of the EA community”, though I acknowledge that some such people wouldn’t use that label for themselves.

3. ^{^}

   I see this as mostly just a specific case of the general claim that people will typically achieve their goals better if they have more clarity on what their goals are and what that implies, and they develop theories of change and strategies with that explicitly in mind. 

4. ^{^}

   We didn’t try to think about whether the 2022 Russian invasion of Ukraine should cause me to shift any of the views I expressed in this post, except in that I added in one place the following point: “E.g., maybe the concern is mostly about shrinking or disrupting the EA movement and its work, since that in turn presumably raises existential risk and other issues? If so, perhaps strikes against cities with large numbers of EA orgs or people would be especially problematic and hence especially important to prevent?”

   We also didn’t try to think about whether the New Nuclear Security Grantmaking Programme at Longview Philanthropy should cause me to shift any views expressed in this post, but I’d guess it wouldn’t.

5. ^{^}

   Here are my paraphrased notes on what one of these people said: 

   “Also, in [org’s] experience, nuclear war seems to be a topic that presents compelling engagement opportunities. And those opportunities have a value that goes beyond just nuclear war.

   • This area is quite amenable to [org] getting high-level policymaker attention
     • Nuclear war has always been something that gets top level policymaker attention
     • In contrast, for climate change, it’s too crowded [or something like that - I missed this bit]
     • [...]
     • It’s relatively easy to get to the forefront of the field for nuclear risk work
   • And then you can also leverage those connections for other purposes
   • And then you have a good space to talk about the global catastrophic risk framing in general
   • Also, having skill at understanding how nuclear security works is a useful intellectual background which is also applicable to other risk areas
   • [...] [This person] might even recommend that people who want to work on AI and international security start off by talking about the AI and nuclear intersection
     • That intersection is currently perceived as more credible”

   See also these thoughts from Seth Baum.

6. ^{^}

   I say “credibility” rather than “credentials” because I don’t just mean things like university degrees, but also work experience, a writing portfolio, good references, the ability to speak fluently on a given topic, etc.

7. ^{^}

   See Baum for a somewhat similar claim about “Slaughterbots” specifically. 

8. ^{^}

   It seems possible Open Phil have relevant data from their ​​Open Phil EA/LT Survey 2020, and/or that data on this could be gathered using approaches somewhat similar to that survey.

9. ^{^}

   This connects to the topic of the Value of movement growth.