What are examples where extreme risk policies have been successfully implemented?

By Joris 🔸 @ 2022-05-16T15:37 (+32)

Hi! My name is Joris, I'm a university group organizer at PISE (the group at Erasmus University Rotterdam, the Netherlands), and I’m currently writing my thesis for my bachelor’s degree in PPE (Philosophy, Politics & Economics). I'm helping Rumtin from GCR Policy study practical policy change on x-risks and GCRs, and concretely I'm exploring the conditions and characteristics of successful policy change for extreme risks. 

 

Policy change on low-likelihood-high-impact risks is challenging to achieve, so finding and studying cases where it has been achieved can help inform the work of the x-risk studies field. The purpose of this research is to find patterns across case studies that can tell us how extreme risk policy occurs. By looking at the key characteristics of past policy efforts, we hope to provide insights on how policy for extreme risks occurs, and to provide guidance to researchers if they wish to do policy engagement.

 

I have found a number of case studies from which to draw lessons, but I want to broaden my search. I'm reaching out here to ask the following: what would be useful cases of extreme risk policy to study? These case studies should be for policies that were implemented to understand, govern, or reduce extreme risks (not simply existential or globally catastrophic risks). For example, case studies could include: NASA’s asteroid tracking program; US Defense Department’s UFO detection program; closing the hole in the ozone layer; and the joint US-Soviet smallpox and polio vaccination program[1]. 
 

We want to look at some other examples (risk- and country-agnostic) to identify common factors that led to policy change, such as individual policy leadership, precipitating events, or other shifts in societal or cultural landscape. Our hypothesis is that precipitating events, particularly small-scale versions of the risk, are the critical factor in shaping policy in that area. For example, a quick review of the literature on NASA's asteroid tracking program identifies a number of drivers for its establishment, including: Comet Shoemaker–Levy 9 hitting Jupiter in 1994, scientific consensus forming around an asteroid causing the dinosaur extinction (including with the discovery of the Chicxulub crater)[2], the releases of blockbusters Deep Impact and Armageddon[3], and the efforts of individual linked to Congress[4] 

 

If you know of any significant extreme risk-related policy successes, we would love to hear. Bonus points if you know of literature (or people) that can help us understand why and how this policy came about!

 

If you're not comfortable commenting, feel free to email me at joris [dot] pijpers [at] gmail [dot] com!

  1. ^

    https://journals.plos.org/plosntds/article?id=10.1371/journal.pntd.0005320

  2. ^

    See The Precipice, p. 69-70

  3. ^

    Carl Schulman talks about this in this 80.000 Hours podcast episode: https://80000hours.org/podcast/episodes/carl-shulman-common-sense-case-existential-risks/

  4. ^

    See https://www.boulder.swri.edu/clark/ncarhist.html

lukeprog @ 2022-05-16T17:05 (+21)

Nunn-Lugar; see quick summary here: https://www.openphilanthropy.org/blog/ai-governance-grantmaking

Marcel D @ 2022-05-17T16:10 (+16)

Setting aside whether or not such risks were actually significant, perhaps planetary protection could be an interesting example of where bureaucracies spent time and money to mitigate unknown risks from e.g., extraterrestrial contamination.

michaelB @ 2022-05-16T19:49 (+12)

The 2014 NIH moratorium on funding gain-of-function research (which was lifted in 2017)

michaelB @ 2022-05-16T19:48 (+10)

The Asilomar Conference on Recombinant DNA, which Katja Grace has a report on: https://intelligence.org/2015/06/30/new-report-the-asilomar-conference-a-case-study-in-risk-mitigation/

Yitz @ 2022-05-16T17:17 (+8)

Fixing the Ozone Layer should provide a whole host of important insights here.

jacob @ 2022-05-17T22:10 (+7)

The US decision to end their biological weapon program

The South African decision to end their nuclear weapons program and destroy their nuclear arsenal

The Nuclear Non-Proliferation Treaty (sort of; it's unclear how much of an impact it has had, but these posts [1, 2] provide some insight)

michaelB @ 2022-05-16T19:58 (+4)

Some more examples of risks which were probably not extreme*, but which elicited strong policy responses:

*I'm not really sure how you're defining "extreme risk", but the examples you gave all have meaningfully life-changing implications for >10s of millions of people. These examples are lesser in severity and/or scope, but seem like they still caused strong policy responses due to overestimated risk (though this warrants being careful about ex ante vs. ex post risk) and/or unusually high concern about the topic.

michaelB @ 2022-05-16T20:18 (+2)

Military/weapons technologies, in particular nuclear weapons, biological weapons, chemical weapons, and cyberattacks

michaelB @ 2022-05-16T20:10 (+2)

Several infectious diseases, including COVID-19, Ebola, SARS, MERS, swine flu, HIV/AIDS, etc.

michaelB @ 2022-05-16T20:08 (+2)

Gene-edited humans (see coverage of / responses to the twins modified by He Jiankui)

michaelB @ 2022-05-16T19:46 (+2)

If you want to draw useful lessons for successful risk governance from this research, it also seems pretty important to collect negative examples of the same reference class, i.e. conditions of extreme risk where policies were proposed but not enacted/enforced, or not proposed at all. E.g. (in the spirit of your example of the DoD's UFO detection program), I don't know of policy governing the risk from SETI-style attempts to contact intelligent aliens.

Are you interested only in public policies related to extreme risk, or examples from corporate governance as well? Corporate risk governance likely happens in a way that's meaningfully different from public policy, and might be relevant for applying this research to e.g. AI labs.

Joris P @ 2022-05-20T06:21 (+2)

Thanks Michael, also for the suggestions you made above! You raise good points and I would've loved to study negative examples and examples from corporate governance, but the scope of my thesis unfortunately has to be really limited - hopefully someone else can look at these later! 

Marcel D @ 2022-05-16T18:15 (+1)

I'm not exactly sure what you have in mind for the research, but I think it might be interesting to at least draw parallels or have pseudo-benchmarks with policy responses to non-existential low-probability risks, such as 9/11 (or terrorism more generally) and US mass shootings.

Joris P @ 2022-05-20T08:31 (+1)

Thanks Harrison, we're indeed looking at exactly those "policy responses to non-existential low-probability risks" as there is little material out there  on policy change regarding GC & X-risks. By 'lowering the bar' a bit to what we called 'extreme risks', we hope to include smaller, less deadly risks into our case study candidates. As such, 9/11 is indeed one to consider, thank!