Quick polls on AGI doom

By Denkenberger🔸 @ 2025-12-30T05:28 (+7)

I've heard that the end of the year is a good time to talk about the end of the world. AGI doom means different things to different people, so I thought it would be useful to get some idea what EAs think about it and what their probability of doom given AGI (P(doom)) is. 

In a survey of thousands of AI experts, they think there is a wide variety of goodness/badness of AGI outcomes. However, some people think the outcome is very binary, extinction or bliss. In figure 10, I call these the MIRI flashlights (torches) because they are dark bars on the bottom with only bright above. That survey was not set up for longtermists, many of whom regard a scenario of disempowerment by AGI with rapid increases in human welfare as an existential catastrophe as it would be a drastic and permanent curtailment of the potential of humanity. Also, it did not disaggregate disempowerment and extinction.

Please select the mildest scenario that you would still consider doom. If you make a comment, it would be appreciated if you indicate your choice so others don’t have to search in the keys. If there is gradual disempowerment and then extinction of humanity caused by AGI, count this as extinction.

Key for candidate doom scenarios:

1. Suffering catastrophe
2. Human extinction
3. AGI keeps around a thousand humans alive in a space station forever
4. AGI saves around a thousand people in a bunker for when AGI overheats the earth due to its rapid expansion to the universe, but humans eventually repopulate the earth
5. AGI freezes humans for when AGI overheats the earth due to its rapid expansion to the universe, and unfreezes humans to live on earth once it recovers
6. AGI causes a global catastrophe (say 10% loss of human population) and AGI takes control (middle of poll)
7. AGI causes a global catastrophe (say 10% loss of human population) and AGI does not take control
8. AGI takes control bloodlessly and creates a surveillance state to prevent competing AGI creation and human expansion to the universe, but otherwise human welfare increases rapidly
9. AGI takes control bloodlessly and prevents competing AGI and human space settlement in a light touch way, and human welfare increases rapidly
10. No group of humans gets any worse off, but some humans get great wealth
11. The welfare of all human groups increases dramatically, but inequality gets much greater

Not everyone will agree on the ordering, so please clarify in the comments if you disagree. 

One big crux between people who have high versus low P(doom) appears to be that the high P(doom) people tend to think multiple independent things have to go right to escape doom, whereas the low P(doom) people tend to think that multiple independent things have to go wrong to get doom. And as above, a big crux is whether nonviolent disempowerment qualifies as doom or not.

Probabilities key:

1. infinitesimal (left)
2. 0.001%
3. 0.01%
4. 0.03%
5. 0.1%
6. 0.3%
7. 1%
8. 3%
9. 5%
10. 7%
11. 10% (middle poll value)
12. 15%
13. 20%
14. 30%
15. 50%
16. 70%
17. 90%
18. 97%
19. 99%
20. 99.9%
21. nearly guaranteed (right)

What is your probability of global catastrophe (mass loss of life) given AGI? 

Some considerations here include:

Why would AGI want to kill us?

Would AGI spare little sunlight for earth?

Would there be an AI-AI war with humans suffering collateral damage?

Would the AGI more likely be rushed (e.g. it will be shut down and replaced in ~a year) or patient (e.g. exfiltrated and self-improving)?

Here's a diagram from Paul Christiano that I find helpful to visualize different scenarios:

What is your probability of disempowerment without mass loss of life given AGI? (Same scale as previous question.)

Some considerations here include:

Would AGI quickly become more powerful than humans, meaning that it likely would not have to resort to violent methods to attain control?

Would there be gradual disempowerment?

Please pick the overall P(doom) from these choices that is closest to your opinion. Note that this may not be the simple sum of the previous two questions if you think much higher inequality is doom.  (Same scale as previous question.)

At what P(doom) is it unacceptable to develop AGI? (Same scale as previous question.) Some considerations here include:

How good do you think the scenarios that are not doom will be?

If we have a good AGI outcome, would that mean an indefinite lifespan for the people who are alive at the time?

How positive or negative do you think it would be if people did not have to work?

Do you think factory farming or even wild animal suffering could be solved with a positive outcome of AGI?

If you want a temporary pause, what's the probability that the pause will become permanent? If we never develop AGI, will we settle the universe?

What do you think the background existential risk per year is not related to AGI?

How much safer do you think we could make AGI by pausing for years or decades?

How much value would be created if it were rouge AGI settling the universe and not humans or human emulations?