Is transformative AI the biggest existential risk? Why or why not?

To clarify, is the main source of your skepticism that you don't think TAI has a particularly high chance of leading to an existential catastrophe, or are you also not sure we get TAI soon enough to matter?

Also, I think your post is asking for arguments which directly compare risks. Is there a reason you'd find this particularly compelling? If I tell you a coin has 2 sides, and Alice tells you a die has 6, it feels like you have enough information to work out that getting tails is more likely than rolling a 1, even if I've never met Alice.

The precipice is probably the best place to start if you do want direct comparisons though

Sorry, I totally meant to answer this on the AMA, but ended up missing it.

I don't have particularly strong views on which x-risk is largest, mostly because I haven't thought very much about x-risks other than AI.

(That being said, for many x-risks such as nuclear war you can get some rough bounds by noticing that they haven't occurred yet, whereas we don't have AGI yet so we can't make a similar argument. Anyone who thinks that AI poses, say, >50% x-risk could use these arguments to justify that AI x-risk is larger than most other risks.)

I do think that "all else equal", AI alignment is the most impactful object-level x-risk to work on, because:

1. The absolute level of risk is not small (and is at least comparable to other risks)
2. It's a "single problem", i.e. there's a specific technical-ish problem description and it is plausible that there's a "single solution" that entirely handles it, that we have to find. (This cashes out as higher tractability in the ITN framework, though there are also reasons to expect lower tractability.)
3. It's extremely neglected relative to most other risks.

Great power conflict is generally considered an existential risk factor, rather than an existential risk per se – it increases the chance of existential risks like bioengineered pandemics, nuclear war, and transformative AI, or lock-in of bad values (Modelling great power conflict as an existential risk factor, The Precipice chapter 7).

I can define a new existential risk factor that could be as great as all existential risks combined – the fact that our society and the general populace do not sufficiently prioritize existential risks, for example. So no, I don't think TAI is greater than all possible existential risk factors. But I think addressing this "risk" would involve thinking a lot about its impact mediated through more direct existential risks like TAI, and if TAI is the main one, then that would be a primary focus.

This passage from The Precipice may be helpful:

the threat of great-power war may (indirectly) pose a significant amount of existential risk. For example, it seems that the bulk of the existential risk last century was driven by the threat of great-power war. Consider your own estimate of how much existential risk there is over the next hundred years. How much of this would disappear if you knew that the great powers would not go to war with each other over that time? It is impossible to be precise, but I’d estimate an appreciable fraction would disappear—something like a tenth of the existential risk over that time. Since I think the existential risk over the next hundred years is about one in six, I am estimating that great power war effectively poses more than a percentage point of existential risk over the next century. This makes it a larger contributor to total existential risk than most of the specific risks we have examined.

While you should feel free to disagree with my particular estimates, I think a safe case can be made that the contribution of great-power war to existential risk is larger than the contribution of all natural risks combined. So a young person choosing their career, a philanthropist choosing their cause or a government looking to make a safer world may do better to focus on great-power war than on detecting asteroids or comets.

I think the answer depends on the timeframe you are asking over. I give some example timeframes you might want to ask the question over and plausible answers to the biggest x-risks. 

• 1-3 year: nuclear war
  Reasoning: we are not close enough to building TAI that it will happen in the next few years. Nuclear war this year seems possible.
• 4-20 years: TAI
  Reasoning: Firstly you could say we are a bit closer to TAI than to building x-risk level viruses  (very unsure about that). Secondly the TAI threat is most worrying in scenarios where it happens very quickly and we loose control (a fast risk) and the pandemic threat is most worrying in scenarios where we gradually get more and more ability to produce homebrew viruses (a slow risk).
• 21-50 years: TAI or manmade pandemics (unclear)
  Reasoning: As above TAI is less worrying if we have lots of time to work on alignment.
• 51-100 years: unknown unknown risks
  Reasoning: Imagine trying to predict the biggest x-risks today from 50 years ago. The world is changing too fast. There are so many technologies that could be transformative and potentially pose x-risk level threats. To think that the risks we think are biggest today will still be biggest in 50+ years is hubris.

I think as a community we could do more to map out the likelihood of different risks on different timeframes and to consider strategies for addressing unknown unknow risks

It's important to distinguish existential risk (x-risk) from global catastrophic risk (GCR). Nuclear war and extreme climate change, for example, are much more likely to have survivors, so are mostly GCRs rather than x-risks. Similarly with engineered pandemics - it seems like they are more likely to be survivable by some fraction of humanity, down to the relatively slow speed of spread, and the possibility of countermeasures (you are only up against human level intelligence), compared to an unaligned AGI (you are up against superintelligence which could wipe out the human race in minutes).