Is rushing friendly AI our best hope of preventing “grey goo” nanotech doom?

I agree that this illustrates a counterpoint to longtermism-style arguments that is underappreciated.

As someone who believes that there are valid reasons to be concerned about the effects advanced AI systems will have and therefore that general "AI risk" ideas contain important insights and are worthy of consideration, I will offer my perspective on why this post aptly demonstrates an important point.

I think there is something of a pattern in discussions around AI risk that conflate formal, high reliability methods with less formal conceptual arguments that have some similarity to the more formal methods. This causes AI risk advocates to have an inaccurate impression of how compelling these arguments will be to people who are more skeptical. I think AI risk advocates sometimes implicitly carry over some of the high reliability and confidence of formal methods to the less formal conceptual arguments, and as a result can end up surprised and/or frustrated when skeptics don't find these arguments to be as persuasive or warrant as a high a level of confidence as AI risk advocates sometimes have.

This post effectively demonstrates this dynamic in two areas where I have also noted this myself in the past: track prediction track records and high impact/low probability reasoning.

As an example of the prediction track record case, consider this from an interview with Will MacAskill on 80,000 hours:

And since 2017 when I wrote that, I’ve kind of informally just been testing — I wish I had done it more formally now — but informally just seeing which one of these two perspectives [inside vs outside view] are making the better predictions about the world. And I do just think that that inside view perspective, in particular from a certain number of people within this kind of community, just has consistently had the right answer.

Formally tracking predictions or returns from bets that are made by members of a specific community and showing they are often correct/realize high returns would indeed be a compelling reason to give those views serious weight.

However, it is much more difficult to know how much to credit this kind of reasoning when the testing is informal. As an example, you can have a cherry-picking or selective memory issue. If AI risk inside view advocates often remember or credit flashy cases where members of a community made a good prediction or bet, but don't similarly recall inaccurate predictions, then this informal testing may not be as compelling, and skeptics are likely to be justifiably more suspicious of this possibility compared to people who already find arguments for AI risk convincing.

As an example of the high impact/low probability reasoning case, consider this post by Richard Chappell:

Even just a 1% chance of extremely high stakes is sufficient to establish high stakes in expectation. So we should not feel assured of low stakes even if a highly credible model—warranting 99% credence—entails low stakes. It hardly matters at all how many credible models entail low stakes. What matters is whether any credible model entails extremely high stakes. If one does—while warranting just 1% credence—then we have established high stakes in expectation, no matter what the remaining 99% of credibility-weighted models imply (unless one inverts the high stakes in a way that cancels out the other high-stakes possibility).

David Thorstad provides some counterargument in this post. Commenting on Thorstad's article on the EA forum, Chappell says this: 

Saying that my "primary argumentative move is to assign nontrivial probabilities without substantial new evidence" is poor reading comprehension on Thorstad's part. Actually, my primary argumentative move was explaining how expected value works. The numbers are illustrative, and suffice for anyone who happens to share my priors (or something close enough). Obviously, I'm not in that post trying to persuade someone who instead thinks the correct probability to assign is negligible. Thorstad is just radically misreading what my post is arguing.

My reading of this exchange is that it demonstrates the formal/informal conflation that I claim exists in these types of discussions. To my mind, the "explaining how expected value works" part suggests an implicit believe that the underlying argument carries the strength and confidence approaching that of a mathematical proof. Although the argument itself is conceptual, it experiences some amount of spillover of reliability/confidence because the concepts involved are mathematical/formal, even though the argument itself is not.

I think this dynamic can cause AI risk advocates to overestimate how convincing skeptics will (or perhaps should) find these arguments. It seems to me like this often leads to acrimony and frustration on both sides. My preferred approach to arguing for AI risk would acknowledge some of the ambiguity/uncertainty and also focus on a different set of concepts than those that often have the focus in discussions about AI risk. 

Executive summary: The author reviews early transhumanist arguments that rushing to build friendly AI could prevent nanotech “grey goo” extinction, and concludes—largely by reductio—that expected value reasoning combined with speculative probabilities can be used to justify arbitrarily extreme funding demands without reliable grounding.

Key points:

1. Eliezer Yudkowsky, Nick Bostrom, Ray Kurzweil, and Ben Goertzel argued that aligned AGI should be developed as quickly as possible to defend against catastrophic nanotechnology risks such as self-replicating “grey goo.”
2. Yudkowsky made concrete forecasts around 1999–2000, assigning a 70%+ extinction risk from nanotechnology and predicting friendly AI within roughly 5–20 years, contingent on funding.
3. Bostrom argued that superintelligence is uniquely valuable as a defensive technology because it could shorten the vulnerability window between dangerous nanotech and effective countermeasures.
4. The post applies expected value reasoning to argue that even astronomically small probabilities of preventing extinction can dominate moral calculations when multiplied by extremely large numbers of potential future lives.
5. Using GiveWell-style cost-effectiveness estimates, the author shows how this logic can imply spending quadrillions of dollars—or even infinite resources—on rushing friendly AI development.
6. The author illustrates the implausibility of this reasoning by humorously proposing that, given sufficiently small but nonzero probabilities, funders should rationally support the author’s own friendly AI project.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

I thought Altman and the Amodeis had already altruistically devoted their lives to saving us from grey goo. Since they're going to do this before 2027 you may already be too late

Peter Thiel wants to know if your AI can be unfriendly enough to make a weapon out of it.

Yarrow Bouchard 🔸 @ 2026-01-16T21:10 (+3)

2027 is far too conservative. You can clearly see in this graph from the scientific report "Situational Awareness" by Leopold Aschenbrenner that AGI was deployed in 2023: 

I believe that GPT-5 is already an ASI, but we will need a super-ASI to defend against bad nanobots. The first super-ASI will be Gronk 5, which Elon Musk says is coming out in 2 months. 

Oh man, I remember the days when Eliezer still called it Friendly and Unfriendly AI. I actually used one of those terms in a question when I was at a Q&A after a tutorial by the then less famous Yoshua Bengio at the 27th Canadian Conference on AI in 2014. He jokingly replied by asking if I was a journalist, before giving a more serious answer saying we were so far away from having to worry about that kind of thing (AI models back then were much more primitive, it was hard to imagine an object recognizer being dangerous). Fun times.

I appreciate the irony and see the value in this, but I'm afraid that you're going to be downvoted into oblivion because of your last paragraph.