Pausing AI Developments Isn't Enough. We Need to Shut it All Down by Eliezer Yudkowsky

In light of this discussion about whether people would find this article alienating, I sent it to four very smart/reasonable friends who aren't involved in EA, don't work on AI, and don't live in the Bay Area (definitely not representative TIME readers, but maybe representative of the kind of people EAs want to reach). Given I don't work on AI/have only ever discussed AI risk with one of them, I don't think social desirability bias played much of a role. I also ran this comment by them after we discussed. Here's a summary of their reactions:

Friend 1: Says it's hard for them to understand why AI would want to kill everyone, but acknowledges that experts know much more about this than they do and takes seriously that experts believe this is a real possibility. Given this, they think it makes sense to err on the side of caution and drastically slow down AI development to get the right safety measures in place.

Friend 2: Says it's intuitive that AI being super powerful, not well understood, and rapidly developing is a dangerous combination. Given this, they think it makes sense to implement safeguards. But they found the article overwrought, especially given missing links in the argument (e.g., they think it's unclear whether/why AI would want our atoms, given immense uncertainty about what AI would want; compared their initial reaction to this argument to their initial reaction to Descartes' ontological argument).

Friend 3: Says they find this article hard to argue with, especially because they recognize how little they know on the topic relative to EY; compared themselves disagreeing with it to anti-vaxxers arguing with virologists. Given the uncertainty about risks, they think it's pretty obvious we ought to slow down.

Friend 4: Says EY knows vastly more about this issue than they do, but finds the tone of the article a little over the top, given missing links. Remains optimistic AI will make the world better, but recognizes possible optimism bias. Generally agrees there should be more safeguards in place, especially given there are ~none.

Anyways, I would encourage others to add their own anecdata to the mix, so we can get a bit more grounded on how people interpret articles like this one, since this seems important to understand and we can do better than just speculate.

It made it to the White House Press Briefing. This clip is like something straight out of the film Don't Look Up. Really hope that the ending is better (i.e. the warning is actually heeded).

When a very prominent member of the community is calling for governments to pre-commit to pre-emptive military strikes against countries allowing the construction of powerful AI in the relatively near-term, including against nuclear powers*, it's really time for people to actually take seriously the stuff about rejecting naive utilitarianism where you do crazy-sounding stuff if a quick expected value calcualtion makes it look maximizing. 

*At least I assume that's what he means by being prepared to risk a higher chance of nuclear war.

Very hard hitting and emotional. I'm feeling increasingly like I did in February/March 2020, pre-lockdown. Full on broke down to tears after reading this. Shut it all down.

Strong agree, hope this gets into the print version (if it hasn't already).

Here’s a comment I shared on my LessWrong shortform.

——

I’m still thinking this through, but I am deeply concerned about Eliezer’s new article for a combination of reasons:

• I don’t think it will work.
• Given that it won’t work, I expect we lose credibility and it now becomes much harder to work with people who were sympathetic to alignment, but still wanted to use AI to improve the world.
• I am not convinced as he is about doom and I am not as cynical about the main orgs as he is.

In the end, I expect this will just alienate people. And stuff like this concerns me.

I think it’s possible that the most memetically powerful approach will be to accelerate alignment rather than suggesting long-term bans or effectively antagonizing all AI use.

I'm not particularly well informed about current EA discourse on AI alignment, but I imagine that two possible strategies are

1. accelerating alignment research and staying friendly with the big AI companies
2. getting governments to slow AI development in a worldwide-coordinated way, even if this angers people at AI companies.

Yudkowsky's article helps push on the latter approach. Making the public and governments more worried about AI risk does seem to me the most plausible way of slowing it down. If more people in the national-security community worry about AI risks, there could be a lot more attention to these issues, as well as the possibility of policies like limiting total computing power for AI training that only governments could pull off.

I expect a lot of AI developers would be angry about getting the public and governments more alarmed, but if the effort to raise alarm works well enough, then the AI developers will have to comply. OTOH, there's also a possible "boy who cried wolf" situation in which AI progress continues, nothing that bad happens for a few years, and then people assume the doomsayers were overreacting -- making it harder to ring alarm bells the next time.

In any social policy battle (climate change, racial justice, animal rights) there will be people who believe that extreme actions are necessary. It's perhaps  unusual on the Al front that one of the highest profile experts is on that extreme, but it's still not an unusual situation. A couple of points in favour of this message having a net positive effect.

1. I don't buy the debate that extreme arguments alienate people about the cause in general. This is a common assumption, but the little evidence we have suggests that extreme actions or talk might actually both increase visibility of the cause and increase support to more moderate groups. Anecdotally on the AI front @lilly seems to be seeing something similar too.
    
2. On a rational front, if he is this sure of doom,  his practical solution seems to make the most sense. It shows intellectual integrity. We can't expect someone to have a pdoom of 99% given the status quo, then just suggest better alignment strategies. From a scout mindset perspective, we need to put ourselves in the 99% doom shoes before dismissing this opinion as irrational, even if we strongly disagree with his pdoom.
    
3. (Related to 1), I feel like AI risk is still perhaps at the "Any publicity is good publicity" stage as many people are still completely unaware of it. Anything a bit wild like this which attracts more attention and debate is likely to be good. Within a few months/years this may change though as AI risk becomes truly mainstream. Outside tech bubbles it certainly isn't yet.

Archive link

(More of a meta point somewhat responding to some other comments.)

It currently seems unlikely there will be a unified AI risk public communication strategy. AI risk is an issue that affects everyone, and many people are going to weigh in on it. That includes both people who are regulars on this forum and people who have never heard of it.

I imagine many people will not be moved by Yudkowsky's op ed, and others will be. People who think AI x-risk is an important issue but who still disagree with Yudkowsky will have their own public writing that may be partially contradictory. Of course people should continue to talk to each other about their views, in public and in private, but I don't expect that to produce "message discipline" (nor should it).

The number of people concerned about AI x-risk is going to get large enough (and arguably already is) that credibility will become highly unevenly distributed among those concerned about AI risk. Some people may think that Yudkowsky lacks credibility, or that his op ed damages it, but that needn't damage the credibility of everyone who is concerned about the risks. Back when there were only a few major news articles on the subject, that might have been more true, but it's not anymore. Now everyone from Geoffrey Hinton to Gary Marcus (somehow) to Elon Musk to Yuval Noah Harari are talking about the risks. While it's possible everyone could be lumped together as "the AI x-risk people," at this point, I think that's a diminishing possibility.

I hope that this article sends the signals that pausing the development of the largest AI-models is good, informing society about AGI xrisk is good, and that we should find a coordination method (regulation) to make sure we can effectively stop training models that are too capable.

What I think we should do now is:

1) Write good hardware regulation policy proposals that could reliably pause the development towards AGI.
2) Campaign publicly to get the best proposal implemented, first in the US and then internationally.

This could be a path to victory.

I appreciate Eliezer's honesty and consistency in what he is is calling for. This approach makes sense if you believe, as Eliezer does, that p(doom | business as usual)>99%. Then it is worth massively increasing the risk of a nuclear war. If you believe, as I do and as most AI experts do, that p(doom | business as usual) <20%, this plan is absolutely insane. 

This line of thinking is becoming more and more common in EA. It is going to get us all killed if it has any traction. No, the U.S. should not be willing to bomb Chinese data centers and risk a global nuclear war. No, repeatedly bombing China for pursing something that is a central goal of the CCP that has dangers that are completely illegible to 90% of the population is not a small, incremental risk of nuclear war on the scale of aiding Ukraine as some other commenters are suggesting. This is insane. 

By all means, I support efforts for international treaties. Bombing Chinese data centers is suicidal and we all know it. 

I say this all as someone who is genuinely frightened of AGI. It might well kill us, but not as quickly or surely as implementing this strategy will.

Edited to reflect that upon further thought, I probably do not support bombing the data centers of less powerful countries either.

We can still have this ending (cf. Terminator analogies): 

Yudkowsky's suggestions seem entirely appropriate if you truly believe, like him, that AI x-risk is probability ~100%. 

However, that belief is absurdly high, based on unproven and unlikely assumptions, like that an AI could build nanofactories by ordering proteins to be mixed over email. 

In the actual world, where the probability of extinction is signficantly less than 100%, are these proposals valuable? It seems like they will just get everyone else labelled luddites and fearmongerers, especially if years and decades go by with no apocalypse in sight. 

This proposal seems to have become extremely polarizing, more so and for different reasons than I would have expected after first reading this. I am more on the “this is pretty fine” side of the spectrum, and think some of the reasons it has been controversial are sort of superficial. Given this though, I want to steelman the other side (I know Yudkowsky doesn’t like steelmanning, too bad, I do), with a few things that are plausibly bad about it that I don’t think are superficial or misreadings, as well as some start of my reasons for worrying less about them:

1. “While it’s true that this isn’t ‘the same as’ calling for outright violence, if we are at least a little bit on Orwell’s side on political violence, surely the position that significantly risking nuclear war is worse than a few terrorists bombing a GPU center seems quite silly. If he supports the former but not the latter, that is quite an extreme position!”:

I’m sympathetic to this, in no small part because I lean Orwell on state violence in many cases, but I think it misunderstands Yudkowsky’s problem with the terrorists. It’s not that the fact that this is terrorism adds enough intrinsic badness to outweigh a greater chance of literal nuclear war, it’s that legitimate state authority credibly being willing to go to nuclear war is likely to actually work, while terrorism is just a naïve tactic which will likely backfire. In fact nothing even rests on the idea that nuclear war is worth preventing AI (though in a quite bad and now deleted tweet Yudkowsky does argue for this, and given that he expects survivors of nuclear war but not AI misalignment nothing about this judgement rests on his cringe “reaching the stars” aside). If a NATO country is invaded, letting it be invaded is surely not as bad as global nuclear war, but supporters of Article 5 tacitly accept the cost of risking this outcome, because non-naïve consequentialism cares about credibly backing certain important norms, even when, in isolation, the cost of going through with them doesn’t look worth it.

2. “While there are international resolutions that involve credibly risking nuclear war, like Article 5, and there are international resolutions that involve punishing rogue states, like ones governing the development of weapons of mass destruction, the combination of these two is in practice not really there, so pointing to each in isolation fails to recognize the way this proposal pushes a difference in degrees all the way to basically a difference in kind”:

I am again sympathetic to this. What Yudkowsky is proposing here is kind of a big deal, and it involves a stricter international order than we have ever seen before. This is very troubling! It isn’t clear that there is a single difference in kind (except perhaps democracy) between Stalinism and a state that I would be mostly fine with. It’s largely about pushing state powers and state flaws that are tolerable at some degree to a point where they are no longer tolerable. I think I’m just not certain if his proposal reaches this crucial level for me. One reason is that I’m just not sure what level of international control really crosses that line for me, and risking war to prevent x-risk seems like a candidate okay think for countries to apply a unique level of force to. Certainly if you believe the things Yudkowsky does. The second reason however, is that his actual proposal is ambiguous in crucial ways that I will cover in point 3, so I would probably be okay with some but not other versions of it.

3. “Yes there is a difference between state force and random acts of violence, but it isn’t clear what general heuristic we can use to distinguish the two other than ‘one is carried out by a legitimate state authority and the other isn’t’. We know what this looks like at the country level because the relevant state authority is usually pretty clear, but on the international stage this is just not something where the difference between legitimate and illegitimate governance is super obvious! How many countries have to sign up? What portion of the world population do they have to represent? How powerful on the international stage do they already have to be? What agreement mechanism/mediating body is needed to grant this authority? Surely there is a difference in the types of ways Yudkowsky cares about between violence carried out by the mafia versus the local government, but on the international stage this sort of difference is very foggy. Given the huge difference he places on illegitimate versus legitimate force, Yudkowsky should have been specific about what it would take for such a governing agreement to be legitimate. Otherwise people can fill in the details however they think one should answer this question, and Yudkowsky specifically is shielded from the most relevant sort of criticism he could face for his proposal”:

This is the objection I am most sympathetic to, and the place I wish critics would focus most of their attention. If NATO agrees to this treaty, does that give them legitimate authority to threaten China with drone strikes that isn’t just like the mafia writing threatening letters to AI developers? What if China joins in with NATO, does this grant the authority to threaten Russia? Probably not for both, but while the difference is probably at an ambiguous threshold of which countries sign up, it’s pretty clear when a country-wide law becomes legitimate, because there’s an agreed upon legitimate process for passing it. These are questions deeply tied to any proposal like this and it does bug me how little Yudkowsky has spelled this out. That said, I think this is sort of a problem for everyone? As I’ve said, and Yudkowsky has said, basically everyone distinguishes between state enforcement and random acts of civilian violence, and aside from this, basically everyone seems confused about how to apply this at the international scale on ambiguous margins. Insofar as you want to apply something like this to the international scale sometimes, you have to live with this tension, and probably just remain a bit confused.

Yudkowsky claims that AI developers are plunging headlong into our research in spite of believing we are about to kill all of humanity. He says each of us continues this work because we believe the herd will just outrun us if any one of us were to stop.
 The truth is nothing like this. The truth is that we do not subscribe to Yudkowsky’s doomsday predictions. We work on artificial intelligence because we believe it will have great benefits for humanity and we want to do good for humankind.
 We are not the monsters that Yudkowsky makes us out to be.

I’m going to go against the grain here, and explain how I truly feel about this sort of AI safety messaging.

As others have pointed out, fearmongering on this scale is absolutely insane to those who don’t have a high probability of doom. Worse, Elizier is calling for literal nuclear strikes and great power war to stop a threat that isn’t even provably real! Most AI researchers do not share his views, neither do I.

I want to publicly state that pushing this maximized narrative about AI x risk will lead to terrorist actions against GPU clusters or individuals involved in AI. These types of acts follow from the intense beliefs of those who agree with Elizier, and have a doomsday cult style of thought.

Not only will that sort of behavior discredit AI safety and potentially EA entirely, it could hand the future to other actors or cause governments to lock down AI for themselves, making outcomes far worse.

The full text of the TIME piece is now available on the EA Forum here, with two clarifying notes by Eliezer added at the end.

What if actors with bad intentions don't stop (and we won't be able to know about that), and they create a more powerful AI than what we have now?