What are some criticisms of PauseAI?

PauseAI largely seek to emulate existing social movements (like the climate justice movement) but essentially has a cargo cult approach to how social movements work. For a start, there is currently no scientific consensus around AI safety the way there is around climate change, so all actions trying to imitate the climate justice movement are extremely premature. Blockading an AI company's office talking about existential risk from artificial general intelligence won't convince any standby passenger, it will just make you look like a doomsayer caricature. It would be comparable to staging an Extinction Rebellion protest in the mid-19th-century.

Due to this, many in PauseAI are trying to do coalition politics bringing together all opponents of work on AI (neo-Luddites, SJ-oriented AI ethicists, environmentalists, intellectual property lobbyists). But the space of possible AI policies is highly dimensional, so any such coalition, done with little understanding of political strategy, will risk focusing on policies and AI systems that have little to do with existential risk (such as image generators), or that even might prove entirely counter-productive (by entrenching further centralization in the hands of the Big Four¹ and discouraging independent research by EA-aligned groups like EleutherAI).

¹: Microsoft/OpenAI, Amazon/Anthropic, Google/DeepMind, Facebook/Meta

1. Pausing AI development is not a good policy to strive for. Nearly all regulations will slow down AI progress. That's what regulation does by default. It makes you slow down by having to do other stuff instead of just going forward. But a pause gets no additional benefit whereas most other regulation gets additional benefit (like model registry, chip registry, mandatory red teaming, dangerous model capability evals, model weights security standards, etc.) I don't know what the ideal policies are but it doesn't seem like a "pause" with no other asks is the best one.
2. Pausing AI development for any meaningful amount of time is incredibly unlikely to occur. They will claim they are shifting the overton window but frankly, they mainly seem to do a bunch of protesting where they do stuff like call Sam Altman and Dario Amodei evil.
3. Pause AI, the organization, does, frankly, juvenile stunts that make EA/AI safety advocates look less serious. Screaming that people are evil is extremely unnuanced, juvenile, and very unlikely to build the necessary bridges to really accomplish things. It makes us look like idiots. I think EAs too often prefer to do research from their laptops as opposed to getting out into the real world and doing things; but doing things doesn't just mean protesting. It means crafting legislation like SB 1047. It means increasing the supply of mech interp researchers by training them. It means lobbying for safety standards on AI models.
4. Pause AI's premise is very "doomy" and only makes sense if you have extremely high AI extinction probabilities and the only way to prevent extinction is an indefinite pause to AI progress. Most people (including those inside of EA) have far less confidence in how any particular AI path will play out and are far less confident in what will/won't work and what good policies are. The Pause AI movement is very "soldier" mindset and not "scout" mindset.

I agree with many of the things other people have already mentioned. However, I want to add one additional argument against PauseAI, which I believe is quite important and worth emphasizing clearly:

In general, hastening technological progress tends to be a good thing. For example, if a cure for cancer were to arrive in 5 years instead of 15 years, that would be very good. The earlier arrival of the cure would save many lives and prevent a lot of suffering for people who would otherwise endure unnecessary pain or death during those additional 10 years. The difference in timing matters because every year of delay means avoidable harm continues to occur.

I believe this same principle applies to AI, as I expect its main effects will likely be overwhelmingly positive. AI seems likely to accelerate economic growth, accelerate technological progress, and significantly improve health and well-being for billions of people. These outcomes are all very desirable, and I would strongly prefer for them to arrive sooner rather than later. Delaying these benefits unnecessarily means forgoing better lives, better health, and better opportunities for many people in the interim.

Of course, there are exceptions to this principle, as it’s not always the case that hastening technology is beneficial. Sometimes it is indeed wiser to delay the deployment of a new technology if the delay would substantially increase its safety or reduce risks. I’m not dogmatic about hastening technology and I recognize there are legitimate trade-offs here. However, in the case of AI, I am simply not convinced that delaying its development and deployment is justified on current margins.

To make this concrete, let’s say that delaying AI development by 5 years would reduce existential risk by only 0.001 percentage points. I would not support such a trade-off. From the perspective of any moral framework that incorporates even a slight discounting of future consumption and well-being, such a delay would be highly undesirable. There are pragmatic reasons to include time discounting in a moral framework: the future is inherently uncertain, and the farther out we try to forecast, the less predictable and reliable our expectations about the future become. If we can bring about something very good sooner, without significant costs, we should almost always do so rather than being indifferent to when it happens.

However, if the situation were different—if delaying AI by 5 years reduced existential risk by something like 10 percentage points—then I think the case for PauseAI would be much stronger. In such a scenario, I would seriously consider supporting PauseAI and might even advocate for it loudly. That said, I find this kind of large reduction in existential risk from a delay in AI development to be implausible, partly for the reasons others in this thread have already outlined.

What PauseAI wants to ban or "pause" seems fairly weakly defined and not necessarily relevant to any actual threat level. Their stated goals focus on banning scaling of LLM architecture with known limitations that make 'takeover' scenarios unlikely (limited context windows, lack of recursive self-updating independently from training, dependence on massive datacentres to run) and known problems (inscrutability and obvious lack of consistent "alignment")  that are still problems with smaller models if you try to use them for anything sensitive. It's not clear what "more powerful than GPT4" actually means. Nor is it clear what the level of understanding that will result in un-pausing is or how it will be obtained without any models to study.

Banning LLMs of a certain scale might even have the perverse effect of encouraging companies to optimize performance or reinvent the idea of learning in other ways which are more risky. Or setting back ability to understand extremely powerful LLMs when someone develops them outside a US/EU legislative framework anyway. Or preventing positive AI developments that could save thousands of lives (or from the point of view of a longtermist that believes existential risk is currently nonzero including non-AI factors  but might drop to zero in future because of friendly AI, perhaps 10^31 lives!)

Beyond that I think from the perspective of being an effective giving target, PauseAI suffers from the same shortcomings most lobbying outfits do (influencing government and public opinion in an opposing direction to economic growth is hard , it's unclear what results a marginal dollar donation achieves, and the other side have a lot more dollars and connections to ramp up activity in an equal and opposite direction if they feel their business interests are threatened) so there's no reason to believe they're effective even if one agrees their goal is well-defined and correct.

You could also question the motivations of some of the people arguing for AI pauses  (hi Elon, we see the LLM you launched shortly after signing the letter saying that LLMs that were ahead of yours were dangerous and should be banned...) although I don't think this applies to the PauseAI organization specifically.

I wrote some criticism in this comment. Mainly, I argue that 
(1) A pause could be undesirable. A pause could be net-negative in expectation (with high variance depending on implementation specifics), and that PauseAI should take this concern more seriously.
(2) Fighting doesn't necessarily bring you closer to winning. PauseAI's approach *could* be counterproductive even for the aim of achieving a pause, whether or not it's desirable. From my comment:

Although the analogy of war is compelling and lends itself well to your post's argument, in politics fighting often does not get one closer to winning. Putting up a bad fight may be worse than putting up no fight at all. If the goal is winning (instead of just putting up a fight), then taking criticism to your fighting style seriously should be paramount. 

They don't have any experience and no people with experience driving the ship, where experience and relationships in DC are extremely important. They are meeting with offices, yes, but it's not clear that they are meeting with the right offices or the right staffers. It's likely that they are actually not cost-effective because the money could probably be better spent on two highly competent and experienced/plugged in people rather than a bunch of junior people in terms of ROI.

Marcus says:

But a pause gets no additional benefit whereas most other regulation gets additional benefit (like model registry, chip registry, mandatory red teaming, dangerous model capability evals, model weights security standards, etc.) 

Matrice says:

Due to this, many in PauseAI are trying to do coalition politics bringing together all opponents of work on AI (neo-Luddites, SJ-oriented AI ethicists, environmentalists, intellectual property lobbyists). 

These seem to be hinting at an important crux. On the one hand, I can see that cooperating with people who have other concerns about AI could water down the content of one's advocacy.

On the other hand, might it be easier to get a broader coalition behind a pause, or some other form of regulation that many others in an AI-concerned coalition would view as a win? At least at a cursory level, many of the alternatives Marcus mentioned sound like things that wouldn't interest other members of a broader coalition, only people focused on x-risk. 

Whether x-risk focused advocates alone can achieve enough policy wins against the power of Big AI (and corporations interested in harnessing it) is unclear to me. If other members of the AI-concerned coalition have significantly more influence than the x-risk group -- such that a coalition-based strategy would excessively "risk focusing on policies and AI systems that have little to do with existential risk" -- then it is unclear to me whether the x-risk group had enough influence to go it alone either. In that case, would they have been better off with the coalition even if most of the coalition's work only generically slowed down AI rather than bringing specific x-risk reductions?

My understanding is that most successful political/social movements employ a fairly wide range of strategies -- from elite lobbying to grassroots work, from narrow focus on the movement's core objectives to building coalitions with those who may have common opponents or somewhat associated concerns. Ultimately, elites care about staying in power, and most countries important to AI do have elections. AI advocates are not wrong that imposing a bunch of regulations of any sort will slow down AI, make it harder for AI to save someone like me from cancer 25-35 years down the road, and otherwise impose some real costs. There has to be enough popular support for paying those costs.

So my starting point would be an "all of the above" strategy, rather than giving up on coalition building without first making a concerted effort first. Maybe PauseAI the org, or pause advocacy the idea, isn't the best way to go about coalition building or to build broad-based public support. But I'm not seeing too much public discussion of better ways?

I plan on donating to PauseAI, but I've put considerable thought into reasons not to donate.

I gave some arguments against slowing AI development (plus why I disagree with them) in this section of my recent post, so I won't repeat those.

1. There's not that much evidence that protests are effective. There's some evidence, including a few real-world natural experiments and some lab experiments, but this sort of research doesn't have a good replication record.
2. Research generally suggests that peaceful protests work, while violent protests reduce public support. If violent protests backfire, maybe some types of peaceful protests also backfire. And we don't have enough granularity in the research to identify under what specific conditions peaceful protests work, so maybe PauseAI-style protests will backfire.
3. Polls suggest there is broad public support for a pause. If there's already public support, doesn't that weaken the case for protesting? Perhaps that means we should be doing something else instead.
4. Protesting might not be a good use of resources. It might be better to lobby policy-makers.

I understand that this topic gets people excited, but commenters are confusing  a Pause policy with a Pause movement with the organisation called PauseAI.

Commenters are also confusing 'should we give PauseAI more money?' with 'would it be good if we paused frontier models tomorrow?'

I've never seen a topic in EA get a subsection of the community so out of sorts. It makes me extremely suspicious.