We're Not Ready: thoughts on "pausing" and responsible scaling policies

By Holden Karnofsky @ 2023-10-27T15:19 (+150)

This is a crosspost, probably from LessWrong. Try viewing it there.

null
Geoffrey Miller @ 2023-10-27T20:52 (+58)

Holden - these are reasonable points. But I have two quibbles.

First, the recent surveys of the general public's attitudes towards AI risk suggest that a strongly enforced global pause would actually get quite a bit of support. It's not outside the public's Overton Window. It might be considered an 'extreme solution' by AI industry insiders and e/acc cultists. But the public seems to understand that it's just fundamentally dangerous to invent Artificial General Intelligence that's as smart as smart humans (and much, much faster), or to invent Artificial Superintelligence. AI experts might patronize the public by claiming they're just reacting to sensationalized Hollywood depictions of AI risk. But I don't care. If the public understands the potential risks, through whatever media they've been exposed to, and if it leads them to support a pause, we might as well capitalize on public sentiment.

Second, I worry that EAs generally have a 'policy fetish', in assuming that the only way to slow down a technological field is through formal, government-sanctioned regulation and 'good policy' solutions. I think this is incorrect, both historically and logically. In this piece on moral stigmatization of AI, I argued that an informal, grass-roots, public moral backlash against the AI industry could accomplish almost everything formal regulation can accomplish, without many of the loopholes and downsides that regulation would face. If the general public realizes that AGI-directed research is just fundamentally stupid and reckless and a huge extinction risk, they can stigmatize AI researchers, funders, suppliers, etc in ways that shut down the industry -- potentially for decades. If that public stigmatization goes global, the AI industry globally could be put on 'pause' for quite a while. Sure, we might delay some potential benefits from some narrow AI applications. But that's a tradeoff most reasonable people would be willing to accept. (For example, if my generation misses out on AI-created longevity treatments, and we die, but our kids survive, without facing AGI-imposed extinction risks, that's fine with me -- and I think it would be OK with most parents.)

I understand that harnessing the power of moral stigmatization to shut down a promising-but-dangerous technology like AI isn't the usual EA style, but at this point, it might be the only practical solution to pausing dangerous AI development.

Greg_Colbourn @ 2023-10-28T18:11 (+8)

Fully agree. A potential taboo on AGI is something that is far too often overlooked by people who worry about pauses not working well (e.g. see also Scott Alexander, Matthew Barnett, Nora Belrose).

Zed Tarar @ 2023-12-01T13:45 (+1)

This is true--it's the same tactic anti-GMO lobbies, the NRA, NIMBYs, and anti-vaxxers have used. The public as a whole doesn't need to be anti-AI, even a vocal minority will be enough to swing elections and ensure an unfavorable regulatory environment. If I had to guess, AI would end up like nuclear fission--not worth the hassle, but with no off-ramp, no way to unring the alarm bell. 

Ryan Greenblatt @ 2023-10-30T03:36 (+4)

First, the recent surveys of the general public's attitudes towards AI risk suggest that a strongly enforced global pause would actually get quite a bit of support. It's not outside the public's Overton Window. It might be considered an 'extreme solution' by AI industry insiders and e/acc cultists. But the public seems to understand that it's just fundamentally dangerous to invent Artificial General Intelligence that's as smart as smart humans (and much, much faster), or to invent Artificial Superintelligence. AI experts might patronize the public by claiming they're just reacting to sensationalized Hollywood depictions of AI risk. But I don't care. If the public understands the potential risks, through whatever media they've been exposed to, and if it leads them to support a pause, we might as well capitalize on public sentiment.

I think the public might support a pause on scaling, but I'm much more skeptical about the sort of hardware-inclusive pause that Holden discusses here:

global regulation-backed pause on all investment in and work on (a) general3 enhancement of AI capabilities beyond the current state of the art, including by scaling up large language models; (b) building more of the hardware (or parts of the pipeline most useful for more hardware) most useful for large-scale training runs (e.g., H100’s); (c) algorithmic innovations that could significantly contribute to (a)

A hardware-inclusive pause which is sufficient for pausing for >10 years would probably effectively dismantle companies like nvidia and would be at least a serious dent in TSMC. This would involve huge job loss and a large hit to the stock market. I expect people would not support such a pause which effectively requires dismantling a powerful industry.

It's possible I'm overestimating the extent to which hardware needs to be stopped for such a ban to be robust and an improvement on the status quo.

Nick K. @ 2023-10-30T16:54 (+1)

I'm not an expert but economic damage seems to me plausibly like a question of implementation details. E.g. if you ask for a stop in hardware improvements at the same time as implementing hardware-level compute monitoring, this likely requires development of new technology to do efficiently which may allow the current companies to maintain their leading position. 

Of course, restrictions are going to have some effect, and plausibly may hit Nvidia's valuation but it is not at all clear that the economic consequences would necessarily be dramatic (the situation of the car industry and switching to E.V.'s might be vaguely analogous).  

kokotajlod @ 2023-10-29T13:26 (+3)

I think the tech companies -- and in particular the AGI companies -- are already too powerful for such an informal public backlash to slow them down significantly.

Geoffrey Miller @ 2023-10-29T20:27 (+26)

Disagree. Almost every successful moral campaign in history started out as an informal public backlash against some evil or danger.

The AGI companies involve a few thousand people versus 8 billion, a few tens of billions of funding versus 360 trillion total global assets, and about 3 key nation-states (US, UK, China) versus 195 nation-states in the world. 

Compared to actually powerful industries, AGI companies are very small potatoes. Very few people would miss them if they were set on 'pause'.

kokotajlod @ 2023-10-30T13:40 (+2)

I hope you are right.

Greg_Colbourn @ 2023-10-29T13:30 (+4)

I imagine it going hand in hand with more formal backlashes (i.e. regulation, law, treaties).

Akash @ 2023-10-27T21:56 (+23)

I think it’s good for proponents of RSPs to be open about the sorts of topics I’ve written about above, so they don’t get confused with e.g. proposing RSPs as a superior alternative to regulation. This post attempts to do that on my part. And to be explicit: I think regulation will be necessary to contain AI risks (RSPs alone are not enough), and should almost certainly end up stricter than what companies impose on themselves.

Strong agree. I wish ARC and Anthropic had been more clear about this, and I would be less critical of their RSP posts if they were upfront and clear about this stance. I think your post is strong and clear (you state multiple times, unambiguously, that you think regulation is necessary and that you wish the world had more political will to regulate). I appreciate this, and I'm glad you wrote this post.

I think it’d be unfortunate to try to manage the above risk by resisting attempts to build consensus around conditional pauses, if one does in fact think conditional pauses are better than the status quo. Actively fighting improvements on the status quo because they might be confused for sufficient progress feels icky to me in a way that’s hard to articulate.

A few thoughts:

  1. One reason I'm critical of the Anthropic RSP is that it does not make it clear under what conditions it would actually pause, or for how long, or under what safeguards it would determine it's OK to keep going. It is nice that they said they would run some evals at least once every 4X in effective compute and that they don't want to train catastrophe-capable models until their infosec makes it more expensive for actors to steal their models. It is nice that they said that once they get systems that are capable of producing biological weapons, they will at least write something up about what to do with AGI before they decide to just go ahead and scale to AGI. But I mostly look at the RSP and say "wow, these are some of the most bare minimum commitments I could've expected, and they don't even really tell me what a pause would look like and how they would end it."
  2.  Meanwhile, we have OpenAI (that plans to release an RSP at some point), DeepMind (rumor has it they're working on one but also that it might be very hard to get Google to endorse one), and Meta (oof). So I guess I'm sort of left thinking something like "If Anthropic's RSP is the best RSP we're going to get, then yikes, this RSP plan is not doing so well." Of course, this is just a first version, but the substance of the RSP and the way it was communicated about doesn't inspire much hope in me that future versions will be better.
  3. I think the RSP frame is wrong, and I don't want regulators to use it as a building block. My understanding is that labs are refusing to adopt an evals regime in which the burden of proof is on labs to show that scaling is safe. Given this lack of buy-in, the RSP folks concluded that the only thing left to do was to say "OK, fine, but at least please check to see if the system will imminently kill you. And if we find proof that the system is pretty clearly dangerous or about to be dangerous, then will you at least consider stopping" It seems plausible to me that governments would be willing to start with something stricter and more sensible than this "just keep going until we can prove that the model has highly dangerous capabilities" regime. 
  4. I think some improvements on the status quo can be net negative because they either (a) cement in an incorrect frame or (b) take a limited window of political will/attention and steer it toward something weaker than what would've happened if people had pushed for something stronger. For example, I think the UK government is currently looking around for substantive stuff to show their constituents (and themselves) that they are doing something serious about AI. If companies give them a milktoast solution that allows them to say "look, we did the responsible thing!", it seems quite plausible to me that we actually end up in a worse world than if the AIS community had rallied behind something stronger. 
  5. If everyone communicating about RSPs was clear that they don't want it to be seen as sufficient, that would be great. In practice, that's not what I see happening. Anthropic's RSP largely seems devoted to signaling that Anthropic is great, safe, credible, and trustworthy. Paul's recent post is nuanced, but I don't think the "RSPs are not sufficient" frame was sufficiently emphasized (perhaps partly because he thinks RSPs could lead to a 10x reduction in risk, which seems crazy to me, and if he goes around saying that to policymakers, I expect them to hear something like "this is a good plan that would sufficiently reduce risks"). ARC's post tries to sell RSPs as a pragmatic middle ground and IMO pretty clearly does not emphasize (or even mention?) some sort of "these are not sufficient" message. Finally, the name itself sounds like it came out of a propaganda department– "hey, governments, look, we can scale responsibly". 
  6. At minimum, I hope that RSPs get renamed, and that those communicating about RSPs are more careful to avoid giving off the impression that RSPs are sufficient.
  7. More ambitiously, I hope that folks working on RSPs seriously consider whether or not this is the best thing to be working on or advocating for. My impression is that this plan made more sense when it was less clear that the Overton Window was going to blow open, Bengio/Hinton would enter the fray, journalists and the public would be fairly sympathetic, Rishi Sunak would host an xrisk summit, Blumenthal would run hearings about xrisk, etc. I think everyone working on RSPs should spend at least a few hours taking seriously the possibility that the AIS community could be advocating for stronger policy proposals and getting out of the "we can't do anything until we literally have proof that the model is imminently dangerous" frame. To be clear, I think some people who do this reflection will conclude that they ought to keep making marginal progress on RSPs. I would be surprised if the current allocation of community talent/resources was correct, though, and I think on the margin more people should be doing things like CAIP & Conjecture, and fewer people should be doing things like RSPs. (Note that CAIP & Conjecture both impt flaws/limitations– and I think this partly has to do with the fact that so much top community talent has been funneled into RSPs/labs relative to advocacy/outreach/outside game).
evhub @ 2023-10-27T23:36 (+9)

Cross-posted from LessWrong.

One reason I'm critical of the Anthropic RSP is that it does not make it clear under what conditions it would actually pause, or for how long, or under what safeguards it would determine it's OK to keep going.

It's hard to take anything else you're saying seriously when you say things like this; it seems clear that you just haven't read Anthropic's RSP. I think that the current conditions and resulting safeguards are insufficient to prevent AI existential risk, but to say that it doesn't make them clear is just patently false.

The conditions under which Anthropic commits to pausing in the RSP are very clear. In big bold font on the second page it says:

Anthropic’s commitment to follow the ASL scheme thus implies that we commit to pause the scaling and/or delay the deployment of new models whenever our scaling ability outstrips our ability to comply with the safety procedures for the corresponding ASL.

And then it lays out a serious of safety procedures that Anthropic commits to meeting for ASL-3 models or else pausing, with some of the most serious commitments here being:

  • Model weight and code security: We commit to ensuring that ASL-3 models are stored in such a manner to minimize risk of theft by a malicious actor that might use the model to cause a catastrophe. Specifically, we will implement measures designed to harden our security so that non-state attackers are unlikely to be able to steal model weights, and advanced threat actors (e.g. states) cannot steal them without significant expense. The full set of security measures that we commit to (and have already started implementing) are described in this appendix, and were developed in consultation with the authors of a forthcoming RAND report on securing AI weights.
  • Successfully pass red-teaming: World-class experts collaborating with prompt engineers should red-team the deployment thoroughly and fail to elicit information at a level of sophistication, accuracy, usefulness, detail, and frequency which significantly enables catastrophic misuse. Misuse domains should at a minimum include causes of extreme CBRN risks, and cybersecurity.
    • Note that in contrast to the ASL-3 capability threshold, this red-teaming is about whether the model can cause harm under realistic circumstances (i.e. with harmlessness training and misuse detection in place), not just whether it has the internal knowledge that would enable it in principle to do so.
    • We will refine this methodology, but we expect it to require at least many dozens of hours of deliberate red-teaming per topic area, by world class experts specifically focused on these threats (rather than students or people with general expertise in a broad domain). Additionally, this may involve controlled experiments, where people with similar levels of expertise to real threat actors are divided into groups with and without model access, and we measure the delta of success between them.

And a clear evaluation-based definition of ASL-3:

We define an ASL-3 model as one that can either immediately, or with additional post-training techniques corresponding to less than 1% of the total training cost, do at least one of the following two things. (By post-training techniques we mean the best capabilities elicitation techniques we are aware of at the time, including but not limited to fine-tuning, scaffolding, tool use, and prompt engineering.)

  1. Capabilities that significantly increase risk of misuse catastrophe: Access to the model would substantially increase the risk of deliberately-caused catastrophic harm, either by proliferating capabilities, lowering costs, or enabling new methods of attack. This increase in risk is measured relative to today’s baseline level of risk that comes from e.g. access to search engines and textbooks. We expect that AI systems would first elevate this risk from use by non-state attackers. Our first area of effort is in evaluating bioweapons risks where we will determine threat models and capabilities in consultation with a number of world-class biosecurity experts. We are now developing evaluations for these risks in collaboration with external experts to meet ASL-3 commitments, which will be a more systematized version of our recent work on frontier red-teaming. In the near future, we anticipate working with CBRN, cyber, and related experts to develop threat models and evaluations in those areas before they present substantial risks. However, we acknowledge that these evaluations are fundamentally difficult, and there remain disagreements about threat models.
  2. Autonomous replication in the lab: The model shows early signs of autonomous self-replication ability, as defined by 50% aggregate success rate on the tasks listed in [Appendix on Autonomy Evaluations]. The appendix includes an overview of our threat model for autonomous capabilities and a list of the basic capabilities necessary for accumulation of resources and surviving in the real world, along with conditions under which we would judge the model to have succeeded. Note that the referenced appendix describes the ability to act autonomously specifically in the absence of any human intervention to stop the model, which limits the risk significantly. Our evaluations were developed in consultation with Paul Christiano and ARC Evals, which specializes in evaluations of autonomous replication.

This is the basic substance of the RSP; I don't understand how you could have possibly read it and missed this. I don't want to be mean, but I am really disappointed in these sort of exceedingly lazy takes.

NickLaing @ 2023-10-28T04:59 (+24)

It think calling a take "lazy", which could indeed be considered "mean" is not avery helpful approach, you could have made your point without that kind of derision. There are going to be a lot of misunderstandings and hot takes around RSPs, and I think AI company employees especially should err heavily on the side of patience and kind understanding it they want to avoid people becoming more adversarial towards them.

Live by the sword, die by the sword.

Akash said...

"that it does not make it clear under what conditions it would actually pause, or for how long, or under what safeguards it would determine it's OK to keep going. It"

I agree the conditions from the RSP you started are clearer than I would have expected reading Akash's above comment, but to be fair to Akash, from those paragraphs you posted above, only the last one seems to state a clear and specific condition for pausing, the others seem to say "refer to experts" which could be considered unclear, to give Akash the benefit of the doubt.

And they don't say how long the pause would be out conditions for restarting either.

Greg_Colbourn @ 2023-10-28T18:26 (+12)

Overall I don’t have settled views on whether it’d be good for me to prioritize advocating for any particular policy.5 At the same time, if it turns out that there is (or will be) a lot more agreement with my current views than there currently seems to be, I wouldn’t want to be even a small obstacle to big things happening, and there’s a risk that my lack of active advocacy could be confused with opposition to outcomes I actually support.

You have a huge amount of clout in determining where $100Ms of OpenPhil money is directed toward AI x-safety. I think you should be much more vocal on this - at least indirectly by OpenPhil grant making. In fact I've been surprised at how quiet you (and OpenPhil) have been since GPT-4 was released!

Greg_Colbourn @ 2023-10-28T18:20 (+4)
  • There’s a serious (>10%) risk that we’ll see transformative AI2 within a few years.
  • In that case it’s not realistic to have sufficient protective measures for the risks in time.
  • Sufficient protective measures would require huge advances on a number of fronts, including information security that could take years to build up and alignment science breakthroughs that we can’t put a timeline on given the nascent state of the field, so even decades might or might not be enough time to prepare, even given a lot of effort.

If it were all up to me, the world would pause now

Reading the first half of this post, I feel that your views are actually very close to my own. It leaves me wondering how much your conflicts of interest - 

I am married to the President of Anthropic and have a financial interest in both Anthropic and OpenAI via my spouse.

- are factoring into why you come down in favour of RSPs (above pausing now) in the end.

Peter Wildeford @ 2023-10-27T16:34 (+3)

I’m guessing stopping scaling by US POTUS executive order is not even legally possible though? So I don’t think we’d have to worry about that.

dan.pandori @ 2023-10-27T17:33 (+9)

Legal or constitutional infeasibility does not always prevent executive orders from being applied (or followed). I feel like the US president declaring a state of emergency related to AI catastrophic risk (and then forcing large AI companies to stop training large models) sounds at least as constitutionally viable as the attempted executive order for student loan forgiveness.

I agree that this seems fairly unlikely to happen in practice though.

Zed Tarar @ 2023-12-01T13:41 (+1)

I think you put it well when you said: 

"Some people think that the kinds of risks I’m worried about are far off, farfetched or ridiculous."

If I made the claim that we had 12 months before all of humanity is wiped by an asteroid, you'd rightly ask me for evidence. Have I picked up a distant rock in space using radio telescopes? Some other tangible proof? Or is it a best-guess, since, hey, it's technically possible that we could be hit with an asteroid on any given year. Then imagine if I advocate we spend two percent of global GDP preparing for this event. 

That's where the state of AGI fear is--all scenarios depend on wild leaps of faith and successive assumptions that build on each other. 

I've attempted to put this all in one place with this post.