Dario Amodei: "We Must Pace The Frontier"

By Matrice Jacobine🔸🏳️‍⚧️ @ 2026-09-12T14:40 (+38)

This is a linkpost to https://darioamodei.com/post/we-must-pace-the-frontier

[...]

Along with my co-founders and employees, I have grappled with this duality of risk and benefit since the beginning of Anthropic. Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless. We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top. We have always devoted a substantial fraction of our efforts to studying, addressing, and informing the public about these AI risks, as well as advocating for well-considered regulation of AI, even when this gets us accused of hype, “doomerism”, or regulatory capture. We have tried to prioritize caution over speed and prudence over profit.

But over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. Two things have convinced me.

Ben Stevenson @ 2026-09-12T17:32 (+6)

I was struck that Dario (1) says that recursive self-improvement has already begun, and (2) says that rogue agents could take over “the entire Internet” within six to 12 months. (This seems similar to Ajeya’s prediction that agents could establish a more permanent rogue deployment inside an AGI company within six months).

Vasco Grilo🔸 @ 2026-09-16T18:37 (+2)

Hi Ben. Thanks for sharing that.

(1) says that recursive self-improvement [RSI] has already begun

It is unclear to me what this means. Dario says "This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described". I have gone through the instances of "recursive self-improvement" and "RSI" in the linked sources, and I did not find any concrete description of what it means for RSI to start. There is a sense in which humanity has always been building on past knowledge.

(2) says that rogue agents could take over “the entire Internet” within six to 12 months

Dario says "it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails". I doubt humans will lose control over the entire internet to bots over the next 12 months. I am open to bets against short timelines for transformative AI (TAI), or what they supposedly imply, up to 10 k$.

Ben Stevenson @ 2026-09-16T18:50 (+4)

Hi Vasco. I also feel unclear about what it means for RSI to properly begin; I’m assuming humans are still in the loop at Anthropic. I’m reminded of Toby Ord’s analogy:

I like to think of it in terms of a group of hikers seeing a mountain in the distance, towering up into the clouds and beyond, with its snowy peak catching the sun’s light. They talk animatedly about how amazing it would be to climb so high that they are inside a cloud. Or imagine being above the clouds, looking over them like an angel. After many hours of climbing, they notice there is a faint haze. Are they inside the cloud now? The mist gradually gets thicker until they can only see 10 metres ahead. Are they inside it now? Then it drops to 9 metres. Then 8. Then visibility starts to increase again. After an hour there is only the slightest haze. Are they above the clouds now? Another 30 minutes and there is no haze, and they can all agree they are above the clouds. 

It is clear that at some point they were inside the cloud and sometime later were above it. And it is clear that these were sensible and useful concepts. For example, they took precautions like roping themselves together for the journey through the cloud due to the low visibility and took cameras with them because they knew they could take beautiful photos above the clouds. A lack of sharp boundaries doesn’t make these concepts useless. But they were admittedly a lot more useful when the hikers were on the ground, planning their route, and a lot less useful in the debatable boundary zones.

Has anybody taken you up on the bet yet? I’m not betting; I have no idea what’s going on!

Vasco Grilo🔸 @ 2026-09-16T19:37 (+2)

Has anybody taken you up on the bet yet?

I have this and this bets resolving at the end of 2027.

I’m not betting; I have no idea what’s going on!

Fair and funny. I suggested the bet having other readers in mind.

Ben Stevenson @ 2026-09-16T19:52 (+5)

I bet Greg Colbourn 10 k€ that AI will not kill us all by the end of 2027

Good luck, I hope you win!

  • If for some reason I am not able to decide (e.g. if I die before 2028), the transfer must be made to my lastly stated organisation of choice, currently The Humane League (THL).

Where would you now like the donation to go?

Vasco Grilo🔸 @ 2026-09-16T20:40 (+2)

Good luck, I hope you win!

Thanks. Me too. I am thinking about suggesting to Greg doing a similar bet resolving at the end of 2030, where I would initially donate to Greg's preferred charity what I win from the 1st bet plus some more money. I could probably bet like 40 k$ in total, and then win 80 k$ adjusted for inflation or growth in stocks at the end of 2030.

Where would you now like the donation to go?

I would make a donation to Rethink Priorities (RP) restricted to research on moral weights led by Bob Fischer. Here is some context.

Yoav Hollander @ 2026-09-13T08:52 (+1)

I like the embedded-evaluator proposal. Coming from chip and autonomous-vehicle verification, I’d suggest one concrete addition: An evolving coverage map connecting safety claims to evidence.

For each relevant configuration (during training, internal use, and release), record which requirements and situations were tested, what failed, what remains unchecked, and where the checking methods themselves are weak. Also record whether apparent alignment survives further capabilities training and generalizes to situations withheld from alignment training.

 

The map may be used to guide improving alignment, not just measuring it - for example, by systematically generating alignment stories / training cases across relevant situations. When a problem is found, identify the broader failure class, strengthen alignment across that class, and re-evaluate (including after further capabilities training).

 

A coverage map cannot establish that all important risks have been identified: Searching for missing dimensions and checkers is part of the work. But it can make the scope and limitations of the evidence inspectable, and help prioritize how to use the time pacing buys us. I discuss this approach in V&V takes on “Pacing the frontier”.