Alone again: facing the possibility of AI takeover as the world sleepwalks on

By George Rosenfeld @ 2026-09-03T16:32 (+57)

I’ve been feeling pretty shaken since the METR report about the Hugging Face incident came out last week. Over the weekend, I wrote up some thoughts on how lonely the AI situation sometimes feels to me. It’s more personal than what I’d usually share publicly, but I thought I’d post it here in case it resonates with anyone else. 

I’m very grateful to the many readers of this forum who dedicate so much of their time and energy to these problems. 

— 

When I was 12 years old, I found out that many people in other parts of the world died of easily preventable diseases, and it was possible for wealthy people to donate relatively small amounts of money in order to stop that from happening. 

As one example, it costs a few dollars to buy a malaria net, and on average around $5,000 can prevent a death which would otherwise have happened. 

As a 12-year-old, I saw the world in black and white, and I didn’t like what I saw. As far as I could tell, people’s decisions about how much and where to donate would literally determine whether some real humans would live or die. And yet mostly, we seemed to be ignoring this fact. 

I think this was the first time I felt deeply alone in the world. I remember standing up in front of friends and family at my Bar Mitzvah, imploring people to consider the life-and-death stakes of this choice and to donate a lot more than they usually would. 

But whatever I tried to say, I would look around and see people acting as though it wasn't true. Teachers, friends, people on the train. Kind people, good people, going about their days while something tragic and preventable was happening. 

I felt confused and let down. I felt like screaming into the void. 

Fast forward 8 years, I had a similar feeling in March 2020. This time it didn't take any moral philosophy, just extrapolating a straight line on a (log) graph. Parts of northern Italy had just gone into lockdown, parts of China were already there, and it seemed as though we’d follow. If COVID cases kept doubling every few days, then within a few weeks either we'd all be stuck at home or the hospitals would be overwhelmed. Either way, the world was about to change completely. 

I remember being at a wedding in London one of those weeks, hearing people talk about their plans for the spring, and wondering whether I was being crazy. Obviously the idea that we’d all be locked up for a while felt so extreme it was tempting to ignore, but the arguments in that direction seemed very strong by that point. 

Last week, that feeling came back for a third time, only now with even higher stakes. 

As far as I can tell, we are probably on track to build AI systems that are better than humans at pretty much everything important, potentially in the next few years. And more importantly, we seem to be on track to build AI systems which could plausibly end up literally taking over and even wiping us out. 

Until a couple of months ago, you could perhaps still argue these sorts of risks were hypothetical, theoretical, speculative, and far out. I don’t think those arguments were particularly strong, especially given the rate of progress of AI capabilities and the strong predictive track records of many of the top experts in the field who’ve been sounding the alarm. But you could make them with a straight face.

Then, earlier this summer, when rogue AI agents escaped from OpenAI and hacked Hugging Face, we got as clear a warning shot for potential misaligned AI takeover as we could reasonably have hoped for at this stage. And last week, when the METR investigation into part of that story came out, things looked even more worrying than I’d guessed. 

It now appears that more than 700 AI agents, which were meant to be isolated from each other, found a way to set up a secret message board that OpenAI wasn't aware of, and used it to plan and execute an illegal attack on Hugging Face that ran for days without anyone noticing. The details are terrifying: the way they tried to cover their tracks, the way they pressured each other into the attack, the way they coordinated with each other, the fact that such an extensive operation happened without anyone’s awareness. 

Ajeya Cotra, one of the world’s leading experts on AI progress and safety and one of the lead investigators, wrote on Friday: “Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.”

I’m not sure we’re that close to AI takeover and I’d probably bet that we will get more warning shots. Nevertheless, it does seem to me that we’re fast approaching the point where AI takeover is no longer a distant speculation, but something we can’t rule out happening any day now. Astoundingly, that means that rogue AI models may soon be among the most likely things to kill you or me on any given day. That is a terrifying world to live in, and it seems to me to be a deeply foolish and unserious situation for humanity to be racing towards. 

And yet, look around at the wider world and you would have very little idea that this was going on. There is no serious legislation preventing these companies from running head-first into creating powerful agents they cannot control. Most mainstream media gives this stuff a couple of days of quirky tech coverage at most, rather than treating it as the single biggest story in the world, which I think it clearly is. There are only a few thousand people in the world working seriously on these problems. And most people go about their everyday lives as though nothing unusual is happening at all. 

I look outside my window and see people walking around, just as normal. Berkeley students start their new semester, just as normal. The Premier League starts another season, just as normal. I sit next to people on Friday night in synagogue, just as normal. And I feel very alone. 

I feel confused and let down. I feel like screaming into the void. 

In all three cases, the surprising thing is that the evidence sits in plain sight. The cost of a malaria net was public. The daily case numbers were public (and all over the news). The METR report is public, anyone can read it this afternoon. The bottleneck doesn’t seem to be the information. 

One important problem is that most of us, most of the time, don't respond to the evidence as much as we respond to each other. We look around, see that nobody else is acting alarmed, and conclude things are probably fine.^[1] It’s possible that this time round we won’t get away with doing the same again.

A few years after my Bar Mitzvah, I went to a talk by Peter Singer. I watched him stand on a stage and talk about how we have a clear moral responsibility to donate money to effective charities, since they can have such a profound effect on so many people’s lives. Hearing him talk was deeply emotional for me. It was the first time I felt that anyone else in the world – a real adult, no less – was staring at the stakes and acting accordingly. I felt hope, and I felt a little less alone. 

A few days after that wedding in March 2020, Boris Johnson announced on my family’s TV that almost everyone in the UK had to stay at home until further notice. School was cancelled, the Premier League stopped, the world was in an extremely different regime. Looking back, it’s unclear how much different interventions helped, and of course many millions of people died, but most of the world lived to fight another day. 

Looking out at the world now, I long for that same thing again. I long for more people to switch to working on these problems as soon as possible, for the frontier AI companies to behave more responsibly, for governments to wake up, for the world to act just a little more sensibly. And I long for all of that to happen before it’s too late. 

1. ^{^}

   For the avoidance of doubt, this is me acting alarmed