Speculating on Secret Intelligence Explosions

By calebp @ 2025-06-05T13:55 (+20)

Cross-posted from Substack. This is a "takes" post and may well not meet your standard for rigour.


In the next few years, I expect AI companies to build AI systems that dramatically speed up their AI research, creating a period of extremely fast growth in capabilities, where AI systems surpass human intelligence in every interesting domain (i.e. an intelligence explosion).[1]

If this happens, I think it will be viable and potentially desirable for an AI company to keep its intelligence explosion secret, even if the project is not securitised. I’m concerned that people outside of AI companies won’t know that an intelligence explosion is happening, they’ll be cut out of high-stakes decisions, and they won’t be able to orient appropriately to a world with superintelligence.

In this post, I’ll discuss:

Overall, I think that a secret intelligence explosion is somewhat unlikely, but plausible and concerning.

Why might an AI company want to keep its intelligence explosion secret?

AI companies rarely have internal models that are much better than public models, so why might this change as AI systems become increasingly powerful?

If AIs automating AI research produce extremely powerful AI systems after a few months, then it’s at least plausible that AI companies will want to do it in secret.

Common Objections to the Feasibility of Keeping an Intelligence Explosion Secret

One of the Stargate data centre megasites currently under construction from @nunosempere. The green thing in the top right is the size of a football field.

These are all reasonable objections and make me think a secret intelligence explosion is less likely, but I don’t think any of them are damning.

Three can keep a secret if two are AIs

I occasionally speak to people who think AI companies couldn’t do an intelligence explosion secretly, even if they wanted to. I think their model is roughly:

  1. An AI company would need to invest substantial resources to create an intelligence explosion.
  2. So internally, a lot of people would need to be aware that the company is massively accelerating.
  3. AI company employees don’t take secrecy or security particularly seriously, and it’s really hard to keep a secret if hundreds of people are aware of it.

This seems wrong if you think (as I do) that an intelligence explosion will be AI-driven, i.e. making substantive use of drop-in replacements for ML engineers and researchers.

An AI workforce may be better at keeping secrets than a human one

Imagine that you are a researcher in North Korea working on an extremely secret military project as part of a large team.[4] The project’s leadership can take any actions they deem necessary to prevent the US from discovering the project.

Working there is frustrating:

I think many autocracies would eagerly adopt these extreme measures if they were affordable and didn’t incur large productivity losses. I expect that all of these measures and more will be relatively inexpensive with AIs because we don’t grant AI systems rights, and because their software nature provides them with many secrecy-preserving features.

For example, AI systems are currently unable to leave their workplace, can be interrogated much more invasively than humans, can be comprehensively monitored, erasing their memories is trivial, and they’ll do whatever kinds of malign or uncomfortable tasks you want.[5][6]

Suppose that an AI lab wanted to keep an intelligence explosion secret, what might they do?

I’m sure that there are many other creative ways you could use the features of an AI workforce to increase secrecy. If you're struggling to come up with strategies, try prompting an LM for ideas.

How will AI companies keep the GPUs on?

So why aren’t AI companies building in secret right now? Some hypotheses:

  1. Founder preferences - I think Sam Altman and Mark Zuckerberg are genuinely into building in public, moving fast, shipping cool stuff etc.
  2. Hiring - having a public and successful product makes it much easier to hire people. The talent density at AI companies is really high; most people can get a decent salary elsewhere if they want to, but there’s a kind of status that working at a company building something important and well-known gets you. A more straightforward explanation is that people are just more likely to remember to have a look at the jobs board when they are looking to move on.
  3. Fundraising - investors care a lot about user growth and revenue.[9] Most AI companies are hoping to raise billions of dollars. It’s hard to have 100M users if your product is a secret.

I’m not sure what will happen to founder preferences at the start of an intelligence explosion. I think it’s likely that founders change their minds, given the large pressures to be more secretive and the allure of potentially moving more quickly if they adopt a more secretive posture. Hiring will be less important if increasingly complex tasks are able to be delegated to AI systems, which leaves fundraising. I think the story here is more confusing.

Governments can clearly finance megaprojects when they are sufficiently bought in (e.g. bridges, nuclear power plants). If the US government wants to bankroll an ASI megaproject, I expect they’d be able to find the capital. If there’s sufficient justification to do this, then they’ll likely also be sufficient justification to secure the project and get some more insight into its operations, which will result in the public and some parts of government having some visibility into the AI project. That said, without significant action, I’d be surprised if all important government decision-makers were well-informed, particularly if the project is securitised.

So, how viable will it be for companies to raise billions of dollars in private capital, without releasing or publicly disclosing their best models?

I’m not sure how much an intelligence explosion will cost, but prices in the range of $70M to $2T seem quite plausible to me from holding training costs fixed and naively extrapolating. Let's take $10B-100B, which is somewhere in the middle (in log space), as our mainline scenario.

Two key questions are:

I had a quick look at privately funded megaprojects, some of the larger infrastructure projects included Global telecommunications networks ($50+ billion each),[10] the UK-France Channel Tunnel ($21 billion adjusted to current value),[11] and the Gorgon LNG project ($54 billion).[12] Some record-breaking financing includes Saudi Aramco IPO ($30B raised) and Verizon’s 2013 mega-bond ($3B).[13]

Whilst the value proposition is less clear than traditional infrastructure projects, it’s not a mystery either. Millions of people pay for AI products, and code assistants are significantly improving the productivity of software engineers. It’s perhaps not so surprising that Open AI, which released the fastest-growing app of all time, were able to raise $40B in March this year.

They could, of course, continue to charge for less capable models, which might be much cheaper to run and require far lower levels of security than intelligence explosion enabling models. Alternatively, they could partner with industries that have an extremely high ROI on intellectual labour per person (e.g., quantitative finance) and provide them with limited access to frontier models or specialised but less generally capable models, or build products or even compete directly in those industries.

How easily can an employee raise the alarm?

It may be that staff at AI companies are so wrapped up in NDAs and social incentives (particularly if AI is taken seriously as a threat or asset to national security) that they don’t blow the whistle, but let's suppose that they do want to blow the whistle.

Suppose that an AI researcher tweets, “The intelligence explosion has started, this is crazy, I’ve just seen x”. What are some candidates for x that would convince interested members of the public? I could see claims like “our AI can now design bioweapons” or “our AI is capable of massively speeding up research” to be brushed aside as AI hype, given the steady stream of people overhyping AI capabilities.[14] I’m a little more optimistic about claims like “our AI caused harm in the real world” being taken seriously. Claims like “look at the massive data centers being built” sound compelling, but massive data centres are already being built, I’m not sure what we expect to see change - maybe someone crunches the numbers and notices that there is way too much inference compute to justify the amount of publicly served models?[15]

If the intelligence explosion only needs to last a few months, will society really notice and reorient in time?[16] Most of the data centre build-out would happen before the intelligence explosion starts. The data centres used for serving weak models to consumers and the data centres used for powerful internal AI agents look very similar. Maybe you think that evals will make it very clear the intelligence explosion has started, but AI companies might simply choose not to run them, or sandbag on them, and there likely won’t be a tonne of public outrage if the model hasn’t been publicly released - how would we even know to be outraged?

  1. ^

    In this post I’m not arguing that there will be an intelligence explosion in the next few years, the main take is if there is an intelligence explosion in the next few years it may well be done secretly. I like this detailed illustration of intelligence explosion happening in the next few years.

  2. ^

    E.g. because of preferences of the CEO or employees. Note that CEO preferences also sometimes push towards transparency.

  3. ^

    So long as they have enough inference compute to run the model.

  4. ^

    To be clear this is just a hypothetical, I know very little about how covert and complex military technologies are developed in North Korea.

  5. ^

    In the sense that all inputs, internal states, and outputs can be stored and looked at, not that we really understand what’s going on in their minds.

  6. ^

    Insofar as you can check they are “actually” doing those tasks.

  7. ^

    Even if people are just fired I’m not sure that that intelligence explosion alarm is raised when there are easily preventable counter-narratives like “we wanted to strip our team back and increase talent density like Elon did at Twitter/X”.

  8. ^

    One reviewer was sceptical because they thought that labs would be respectful of employee preferences, and companies have high inertia preventing them for pivoting to this agent focussed strategy. I think these are reasonably counterarguments but they are fairly small updates for me on the viability of setting up a large AI agent powered team with a substantial fraction of total compute if senior AI company leadership want to do that.

  9. ^
  10. ^

    Major telecommunications companies (AT&T, Verizon, and Vodafone) built extensive global networks primarily financed through corporate debt, equity offerings, and reinvested company profits.

  11. ^

    One of history's most ambitious privately funded infrastructure endeavors funded by equity investments and commercial bank loans. Developed without direct government funding, though government guarantees were provided to support the project.

  12. ^

    Entirely funded by energy companies (Chevron, ExxonMobil, and Shell) via corporate resources and private debt arrangements. No direct government financial contribution.

  13. ^

    I haven’t looked into any of these examples deeply, maybe a subject for a future post.

  14. ^

    To be clear, I don’t think that AI is overhyped on net, just that it’s easy to dismiss comments that are very bullish on AI.

  15. ^

    Note that I think this will be very hard as the public would need to know how much inference compute models need to run, how much models are being run for public use. If a company is doing a secret intelligence explosion presumably they are also able to significantly push the frontier and serve small, cheap to run models that are competitive with public frontier models and use most of their inference compute for an intelligence explosion.

  16. ^

    Note that there are a lot of simple strategies like distilling a model to save inference compute and serving that model, just saying that your model is being very reward hacky so you’ve decided not to serve it for a while until you get more of a handle on it etc.


Ozzie Gooen @ 2025-06-05T15:31 (+6)

Very quickly:

1. I think that "secret intelligence explosions" are something that would be quite scary and disruptive. So it seems worth spending some attention on, like this writing.

2. I feel like there are some key assumptions here. I'd naively guess:

a. An intelligence explosion like you're describing doesn't seem very likely to me. It seems to imply a discontinuous jump (as opposed to regular acceleration), and also implies that this resulting intelligence would have profound market value, such that the investments would have some steeply increased ROI at this point. 

b. This model also implies that it might be feasible for multiple actors, particularly isolated ones, to make an "intelligent explosion." I'd naively expect there to be a ton of competition in this area, and I'd expect that competition would greatly decrease the value of the marginal intelligence gain (i.e. cheap LLMs can do much of the work that expensive LLMs do). I'd naively expect that if there are any discontinuous gains to be made, they'll be made by the largest actors.


I agree that abstract arguments about the cost/benefits of privacy don't make either case certain. But I think the empirical evidence is quite positive so far, against secrecy. That said, of course I expect that the situation could change quickly as major circumstances change. 

calebp @ 2025-06-05T15:44 (+2)

a. An intelligence explosion like you're describing doesn't seem very likely to me. It seems to imply a discontinuous jump (as opposed to regular acceleration), and also implies that this resulting intelligence would have profound market value, such that the investments would have some steeply increased ROI at this point. 

I'm not exactly sure what you mean by discontinuous jump. I expect the usefulness of AI systems to be pretty "continuous" inside AI companies and "discontinuous" outside AI companies. If you think that:
1. model release cadence will stay similar 
2. but, capabilities will accelerate
3. then you should also expect external AI progress to be more "discontinuous" than it currently is. 

I gave some reasons why I don't think AI companies will want to externally deploy their best models (like less benefit from user growth), so maybe you disagree with that, or do you disagree with 1,2, or 3?

b. This model also implies that it might be feasible for multiple actors, particularly isolated ones, to make an "intelligent explosion." I'd naively expect there to be a ton of competition in this area, and I'd expect that competition would greatly decrease the value of the marginal intelligence gain (i.e. cheap LLMs can do much of the work that expensive LLMs do). I'd naively expect that if there are any discontinuous gains to be made, they'll be made by the largest actors.

I do think that more than one actor (e.g. 3 actors) may be trying to IE at the same time, but I'm not sure why this is implied by my post. I think my model isn't especially sensitive to single vs multiple competing IEs, but possible you're seeing something I'm not. I don't really follow

competition would greatly decrease the value of the marginal intelligence gain (i.e. cheap LLMs can do much of the work that expensive LLMs do)

Do you expect competition to increase dramatically from where we are at rn? If not then I think current level of competition empirically do lead to people investing a lot in AI development so I'm not sure I quite follow your line of reasoning.

Ozzie Gooen @ 2025-06-05T17:43 (+2)

I gave some reasons why I don't think AI companies will want to externally deploy their best models (like less benefit from user growth), so maybe you disagree with that, or do you disagree with 1,2, or 3?

I understand that there are some reasons that companies might do this. One 1/2/3, I'm really unsure about the details of (2). If capabilities accelerate, but predictably and slowly, I assume this wouldn't feel very discontinuous.

Also, there's a major difference between AIs getting better and them becoming more useful. Often there are diminishing returns to intelligence. 

> I do think that more than one actor (e.g. 3 actors) may be trying to IE at the same time, but I'm not sure why this is implied by my post. I think my model isn't especially sensitive to single vs multiple competing IEs, but possible you're seeing something I'm not.
 

Sorry, I may have misunderstood that. But if there is only one or two potential actors, that does seem to make the situation far easier. Like, it could be fairly clear to many international actors that there are 1-2 firms that might be making major breakthroughs. In that case, we might just need to worry about policing these firms. This seems fairly possible to me (if we can be somewhat competent). 

Do you expect competition to increase dramatically from where we are at rn? If not then I think current level of competition empirically do lead to people investing a lot in AI development so I'm not sure I quite follow your line of reasoning.

I'd expect that market caps of these companies would be far higher if it were clear if there would be less competition later, and I'd equivalently expect these companies to do (even more) R&D. 

I'm quite sure investors are quite nervous about the monopoly abilities of LLM companies. 

Right now, I don't think it's clear to anyone where OpenAI/Anthropic will really make money 5+ years from now. It seems like [slightly worse AIs] often are both cheap / open-source and good enough. I think that both companies are very promising, but just that the future market value is very unclear.

I've heard that some of the Chinese strategy is, "Don't worry too much about being on the absolute frontier, because it's far cheaper to just copy from 1-2 steps behind."

I wasn't saying that "competition would greatly decrease the value of the marginal intelligence gain" in the sense of "things will get worse from where we are now", but in the sense of "things are generally worse from where they would be without such competition"

Thane Ruthenis @ 2025-06-06T13:01 (+3)

I could see claims like “our AI can now design bioweapons” or “our AI is capable of massively speeding up research” to be brushed aside as AI hype, given the steady stream of people overhyping AI capabilities.

Which means that if an AGI lab wants to make their employees less able to successfully whistleblow, they should arrange for fake "leaks" about intelligence explosions to be happening all the time, crying wolf and poisoning the epistemic environment.

Is this what that kerfuffle was about? See also this, and the rest of that account's activity.

Those may or may not be just random shitposters, but I now see a clear motivation for OpenAI to actually run those psyops.

What are some candidates for x that would convince interested members of the public?

I think the whistleblower would need to grab some actual proof of their claims: outputs of the models' R&D/bioweapons research, or logs displaying the model's capability as it autonomously researches stuff...

A copy of internal correspondence discussing all of this might also work, if there's sufficient volume of it. Or copies of internal papers which confirm/enable the breakthroughs.

calebp @ 2025-06-06T15:52 (+4)

I have updated upwards a bit on whistleblowers being able to make credible claims on IE. I do think that people in positions with whistleblowing potential should probably try and think concretely about what they should do, what they'd need to see to do it, and who specifically they'd get in contact with, and what evidence might be compelling to them (and have a bunch of backup plans).