Should there be just one western AGI project?

By rosehadshar, Tom_Davidson @ 2024-12-04T14:41 (+49)

This is a linkpost to https://www.forethought.org/research/should-there-be-just-one-western-agi-project

Tom Davidson did the original thinking; Rose Hadshar helped with later thinking, structure and writing.

Some plans for AI governance involve centralising western AGI development.[1] Would this actually be a good idea? We don’t think this question has been analysed in enough detail, given how important it is. In this post, we’re going to:

  1. Explore the strategic implications of having one project instead of several
  2. Discuss what we think the best path forwards is, given that strategic landscape

(If at this point you’re thinking ‘this is all irrelevant, because centralisation is inevitable’, we disagree! We suggest you read the appendix, and then consider if you want to read the rest of the post.)

On 2, we’re going to present:

Overall, we think the best path forward is to increase the chances we get to good versions of either a single or multiple projects, rather than to increase the chances we get a centralised project (which could be good or bad). We’re excited about work on:

What are the strategic implications of having one instead of several projects?

What should we expect to vary with the number of western AGI development projects?

At a very abstract level, if we start out with some blobs, and then mush them into one blob, there are a few obvious things that change:

Summary table

VariableImplications of one projectUncertainties[2]
Race dynamics

Less racing between western projects

- No competing projects

 

 

 

 

 

Unclear implications for racing with China:

- US might speed up or slow down

- China might speed up too

Do ‘races to the top’ on safety outweigh races to the bottom?

How effectively can government regulation reduce racing between multiple western projects?

 

Will the speedup from compute amalgamation outweigh other slowdowns for the US?

How much will China speed up in response to US centralisation?

How much stronger will infosecurity be for a centralised project?

Power concentration

Greater concentration of power:

- No other western AGI projects

- Less access to advanced AI for the rest of the world

- Greater integration with USG

How effectively can a single project make use of:

- Market mechanisms?

- Checks and balances?

How much will power concentrate anyway with multiple projects?

Infosecurity

Unclear implications for infosecurity:
- Fewer systems but not necessarily fewer security components overall

- More resources, but USG provision or R&D breakthroughs could mitigate this for multiple projects

- Might provoke larger earlier attacks

How much bigger will a single project be?

How strong can infosecurity be for multiple projects?

Will a single project provoke more serious attacks?

Race dynamics

One thing that changes if western AGI development gets centralised is that there are fewer competing AGI projects.

When there are multiple AGI projects, there are incentives to move fast to develop capabilities before your competitors do. These incentives could be strong enough to cause projects to neglect other features we care about, like safety.

What would happen to these race dynamics if the number of western AGI projects were reduced to one?

Racing between western projects

At first blush, it seems like there would be much less incentive to race between western projects if there were only one project, as there would be no competition to race against.

This effect might not be as big as it initially seems though:

Also, competition can incentivise races to the top as well as races to the bottom. Competition could create incentives to:

It’s not clear how races to the top and races to the bottom will net out for AGI, but the possibility of races to the top is a reason to think that racing between multiple western AGI projects wouldn’t be as negative as you’d otherwise think.

Having one project would mean less racing between western projects, but maybe not a lot less (as the counterfactual might be well-regulated projects with races to the top on safety).

Racing between the US and China

How would racing between the US and China change if the US only had one AGI project?

The main lever that could change the amount of racing is the size of the lead between the US and China: the bigger the US’s lead, the less incentive there is for the US to race (and the smaller the lead, the more there’s an incentive).[3]

Somewhat paradoxically, this means that speeding up US AGI development could reduce racing, as the US has a larger lead and so can afford to go more slowly later.

Speeding up US AGI development gives the US a bigger lead, which means they have more time to pause later and can afford to race less.

At first blush, it seems like centralising US AGI development would reduce racing with China, because amalgamating all western compute would speed up AGI development.

However, there are other effects which could counteract this, and it’s not obvious how they net out:

So it’s not clear whether having one project would increase or decrease racing between the US and China.

Why do race dynamics matter?

Racing could make it harder for AGI projects to:

This would increase AI takeover risk, risks from proliferation, and the risk of coups (as mitigating all of these risks takes time and investment).

It might also matter who wins the race, for instance if you think that some projects are more likely than others to:

Many people think that this means it’s important for the US to develop AGI before China. (This is about who wins the race, not strictly about how much racing there is. But these things are related: the more likely the US is to win a race, the less intensely the US needs to race.[4])

Power concentration

If western AGI development gets centralised, power would concentrate: the single project would have a lot more power than any individual project in a multiple project scenario.

There are a few different mechanisms by which centralising would concentrate power:

If multiple projects compete to sell AI services to the rest of the world, the rest of the world will be more empowered.

With multiple projects there would be more independent centres of power (red diamonds).

How much more concentrated would power be if western AGI development were centralised?

Partly, this depends on how concentrated power would become in a multiple project scenario: if power would concentrate significantly anyway, then the additional concentration from centralisation would be less significant. (This is related to how inevitable a single project is - see this appendix.)

And partly this depends on how easy it is to reduce power concentration by designing a single project well.[5] A single project could be designed with:

But these mechanisms would be less robust than having multiple projects at reducing power concentration: any market mechanisms and checks and balances would be a matter of policy, not competitive survival, so they would be easier to go back on.

Having one project might massively increase power concentration, but also might just increase it a bit (if it’s possible to have a well-designed centralised project with market mechanisms and checks and balances).

Why does power concentration matter?

Power concentration could:

Infosecurity

Another thing that changes if western AGI development gets centralised is that there’s less attack surface:

Some attack surface scales with the number of projects.

At the same time, a single project would probably have more resources to devote to infosecurity:

So all else equal, it seems that centralising western AGI development would lead to stronger infosecurity.

But all else might not be equal:

If a single project is big enough, it would have more attack surface than multiple projects (as some attack surface scales with total size).

It’s not clear whether having one project would reduce the chance that the weights are stolen. . We think that it would be harder to steal the weights of a single project, but the motivation to do so would also be stronger – it’s not clear how these balance out.

Why does infosecurity matter?

The stronger infosecurity is, the harder it is for:

If we’re right that centralising western AGI development would make it harder to steal the weights, but also increase the motivation to do so, then the effect of centralising might be more important for reducing proliferation risk than for preventing China stealing the weights:

What is the best path forwards, given that strategic landscape?

We’ve just considered a lot of different implications of having a single project instead of several. Summing up, we think that:

So, given this strategic landscape, what’s the best path forwards?

Our overall take

It’s very unclear whether centralising would be good or bad.

It seems to us that whether or not western AGI development is centralised could have large strategic implications. But it’s very hard to be confident in what the implications will be. Centralising western AGI development could:

It’s also unclear what the relative magnitudes of the risks are in the first place. Should we prefer a world where the US is more likely to beat China but also more likely to slide into dictatorship, or a world where it’s less likely to beat China but also less likely to become a dictatorship? If centralising increases AI takeover risk but only by a small amount, and greatly increases risks from power concentration, what should we do? The trade-offs here are really hard.

We have our own tentative opinions on this stuff (below), but our strongest take here is that it’s very unclear whether centralising would be good or bad. If you are very confident that centralising would be good — you shouldn’t be.

Our current best guess

We think that the overall effect of centralising AGI development is very uncertain, but it still seems useful to put forward concrete best guesses on the object-level, so that others can disagree and we can make progress on figuring out the answer.

Our current best guess is that centralisation is probably net bad because of risks from power concentration.

Why we think this:

But because there’s so much uncertainty, we could easily be wrong. These are the main ways we are tracking that our best guess could be wrong:

Overall conclusion

Overall, we think the best path forward is to increase the chances we get to good versions of either a single or multiple projects, rather than to increase the chances we get a centralised project (which could be good or bad). 

The variation between good and bad versions of these projects seems much more significant than the variation from whether or not projects are centralised.

A centralised project could be:

A multiple project scenario could be:

It’s hard to tell whether a centralised project is better or worse than multiple projects as an overall category; it’s easy to tell within categories which scenarios we’d prefer.

We’re excited about work on:

For extremely helpful comments on earlier drafts, thanks to Adam Bales, Catherine Brewer, Owen Cotton-Barratt, Max Dalton, Lukas Finnveden, Ryan Greenblatt, Will MacAskill, Matthew van der Merwe, Toby Ord, Carl Shulman, Lizka Vaintrob, and others.

Appendix: Why we don’t think centralisation is inevitable

A common argument for pushing to centralise western AGI development is that centralisation is basically inevitable, and that conditional on centralisation happening at some point, it’s better to push towards good versions of a single project sooner rather than later.

We agree with the conditional, but don’t think that centralisation is inevitable.

The main arguments we’ve heard for centralisation being inevitable are:

These arguments don’t convince us:

So, while we still think that centralisation is plausible, we don’t think that it’s inevitable.

  1. ^

    Centralising: either merging all existing AGI development projects, or shutting down all but the leading project. Either of these would require substantial US government (USG) involvement, and could involve the USG effectively nationalising the project (though there’s a spectrum here, and the lower end seems particularly likely).

    Western: we’re mostly equating western with US. This is because we’re assuming that:

    • Google DeepMind is effectively a US company because most of its data centres are in the US.
    • Timelines are short enough that there are no plausible AGI developers outside the US and China.

    We don’t think that these assumptions change our conclusions much. If western AGI projects were spread out beyond the US, then this would raise the benefits of centralising (as it’s harder to regulate racing across international borders), but also increase the harms (as centralising would be a larger concentration of power on the counterfactual) and make centralisation less likely to happen.

  2. ^

     An uncertainty which cuts across all of these variables is what version of a centralised project/multiple project scenario we would get.

  3. ^

     This is more likely to be true to the extent that:

    1. There are winner-takes-all dynamics.
    2. The actors are fully rational.
    3. The perceived lead matches the actual lead.

    It seems plausible that 2 and 3 just add noise, rather than systematically pushing towards more or less racing.

  4. ^

     Even if you don’t care who wins, you might prefer to increase the US lead to reduce racing. Though as we saw above, it’s not clear that centralising western AGI development actually would increase the US lead.

  5. ^

     There are also scenarios where having a single project reduces power concentration even without being well-designed:  if failing to centralise would mean that US AGI development was so far ahead of China that the US was able to dominate, but centralising would slow the US down enough that China would also have a lot of power, then having a single project would reduce power concentration by default.

    There are a lot of conditionals here, so we’re not currently putting much weight on this possibility. But we’re noting it for completeness, and in case others think there are reasons to put more weight on it.

  6. ^

     By ‘secret loyalties’, we mean undetected biases in AI systems towards the interests of their developers or some small cabal of people. For example, AI systems which give advice which subtly tends towards the interests of this cabal, or AI systems which have backdoors.

  7. ^

     A factor which might make it easier to install secret loyalties with multiple projects is racing: CEOs might have an easier time justifying moving fast and not installing proper checks and balances, if competition is very fierce.

  8. ^

     Though these standards might be hard to audit, which would make compliance harder to achieve.

  9. ^

     There are a few ways that making it harder for China to steal the model weights might not reduce racing:

    • Centralising might simultaneously cause China to speed up its AGI development, and make it harder to steal the weights. It’s not clear how these effects would net out.
    • What matters is the perceived size of the lead. The US could be poorly calibrated about how hard it is for China to steal the weights or about how that nets out with China speeding up AGI development, such that the US doesn’t race less even though it would be rational to do so.
    • If it were very easy for China to steal the weights, this would reduce US incentives to race. (Note that this would probably be very bad for proliferation risk, and so isn’t very desirable.)

    We still think that making the weights harder to steal would probably lead to less racing, as the US would feel more secure - but this is a complicated empirical question.

  10. ^

     Bostrom defines DSA as “a level of technological and other advantages sufficient to enable it to achieve complete world domination” in Superintelligence. Tom tends to define having a DSA as controlling >99% of economic output, and being able to do so indefinitely.


Xavier_ORourke @ 2024-12-05T05:18 (+5)

I think an important consideration being overlooked is how comptetntly a centralised project would actually be managed.

In one of your charts, you suggest worlds where there is a single project will make progress faster due to "speedup from compute almagamation". This is not necessarily true. It's very possible that different teams would be able to make progress at very different rates even if both given identical compute resources.

At a boots-on-the-ground level, the speed of progress an AI project makes will be influenced by thosands of tiny decisions about how to:
 

The list goes on!

Even seemingly minor decisions like coding standards, meeting structures and reporting processes might compound over time to create massive differences in research velocity. A poorly run organization with 10x the budget might make substantially less progress than a well-run one.

If there was only one major AI project underway it would probably be managed less well than the overall best-run project selected from a diverse set of competing companies.

Unlike the Manhattan project - there's already sufficently strong commercial incentives for private companies to focus on the problem, it's not already clear exactly how the first AGI system will work, and capital markets today are more mature and capable of funding projects at much larger scales. My gut feeling is if AI was fully consolidated tomorrow - this is more likely to slow things down than speed them up.

rosehadshar @ 2024-12-05T13:33 (+5)

I agree that it's not necessarily true that centralising would speed up US development!

(I don't think we overlook this: we say "The US might slow down for other reasons. It’s not clear how the speedup from compute amalgamation nets out with other factors which might slow the US down:

  • Bureaucracy. A centralised project would probably be more bureaucratic.
  • Reduced innovation. Reducing the number of projects could reduce innovation.")

Interesting take that it's more likely to slow things down than speed things up. I tentatively agree, but I haven't thought deeply about just how much more compute a central project would have access to, and could imagine changing my mind if it were lots more.

SummaryBot @ 2024-12-04T22:23 (+1)

Executive summary: While centralizing Western AGI development into a single project could have major strategic implications, the authors argue it's unclear whether this would be beneficial overall and tentatively conclude it would be net negative due to risks from power concentration.

Key points:

  1. Race dynamics: Centralizing would reduce competition between Western projects but has unclear effects on US-China competition, potentially intensifying rather than reducing that race.
  2. Power concentration is a major concern: A single project would concentrate unprecedented power, reducing pluralism and increasing risks of coups/dictatorship, though good governance design could partially mitigate this.
  3. Information security implications are ambiguous: While fewer projects means less attack surface, a single project might attract more serious attacks and earlier attempts at theft.
  4. Rather than pushing for centralization specifically, efforts should focus on improving outcomes under either scenario through robust safeguards and governance structures.
  5. The authors reject arguments that centralization is inevitable, noting multiple projects could remain economically viable and government involvement doesn't necessitate full centralization.

 

 

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.