AGI Timelines in Governance: Different Strategies for Different Timeframes

Thanks for your comment! 
That's an important point that you're bringing up. 

My sense is that at the movement level, the consideration you bring up is super important. Indeed, even though I have fairly short timelines, I would like funders to hedge for long timelines (e.g.  fund stuff for China AI Safety). Thus I think that big actors should have in mind their full distribution to optimize their resource allocation. 

That said, despite that, I have two disagreements: 

1. I feel like at the individual level (i.e. people working in governance for instance, or even organizations), it's too expensive to optimize over a distribution and thus you should probably optimize with a strategy of "I want to have solved my part of the problem by 20XX". And for that purpose, identifying the main characteristics of the strategic landscape at that point (which this post is trying to do) is useful.
2. "the EV gained in the worlds where things move quickly is not worth the expected cost in worlds where they don't." I disagree with this statement, even at the movement level. For instance I think that the trade-off of "should we fund this project which is not the ideal one but still quite good?" is one that funders often encounter and I would expect that funders have more risk adverseness than necessary because when you're not highly time-constrained, it's probably the best strategy (i.e. in every fields except in AI safety, it's probably a way better strategy to trade-off a couple of years against better founders).

Finally, I agree that "the best strategies will have more variance" is not a good advice for everyone. The reason I decided to write it rather than not is because I think that the AI governance community tends to have a too high degree of risk adverseness (which is a good feature in their daily job) which penalizes mechanically a decent amount of actions that are way more useful under shorter timelines. 

There's a balance here for communication purposes - concrete potential timelines are easier for some of us to understand than distributions. Perhaps both could be used?

Thanks for your comment!

A couple of remarks: 

1. Regulations that cut X-risks are strong regulations: My sense is that regulations that really cut X-risks at least a bit are pretty "strong", i.e. in the reference class of "Constrain labs to airgap and box their SOTA models while they train them" or "Any model which is trained must be trained following these rules/applying these tests".  So what matters in terms of regulation is "will governments take such actions?" and my best guess is no, at least not without the public opinion caring a lot about that. Do you have in mind an example of regulation which is a) useful and b) softer than that? 
2. Additional state funding would be bad by default: I think that the default of "more funding goes towards AGI" is that it accelerates capabilities more (e.g backing some labs so that they move faster than China, things like that). Which is why I'm not super excited about increasing the amount of funding governments put into AGI. But to the extent that there WILL be some funding, then it's nice to steer it towards safety research. 

And finally, I like the example you gave on cyber. The point I was making was something like "Your theories of change for pre-2030 timelines shouldn't rely too much on national government policy" and my understanding of what you're saying is something like "that may be right, but national governments are still likely to have a lot of (bad by default) influence, so we should care about them". 

I basically had in mind this kind of scenario where states don't do the research themselves but are backing some private labs to accelerate their own capabilities, and it makes me more worried about encouraging states to think about AGI. But I don't put that much weight on these scenarios yet.

How confident are you that governments will get involved in meaningful private-public collaboration around AGI by 2030? A way of operationalizing that could be "A national government spends more than a billion $ in a single year on a collaboration with a lab with the goal to accelerate research on AGI". 

If you believe that it's >50%, that would definitely update me towards "we should still invest a significant share of our resources in national policy, at least in the UK and the US so that they don't do really bad moves". 

Hey Misha! Thanks for the comment!

I am quite confused about what probabilities here mean, especially with prescriptive sentences like "Build the AI safety community in China" and "Beware of large-scale coordination efforts."

As I wrote in note 2, I'm here claiming that this claim is more likely to be true under these timelines than the other timelines. But how could I make it clearer without bothering too much? Maybe putting note 2 under the table in italic?

I also disagree with the "vibes" of probability assignment to a bunch of these, and the lack of clarity on what these probabilities entail makes it hard to verbalize these.

I see, I hesitated in the trade-off (1) "put no probabilities" vs (2) "put vague probabilities" because I feel like that the second gives a lot more signal on how confident I am in what I say and allow people to more fruitfully disagree but at the same time it gives a "seriousness" signal which is not good when the predictions are not actual predictions.

Do you think that putting no probabilities would have been better? 

By "I also disagree with the vibes of probability assignment to a bunch of these", do you mean that it seems over/underconfident in a bunch of ways when you try to do a similar exercise? 

Ah ah you probably don't realize it but "you" is actually 4 persons: Amber Dawn for the first draft of the post, me (Simeon) for the ideas, the table and the structure of the post, and me, Nicole Nohemi & Felicity Riddel for the partial rewriting of the draft to make it clearer.

So the credits are highly distributed! And thanks a lot, it's great to hear that! 

Thanks for your comment! 

First, you have to have in mind that when people are talking about "AI" in industry and policymaking, they usually have mostly non-deep learning or vision deep learning techniques in mind simply because they mostly don't know the ML academic field but they have heard that "AI" was becoming important in industry. So this sentence is little evidence that Russia (or any other country) is trying to build AGI, and I'm at ~60% Putin wasn't thinking about AGI when he said that. 

If anyone who could play any role at all in developing AGI (or uncontrollable AI as I prefer to call it) isn't trying to develop it by now, I doubt very much that any amount of public communication will change that. 

I think that you're deeply wrong about this. Policymakers and people in industry, at least till ChatGPT had no idea what was going on (e.g at the AI World Summit, 2 months ago very few people even knew about GPT-3). SOTA large language models are not really properly deployed, so nobody cared about them or even knew about them (till ChatGPT at least). The level of investment right now in top training runs probably doesn't go beyond $200M. The GDP of the US is 20 trillion. Likewise for China. Even a country like France could unilaterally put $50 billion in AGI development and accelerate timelines quite a lot within a couple of years. 

Even post ChatGPT, people are very bad at projecting what it means for next years and still have a prior on the fact that human intelligence is very specific and can't be beaten which prevents them from realizing all the power of this technology.

I really strongly encourage you to go talk to actual people from industry and policy to get a sense of their knowledge on the topic. And I would strongly recommend not publishing your book as long as you haven't done that. I also hope that a lot of people who have thought about these issues have proofread your book because it's the kind of thing that could really increase P(doom) substantially.

I think that to make your point, it would be easier to defend the line that "even if more governments got involved, that wouldn't change much". I don't think that's right because if you gave $10B more to some labs, it's likely they'd move way faster. But I think that it's less clear. 

a common, clear understanding of the dangers

I agree that it would be something good to have. But the question is: is it even possible to have such a thing? 

I think that within the scientific community, it's roughly possible (but then your book/outreach medium must be highly targeted towards that community). Within the general public, I think that it's ~impossible. Climate change, which is a problem which is much easier to understand and explain is already way too complex for the general public to have a good idea of what are the risks and what are the promising solutions to these risks (e.g. a lot people's top priorities is to eat organic food, recycle and decrease plastic consumption).

I agree that communicating with the scientific community is good, which is why I said that you should avoid publicizing only among "the general public". If you really want to publish a book, I'd recommend targeting the scientific community, which is not at all the same public as the general public. 

"On the other hand, if most people think that strong AI poses a significant risk to their future and that of their kids, this might change how AI capabilities researchers are seen, and how they see themselves"

I agree with this theory of change and I think that it points a lot more towards "communicate in the ML community" than "communicate towards the general public". Publishing great AI capabilities is mostly cool for other AI researchers and not that much for the general public. People in San Francisco (where most of the AGI labs are) also don't care much about the general public and whatever it thinks ; the subculture there and what is considered to be "cool" is really different from what the general public thinks is cool. As a consequence, I think they mostly care about what their peers are thinking about them. So if you want to change the incentives, I'd recommend focusing your efforts on the scientific & the tech community.