My thoughts on the social response to AI risk

Similary, 'Politics is the Mind-Killer' might be the rationalist idea that has aged worst - especially for its influences on EA.

What influence are you thinking about? The position argued in the essay seems pretty measured.

Politics is an important domain to which we should individually apply our rationality—but it’s a terrible domain in which to learn rationality, or discuss rationality, unless all the discussants are already rational. [...]

I’m not saying that I think we should be apolitical, or even that we should adopt Wikipedia’s ideal of the Neutral Point of View. But try to resist getting in those good, solid digs if you can possibly avoid it. If your topic legitimately relates to attempts to ban evolution in school curricula, then go ahead and talk about it—but don’t blame it explicitly on the whole Republican Party; some of your readers may be Republicans, and they may feel that the problem is a few rogues, not the entire party.

The Bay Area rationalist scene is a hive of techno-optimisitic libertarians.^[1] These people have a negative view of state/government effectiveness at a philosophical and ideological level, so their default perspective is that the government doesn't know what it's doing and won't do anything

The attitude of expecting very few regulations made little sense to me, because -- as someone who broadly shares these background biases -- my prior is that governments will generally regulate a new scary technology that comes out by default. I just don't expect that regulations will always be thoughtful, or that they will weigh the risks and rewards of new technologies appropriately.

There's an old adage that describes how government sometimes operates in response to a crisis: "We must do something; this is something; therefore, we must do this." Eliezer Yudkowsky himself once said,

So there really is a reason to be allergic to people who go around saying, "Ah, but technology has risks as well as benefits".  There's a historical record showing over-conservativeness, the many silent deaths of regulation being outweighed by a few visible deaths of nonregulation.  If you're really playing the middle, why not say, "Ah, but technology has benefits as well as risks"?

The Bay Area rationalist scene is a hive of techno-optimisitic libertarians.[1] These people have a negative view of state/government effectiveness at a philosophical and ideological level, so their default perspective is that the government doesn't know what it's doing and won't do anything. [edit: Re-reading this paragraph it comes off as perhaps mean as well as harsh, which I apologise for]

Yeah, I kinda of have to agree with this, I think the Bay Area rationalist scene underrates government competence, though even I was surprised at how little politicking happened, and how little it ended up being politicized.

Similary, 'Politics is the Mind-Killer' might be the rationalist idea that has aged worst - especially for its influences on EA. EA is a political project - for example, the conclusions of Famine, Affluence, and Morality are fundamentally political.

I think that AI was a surprisingly good exception to the rule that politicizing something would make it harder to get, and I think this is mostly due to the popularity of AI regulations. I will say though that there's clear evidence that at least for now, AI safety is in a privileged position, and the heuristic no longer applies.

Overly-short timelines and FOOM. If you think takeoff is going to be so fast that we get no firealarms, then what governments do doesn't matter. I think that's quite a load bearing assumption that isn't holding up too well

Not just that though, I also think being overly pessimistic around AI safety sort of contributed, as a lot of people's mental health was almost certainly not great at best, making them catastrophize the situation and being ineffective.

This is a real issue in the climate change movement, and I expect that AI safety's embrace of pessimism was not good at all for thinking clearly.

Thinking of AI x-risk as only a technical problem to solve, and undervaluing AI Governance. Some of that might be comparative advantage (I'll do the coding and leave political co-ordination to those better suited). But it'd be interesting to see x-risk estimates include effectiveness of governance and attention of politicians/the public to this issue as input parameters.

I agree with this, at least for the general problem of AI governance, though I disagree if we talk about AI alignment, though I agree that rationalists underestimate the governance work required to achieve a flourishing future.

From the report, the method is a multi-step process with this sample:

over 8,100 think tanks and approximately 12,800 journalists, public and private donors, and policymakers from around the world.

I wouldn't lean too much on this though? I'm not that familiar with the space, but a bunch of somewhat unknown institutes are pretty high up.

I do agree with your general point though: EA has done a lot of leg work to give credibility to AI X-risk concerns and specific issues to focus on (let's not forget CSET). This made it easy for other credible people like Bengio and Hinton to read up on the arguments and be open with their concerns. Without that leg work, things would probably have looked very differently.

The link is dead. Is it available anywhere else?

Agree, but I also think that insufficient "security mindset" is still a big problem. From OP:

it still remains to be seen whether US and international regulatory policy will adequately address every essential sub-problem of AI risk. It is still plausible that the world will take aggressive actions to address AI safety, but that these actions will have little effect on the probability of human extinction, simply because they will be poorly designed. One possible reason for this type of pessimism is that the alignment problem might just be so difficult to solve that no “normal” amount of regulation could be sufficient to make adequate progress on the core elements of the problem—even if regulators were guided by excellent advisors—and therefore we need to clamp down hard now and pause AI worldwide indefinitely.

Matthew goes on to say:

That said, I don't see any strong evidence supporting that position.

I'd argue the opposite. I don't see any strong evidence opposing that position (given that doom is the default outcome of AGI). The fact that a moratorium was off the table at the UK AI Safety Summit was worrying. Matthew Syed, writing in The Times, has it right:

The one idea AI won’t come up with for itself — a moratorium

The Bletchley Park summit was an encouraging sign, but talk of regulators and off switches was delusional

Or, as I recently put it on X. It's

Crazy that accepted levels of [catastrophic] risk for AGI [~10%] are 1000x higher (or more) than for nuclear power. Any sane regulation would immediately ban the construction of ML-based AGI.

Two key points I want to add to this summary:

1. I think these arguments push against broad public advocacy work, in favor of more cautious efforts to target regulation well, and make sure that it's thoughtful. Since I think we'll likely get strong regulation by default, ensuring that the regulation is effective and guided by high-quality evidence should be the most important objective at this point.
2. Policymakers will adjust policy strictness in response to evidence about the difficulty of alignment. The important question is not whether the current level of regulation is sufficient to prevent future harm, but whether we have the tools to ensure that policies can adapt appropriately according to the best evidence about model capabilities and alignment difficulty at any given moment in time.