Warning Shots Probably Wouldn't Change The Picture Much

I have some sympathy for the second view, although I'm skeptical that sane advisors have significant real impact. I'd love a way to test it as decisively as we've tested the "government (in its current form) responds appropriately to warning shots" hypotheses.

On my own models, the "don't worry, people will wake up as the cliff-edge comes more clearly into view" hypothesis has quite a lot of work to do. In particular, I don't think it's a very defensible position in isolation anymore....if you want to argue that we do need government support but (fortunately) governments will start behaving more reasonably after a warning shot, it seems to me like these days you have to pair that with an argument about why you expect the voices of reason to be so much louder and more effectual in 2041 than they were in 2021.

(Which is then subject to a bunch of the usual skepticism that applies to arguments of the form "surely my political party will become popular, claim power, and implement policies I like".)

I think the second view is basically correct for policy in general, although I don't have a strong view yet of how it applies to AI governance specifically. One thing that's become clear to me as I've gotten more involved in institution-focused work and research is that large governments and other similarly impactful organizations are huge, sprawling social organisms, such that I think EAs simultaneously underestimate and overestimate the amount of influence that's possible in those settings. The more optimistic among us tend to get too excited about isolated interventions (e.g., electing a committed EA to Congress, getting a voting reform passed in one jurisdiction) that, even if successful, would only address a small part of the problem. On the other hand, skeptics see the inherent complexity and failures of past efforts and conclude that policy/advocacy/improving institutions is fundamentally hopeless, neglecting to appreciate that critical decisions by governments are, at the end of the day, made by real people with friends and colleagues and reading habits just like anyone else.

Viewed through that lens, my opinion and one that I think you will find is shared by people with experience in this domain is that the reason we have not seen more success influencing large-scale bureaucratic systems is that we have have been under-resourcing it as a community.  By "under-resourcing it" I don't just mean in terms of money, because as the Flynn campaign showed us it's easy to throw millions of dollars at a solution that hits rapidly diminishing returns. I mean that we have not been investing enough in strategic clarity, a broad diversity of approaches that complement one another and collectively increase the chances of success, and the patience to see those approaches through. In the policy world outside of EA, activists consider it normal to have a 6-10 year timeline to get significant legislation or reforms enacted, with the full expectation that there will be many failed efforts along the way. But reforms do happen -- just look at the success of the YIMBY movement, which Matt Yglesias wrote about today, or recent legislation to allow Medicare to negotiate prescription drug prices, which was in no small part the result of an 8-year, $100M campaign by Arnold Ventures.

Progress in the institutional sphere is not linear. It is indeed disappointing that the United States was not able to get a pandemic preparedness bill passed in the wake of COVID, or that the NIH is still funding ill-advised research. But we should not confuse this for the claim that we've been able to do "approximately nothing." The overall trend for EA and longtermist ideas being taken seriously at increasingly senior levels over the past couple of years is strongly positive. Some of the diverse factors include the launch of the Future Fund and the emergence of SBF as a key political donor; the publication of Will's book and the resulting book tour; the networking among high-placed government officials by EA-focused or -influenced organizations such as Open Philanthropy,  CSET, CLTR, the Simon Institute, Metaculus, fp21, Schmidt Futures, and more; and the natural emergence of the initial cohort of EA leaders into the middle third of their careers. Just recently, I had one senior person tell me that Longview Philanthropy's hiring of Carl Robichaud, a nuclear security grantmaker with 20 years of experience, is what got them to pay attention to EA for the first time. All of it is, by  itself, not enough to make a difference, and judged on its own terms will look like a failure. But all of it combined is what creates the possibility that more can be accomplished the next time around, and all of the time in between.

And—despite valiant effort!—we've been able to do approximately nothing.

Why not?

I apologize for an amateur question but: what all have we tried and why has it failed?

This post makes the case that warning shots won't change the picture in policy  much, but I could imagine a world where  some warning shot makes the leading AI labs decide to focus more on safety, or agree to slow down their deployment, without policy change occurring. Maybe this could buy a couple of years time for safety researchers?

This isn't a well developed thought, just something that came to mind while reading.

To be honest, Nate’s analysis about the hope for government action sometimes comes across as if it’s from someone who never studied political economy or other poli-sci topics, assumed governments would act rational, and then concludes governments are hopeless when that assumption is flawed.

If you started out with the assumption that governments are nationally rational and fast-acting, then yes you obviously need to update away from that in light of the lack of COVID response (and many other examples predating COVID). And yes you do need to have some plausible causal chains for success.

But COVID definitely shouldn’t be assumed as proof of hopelessness (or other pessimistic claims like “Warning Shots Probably Wouldn't Change The Picture Much”), if only given examples of where government did take sensible or at least strong actions in response to threats/events (sometimes without them even occurring). See for example: 9/11, Y2K, Nunn-Lugar Cooperative Threat Reduction program, the asteroid tracking system, etc. Some of these have even been analyzed in posts here on the EA Forum! (Plausibly also worth learning about MADD: Mothers Against Drunk Driving)

Ultimately, factors such as “Probability of affecting the decision-makers personally,” “existing options for response,” “temporal sharpness of harm,” “good political narratives,” “influential advisors,” etc. matter. Are those things guaranteed for AI? No, but I don’t think this COVID situation is a good/sufficient reason to assume that we can’t act rationally in response to a clear warning shot for AI.

Models of success should reflect this uncertainty clearly in their reasoning/modeling, in case later analysis shows this to be false (i.e., warning shots for AI probably won’t matter).

I would be curious how much the pandemic preparedness stuff is actually a crux. E.g. if gain of function research is restricted within the next year, would that noticeably change your estimate of how helpful a warning shot will be?

I think this is kind of testing your argument (2) – EA advisors might possibly become more influential within the next year. (And also it just takes forever to do anything in policy.)

Random meta point: You can now crosspost posts to the EA Forum from LW and vice-versa, which automatically adds a link to the crosspost to the top, and adds a link to the comment section on the other side to the bottom of the comment section (together with a counter of the number of comments). Seems like this would have been a bit nicer for this case.

Readers might be interested in the comments over here, especially Daniel K.'s comment:

The only viable counterargument I've heard to this is that the government can be competent at X while being incompetent at Y, even if X is objectively harder than Y. The government is weird like that. It's big and diverse and crazy. Thus, the conclusion goes, we should still have some hope (10%?) that we can get the government to behave sanely on the topic of AGI risk, especially with warning shots, despite the evidence of it behaving incompetently on the topic of bio risk despite warning shots.

Or, to put it more succinctly: The COVID situation is just one example; it's not overwhelmingly strong evidence.

I just wanted to share as my experience was so radically different from yours. Based in the UK during the pandemic  I felt like:

• No one in was really doing anything to try to "make the government sane around biorisk". I published a paper targeted at government on managing risks. I remember at the time (in 2020) it felt like no one else was shifting to focus on policy change on lessons learned from COVID.
• When I tried doing stuff it went super well. As mentioned here  (and here) this work went much better than expected. The government seemed willing to update and commit to being better in future.

 I came away from the situation with a feeling that influencing policy was easy and impactful and neglected and hopefully about what policy work could achieve – but just disappointed that not more was being done to  "make the government sane around biorisk".
 

This leads me to questions Why are our experiences so different? Some hypothesis that I have are:

• Luck / randomness – maybe I was lucky or US advocates were unlucky and we should assume the truth lies in the middle.
• Different country – the US is different, harder to influence, or less sane than some (or many) other places.
• Different methodology – The standard policy advocacy sector really sucks, it is not evidence based and there is little M&E. It might be that advocacy run in an impact-focused way (like was happening in the UK) is just much better than funding standard advocacy organisations (which I guess was happening in the US). See discussion on this here.
• Different amount of work – your post mentions a "valiant effort" was made, but does not evidence this. This makes it hard to form an opinion on what works and why. Would be great to get an answer to this (see Susan's comment) e.g. links to a few campaigns in this space. 

Grateful for your views.

Can't we produce really good text-to-image educational videos? E.g. Eliezer's fictional writings are really fun, and have introduced many of us into this topic. Bonus point if these videos accurately predict the future, gaining us some sort of reputation.

I'll just note that I have a prediction market on this here, which is currently at a 7% chance of some prominent event causing mainstream AI capabilities researchers to start taking the risk more seriously by 2028.

And—despite valiant effort!—we've been able to do approximately nothing.

This should update judgements on whether GOF research is as easy to influence as was thought in 2021.

Some resources I recommend on GOF research are the first two chapters of Mearshimer's Tragedy of Great Power Politics (2014) and the first two chapters of Schelling's Arms and Influence (1966).