Why don't governments seem to mind that companies are explicitly trying to make AGIs?

This isn't intended to be a complete response to your post, but for comparison, here are some other things that ambitious tech companies have serious plans to accomplish:

• SpaceX has a pretty credible plan to create a revolutionary space-launch system (Starship) and use it to literally colonize another planet. The USA doesn't seem as excited about this as it rationally ought to be, nor do China or other countries seem as concerned as one might expect.

• Various crypto companies and projects have big plans to literally replace fiat currency, undermining one of the fundamental ways that governments wield power. Crypto does seem to be on the radar of most governments, but still less than one might expect given the long-term stakes.

• Nuclear fusion companies like Helion, General Fusion, and others have collectively raised billions of dollars to make "energy too cheap to meter" a reality. You'd think governments would be more excited about subsidizing potentially incredible energy sources like this, but maybe they've been burned too many times before or maybe they're just slow.

• There is a lot we could straightforwardly do to be better prepared for future pandemics, but we're mostly not doing that stuff (hugely scaled-up metagenomic sequencing, improved ventilation and PPE, making vacaccines in advance for all potential pandemic viruses, etc) even though it's totally legible, it doesn't require any risky breakthroughs, and it's extremely salient due to covid.

Obviously the consequences of even Mars colonization, full cryptoization of the economy, abundant power from fusion, and significantly mitigated biorisk pale in comparison to the transformative power of AGI. But from a government's perspective they might all seem to be in the same reference class. Yet it's surprisingly close to "crickets" on all counts. (I admit that AGI might be especially neglected even here, though -- SpaceX at least gets normal NASA contracts, etc.)

Governments are concerned/interested in near-term AI. See EU, US, UK and Chinese regulation and investment. They're maybe about as interested in it as like clean tech and satellites, more than lab-grown meat.

Transformative AI is several decades away, governments aren't good at planning for possibilities over long time periods. If/when we get closer to transformative capabilities, governments will pay more attention. See: nuclear energy + weapons, bioweapons + biotech, cryptography, cyberweapons, etc etc. 

Jade Leung's thesis is useful on this. So to is Jess Whittlestone's conceptual clarifications of near/long distinctions (with Carina Prunkl) and on transformative AI (with Ross Gruetzemacher)

This interview with Obama Allan Defoe pointed to once is pretty instructive around these questions. On reflection, reasonable government actors see the case, it's just really hard to prioritize given short-run incentives ("maybe in 20 years the next generation will see things coming and deal with it").

My basic model is that government actors are all tied up by the stopping problem. If you ever want to do something good you need to make friends and win the next election. The voters and potential allies are even more short-termist than you. Availability bias explains why people would care about private nuclear weapons. Superintelligence codes as dinner party/dorm room chat. It will sell books, but it's not action relevant.  

"The average age of Members of the House at the beginning of the 117th Congress was 58.4 years; of Senators, 64.3 years."

What do you think it would look like if the US government was minding companies explicitly making AGIs?

One of EA’s most important and unusual beliefs is that superintelligent AGI is imminently possible. While ideally effective altruism is just an ethical framework that can be paired with any set of empirical beliefs, it is a very important fact that people in this community hold extremely unusual beliefs about the empirical question of AI progress.

Nobody I have ever met outside of the EA sphere seriously believes that superintelligent computer systems could take over the world within decades. I study computer science in school, I work in the field of data science, and everybody I know anticipates progress-as-usual for the foreseeable future. GPT-3 is a cool NLP algorithm, but it doesn’t spell world takeover anytime soon. The stock market would arguably agree, with DeepMind receiving a valuation of only $400M in 2014, though more recent progress within Google and Facebook has not received public financial valuations. The AI Impacts investigation into the history of technological progress revealed just how rare it is for a single innovation to bring decades worth of progress on an important metric. Much more likely in my opinion is the gradual and progressive acceleration of progress in AI and ML systems where the 21st century sees a booming Silicon Valley, but no clear “takeoff point” of discontinuous progress, with the possibility that the supposed impacts of AGI (such as automating most of the labor force or >2xing the global GDP growth rate) do not emerge for a century or centuries.

To be clear, I agree that unprecedented AI progress is possible and important. There are some strong object-level arguments, particularly Ajeya’s OpenPhil analysis of the size of the human brain vs. the size of our biggest computers. These arguments have helped convince influential experts to write books, conduct research, and bring attention to the problem of AGI safety. Perhaps the more persuasive argument is that no matter how slim the chances are, the chance cannot be disproven, and the impact of such a transformation would be so great that a group of people should be seriously thinking about. But it shouldn’t be a surprise when other groups do not take the superintelligence revolution seriously, nor should it be a surprise if the revolution does not come this century.

Epistemic Status: Possibly overstated.

EDIT: Here’s a better summary of my views. https://forum.effectivealtruism.org/posts/7ZZpWPq5iqkLMmt25/aogara-s-shortform?commentId=xZFEv84LGqbRFwt4G

You're modeling government as a single coherent actor - and I think that's the most critical mistake. That's not to say they are incompetent, just that governments aren't actually looking at what companies do to decide how to respond. (And many would say this is a feature, not a bug!)

Are you sure that they don't mind? I would be surprised if intelligence agencies weren't keeping some track of the technical capabilities of foreign entities, and I'd be unsurprised if they were also keeping track of domestic entities as well. If they thought we were six months away from transformative AGI, they could nationalize it or shut it down.

AGI concerns are outside the overtone-window and are often considered actively harmful. The narrative "The whole debate about existential risks AI poses to humanity in the far off future is a huge distraction" (as illustrated in this post https://www.skynettoday.com/editorials/dont-worry-agi/) is quite wide-spread in the AI policy community. 

In this situation, actors who raise AGI concerns thus additionally risk being portrayed as working against the public interest.

https://www.ai.gov/ What do you make of this? 

Charles He @ 2021-12-24T04:36 (+8)

My guess is that this site focuses on the prosaic, mainstream sense of AI harms, e.g. automation, privacy, competition, what Acemoglu means here.

 

By the way, of the content on the webpage "advancing trustworthy AI" seems like it could be the most relevant to AGI/ASI risk.  But the link is broken, which is really on the nose!

 

Charles_Guthmann @ 2021-12-24T12:28 (+7)

The link for the trustworth AI wasn't broken for me? https://www.ai.gov/strategic-pillars/advancing-trustworthy-ai/#Use-of-AI-by-the-Federal-Government

But unsurprisingly, it mostly seems like they are talking about bigoted algorithms and not singularity. 

However it did link this:

https://www.nscai.gov/

Find their abriged 2021 report here:

https://reports.nscai.gov/final-report/table-of-contents/ 

https://reports.nscai.gov/final-report/chapter-7/ 

Personally, this looked more promising than anything else I had seen. There was a section titled adversarial AI, which I thought might be about AGI, but upon further reading, it wasn't. So this appears to also be in the vein of what Ozzie is saying.  However, It seems they have events semi-frequently. I think someone from EA should really try to go to they are allowed.  The second link is the closest chapter in the report to AGI stuff if anyone wants to take a look- again though it's not that impressive. 

And Also I found this: https://www.dod-coe4ai-ml.org/leadership-members

But I can't really tell if this is the DODs org or Howard universities; it seems like they only hire Howard professors and students so probably the latter. 

Closest paper I could find from them to anything AGI related: https://www.techrxiv.org/articles/preprint/Recent_Advances_in_Trustworthy_Explainable_Artificial_Intelligence_Status_Challenges_and_Perspectives/17054396/1

Ozzie Gooen @ 2021-12-24T05:49 (+4)

Yep; The US government is definitely taking some actions to progress AI development in general. 

Its work to promote AI safety, and particularly, regulate or at least discuss what to do about AGI, seems to be much more lacking.

Evan R. Murphy @ 2021-12-30T19:51 (+1)

Indeed, ai.gov doesn't have even a single mention of the term "AGI".

There are two governance-related proposals in the second EA megaprojects thread. One is to create a really large EA-oriented think tank. The other is essentially EA lobbying, i.e. to put major funding behind political parties and candidates who agree to take EA concerns seriously.

Making one of these megaprojects a reality could get officials in governments to take AGI more seriously and/or get it more into the mainstream political discourse.

Andrew Yang made transformative AI a fairly central part of his 2020 presidential campaign. To the OP's point though, I don't recall him raising any alarms about the existential risks of AGI.

One possibility is that either the plausibility of AGI being developed soon is smaller than we think, or the danger it imposes is smaller than we think. This is far from the only explanation though.

I think governments are not aware of the stop button problem and they think in case of emergency they can just shut down the company / servers running the AGI using force. That's what happened in the past with digital currencies (which Jackson Wagner mentions here as a plausible member of the same reference class as AGI for governments) before bitcoin - they either failed on their own, or if successful, were shut down by government (https://en.wikipedia.org/wiki/Digital_currency#History).