What does it mean for an AGI to be 'safe'?

I don't think it's a good plan to build an AI that enacts some pivotal act ensuring that nobody ever builds a misaligned AGI. See Critch here and here.

My reply to Critch is here, and Eliezer's is here and here.

I'd also point to Scott Alexander's comment, Nate's "Don't leave your fingerprints on the future", and my:

"Also, it's a little odd if you end up making decisions that have huge impacts on the rest of humanity that they had no say in; from a certain perspective that is inappropriate for you to do."

Keep in mind that 'this policy seems a little odd' is a very small cost to pay relative to 'every human being dies and all of the potential value of the future is lost'. A fire department isn't a government, and there are cases where you should put out an immediate fire and then get everyone's input, rather than putting the fire-extinguishing protocol to a vote while the building continues to burn down in front of you. (This seems entirely compatible with the OP to me; 'governments should be involved' doesn't entail 'government responses should be put to direct population-wide vote by non-experts'.)

Specifically, when I say 'put out the fire' I'm talking about 'prevent something from killing all humans in the near future'; I'm not saying 'solve all of humanity's urgent problems, e.g., end cancer and hunger'. That's urgent, but it's a qualitatively different sort of urgency. (Delaying a cancer cure by two years would be an incredible tragedy on a human scale, but it's a rounding error in a discussion of astronomical scales of impact.)

What, concretely, do you think humanity should do as an alternative to "build an AI that enacts some pivotal act ensuring that nobody ever builds a misaligned AGI"? If you aren't sure, then what's an example of an approach that seems relatively promising to you? What's a concrete scenario where you imagine things going well in the long run?

To sharpen the question: Eventually, as compute becomes more available and AGI techniques become more efficient, we should expect that individual consumers will be able to train an AGI that destroys the world using the amount of compute on a mass-marketed personal computer. (If the world wasn't already destroyed before that.) What's the likeliest way you expect this outcome to be prevented, or (if you don't think it ought to be prevented, or don't think it's preventable) the likeliest way you expect things to go well if this outcome isn't prevented?

(If your answer is "I think this will never happen no matter how far human technology advances" and "in particular, the probability seems low enough to me that we should just write off those worlds and be willing to die in them, in exchange for better focusing on the more-likely world where [scenario] is true instead", then I'd encourage saying that explicitly.)

When I think about building AI that is safe, I think about multiple layers of safety including monitoring, robustness, alignment, and deployment.

At that level of abstraction, I'd agree! Dan defines robustness as "create models that are resilient to adversaries, unusual situations, and Black Swan events", monitoring as "detect malicious use, monitor predictions, and discover unexpected model functionality", alignment as "build models that represent and safely optimize hard-to-specify human values", and systemic safety as "use ML to address broader risks to how ML systems are handled, such as cyberattacks". All of those seem required for a successful AGI-mediated pivotal act.

If this description is meant to point at a specific alternative approach, or meant to exclude pivotal acts in some way, then I'm not sure what you have in mind.

Safety is not a single system that doesn't destroy the world; it's an ongoing process that prevents bad outcomes.

I agree on both fronts. Destroying the world is insufficient (you need to save the world; we already know how to build AI systems that don't destroy the world), and a pivotal act fails if it merely delays doom, rather than indefinitely putting a pause on AGI proliferation (an "ongoing process", albeit one initiated by a fast discrete action to ensure no one destroys the world tomorrow).

But I think you mean to gesture at some class of scenarios where the "ongoing process" doesn't begin with a sudden discrete phase shift, and more broadly where no single actor ever uses AI to do anything sudden and important in the future. What's a high-level description of how this might realistically play out?

See Hendrycks here and here. 

You linked to the same Hendrycks paper twice; is there another one you wanted to point at? And, is there a particular part of the paper(s) you especially wanted to highlight?

Economic degrowth/stagnation is another example of something that prevents AI doom but will be very bad to have in the long run.

I like the term AGI x-safety

A lot of people misunderstand "existential risk" as meaning something like "extinction risk", rather than as meaning 'anything that would make the future go way worse than it optimally could have'. Tacking on "safety" might contribute to that impression; we're still making it sound like the goal is just to prevent bad things (be it at the level of an individual AGI project, or at the level of the world), leaving out the "cause good things" part. That said, "existential safety" seems better than "safety" to me.

(Nate's thoughts here.)

And "AI Alignment" is only part of the problem. Governance is also required to implement aligned AI and prevent unaligned AI.

I don't know what you mean by "governance". The EA Forum wiki currently defines it as:

AI governance (or the governance of artificial intelligence) is the study of norms, policies, and institutions that can help humanity navigate the transition to a world with advanced artificial intelligence. This includes a broad range of subjects, from global coordination around regulating AI development to providing incentives for corporations to be more cautious in their AI research.

... which makes it sound like governance ignores plans like "just build a really good company and save the world". If I had to guess, I'd guess that the world is likeliest to be saved because an adequate organization existed, with excellent internal norms, policies, talent, and insight. Shaping external incentives, regulations, etc. can help on the margin, but it's a sufficiently blunt instrument that it can't carve orgs into the exact right shape required for the problem structure.

It's possible that the adequate organization is a government, but this seems less likely to me given the absolute number of exceptionally competent governments in history, and given that govs seem to play little role in ML progress today.

Open Phil's definition is a bit different:

By AI governance we mean local and global norms, policies, laws, processes, politics, and institutions (not just governments) that will affect social outcomes from the development and deployment of AI systems.

Open Phil goes out of its way to say "not just governments", but its list ("norms, policies, laws, processes, politics, and institutions") still makes it sound like the problem is shaped more like 'design a nuclear nonproliferation treaty' and less like 'figure out how to build an adequate organization', 'cause there to exist such an organization', or the various activities involved in actually running such an organization and steering it to an existential success.

Both sorts of activities seem useful to me, but dividing the problem into "alignment" and "governance" seems weird to me on the above framings—like we're going out of our way to cut skew to reality.

On my model, the proliferation of AGI tech destroys the world, as a very strong default. We need some way to prevent this proliferation, even though AGI is easily-copied software. The strategies seem to be:

1. Using current tech, limit the proliferation of AGI indefinitely. (E.g., by creating a stable, long-term global ban on GPUs outside of some centralized AGI collaboration, paired with pervasive global monitoring and enforcement.)
2. Use early AGI tech to limit AGI proliferation.
3. Develop other, non-AGI tech (e.g., whole-brain emulation and/or nanotech) and use it to limit AGI proliferation.

1 sounds the most like "AGI governance" to my ear, and seems impossible to me, though there might be more modest ways to improve coordination and slow progress (thus, e.g., buying a little more time for researchers to figure out how to do 2 or 3). 2 and 3 both seem promising to me, and seem more like tech that could enable a long (or short) reflection, since e.g. they could also help ensure that humanity never blows itself up with other technologies, such as bio-weapons.

Within 2, it seems to me that there are three direct inputs to things going well:

• Target selection: You've chosen a specific set of tasks for the AGI that will somehow (paired with a specific set of human actions) result in AGI nonproliferation.
• Capabilities: The AGI is powerful enough to succeed in the target task. (E.g., if the best way to save the world is by building fast-running WBE, you have AGI capable enough to do that.)
• Alignment: You are able to reliably direct the AGI's cognition at that specific target, without any catastrophic side-effects.

There's then a larger pool of enabling work that helps with one or more of those inputs: figuring out what sorts of organizations to build; building and running those organizations; recruiting, networking, propagating information; prioritizing and allocating resources; understanding key features of the world at large, like tech forecasting, social dynamics, and the current makeup of the field; etc.

"In addition to alignment, you also need to figure out target selection, capabilities, and (list of enabling activities)" seems clear to me. And also, you might be able to side-step alignment if 3 (or 1) is viable. "Moreover, you need a way to hand back the steering wheel and hand things off to a reasonable decision-making process" seems clear to me as well. "In addition to alignment, you also need governance" is a more opaque-to-me statement, so I'd want to hear more concrete details about what that means before saying "yeah, of course you need governance too".

Perhaps ASI x-safety would be even better though (the SI being SuperIntelligent), if people are thinking “we win if we can build a useless-but-safe AGI”.