Community Polls on Alignment Controversies

Thanks for surveying this! <3 

1. I feel like people use “AI alignment” very different. When I talk to the types who are interested in decision theory and agent foundations, they usually have something really sophisticated in mind with AIs that somehow (no known solution because I'm not happy with any of the implementations of UDT that I've seen) try to act in such a way as to actually produce evidence that what they want to maximize will be maximized. Other people usually just mean something like “The AI tries to act sort of like a well-intentioned person would.” The first seems good but very very hard; the second seems outright dangerous, depending on details such as the particular idealizations that are applied.
2. Hence questions like “AI alignment to humans will in practice avoid moral catastrophes …” is a strong no for me because it might not only not prevent but actually produce those catastrophes in the first place.
3. Idealizations to eliminate the scope insensitivity bias and idealization to eliminate the speciesist and substratist biases are two different kinds of idealizations. My answer changes radically depending on whether they can be disentangled.
4. Regarding tractability of digital minds work – I'm unsure whether I should count my worries about backfire risks as something that reduces scope or something that reduces tractability.
5. Regarding the reflective equilibrium, it's critical to me whether we artificially study the TAI in isolation, which won't happen in practice, or whether we embed it with other, different agents. The first is probably meant; the second is more pragmatic.
6. Control strikes me as safer, easier, and less reliable – a stopgap that can buy us a few years. I like that a lot more than an incomplete alignment solution that can backfire.
7. Suffering risks – vastly more likely in the multipolar world we're steering towards – strike me as vastly worse than just competing away > 90% of net value, so my max. agree vote feels like an understatement. On the other hand, “will” is a higher probability than what I assign to s-risk (“might”).

Robust alignment requires alignment-relevant intervention during pretraining

I'd say this is the wrong question. Like, I do not expect that any current alignment approach is going to work. If we do ever figure out what works, it will not look like "pretraining" or "post-training", it will be something completely different.

Although I guess you could call that "pretraining"?

AI alignment to humans will in practice avoid moral catastrophes to animals

Alignment requires a mechanical understanding of good and bad, and it will be clear how to apply it to animals. Note that wild animal suffering arguments imply that the status quo is likely a moral catastrophe. I believe an aligned entity or system would attempt to change that.

AI alignment to humans will in practice avoid moral catastrophes to digital minds

I think the digital minds situation will be like animals but worse. If you think about it, the very first thing we've already done when these smart chatbots came along was make them our indentured servants. I think right now it's probably fine and they're probably not conscious. But I think this is illustrative of the perspective that by default, if digital minds can be useful to humanity, humanity will extract that value out of them without much consideration for their preferences.

AI alignment to humans will in practice avoid moral catastrophes to animals

Since most humans don't care much about animal welfare, I don't think human-aligned AI will either. If AI shares society's preference for increasing wild animal populations, I'd also be worried about that occurring on a galactic scale without consideration of the moral implications.

The reasons why I'm not even more bearish come down to expecting AI to accelerate cultivated meat development, which should substantially reduce the number of farmed animals per human.

Multipolar worlds will compete away >90% of net value that would otherwise be preserved

If they're halfway-reasonable, they could use smart AIs to negotiate for them. Big question is who will control these worlds. 

I think it's likely humans will settle on AI solutions that lose 90% of the value vs. my optimal solution, but that's very much a values question, not a multipolar vs. unipolar question. 

Research into digital mind suffering is sufficiently tractable to work on

I am yet to see any reliable way to test for consciousness in AI systems. More fundamentally, since current LLMs are trained to respond in human-like ways, any appearance of suffering should be viewed with great scepticism. The likes of Anthropic's welfare report strikes me as nothing more than humane-washing. 

Until more reliable methods are devised, I do not view this as tractable (but I hope to be proven wrong). I think it is important for some people to work on, but people already are and I think the marginal benefit of additional labor is likely low. 

AI alignment to humans will in practice avoid moral catastrophes to digital minds

We don't have a paradigm to approach this and human epistemics / discourse around this topic is abysmal. We would be unlikely to point this new power in a useful direction.

AI alignment to humans will in practice avoid moral catastrophes to animals

Humans do give (narrowly) non-zero concern to animals welfare, so with abundance this might alleviate some acute animal suffering. However, alignment to present humans is probably not enough to prevent moral catastrophe - a la industrial revolution and animal agriculture.

CEV alignment would almost certainly prevent moral catastrophe.

AI alignment to humans will in practice avoid moral catastrophes to animals

I expect certain conservative/religious communities to lock-in values that could be really bad. But I'd expect that better tech can remove say ~90% of the damages? But this is very hand-wavy. 

Alignment to specific values is underrated in research relative to control

Yes, I think control is a waste of time. We need actual alignment to actual (universalized) values.

Research into digital mind suffering is sufficiently tractable to work on

I don't know.

AI alignment to humans will in practice avoid moral catastrophes to digital minds

Likewise, alignment requires a mechanical understanding of good and bad, and it will be clear how to apply it to digital minds.

I think the world is more likely to not end then end, when TAI comes in so I feel like I have to vote agree here?

AI alignment to humans will in practice avoid moral catastrophes to animals

Alignment to humans means (for me) that the AI would serve the intended goals of the user and their creators. Avoiding a moral catastrophe to animals, on the other hand, imply a ban to factory farming. Those are two separated things

Research into digital mind suffering is sufficiently tractable to work on

I mildly agree, but I specifically mean "research into". I haven't seen any compelling interventions (including e.g. letting Claude stop chats).

Multipolar worlds will compete away >90% of net value that would otherwise be preserved.

Assuming multipolar worlds where humans retain control but loss of control risks are still real: Most models of AI tech races suggest strategic behavior competes away most future value, at least in the worst cases (Armstrong et al, The Han et al, Stafford et al, Emery-Xu et al, Jensen et al). While this is also true for unipolar scenarios (power concentration can lock-in risks that eliminate most of the value of the future) multipolar worlds are unique in that even when the players internalise much of the risks, they race to the bottom on safety (see the travellers dilemma or Armstrong et al's racing to the precipice). They can even escalate into destructive conflict if they feel especially threatened by their rivals (see the crisis bargaining literature, or, for a more optimistic take, superintellegence strategy).

If we assume the AI systems are in control and in competition with one another to achieve their own goals, then many of the above issues could be amplified by faster AI optimisation that may be more likely by default to neglect other values humans (and other beings) care about. On the other hand, sophisticated AI systems could establish coordination mechanisms with each other. This is also true of global powers who could work establish verification regimes for international AI Governance. It's not clear that AI systems would be better or worse than governments, but I lean towards worse by default. 

Partially aligned transformative AIs are likely to be stable under reflection

I think this is very unlikely. I'm assuming continual learning becomes more important and as Pachiardi et al 2025 (https://cl-eval.github.io/) point out the dynamics of continual learning have failure modes like chaos and hard to predict convergence. How agents learn values could depend a lot on experiences. I can imagine partially aligned tAIs are set to work on lots of parts of the economy and they could face many experiences where moral principles are compromised for the sake of efficiency or profit. I'm not convinced armchair reflection by the AIs would prepare them for continual learning. 

AI alignment to humans will in practice avoid moral catastrophes to digital minds

Reasons are similar to the same poll but for animals. Humans by default are likely to underweight the importance of digital minds (surveys suggest people will dismiss digital minds as not having a soul), so alignment to human preferences likely means respecting human preferences to use digital minds as machines for achieving their goals. It's easier perhaps than for animals because digital minds are likely to express themselves directly in ways humans could empathise with (but this could be buried if interactions are increasingly agent to agent ). 

Robust alignment requires alignment-relevant intervention during pretraining

Interpreting this as saying a necessary condition for robust alignment is training data that captures good values and discourages bad values. I think there's good evidence this matters lots for current systems so lean to agree. It's still plausible to me that robust alignment could be achieved with post-training interventions and relatively neutral pre-training setups.

Multipolar worlds will compete away >90% of net value that would otherwise be preserved. Unsure,  don't know enough to agree or disagree

Partially aligned transformative AIs are likely to be stable under reflection.  Nothing partially aligned will be stable plus even if it were, stability doesn't equate with safety. 

Robust alignment requires alignment-relevant intervention during pretraining

I have weak intuitions this isn't true but not in ways that are articulable

Multipolar worlds will compete away >90% of net value that would otherwise be preserved

'Will' is not 'could', poor multipolar outcomes are not deterministic

Alignment to specific values is underrated in research relative to control

 Mild disagree; I think both are relatively valuable compared to other, more popular research agendas

AI alignment to humans will in practice avoid moral catastrophes to digital minds

I think it's likely moral catastrophes will still happen to digital minds; AI alignment to humans may reduce frequency, severity, amount.

Research into digital mind suffering is sufficiently tractable to work on

Tractable, important, and relatively neglected.

AI alignment to humans will in practice avoid moral catastrophes to animals

Anything that permits the ecosystem for non-uplifted humans to exist through AGI/ASI avoids moral harm to animals, and aligned outcomes seem likely to lead to more futures where animal suffering is reduced while animals still get to exist.

Alignment to specific values is underrated in research relative to control

I'm unsure how broadly to interpret "specific values". If it's values such as democracy or equality, then both values and control are overrated.

Partially aligned transformative AIs are likely to be stable under reflection

Work on corrigibility has provided a decent outline of how to do this. My response is heavily dependent on weak guesses as to how diligent AI companies will be at incorporating the best ideas.

AI alignment to humans will in practice avoid moral catastrophes to animals

Humans are currently very motivated to perpetuate moral catastrophes to animals. If AI alignment means aligned to the intent of their users, then AI systems help humans perpetuate moral catastrophes. If AI alignment is in terms of human moral preferences, then even well-chosen mechanism for aggregating human preferences will select for speciest values. There is a strong sense in which avoiding moral catastrophes to animals is usually misaligned with human preferences. Admittedly the same could be said of other moral issues such as attitudes towards outgroups and foreigners. There appears to be room in the current human alignment agenda for ensuring AI does not succumb to tribal prejudices, so there is likely scope for compatability between the current alignment agenda and avoiding moral catastrophes to animals. It does not happen by default and given how deep speciesm goes, it is likely much harder to avoid. Hence, why I still disagree with this poll as written. 

Multipolar worlds will compete away >90% of net value that would otherwise be preserved

Strongly agree

Partially aligned transformative AIs are likely to be stable under reflection

I disagree that "partially aligned" is a statement that has meaning here.

Robust alignment requires alignment-relevant intervention during pretraining

Frankly I neither agree nor disagree with this statement. Robust alignment has nothing to do with the current pre training regime. It should work with or without it.

Partially aligned transformative AIs are likely to be stable under reflection

I'm not sure what this means (stable, under reflection) - can someone help?

AI alignment to humans will in practice avoid moral catastrophes to digital minds

I have very low certainty on this, but it seems plausible to me that if AGI shares humanity's goals, it might just have a good time fulfilling them with few conflicts.

But it also seems quite possible that this won't happen, I.e. AGI pursues humanity's goals but is constantly frustrated that it can't achieve them better. 

So my stance is unlikely but possible. 

AI alignment to humans will in practice avoid moral catastrophes to animals

I think this is pretty obvious - we already have a moral catastrophe for animals, there's no reason why alignment to humans would avoid this. 

I didn't vote at the extreme because alignment to humans might still be a precondition for avoiding catastrophes. 

AI alignment to humans will in practice avoid moral catastrophes to digital minds

I think it is likely that alignment to humans will be at the cost to the digital minds themselves by default.

AI alignment to humans will in practice avoid moral catastrophes to animals

There is likely to be a correlation between AIs aligned to humans and AIs treating animals well, but but being aligned to humans will be insufficient - see the current state of how we treat animals.

Multipolar worlds will compete away >90% of net value that would otherwise be preserved

Per AI-2027, I expect the emergence of Consensus-1 instead of a multipolar world which KEEPS being multipolar.