How much AI suffering matters does not depend (much) on how we count AI systems

By LeonardDung @ 2026-08-29T07:29 (+4)

Leonard Dung

Confidence: I am confident in the part explaining why questions of AI identity are not quite as important as one may think. I am somewhat less confident in the brief thoughts about how to count/individuate constituents of AI suffering - the philosophical issues there are complex and little explored. I am unsure how common the view I object to really is.

The central case for thinking that the risk of creating AI systems that can suffer (e.g. are in states that are intrinsically bad for them, e.g. unpleasant experiences) is an important priority is based on the idea that there could be very many of these systems (importance). For example, people often think that trillions of AI systems capable of suffering could be created very quickly after the first such AI systems are created (for example) because it may be cheap and simple to copy them. 

However, if this is the reasoning, it seems like the importance of AI suffering risk depends centrally on questions of AI individuation: what the criteria are to count how many AI systems there are. For example, one may think, if all agents powered by GPT-5 count as one system (as parts of the GPT-5 model) there would be relatively few AI moral patients, whereas if each conversation constitutes a distinct AI moral patient there would be many. This thought is, for example, expressed in this recent paper (p. 25, cited from the preprint): "First, this objection shows that, when calculating expected utility of AI welfare, what is doing the overwhelming majority of the work is the presumption that there could be enormous amounts of AI (dis)utility. But, second, the plausibility of this presumption depends on how we are to individuate artificial conscious minds, and it’s far from clear how to do this.“

I think this is importantly overstated. It is true that some important ethical questions likely depend on AI individuation. For example, if death is harmful for AI individuals, then whether they are harmed by ending a conversation depends on whether this is morally analogous to dying for them which depends on the identity question, or something like it. However, we should first note that for some ethical views (most notably, totalist hedonist utilitarianism) questions about the individuation of persons/moral patients don’t (at least more or less) matter to our moral obligations. This is famously why many people say that these views neglect “the separateness of persons” or something like that. Second, and crucially, most ethical views would give some weight to the things that totalist utilitarians find important, just that they think things besides total utility are also important. That is, a central ethical question - even though not the only one - is how much total suffering is created by creating suffering AI systems and this question does not obviously depend on an answer to the question of AI individuation. If - for example - we think that GPT-5 can suffer and that the whole model is the correct unit of individuation, we should think that it instantiates a lot of suffering. If we think specific conversations are the unit of AI individuation, then we should think that each of these short-lived individuals suffers significantly (and somewhat proportionally) less.

What we should plausibly do, methodologically, is try to figure out which states plausibly constitute suffering (e.g. frustrated desires or unpleasant experiences) and then try to count them. There is still a philosophical individuation question here, but it is different from the question how to individuate persons/individuals/moral patients. And, for my part (more work here would be very useful!), I think it is very plausible that we should individuate suffering-constituting states in fine-grained ways, no matter what we think about identity. The entire (e.g. GPT-5) model at any given moment can power billions of agents that are motivated to do and say lots of different and conflicting things - it would seem very strange and lose all connection to our typical welfare measures (chiefly, self-report, revealed preferences, and internal activation patterns) to say that there is only one experience or desire state there. Similarly, even if we think AI moral patients should be individuated on the level of personas, since the same assistant persona can express very different and conflicting states - as expressed in behavior and activations - in different contexts, it would be very strange to say that experiences or desire-frustration events are individuated at the persona-level. If this reasoning is true and generalizes to other accounts of identity, then the case that AI suffering would be very important does not depend on any contentious metaphysical questions about individuation.