Notes on the possibility of moral progress

By MichaelDickens @ 2026-08-05T16:05 (+23)

To achieve the best possible future, we must know what that future looks like. In other words, we need to solve ethics.^[1]

The problem of solving ethics is so large and abstract that it’s difficult to say useful things about. In lieu of any structured analysis, herein lies a collection of thoughts about the problem.

Cross-posted from my website.

Acting in the face of moral uncertainty

• If we have persistent moral uncertainty between maximizing and satisficing^[2] moral theories, then it’s not difficult to decide what to do in practice: allocate a tiny portion of the universe to satisfying the non-maximizing moral theories, and allocate the rest to the maximizing moral theories.

  • Example: If theory A says we should fill the universe with welfareans, and theory B says we should preserve homo sapiens but it doesn’t much matter how many humans there are, then we can near-perfectly satisfy both theories by maintaining humanity in a small segment of the universe and giving the rest to welfareans.

  • However, the distinction between maximizing and satisficing theories may be irrelevant because plausible satisficing theories still hold that it is a good thing to maximize The Good, even though it is not morally obligatory^[3] to do so. Satisficing theories would still want to fill most of the universe with The Good.

• Given uncertainty between mutually incompatible maximizing moral theories, we have to choose. Allocating resources incorrectly would be catastrophically bad.

Can we discover facts that resolve moral uncertainty?

I believe so. I expect that we can eventually eliminate almost all moral disagreements purely by discovering facts. The fact-value distinction implies that we cannot 100% determine what we ought to do by discovering facts, but most of what look like terminal values disagreements are not truly terminal.^[4]

Some examples:

• Right now, we do not know how to weight different people’s experiences against each other. I strongly suspect that there are facts of the matter about how to weight experiences, and that these facts can be discovered empirically.

• The problem of weighting experiences is downstream of the hard problem of consciousness. I likewise suspect that there is a definitive answer to the hard problem,^[5] and that we can, in principle, find that answer.

• What is the nature of personal identity? Certain theories of personal identity rule out classes of moral theories. If personal identity is not metaphysically meaningful, then person-affecting views must be false—there is no relevant distinction between bringing a new person into existence and changing the life trajectory of an already-existing person. Changing a life trajectory creates new person-moments, which is (in this view) metaphysically equivalent to creating a new person.
• There may be some way to salvage person-affecting views, but if so, that possibility would itself be a non-normative fact—i.e., person-affecting views may be permitted or ruled out purely by facts, without any moral stance required.

• Another important (although slightly obscure) question is: given two identical copies of the same mind, are they experiencing “twice as much” as if there were only one copy, or “the same amount”?^[6] This seems like a factual question, not a moral one. I have no idea how we would answer this question, but it seems answerable in principle.

• Harsanyi’s utilitarian theorem (see also Harsanyi (1955)^[7]), which showed that if individuals have VNM utility functions, and if the Pareto principle^[8] holds over groups, then a version of utilitarianism must be true. The Pareto principle is a normative principle, not a factual one; the question of whether individuals ought to care about their own welfare is also a normative one. But I strongly suspect that there is a fact of the matter about whether an individual’s welfare can be described as a utility function, and there is a fact of the matter about how exactly that function is specified. Discovering those facts would get us at least part of the way toward a fully-specified theory of ethics.

• It matters whether the universe is finite or infinite, and whether our actions can have finite or infinite influence. Infinite ethics poses troubling problems, but some (maybe all) of those problems can be resolved factually (or if they’re unsolvable, then their unsolvability is a factual question).

• Note: I don’t think they can be resolved purely empirically. For example, by our understanding of the laws of physics, our actions cannot have infinite influence.^[9] But there is a nonzero probability that we are wrong about the laws of physics and that our actions can have infinite influence after all. No amount of empirical investigation can reduce our uncertainty to zero, but there may nonetheless be a mathematical solution to the problem.

• Standard formulations of deontology may break down in light of the fact that you do not have certainty about the consequences of your actions, so you can never be sure that you’re not doing something impermissible by acting. (See also Nye (2014)^[10].) If so, those flavors of deontology are ruled out purely based on a factual analysis (no normative claims necessary).

Many value disagreements do not purely boil down to a disagreement about facts, but I expect they can be resolved anyway. Some examples:

• People would rather donate money to a single identifiable person than to a much larger, but nebulous, group of people. This preference should break down upon reflection. Suppose Alice is offered the chance to donate to a single identifiable person. Then imagine an alternative world where Alice can donate the same amount of money to help that same single person plus several other people, but the single person is never identified. Surely she would prefer this.

• I believe the disagreement between negative utilitarians and classical utilitarians^[11] would be resolved if we knew how to directly compare experiences / if we solved the hard problem of consciousness.

  • My guess is that negative utilitarianism is a mistake stemming from the fact that maximum suffering in humans far exceeds maximum happiness, and this creates the appearance that suffering is terminally more important than happiness.

• People support animal welfare, but also eat factory-farmed animals. It’s conceivable that people in reflective equilibrium would resolve this inconsistency by throwing out their concern for animal welfare, but that seems unlikely.
• I believe people reject the mere addition paradox due to scope insensitivity—an enormous population of slightly happy people is indeed better than a small population of very happy people. It should be possible to prove that this intuition is the result of scope insensitivity, that scope insensitivity is inconsistent with people’s other values.
  • Alternatively, some argue that an enormous population of slightly happy people is not particularly good because the experiences are too uniform, and two copies of an identical experience is no better than one copy. If there is a fact of the matter about how to consider two copies of an experience, then this alternative view could be proven right.

Our understanding of philosophy is limited. What does it mean to do good philosophy? What qualifies as a good philosophical argument?

We could make progress on those questions. We have already made progress: Descartes innovated on rightly conducting reasoning and seeking truth. The significance of philosophical thought experiments is a recent development—the concept is pre-Socratic, but modern thought experiments are more refined and more useful. (On Wikipedia’s list of notable philosophy thought experiments, two thirds were invented after 1900, and over half were not developed until 1960 or later.^[12]) Most modern concepts in moral philosophy come from the 1700s or later; philosophy of mind primarily comes from the 1900s;^[13] analytic philosophy improved on the methods of its predecessors, and did not emerge until the 1800s. All that suggests that civilization is indeed making progress on philosophy, even if the rate of progress is slow.

Some normative claims evade fact-based analysis

I hold some foundational moral beliefs that seem unrelated to descriptive facts. I cannot conceive of how an empirical or theoretical investigation could provide reasons to believe that these are true or false.

• It seems self-evident that pleasurable experiences are good (and suffering is bad), in the same way it’s self-evident that I am conscious. This is difficult to dispute, and to my knowledge virtually all moral philosophers (and regular people) agree that pleasure is good and suffering is bad.

• I find it hard to see how anything other than good or bad experiences could be good or bad, because where does the goodness or badness come from if it’s not being directly experienced by anyone?^[14] (A.K.A. welfarism.) However, many people believe that non-experiences can be innately good or bad, and I don’t see how we could resolve this dispute.

• Other people’s experiences matter, not just my own. (This claim is uncontroversial, but still, I see no way to prove it, or even give any reason to believe that it’s true.)
• All beings’ experiences matter equally. It does not matter who the experience resides in; all that matters is the intensity of the experience. This view has several corollaries:
  • Welfare aggregates linearly across individuals.
  • The total view of population ethics is correct.
  • Speciesism is wrong—experiences of humans should not be given more moral weight purely due to species membership.
  • If you live in the 1700s, it implies that slavery and misogyny are wrong.

The first and third claims (the goodness of pleasure/badness of suffering, and the principle of altruism) are widely accepted. Many people disagree with me about the second and fourth claims, and there is no visible path to resolving those disagreements—plus, I have internal uncertainty about whether they’re true, which I have no idea how to resolve.

Even though the first and third claims are uncontroversial, they still evade any attempt to explain why they’re true. It could be that we’re all wrong.

And yet, it’s possible to convince people about these sorts of normative claims. (Peter Singer made arguments that convinced many people, including me, of the principle of equal consideration of interests.) What’s going on inside people’s heads when they change their minds about seemingly terminal values? Or, what’s going on when I contemplate two conflicting intuitions and decide that one is more important than the other? We have no theory of what constitutes a good argument for a normative position. We have some idea about the sorts of arguments people find convincing, but not a great understanding of why, and no way of saying that people are right to be convinced by a particular argument.

Implications for how the future goes

This essay has posited that we are making progress in philosophy, and that most (maybe all) moral disagreements can be resolved by learning new facts. If true, what does that imply?

• Smarter-than-human AI should be better than humans at discovering facts. That’s useful insofar as moral disagreements can be resolved by facts.
• The positive vision in the epilogue of AI 2040 has every individual human controlling an equal part of the lightcone. How good an outcome is that? If it’s feasible to converge on moral beliefs, then in that scenario, people (with superintelligent AI assistants) will come to agree on ethics, and will shape the universe in the way it ought to be shaped. Some people may have persistently bad values (like maybe Putin, or maybe not), but if most people converge on good values, then most of the universe will be directed well.
• How well would a Long Reflection work? It would provide more opportunity to discover ethics-relevant facts and to improve our understanding of how to do good philosophy. But it would also give amoral actors more time to seize power. The AI 2040 idea of “give everyone an equal share of the lightcone, and then let people cooperate if they want to” is plausibly better than a Long Reflection, and plausibly worse.

• Over sufficiently long time horizons, natural selection takes over. The dominant ethical belief will be that the right thing to do is to spread one’s own genes at the exclusion of everything else. We need to solve ethics before that happens, or otherwise prevent that from happening.^[15]

• In the scenario where humans control the future, the principle I worry about most is impartial altruism. I worry that most of the people in control will simply not care to devote resources to helping others.
• I worry much less about disagreements between altruistic people (e.g., between negative and classical utilitarians). I expect these disagreements can be resolved factually.

Notes

1. ^{^}

    Or perhaps it would be more accurate to speak of solving axiology, i.e., “what is good?” as opposed to “what is right?”

2. ^{^}

    A maximizing theory holds that the right things to do is to maximize some quantity—usually, to maximize utility, although “utility” can be defined in various ways. A satisficing theory says that there are certain things we ought to do (e.g. don’t commit murder), but as long as we do those, we have “satisfied” our moral obligations.

3. ^{^}

    I’m hesitant to use the word “obligatory” because it creates confusion when talking about consequentialist theories. For more on this, see Richard Y. Chappell’s blog post Deontic Pluralism (2022) or his academic paper Deontic Pluralism and the Right Amount of Good (2020).

4. ^{^}

    There is a class of moral disagreements that can clearly be resolved by factual questions. For example, should I drive a bulldozer through a particular building? That entirely depends on the answer to the factual of “is this an empty run-down building that’s scheduled to be demolished, or it a house that somebody’s living in?” Those sorts of disagreements are not interesting for the purposes of this essay.

5. ^{^}

    Some people claim that there is no fact of the matter about the hard problem of consciousness. I find it hard to comprehend why anyone holds that view. There is, at minimum, a fact of the matter about whether I am conscious (and that fact is “I am conscious”). I find this position about as confusing as the illusionist view (i.e. the view that consciousness is an illusion and in fact there is no such thing as consciousness).

6. ^{^}

    Bostrom, N. (2006). Quantity of experience: brain-duplication and degrees of consciousness.

7. ^{^}

    Harsanyi, J. C. (1955). Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparisons of Utility.

8. ^{^}

    The Pareto principle states that if outcome A is at least as good as outcome B for every person, and outcome A is better for at least one person, then outcome A is better overall.

9. ^{^}

    Sandberg, A. & Manheim, D. (2021) What Is the Upper Limit of Value?

10. ^{^}

     Nye, H. (2014). Chaos and Constraints.

11. ^{^}

     Here, classical utilitaranism refers to any flavor of utilitarianism that gives equal weight to happiness and suffering.

12. ^{^}

     Dates are pulled from Wikipedia and aggregated by Claude Opus 4.8 (chat source). Of the 41 philosophy thought experiments on Wikipedia’s list, there are 13 from pre-1900, 5 from 1900–1959, 20 from 1960–1989, and 3 from 1990–present. I’m taking Wikipedia’s list as a reasonable proxy for notability.

13. ^{^}

     Thomas Nagel’s What Is It Like to Be a Bat?, written in 1974, is a more lucid exploration of consciousness than anything that came before it, and represented important progress. More broadly, much of the best work on philosophy of mind came from people who are still alive today.

14. ^{^}

     The parable of the Hrogmorph’s Heartstone is an attempt at justifying this intuition. (See also the extended parable.)

15. ^{^}

     Natural selection hasn’t taken over yet because humans are adaptation-executors, not fitness-maximizers. We have evolved the intelligence necessary to explicitly optimize for genetic fitness, but our big brains haven’t been around long enough for natural selection to push us in that direction.