"Disappointing Futures" Might Be As Important As Existential Risks

By MichaelDickens @ 2020-09-03T01:15 (+96)

Confidence: Possible

Summary

Cross-posted to my website.

Introduction

As defined by Toby Ord in The Precipice, "An existential catastrophe is the destruction of humanity’s longterm potential." Relatedly, a disappointing future is when humans do not go extinct and civilization does not collapse or fall into a dystopia, but civilization[1] nonetheless never realizes its potential.

The most salient (although perhaps not the most probable) example of a disappointing future: civilization continues to exist in essentially the same form that it has for the past few hundred years or so. If we extrapolate from civilization's current trajectory, we might expect the long-run future to have these features:

Call this the "naively extrapolated future". We could certainly extrapolate other plausible futures from civilization's current trajectory—for example, if we extrapolate the expanding circle of moral concern, we might predict that future humans will care much more about animals' welfare. And if humans become sufficiently technologically powerful, we might decide to end wild animal suffering. I don't actually believe the naively extrapolated future is the most plausible outcome—more on that later—but I do think if you asked most people what they expect the world to look like a thousand years from now, they'd predict something like it.

A note on definitions: Under the definitions of "existential catastrophe" and "disappointing future", it's debatable whether a disappointing future counts as a type of existential catastrophe. But when people talk about interventions to reduce existential risk, they almost always focus on near-term extinction events or global catastrophes. If civilization putters around on earth for a few billion years and then goes extinct when the sun expands, that would qualify as a disappointing future. It might technically count as an existential catastrophe, but it's not what people usually mean by the term. If we want to reduce the risk of a disappointing future, we might want to focus on things other than typical x-risk interventions. Therefore, it's useful to treat disappointing futures as distinct from existential catastrophes.

In this essay, I argue that disappointing futures appear comparably important to existential catastrophes. I do not attempt to argue that effective altruists should prioritize disappointing futures. In terms of the scale/neglectedness/tractability framework, this essay focuses primarily on scale (although I will touch on tractability).

The basic argument:

  1. The importance of a particular future equals its goodness or badness, weighted by its probability of occurring.[2]
  2. We might prefer a disappointing future over one where an existential catastrophe occurs, but both massively under-shoot civilization's potential. Disappointing futures and existential catastrophes look comparably bad in comparison to the best possible outcomes.
  3. Disappointing futures are approximately as probable as existential catastrophes (to within an order of magnitude).
  4. Therefore, disappointing futures are about as important as existential catastrophes.

The rest of this essay is structured as follows:

The first few parts of this essay can be read somewhat independently. Readers who feel they already understand the concept of disappointing futures can skip past "Examples of disappointing futures", and start reading at "Why we should be concerned about disappointing futures". Readers who agree that disappointing futures matter, or who just want to see quantitative predictions, can skip forward to "Comparing disappointing futures to x-risks".

Examples of disappointing futures

My description of the naively extrapolated future paints a fairly specific picture. In this section, I will describe some disappointing futures that don't require making as many predictions about how the future will look.

For each of these possible futures, the claim that it would be bad depends on adopting at least one controversial philosophical premise. I will not attempt to fully justify these premises—entire dissertations could be (and indeed have been) written about them. But you only need to accept the premises of one of these possible futures, or of some other future I didn't list, for my argument to work. Under each example, I will briefly describe the required premises.

This is not meant to be a comprehensive list; it just includes some of the most easily-imaginable disappointing futures.

We never leave the solar system

If we spread throughout the galaxy, we could create many orders of magnitude more happy lives—perhaps on the order of a trillion times more than exist today. If we fail to do this, we reduce the value of the future by a factor of a trillion.

For the sake of illustration, suppose humanity has a 1 in 3 chance of extinction, a 1 in 3 chance of surviving but staying on earth, and a 1 in 3 chance of populating the galaxy. If we have the opportunity to snap our fingers and reduce the chance of one of the two bad outcomes to zero, then we prefer to eliminate the "stay on earth" outcome. And not just slightly prefer it—preventing this outcome is a trillion times better than preventing extinction. If we have the choice between reducing extinction risk to 0 with certainty, or reducing the chance of staying on earth by only 1 in a billion (from 33.3333333% to 33.3333332%), then we should still strongly prefer the latter.[3]

Why might humans never spread throughout the universe?

We have already spread across almost all (land) regions of the earth. Given sufficient time, humans tend to populate habitable areas. However:

  1. Humans spreading across the planet coincided with increasing population, but human population appears to be stabilizing, which reduces the value of new land.
  2. Interstellar travel involves significant hurdles that we have never faced before. Other planets are uninhabitable, and we would have to invest massive efforts into terraforming them. Before colonizing other planets, one would expect humans to colonize Antarctica or the bottom of the ocean (or Nebraska for that matter), and people have not shown much interest in those.

Tyler Cowen spoke on the 80,000 Hours Podcast on why he expects humanity not to populate other planets:[4]

I think space is overrated.

It’s far, there are severe physical strains you're subject to while you’re being transported, communication back and forth takes a very long time under plausible scenarios limited by the speed of light. And what’s really out there? Maybe there are exoplanets, but when you have to construct atmosphere, there’s a risk diversification argument for doing it.

But simply being under the ocean or high up in the sky or distant corners of the earth, we’re not about to run out of space or anything close to it. So I don’t really see what’s the economic reason to have something completely external, say, to the solar system.

The idea that somehow we’re going to be sitting here three million years from now, and I’ll have my galaxy and you’ll have yours, and we’re not even human anymore. It’s not even recognizable as something from science fiction. I would bet against that if we could arrange bets on it.

I would take the other side of that bet, conditional on civilization surviving long enough. But Cowen's prediction is not unreasonable.

Required philosophical premises

The badness of this scenario depends on something like the total view of population ethics with linear aggregation:

  1. It is good to create happy people, and bad not to.
  2. All else equal, each additional happy person matters as much as the last one.

Other views might claim that something can only be good if it is good for someone (the person-affecting view) or that additional people have diminishing marginal value. (For a review of the most important views in population ethics, see Greaves (2017), Population Axiology.)

Under these other views, it is okay not to create astronomical numbers of people in the far future, so it does not much matter if civilization never leaves earth.

However, to believe that existential risk matters overwhelmingly more than short-term causes, you need to adopt (something like) the total view with linear aggregation. Under non-total views, an existential catastrophe is still bad. But most justifications for prioritizing existential risk reduction focus on the overwhelming importance of the far future, which entails the total view with linear aggregation.[5] For more, see Beckstead (2013), On the Overwhelming Importance of Shaping the Far Future, sections 4 and 5.

If we reject either of these premises, we must also reject the overwhelming importance of shaping the far future.

Massive nonhuman animal suffering continues

Even if humans colonize the galaxy, this could be bad if we bring wild animals along with us. There are many more wild animals than humans, and especially so for small animals like insects and nematodes. Depending on assumptions about how consciousness works and on how happiness and suffering trade off, it seems likely that wild animal suffering outweighs human happiness. A galaxy filled with humans and wild animals might not merely be disappointing—it could be worse than extinction.

Most people value nature, and many see a moral duty to preserve it. We might expect future humans to bring nature with them to other planets. But over the last century, human populations have increased much more rapidly than other animals', with the exception of farmed animals; if this trend continues, the balance of welfare might tip toward positive. Even so, a future universe filled with animal suffering would contain much less net welfare than it should, thus qualifying as a disappointing future.[6] (Although a net positive future with small wild animal populations still seems less disappointing than a future where we never leave earth at all.)

Required philosophical premises

This scenario is only bad if you assign non-trivial moral value to non-human animals. for more on this, see Schukraft (2020), How to Measure Capacity for Welfare and Moral Status. The degree of badness of this scenario depends on how bad you believe wild animals' lives are.

Its badness does not depend on any controversial assumptions about population ethics, because while not all widely-held views agree that it is good to create happy beings, they do agree that it is bad to create beings who live miserable lives. In fact, under non-total views of population ethics, wild animal suffering could matter more than under the total view, because creating unhappy wild animals cannot be outweighed by creating happy humans.

We never saturate the universe with maximally flourishing beings

If we accept something like the total view of population ethics with linear aggregation, it follows that we should enrich the universe with as much goodness as possible. That means creating maximum pleasure, or eudaimonia, or whatever it is we consider valuable.

The current structure of the world uses almost all available energy on things that aren't valuable in themselves, like heating and transportation. We could use that energy to make more happy people instead. And the beings capable of the greatest flourishing probably don't look much like humans. By retaining physical human bodies, we would lose out on most of the possible good we could create. (David Pearce writes of the hedonistic imperative to create beings capable of much greater heights of flourishing than we can imagine.) And if we use our resources efficiently, we could create astronomically more flourishing beings than exist today.[7]

This vision does not depend on specific assumptions about what "flourishing" looks like. It could fit the hedonistic utilitarian idea of hedonium—matter that's organized to efficiently produce simple but maximally happy beings, who have no functions other to experience happiness—but it could also look like something else. For almost any conception of what constitutes the good, surely we can find better ways of expressing it than with human bodies living in large, mostly-empty spaces.

But humans tend to like their bodies. Many people want to see the human species continue. Some people have proposed that we leave earth relatively untouched, while filling the rest of the universe with hedonium, achieving a good compromise across multiple value systems. If we don't do this, and simply fill the universe with a human (or post-human) civilization that uses energy about as inefficiently as we use it today, this would represent a major missed opportunity.

Required philosophical premises

Like "We never leave earth", the badness of this scenario presupposes the total view of population ethics.

This scenario does not depend on any particular assumption about what constitutes happiness/flourishing. The only necessary assumption is that the best possible future is far better than a reasonably good one, even one where humans populate the accessible universe.

Why we should be concerned about disappointing futures

In the past, when I've talked to friends about the possibility of disappointing futures, especially friends who prioritize existential risk, they've tended to dismiss it as too unlikely. For example, they argue that wild animal suffering will not continue because technology will advance to the point that we have the power to end wild animal suffering. And even if most people don't care about it, all we need is that a few people care enough to make it happen. Alternatively, I have heard people argue that civilization will stop consisting of biological beings (perhaps because humans upload their brains) and wild animals will cease to exist.

I believe some people are too confident in these arguments.

I find them plausible, at least. But I can also see plausible arguments that we shouldn't worry about existential risks. The case for prioritizing x-risk reduction does not depend on a strong conviction that an existential catastrophe will occur otherwise. It only requires that an existential catastrophe appears to be a reasonable possibility (and might not even require that, if you're willing to take bets with high apparent EV but tiny probabilities of success).

Similarly, the case for caring about disappointing futures does not require that they appear likely, only that they appear to be a reasonable possibility.

My best guess is that marginal work on x-risk reduction does more expected good than marginal work on disappointing futures. (More on this later.) But I do believe disappointing futures matter on roughly the same scale as x-risks (to within an order of magnitude), and have received almost no discussion. So they represent an under-investigated opportunity.

Below, I present a series of arguments for why we might expect the future to be disappointing. I find all of them plausible but not strongly compelling. I can also come up with many arguments to the contrary, on why disappointing futures might not occur. My intention is to show that we cannot easily dismiss the possibility, so I will only present arguments in favor.

Argument from the difficulty of mutually beneficial trades

People's lives have generally gotten better over time, so it seems natural to predict that eventually we will solve all major problems and ensure everyone lives a fulfilling life. This prediction, while plausibly true, is certainly not obviously true.

So far, civilization has improved the lives of most participants because it has allowed people to make mutually beneficial trades. But it has utterly failed to improve the lives of non-participants, most notably non-existent potential humans and non-human animals. In the case of animals, civilization has in fact made their lives much worse, to an extent that perhaps outweighs all the good civilization has done for humans (at least so far).[8][9] And while humans' lives have tended to get better across generations, this has happened more or less accidentally due to economic and technological development—past generations did not focus on growing the economy so they could provide value future generations. And this has only benefited existing people, not nonexistent potential people. While many people want to have children, almost nobody intrinsically cares about bringing new people into existence irrespective of kinship.

Life has gotten better (for humans) through two mechanisms:

  1. The economy allows people to gain from helping others (e.g., by developing new technology and selling it). Increasing globalization makes this process easier.
  2. Most people have a strong evolutionary desire to reproduce, so civilization continues across generations.

At a bird's-eye view, people in society act in their own self-interest (hence the common economic concept of homo economicus). In order to avoid one of the disappointing futures described above, this must fundamentally change. To end wild animal suffering, or to create increasingly many happy people, or to create non-human sentient beings, actors must do things that provide no personal benefit. People do behave selflessly all the time, but (1) not frequently enough to be visible at the civilizational level, and (2) people almost exclusively behave altruistically toward those with whom they have a personal connection. (For example, charitable donations clearly do help people; but improvements in standard of living over the past several hundred years have come almost entirely from people behaving self-interestedly in ways that had positive externalities, not from charity.)

In theory, we might achieve much larger human populations simply due to people's evolutionary motivations to spread their genes as widely as possible. But empirically, this doesn't seem to happen—people in wealthier societies tend to have fewer children.

If civilization will someday spend substantial resources on helping non-participants such as wild animals and non-existent future people, this seems to require a fundamental change in how people behave. Such a change could occur, but a hypothesis that posits a major deviation in human behavior should be viewed with suspicion.

The expanding circle

As Peter Singer and others have observed, over time, civilization has tended to widen its circle of moral concern. If we extrapolate the expanding circle, we might expect society's concern to extend to all morally relevant beings. If so, society will probably aim to create the best possible world for all beings, averting a disappointing future. But we have some reason to doubt that this trend exists.

What is the probability that civilization will eventually come to incorporate the interests of all morally relevant beings? This is a complex sociological question that I am not qualified to answer. My weakly-held impression is that historical instances of moral circle expansion generally only occurred when the people in power could make beneficial trades with the previously-disenfranchised individuals or with those individuals' kin. In any case, I do not believe we can confidently claim that future society will properly recognize the interests of all morally relevant beings.

More writings on this topic:

  1. Aaron Gertler, "The Narrowing Circle (Gwern)"
  2. Grue_Slinky, "The Moral Circle is not a Circle"
  3. Michael Aird, "Moral circles: Degrees, dimensions, visuals"

Why might civilization eventually benefit non-participants?

I have heard a few specific predictions on why we shouldn't worry about moral circle expansion:

  1. We will develop an AI that runs something like Coherent Extrapolated Volition, and it will figure out that we should care about wild animals, future humans, and any other beings that might matter.
  2. A small number of altruists will single-handedly (few-handedly?) do what's necessary to help disenfranchised groups.
  3. There won't be any biological beings in the far future, so there will be no more wild animal suffering.

(The third prediction only matters for wild animal suffering, not for other causes of disappointing futures.)

I won't address any of these in detail. But all of these predictions depend on fairly specific prerequisites. For example, #2 requires that these altruists have sufficient resources to substantially impact the world, and that their goals will not conflict with anyone else's. If altruists decide to, say, eliminate predation, then they might face resistance from people who aesthetically value nature. If altruists want to build Dyson spheres to power tiny maximally-happy brains, that requires taking up lots of resources that other actors might want.[10]

Be wary of predictions that have specific prerequisites. Do not dismiss them entirely, but pay attention to their prerequisites.

Argument from the rarity of values realization

Even when people hold certain values, they typically fail to live up to those values except when it's expedient. Some examples:

  1. In Peter Singer's "drowning child" thought experiment, almost everyone agrees that we have a moral obligation to save the child, and that physical distance is not morally relevant. But almost nobody donates meaningful amounts of money to global poverty alleviation, even though doing so would not require any personal hardship for a large percentage of the population.
  2. A 2015 Gallup poll found that 32% believe animals should have the same rights as people, even though only 5% are vegetarian.[11] (Surveys like this often suffer from social desirability bias and other issues. Nonetheless, 32% vs. 5% seems like such a large difference that it probably overwhelms any methodological flaws.)
  3. Famously, several of America's founding fathers (most notably Thomas Jefferson) opposed slavery even though they owned slaves. Slavery did not end in the developed world until industrialization rendered the practice largely unprofitable.

In Utilitarianism (1863), John Stuart Mill made a similar observation:

All social inequalities which have ceased to be considered expedient, assume the character not of simple inexpediency, but of injustice...The entire history of social improvement has been a series of transitions, by which one custom or institution after another, from being a supposed primary necessity of social existence, has passed into the rank of universally stigmatised injustice and tyranny. So it has been with the distinctions of slaves and freemen, nobles and serfs, patricians and plebeians; and so it will be, and in part already is, with the aristocracies of colour, race, and sex.

Mill paints an optimistic picture of a historical trend of improvements in social equality. But this trend also suggests some reason for pessimism: if we expect a particular injustice to remain expedient, then it seems likely to continue indefinitely.

It seems highly likely that, given continuing technological and economic progress, global poverty will continue to shrink, and society will eventually replace factory farming with more humane methods of producing food. But what about the inequality between existent and non-existent beings? It seems likely that bringing happy beings into existence (outside of one's own offspring or relatives) will never become expedient.[12]

Argument from the impossibility of reflective equilibrium

Brian Tomasik writes:

Reflective-equilibrium views aim to resolve ethical disagreements into a self-consistent whole that society can accept. One example is Eliezer Yudkowsky's coherent extrapolated volition (CEV), which would involve finding convergent points of agreement among humans who know more, are more connected, are more the people they wish they were, and so on.

A main problem with this approach is that the outcome is not unique. It's sensitive to initial inputs, perhaps heavily so. How do we decide what forms of "extrapolation" are legitimate? Every new experience changes your brain in some way or other. Which ones are allowed? Presumably reading moral-philosophy arguments is sanctioned, but giving yourself brain damage is not? How about taking psychedelic mushrooms? How about having your emotions swayed by poignant music or sweeping sermons? What if the order of presentation matters? After all, there might be primacy and recency effects. Would we try all possible permutations of the material so that your neural connections could form in different ways? What happens when -- as seems to me almost inevitable -- the results come out differently at the end?

There's no unique way to idealize your moral views. There are tons of possible ways, and which one you choose is ultimately arbitrary. (Indeed, for any object-level moral view X, one possible idealization procedure is "update to view X".) Of course, you might have meta-views over idealization procedures, and then you could find idealized idealized views (i.e., idealized views relative to idealized idealization procedures). But as you can see, we end up with infinite regress here.

If civilization undertakes a Long Reflection, it might fail to reach reflective equilibrium, or it might land on a bad set of terminal values.

A counter-argument: if we do not believe civilization will identify correct values after a Long Reflection, why should we expect our current values to be correct? Determining whether we should care about this argument essentially requires solving meta-ethics. It is extremely unclear to me how much credence I should put on the outcome of a Long Reflection if it disagrees with my current values. As a result, I consider this argument (the impossibility of reflective equilibrium) fairly weak, but hard to definitively dismiss.

Argument from technological hurdles

I alluded to this argument previously: humans might not leave earth because doing so requires overcoming significant technological hurdles without much obvious advantage.

In the past, when humans migrated to new areas, they were largely motivated by economic opportunity. Other planets do not appear to offer any such opportunities, or at least not as directly. And while migrating on earth is not easy, migrating to another planet is far more difficult.

Right now we don't have the technology to populate other planets, and we can only loosely speculate as to how we might develop it in the future. This holds especially true for interstellar travel. We might someday become sufficiently economically and technologically prosperous that interstellar colonization becomes easy, but then again, we might not.

Interstellar travel, terraforming, and other exceptionally difficult problems might only be solvable by a large unified Manhattan Project-style effort. In the past, people have mustered these sorts of efforts during wartime, or in bids to achieve global dominance, or possibly due to other idiosyncratic motivations that might not repeat themselves in the future. Interstellar colonization might never happen without a major concerted push for it.

Another possibility: interstellar colonization might simply be impossible no matter how advanced our technology becomes. This seems implausible, but if true, it would mean we should not prioritize trying to make it happen, because we can't do anything about it anyway.

For more, see Mike Smith on the difficulty of interstellar travel for humans.

Suppose the ideal future contains beings that experience much greater heights of well-being than humans do. Creating such beings might require substantial technological advances that won't occur without concerted effort. If civilization broadly agrees that this is a desirable goal, then they can probably make it happen. But if we depend on a small number of altruists to create most of the value of the future (a possibility that Paul Christiano explores in "Why might the future be good?"), then they might not have sufficient resources to achieve this goal.

Argument from natural selection

People who use their resources for altruistic purposes might get out-competed by self-interested actors. Or evolution might otherwise select for undesirable properties that make it much less likely that we will achieve a maximally good future. This could happen either with ordinary biological humans or with post-humans.

Undesirable natural selection could lead to catastrophically bad futures, such as if beings who live happy lives get out-competed by those who experience substantial suffering. Or it could lead to merely disappointing outcomes, such as if selection pressures against altruism make people less willing to create new happy beings.

The details of how natural selection might operate in the future can get fairly technical. I will not discuss them in detail. Nick Bostrom's The Future of Human Evolution provides a more detailed perspective.

Argument from status quo

Certain predictions that effective altruists commonly make—such as that we end predation, or that we upload all humans' brains onto computers, or that a superintelligent AI gains control of the world—would sound very strange to most people (and I am wary of unusual predictions that lots of smart people believe[13]). These predictions require something more than naively extrapolating from the status quo.

We should assign significant probability to the hypothesis that the world of the distant future will look basically the same as the world today, at least in the ways that count.

In general, perhaps due to the base rate fallacy, people tend to underestimate the probability that things don't change. Bryan Caplan has famously won 20 out of 20 bets, mostly by betting on the status quo (see this reddit thread for relevant discussion).

Perhaps we shouldn't extrapolate from "things usually don't change over timescales of several years" to "things don't change over centuries." Clearly things change dramatically over centuries. But we should use caution when making predictions about unprecedented events, such as humanity colonizing other planets, or creating non-biological beings. We should not assign too low a probability to the naively extrapolated future (as described in the introduction).

Furthermore, for most of human civilization, change has occurred extremely slowly. (This is even more true if you look at the history of life on earth.) The rapid change over the last few centuries represents a big deviation from this trend. We might reasonably predict a reversion to the status quo of stability across centuries.

Comparing disappointing futures to x-risks

We have some good reasons to expect civilization to achieve its potential. But as argued in the previous section, disappointing futures do not seem particularly unlikely. How likely are disappointing futures compared to existential catastrophes?

Most people who prioritize x-risk don't believe an existential catastrophe is overwhelmingly likely to occur. They use probabilistic reasoning: if this happened, it would be extremely bad; it doesn't seem overwhelmingly unlikely; therefore, it's important to work on. (I'm leaving out some steps in the argument, but that's the basic idea.) The same reasoning applies to disappointing futures.

The main argument I have heard against disappointing futures is that they probably won't happen. But this argument also defeats x-risk as a top priority.

That said, one could give two other arguments against prioritizing disappointing futures that don't apply to x-risk:

  1. A disappointing future is much less likely than an existential catastrophe.
  2. Disappointing futures are harder to prevent.

I will discuss the second argument in its own section. Let's dive into the first argument by thinking about the probabilities of a disappointing future or existential catastrophe.

Probability of an existential catastrophe

Michael Aird's database of existential risk estimates includes three estimates of the probability of existential catastrophe over timescales longer than a century:

  1. Leslie (1996)[14]: "The probability of the human race avoiding extinction for the next five centuries is encouragingly high, perhaps as high as 70 percent."
  2. Bostrom (2002)[15]: "My subjective opinion is that setting [the] probability [that an existential disaster will do us in] lower than 25% would be misguided, and the best estimate may be considerably higher."
  3. Ord (2020)[16] (p. 169): "If forced to guess, I’d say there is something like a one in two chance that humanity avoids every existential catastrophe and eventually fulfills its potential: achieving something close to the best future open to us."

A note on Ord's estimate, because he provides the most detail: it is apparent from the context of this quote that either he believes disappointing futures have negligible probability, or that they count as a form of existential catastrophe.[17]

These estimates put the total probability of existential catastrophe somewhere around 25-50%, and they all agree with each other to within a factor of two. But we only have three estimates, so we should not anchor too strongly to this range.

Arguably, we care about the probabilities of specific x-risks and specific disappointing futures more than we care about the aggregate probability, because we might be able to prevent specific futures (e.g., existential catastrophe due to AI) more effectively than we can prevent an entire class of futures (e.g., existential catastrophe in general). Predictions on specific x-risks over the long term appear even sparser than on x-risk in general. I only know of one directly relevant estimate: Grace et al. (2017)[18] surveyed AI experts and found a median 5% probability that AI results in an "extremely bad (e.g human extinction)" outcome.

Probability of a disappointing future

I do not believe we can reasonably put the probability of a disappointing future at less than about 5%, which means the probabilities of disappointing future and existential catastrophe fall within an order of magnitude of each other. (I would put my subjective probability of a disappointing future at much higher than 5%, although I do not have high confidence in my estimate.) Even if x-risks matter more than disappointing futures, they do not matter overwhelmingly more.

What do other people believe about disappointing futures?

Brian Tomasik provides a list of probability estimates on big questions about the future. Pablo Stafforini gave his probabilities on the same statements. Three years ago, I gave my own probabilities on the comments under Stafforini's post, although today I do not entirely agree with the answers I gave. I have reproduced some of the most relevant statements in the table below. Another individual created a Google Doc to allow more people to provide their own estimates, which a few people did; but nobody provided estimates on all of the statements below, so I did not include them.

Brian Pablo Michael
Human-inspired colonization of space will cause net suffering if it happens 72% 1% 40%
Earth-originating intelligence will colonize the entire galaxy (ignoring anthropic arguments) 50% 10% 20%
Wild-animal suffering will be a mainstream moral issue by 2100, conditional on biological humans still existing 15% 8% 50%
Humans will go extinct within millions of years for some reason other than AGI 5% 10% 75%

Regarding their credibility: Brian Tomasik has written many excellent essays on relevant issues. Pablo Stafforini has a strong record on measurable predictions—as of this writing, he holds rank 16 on Metaculus out of hundreds (thousands?) of active users. I would consider them both far more credible on these predictions than most people.

To my knowledge, those are all the public predictions on disappointing futures that have ever been made. I wanted more predictions, so I conducted a survey on the Effective Altruism Polls Facebook group. The survey received 12 responses—not a large sample, but much better than nothing.

The survey asked the following questions:

  1. What is the % probability that humanity achieves its long-term potential?
  2. What is the % probability that human civilization continues to exist into the long-term future, does not fall into a dystopia, but never achieves its potential?
  3. What is the % probability that the world in the long-term future is better than the world today, but less than 1000 times better?

I asked a fourth, optional question:

  1. What does "humanity achieves its long-term potential" mean to you?

The responses to this fourth question suggested that most participants roughly share the views laid out in this essay as to how good humanity's long-term potential can be.[19]

The second and third questions are meant to give two different interpretations of what a disappointing future means. The second question is less precise but more relevant, while the third question is less relevant but more precise.

This table summarizes the responses to the three main questions:

Q1 Q2 Q3
mean 19% 39% 41%
median 8% 40% 40%
standard deviation 23% 35% 39%
interquartile range 23% 62% 71%

Some key observations:

The mean and median estimates, as well as the high variance in answers, indicate that we should be concerned about the impact of disappointing futures on the expected value of the future.

However, this survey has some important limitations:

What might reduce the risk of disappointing futures?

Suppose we agree that disappointing futures appear likely—possibly even more likely than existential catastrophes. What can we do about it?

  1. Some have proposed that we can increase the probability of positive outcomes via values spreading / moral circle expansion. People who argue for prioritizing values spreading have sometimes relied on the assumption that the long-term future is likely to have negative expected value. But this assumption is not necessary. We only require that the ideal future is much more desirable than the "average" future, and that values spreading can meaningfully increase the probability of achieving this ideal. (Jacy Reese discusses the tractability of this solution.)

  2. We could structure incentives such that people properly account for the interests of beings who have no direct representation. Longtermist Institutional Reform by MacAskill & John (2020) offer some proposals on how to encourage institutions to make decisions that benefit future citizens. These proposals are preliminary and have some substantial limitations. And ensuring the interests of non-citizens (including animals, beings who would only exist if population increased, simulated minds, etc.) seems even more difficult. But the basic concept does appear promising. See also Dullaghan (2019), Deliberation May Improve Decision-Making.

  3. Reducing global catastrophic risk might be the most effective way to decrease the probability of a disappointing future. A global catastrophe that destabilizes civilization without permanently destroying it might damage civilization's propensity toward altruism. Nick Beckstead discusses this possibility in "The long-term significance of reducing global catastrophic risks".

  4. If altruistic actors build the first superintelligent AI, this could increase the chances that the AI gives proper consideration to all morally significant beings.

  5. Some have argued that the best way to improve the long-term future is to increase economic growth (see Tyler Cowen in an interview with 80,000 Hours). This seems unlikely to help us avert a disappointing future—on the margin, economic growth probably cannot (e.g.) provide sufficient resources/motivation for civilization to colonize space if it wasn't going to happen anyway. Marginal economic growth can probably only make things happen sooner, not switch them from not-happening to happening. But it's at least conceivable that increasing economic growth could reduce the risk of a disappointing future.

  6. The best way to reduce the risk of disappointing futures might be to save money. Longtermist altruists can behave more patiently than other actors. If altruists can preserve their assets over a long enough time horizon (and if non-altruists prefer not to do this), then the altruists will slowly gain power. Eventually, they can use that power to bring about an ideal world. If this plan works, it successfully avoids concerns about the impossibility of mutually beneficial trades, because only altruists will inherit the savings of the previous generation's altruists. But this requires ensuring that the money stays in altruistic hands.

These ideas all pose significant challenges. This list only represents a first look at some possible solutions.

Tractability

Existential risks and disappointing futures appear to have similar scale. The choice of which to prioritize depends primarily on tractability. Each of the possible solutions in the previous section have their own tractability issues. We cannot say much about them in general—we need to discuss each potential solution individually. So, while much could be said about each of them, I will leave such discussion out of this essay. In the previous section, I included references to deeper analyses of each solution, some of which address tractability.

I do want to raise one broad consideration on tractability. If future people do not achieve an astronomically good outcome, that means the future people who want to achieve this are unable to. If they can't set civilization on a good trajectory, why should we expect to succeed?

This seems to me like a strong argument, and pushes in favor of investing to give later as the best solution. If we expect future people to have more power to avert a disappointing future, then we should give our resources to them.

But I have three counter-arguments.

  1. Civilization might have inertia: as it grows, efforts to change its trajectory become more difficult. In that case, earlier efforts can expect to have more success, and we should act now. (However, this only applies if inertia grows more quickly than the investment rate of return. Otherwise, future beings will still be able to do more with our invested money than we can today.[20])
  2. We might live at a time of unusually high leverage. People often claim this with respect to x-risk: we face an unusually high risk of extinction, and if we can get through this critical period, we should expect to survive for millennia. Something similar could be true with respect to avoiding disappointing futures, perhaps if we expect values to "lock in" at some point, or for some other reason.
  3. Existential risk could increase over time or remain relatively high for a long time. And perhaps deliberate reductions in x-risk only persist for a few decades or centuries. So this argument against working on disappointing futures also makes working on x-risk look less valuable. Therefore, it does not make disappointing futures look less tractable relative to x-risk.

My subjective probability estimates

Making long-term predictions is hard. I do not expect that I can make reliable predictions about the long-term future of civilization. Still, for the sake of transparency, I will provide my rough probability estimates for the main outcomes discussed in this essay.

A disappointing future is more likely than an existential catastrophe: 2 in 5

Marginal work now to reduce the probability of a disappointing future does more good than marginal work now on existential risk: 1 in 3

Working to determine whether we should prioritize disappointing futures has better marginal expected value than working directly on existential risk: 1 in 3[21]

The effective altruism community under-appreciates the significance of disappointing futures: 2 in 3

On long-term outcomes:

(These add up to more than 100% because I don't want to give too much precision in my estimates.)

Acknowledgments

Thanks to David Moss and Jason Schukraft for providing feedback on drafts of this essay.

  1. I prefer to speak about existential risks with respect to civilization rather than humanity, for three reasons:

    1. We should care about all sentient life, not just humans. Use of the word "humanity" seems to implicate that only humanity matters, in a way that "civilization" doesn't. (I do not mean to suggest that people who use the word "humanity" only care about humans, because I know that is often not the case.)
    2. Civilization is the thing that has potential in the relevant sense, not the human species.
    3. Future civilization might not consist (solely) of humans.
    ↩︎

  2. I will not attempt to justify this premise, but it is generally accepted by longtermists. ↩︎

  3. This assumes that decreasing the probability of one outcome increases the probability of the following outcome. So decreasing the probability of extinction increases the probability that we survive but stay on earth, without changing the probability that we populate the galaxy. ↩︎

  4. Lightly edited to condense. ↩︎

  5. Some other views might still endorse the overwhelming importance of the far future. For example, a form of utilitarianism where the total utility of the world is the square root of the sums of each individual's utilities. But the total view with linear aggregation is the most notable such view. ↩︎

  6. In a future where we successfully implement Dyson spheres of hedonium but still have wild animals for some reason, the animals' suffering would be relatively minor compared to the hedonium, so I wouldn't consider this a disappointing future. ↩︎

  7. In Astronomical Waste, Nick Bostrom estimates that the reachable universe can concurrently support about 10^38 humans. If the earth can support up to 10^15 humans at a time (which seems like a liberal upper estimate), then failing to colonize the universe inhibits our potential by a factor of 10^23. ↩︎

  8. This depends on philosophical issues around how to trade between humans' and animals' welfare, but that's beyond the scope of this essay. ↩︎

  9. At least, this is true if we look at the animals that directly interact with civilization, most of whom live in bad conditions on factory farms. Civilization might have reduced wild animal suffering on balance, but if so, only by accident. ↩︎

  10. Carl Shulman argues that "[s]preading happiness to the stars seems little harder than just spreading." But even if (say) 50% of resources are taken by Eudaimonians and the other 50% are taken by Locusts, then shifting Eudaimonian control to 50% + epsilon still has enormous value. ↩︎

  11. Additionally, according to a survey by Sentience Institute (2017), 49% of US adults support a ban on factory farming and 33% support a ban on animal farming. This and the Gallup survey cover different populations and use different methodologies, so they aren't directly comparable. ↩︎

  12. Over long time horizons, natural selection might favor (post-)humans who try to have as many children as possible. This might be a mechanism by which long-term populations increase greatly, although it doesn't necessarily mean those future populations will be close to as happy as possible, or that they will efficiently convert energy into well-being. ↩︎

  13. The "that lots of smart people believe" clause doesn't make the predictions less likely. The point is that lots of smart people believing something isn't necessarily good evidence that it's true, particularly with regard to predictions about complicated issues. ↩︎

  14. Leslie, J. (2002). The End of the World: the science and ethics of human extinction. Routledge. ↩︎

  15. Bostrom, N. (2002). Existential Risks. ↩︎

  16. Ord, T. (2020). The Precipice. Hachette Books. Kindle Edition. ↩︎

  17. In the context surrounding this quote, Ord writes that he belives the probability of existential catastrophe in the next century is 1/3 the total probability. In chapter 6, he assigns a 1 in 6 chance to existential catastrophe this century, implying a 50% total probability of existential catastrophe. Thus, he believes the only two possibilities are existential catastrophe or humanity realizes its potential (which could be true by definition, if you define an existential catastrophe as merely failing to realize humanity's potential (as opposed to defining it as a specific event that permanently prevents humanity's potential from being realized)).

    Additionally, in chapter 6, he categorizes all existential risks and lists his probability estimates. Disappointing futures can only fit under the "Other anthropogenic risks" category, which he estimates at a 1 in 50 chance in the next century. Multiplying this by 3 gives a 6% total chance.

    However, this arithmetic doesn't really make sense, because a disappointing future can't occur in the next century. A disappointing future is the non-occurrence of humanity realizing its potential, so it doesn't happen at any particular time. Ord does not specify his long-run probability estimates of extinction from various causes, so as an upper bound, he might believe that 100% of the existential risk after the 21st century comes from disappointing futures. ↩︎

  18. Grace, K., et al. (2017). When Will AI Exceed Human Performance? Evidence from AI Experts. ↩︎

  19. For example, some of them mentioned space-faring civilization, or the possibility of creating beings capable of much greater well-being than humans. ↩︎

  20. This is analogous to how, in the Ramsey equation, the social discount rate accounts for the rate of consumption growth. Future consumption is less valuable in proportion to how wealthy people are, much like how future efforts on trajectory change are less valuable in proportion to how much inertia civilization has.

    Technically, we should discount future spending on trajectory changes at the inertia growth rate plus the philanthropic discount rate. ↩︎

  21. I would give a higher probability, but I am mildly pessimistic regarding the ability of meta-level research to solve fundamental problems in cause prioritization. If I thought cause prioritization were more tractable, I would consider it by far the most important thing to work on, given how much uncertainty we currently have. (When I speak of cause prioritization, that includes upstream issues like population ethics and decision theory.) ↩︎

Stefan_Schubert @ 2020-09-03T11:06 (+17)
I don't actually believe the naively extrapolated future is the most plausible outcome—more on that later—but I do think if you asked most people what they expect the world to look like a thousand years from now, they'd predict something like it.

In a recent paper, we asked participants:

Suppose that humanity does not go extinct, but survives for a very long time. How good do you think that the world will become in that future, compared with the present world? 

Results:

Participants believed that provided that humanity will not go extinct, the future is going to be roughly as good as the present (1 = much worse than the present world, 4 = about as good as the present world, 7 = much better than the present world; M = 4.48, SD = 1.57)...
Brian_Tomasik @ 2020-09-04T21:44 (+13)

Nice post. :) My question "Human-inspired colonization of space will cause net suffering if it happens" that I, Pablo, and you answered was worded poorly. I later rewrote it to be more clear: "Human-inspired colonization of space will cause more suffering than it prevents if it happens". As he explains in his post, Pablo (a classical utilitarian) interpreted my original wording to refer to the net balance of happiness minus suffering, while I (a negative utilitarian) meant merely the net balance of suffering. Which way did you read it?

While Pablo gave 1% probability of more suffering than happiness, he gave 99% probability that suffering itself would increase, saying: "But maybe Brian meant that colonization will cause a surplus of suffering relative to the amount present before colonization. I think this is virtually certain; I’d give it a 99% chance."

MichaelDickens @ 2020-09-04T22:00 (+3)

I interpreted it as the balance of happiness minus suffering.

Brian_Tomasik @ 2020-09-06T00:18 (+18)

Ok. :) For that question I might give a slightly lower than 50% chance that human-inspired space colonization would create more suffering than happiness (where the numerical magnitudes of happiness and suffering are as judged by a typical classical utilitarian). I think the default should be around 50% because for a typical classical utilitarian, it seems unclear whether a random collection of minds contains more suffering or happiness. There are some scenarios in which a human-inspired future might either be relatively altruistic with wide moral circles or relatively egalitarian such that selfishness alone can produce a significant surplus of happiness over suffering. However, there are also many possible futures where a powerful few oppressively control a powerless many with little concern for their welfare. Such political systems were very common historically and are still widespread today. And there may also be situations analogous to animal suffering of today in which most of the sentience that exists goes largely ignored.

The expected value of human-inspired space colonization may be less symmetric than this because it may be dominated by a few low-probability scenarios in which the future is very good or very bad, with very good futures plausibly being more likely.

Pablo_Stafforini @ 2020-09-05T10:39 (+8)

Just to say that I would put very little weight on my responses in that post, many of which are highly unstable, and some of which I no longer endorse (including the 1% and 99% estimates quoted above). I hope to revise it soon, adding measures of resilience as Greg Lewis suggests here.

antimonyanthony @ 2020-09-03T17:31 (+8)
If we reject either of these premises, we must also reject the overwhelming importance of shaping the far future.

Perhaps a nitpick (on a post that is otherwise very well done!), but as phrased this doesn't appear true. Rejecting either of those premises only entails rejecting the overwhelming importance of populating the far future with lots of happy lives. You could still consider the far future overwhelmingly ethically important in that you want to prevent it from being worse than extinction, for example.

Kaj_Sotala @ 2020-09-03T07:55 (+8)

We also discussed some possible reasons for why there might be a disappointing future in the sense of having a lot of suffering, in sections 4-5 of Superintelligence as a Cause or Cure for Risks of Astronomical Suffering. A few excerpts:

4.1 Are suffering outcomes likely?

Bostrom (2003a) argues that given a technologically mature civilization capable of space colonization on a massive scale, this civilization "would likely also have the ability to establish at least the minimally favorable conditions required for future lives to be worth living", and that it could thus be assumed that all of these lives would be worth living. Moreover, we can reasonably assume that outcomes which are optimized for everything that is valuable are more likely than outcomes optimized for things that are disvaluable. While people want the future to be valuable both for altruistic and self-oriented reasons, no one intrinsically wants things to go badly.

However, Bostrom has himself later argued that technological advancement combined with evolutionary forces could "lead to the gradual elimination of all forms of being worth caring about" (Bostrom 2004), admitting the possibility that there could be technologically advanced civilizations with very little of anything that we would consider valuable. The technological potential to create a civilization that had positive value does not automatically translate to that potential being used, so a very advanced civilization could still be one of no value or even negative value.

Examples of technology’s potential being unevenly applied can be found throughout history. Wealth remains unevenly distributed today, with an estimated 795 million people suffering from hunger even as one third of all produced food goes to waste (World Food Programme, 2017). Technological advancement has helped prevent many sources of suffering, but it has also created new ones, such as factory-farming practices under which large numbers of animals are maltreated in ways which maximize their production: in 2012, the amount of animals slaughtered for food was estimated at 68 billion worldwide (Food and Agriculture Organization of the United Nations 2012). Industrialization has also contributed to anthropogenic climate change, which may lead to considerable global destruction. Earlier in history, advances in seafaring enabled the transatlantic slave trade, with close to 12 million Africans being sent in ships to live in slavery (Manning 1992).

Technological advancement does not automatically lead to positive results (Häggström 2016). Persson & Savulescu (2012) argue that human tendencies such as “the bias towards the near future, our numbness to the suffering of great numbers, and our weak sense of responsibility for our omissions and collective contributions”, which are a result of the environment humanity evolved in, are no longer sufficient for dealing with novel technological problems such as climate change and it becoming easier for small groups to cause widespread destruction. Supporting this case, Greene (2013) draws on research from moral psychology to argue that morality has evolved to enable mutual cooperation and collaboration within a select group (“us”), and to enable groups to fight off everyone else (“them”). Such an evolved morality is badly equipped to deal with collective action problems requiring global compromises, and also increases the risk of conflict and generally negative-sum dynamics as more different groups get in contact with each other.

As an opposing perspective, West (2017) argues that while people are often willing to engage in cruelty if this is the easiest way of achieving their desires, they are generally “not evil, just lazy”. Practices such as factory farming are widespread not because of some deep-seated desire to cause suffering, but rather because they are the most efficient way of producing meat and other animal source foods. If technologies such as growing meat from cell cultures became more efficient than factory farming, then the desire for efficiency could lead to the elimination of suffering. Similarly, industrialization has reduced the demand for slaves and forced labor as machine labor has become more effective. At the same time, West acknowledges that this is not a knockdown argument against the possibility of massive future suffering, and that the desire for efficiency could still lead to suffering outcomes such as simulated game worlds filled with sentient non-player characters (see section on cruelty-enabling technologies below). [...]

4.2 Suffering outcome: dystopian scenarios created by non-value-aligned incentives.

Bostrom (2004, 2014) discusses the possibility of technological development and evolutionary and competitive pressures leading to various scenarios where everything of value has been lost, and where the overall value of the world may even be negative. Considering the possibility of a world where most minds are brain uploads doing constant work, Bostrom (2014) points out that we cannot know for sure that happy minds are the most productive under all conditions: it could turn out that anxious or unhappy minds would be more productive. [...]

More generally, Alexander (2014) discusses examples such as tragedies of the commons, Malthusian traps, arms races, and races to the bottom as cases where people are forced to choose between sacrificing some of their values and getting outcompeted. Alexander also notes the existence of changes to the world that nearly everyone would agree to be net improvements - such as every country reducing its military by 50%, with the savings going to infrastructure - which nonetheless do not happen because nobody has the incentive to carry them out. As such, even if the prevention of various kinds of suffering outcomes would be in everyone’s interest, the world might nonetheless end up in them if the incentives are sufficiently badly aligned and new technologies enable their creation.

An additional reason for why such dynamics might lead to various suffering outcomes is the so-called Anna Karenina principle (Diamond 1997, Zaneveld et al. 2017), named after the opening line of Tolstoy’s novel Anna Karenina: "all happy families are all alike; each unhappy family is unhappy in its own way". The general form of the principle is that for a range of endeavors or processes, from animal domestication (Diamond 1997) to the stability of animal microbiomes (Zaneveld et al. 2017), there are many different factors that all need to go right, with even a single mismatch being liable to cause failure.

Within the domain of psychology, Baumeister et al. (2001) review a range of research areas to argue that “bad is stronger than good”: while sufficiently many good events can overcome the effects of bad experiences, bad experiences have a bigger effect on the mind than good ones do. The effect of positive changes to well-being also tends to decline faster than the impact of negative changes: on average, people’s well-being suffers and never fully recovers from events such as disability, widowhood, and divorce, whereas the improved well-being that results from events such as marriage or a job change dissipates almost completely given enough time (Lyubomirsky 2010).

To recap, various evolutionary and game-theoretical forces may push civilization in directions that are effectively random, random changes are likely to bad for the things that humans value, and the effects of bad events are likely to linger disproportionately on the human psyche. Putting these considerations together suggests (though does not guarantee) that freewheeling development could eventually come to produce massive amounts of suffering.
Jim Buhler @ 2021-06-30T14:02 (+1)

Michael's definition of risks of disappointing futures doesn't include s-risks though, right? 

a disappointing future is when humans do not go extinct and civilization does not collapse or fall into a dystopia, but civilization[1] nonetheless never realizes its potential.

I guess we get something like "risks of negative (or nearly negative) future" adding up the two types.

Kaj_Sotala @ 2021-06-30T15:11 (+1)

Depends on exactly which definition of s-risks you're using; one of the milder definitions is just "a future in which a lot of suffering exists", such as humanity settling most of the galaxy but each of those worlds having about as much suffering as the Earth has today. Which is arguably not a dystopian outcome or necessarily terrible in terms of how much suffering there is relative to happiness, but still an outcome in which there is an astronomically large absolute amount of suffering.

Jim Buhler @ 2021-06-30T20:55 (+2)

Good point, and it is consistent with CLR's s-risks definition. :) 

WilliamKiely @ 2021-12-01T00:34 (+6)

I just came across this post and wanted to emphasize one point related to:

We never saturate the universe with maximally flourishing beings

The term "disappointing future" is a bit of a misnomer in the sense that there are many possible "disappointing futures" that are astronomically good--and therefore do not seem disappointing at first glance.

E.g. If the intrinsic value of the year-long conscious experiences of the billion happiest people on Earth in 2020 is equal to 10^9 utilons, we could say that an "astronomically good future" is a future worth >10^30 utilons.

While many astronomical good futures are near in value to an optimal future, many are <1% as valuable as an optimal future (and even <<1% as valuable), despite how they might seem like unimaginably good Bostrom utopias to us considering them today.

Such futures may involve life saturating the universe for billions of years with beings that flourish beyond our imagination today because their minds allow for much more valuable experiences than ours do, and yet they may still fall far short of what is optimal and therefore be "disappointing" and even existentially catastrophic (due to the (not-horrible-and-in-fact-extremely-good-)sub-optimality getting locked in).

The central examples of "futures with existential catastrophes" and "disappointing futures" that come to mind are not "astronomically good futures," but thinking about these fantastic futures that still fall far short of our potential may be relevant if we're trying to figure out how to maximize total utility.

WilliamKiely @ 2021-12-01T00:51 (+4)

If an event occurs that permanently locks us in to an astronomically good future that is <X% as valuable as the optimal future, has an existential catastrophe occurred? I'd like to use the language such that the answer is "no" for any value of X that still allows for the future to intuitively seem "astronomically good." If it seems like a future is astronomically good, saying an existential catastrophe has occurred in that future before all the good stuff happened feels wrong.

MichaelStJules @ 2021-12-01T06:00 (+3)

If it feels wrong to you, that's probably a reason against being a risk-neutral expected value-maximizing total utilitarian. It's possible the difference between extinction and an astronomically good default is far smaller than the difference between that default and how good it could realistically be (say by creating far more flourishing sentient beings), and the point of "existential risk"/"existential catastrophe" is to capture the stakes.

Also, moral uncertainty may push away from going for the optimum according to one view, since the optimum according to one view may sacrifice a lot on other views. However, as long as the individuals lead robustly positive existences across views, and you're committed to totalism in general, then more individuals with similar existences is linearly better.

WilliamKiely @ 2021-12-01T01:33 (+2)

I said more on this thought here.

zsp @ 2021-08-28T21:00 (+5)

I found this post after writing a related post. My post takes a different approach to the issue, fortunately. As a high-level comparison between our positions:

I would be excited to discuss possible disappointing futures (in these comments, in the comments on my post, or on a call) if you're interested.

Linch @ 2021-12-30T22:11 (+4)

So I thought about this post a bit more, particularly the we never saturate the universe with maximally flourishing beings and  impossibility of reflective equilibrium sections. 

If we accept something like the total view of population ethics with linear aggregation, it follows that we should enrich the universe with as much goodness as possible. That means creating maximum pleasure, or eudaimonia, or whatever it is we consider valuable.

[...]

This vision does not depend on specific assumptions about what "flourishing" looks like. It could fit the hedonistic utilitarian idea of hedonium—matter that's organized to efficiently produce simple but maximally happy beings, who have no functions other to experience happiness—but it could also look like something else. 

The  situation feels much bleaker to me than that, because for any very specific and plausible definition of "flourishing" X under a total view with linear aggregation, it seems to me that X would likely capture <<1% of the astronomical value under the vast majority of other plausible total/linear definitions of flourishing. 

So it seems to me that if we have a fairly high bar of "existential win", that naively doesn't look too unreasonable (like ">1% of the value of the best possible utopia by Linch's values under reflective equilibrium"), then there's an intuitively compelling worldview that there's a <1% probability of an existential win and thus x-risk is >99%, even if we get the other things right like AI alignment, large-scale coordination, long reflection, etc.

My impression is that this belief is pretty unusual, which leads me to think that I'm missing some important steps. 

Although perhaps the most important consideration here is that all these questions can be deferred to until after the long reflection.

MichaelStJules @ 2021-12-31T01:00 (+2)

How many different plausible definitions of flourishing that differ significantly enough do you expect there to be?

One potential solution would be to divide the future spacetime (not necessarily into contiguous blocks) in proportion to our credences in them (or evenly), and optimize separately for the corresponding view in each. With equal weights, each of n views could get at least about 1/n of what it would if it had 100% weight (taking ratios of expected values), assuming there isn't costly conflict between the views and no view (significantly) negatively values what another finds near optimal in practice. They could potentially do much better with some moral trades and/or if there's enough overlap in what they value positively. One view going for a larger share would lead to zero sum work and deadweight loss as others respond to it.

I would indeed guess that a complex theory of flourishing ("complexity of value", objective list theories, maybe), a preference/desire view and hedonism would assign <1% value to each other's (practical) optima compared to their own. I think there could be substantial agreement between different complex theories of flourishing, though, since I expect them generally to overlap a lot in their requirements. I could also see hedonism and preference views overlapping considerably and having good moral trades, in case most of the resource usage is just to sustain consciousness (and not to instantiate preference satisfaction or pleasure in particular) and most of the resulting consciousness-sustaining structures/activity can shared without much loss on either view. However, this could just be false.

Jim Buhler @ 2021-06-30T14:48 (+2)

Thank you for writing this.

  • According to a survey of quantitative predictions, disappointing futures appear roughly as likely as existential catastrophes. [More]

It looks like that Bostrom and Ord included risks of disappointing futures in their estimates on x-risks, which might  make this conclusion a bit skewed, don't you think?