Rule High Stakes In, Not Out

By Richard Y Chappell🔸 @ 2025-10-21T02:44 (+3)

Scott Alexander argued that we should rule some things (e.g. thinkers) in, not out. Ord et al (2010) explain how, when a model or argument seeks to establish a tiny probability as the best estimate of some high-stakes event (e.g. human extinction), the estimated probability may be “dwarfed by the chance that the argument itself is flawed.” Putting these together yields an important methodological heuristic for thinking about longtermism: given our immense uncertainty about the future, and the wide range of possible models and scenarios over which we should remain reasonably uncertain, we should rule high stakes in, not out.

Even just a 1% chance of extremely high stakes is sufficient to establish high stakes in expectation. So we should not feel assured of low stakes even if a highly credible model—warranting 99% credence—entails low stakes. It hardly matters at all how many credible models entail low stakes. What matters is whether any credible model entails extremely high stakes. If one does—while warranting just 1% credence—then we have established high stakes in expectation, no matter what the remaining 99% of credibility-weighted models imply (unless one inverts the high stakes in a way that cancels out the other high-stakes possibility).

In what follows, I’ll step through some highlights from the Essays on Longtermism collection that hit differently with this methodological dictum in mind.

Ruling-in Longtermist Causes

‘Minimal and Expansive Longtermism’, by Greaves & Tarsney, contrasts a “minimal” view—on which promising longtermist projects are basically exhausted by a handful of x-risk reduction priorities—from a more “expansive” view on which “There is a very wide variety of ways to greatly improve the expected value of the far future.” (And this is apt to remain true for the foreseeable future, almost no matter how many resources are dedicated to longtermism.)

They write:

The problem is that, in general, the project of trying to influence the course of the far future has a strong air of intractability. The further into the future we look, the harder it becomes to predict either what the world will look like in the absence of any intervention on our part, or the effects of any particular present action. Risks of human extinction and other ‘existential catastrophes’ create an exception to these worries about intractability, since each such risk comes with a strong and clear ‘lock-in’ mechanism. But most other ways in which we might hope to improve the far future of humanity can be motivated only via significantly more speculative reasoning concerning very long-term causal chains.

This all seems true, but may not matter so much if the “ruling in” heuristic is apt. For example, the authors later report being “unconvinced” by Tyler Cowen’s arguments for compounding economic growth as a longtermist cause. Granted, it seems perfectly reasonable to doubt that economic growth could both (i) continue so long, and (ii) continue to have significant marginal effects on well-being all the while. But that isn’t determinative; what instead matters is whether we should grant some credence to Cowen’s model of the world. (And I suspect we should!)

Later in the paper, the authors write:

Our impression is that some longtermists believe that the marginal value of the best longtermist interventions exceeds the neartermist benchmark by 10:1 or less, while others believe that this ratio is 10,000:1 or more. And there is more than enough reasonable basis for this wide range of views. In particular, estimates of the expected future population that would be lost to existential catastrophe span tens of orders of magnitude.

I wonder how much of this disagreement is due to people offering different best-guess models rather than “all-models-considered” expected values, where low-end estimates require giving near-zero credence to models implying vastly higher stakes.

Overall, my sense is that the case for “expansive longtermism” may be greatly aided by the “rule high stakes in, not out” principle, together with the possibility of credible-seeming (albeit highly speculative) arguments for a wide range of potential longtermist interventions other than traditional x-risk reduction.^[1]

Changing Population Levels

I especially liked the paper, ‘Is Extinction Risk Mitigation Uniquely Cost-Effective? Not in Standard Population Models’ by Gustav Alexandrie & Maya Eden. They note that standard models “typically imply that any shocks that proportionally decrease all factors of production have proportional, permanent effects on long-run population levels.” This means that disasters (like asteroids or wars) that destroy capital in addition to causing deaths (short of extinction) may be vastly more harmful than a comparably deadly pandemic that leaves capital intact (leading to permanently greater population recovery). Conversely, “the combined intervention of saving a life and increasing the capital stock to offset the decline in the capital-labor ratio” may, if standard models are correct, result in “a permanent, proportional increase in the population.” Given various “extremely simplifying assumptions”, this population-boosting intervention could even turn out to be more cost-effective than standard x-risk reduction interventions.^[2]

The authors responsibly caution against taking their calculations at face value, given all the questionable assumptions involved. Their aim is rather “to illustrate, using an empirically grounded example, that there may indeed be interventions other than extinction risk mitigation that are cost-effective in virtue of having a long-run effect on the size of the global population.” Given my rule high stakes in principle, we should take this possibility seriously so long as we can’t rule out their assumptions (even if we consider the assumptions in question very likely mistaken).

Cluelessness and the Optimism-Pessimism Dilemma

In ‘Longtermist Myopia’, Askell & Neth argue that:

even if you accept that the future matters just as much as the present from a moral point of view, there are important reasons to focus on the near-term consequences of our actions for the purpose of decision-making. These reasons include… epistemic diffusion of our action’s predictable consequences, and both optimism and pessimism about existential risk…

“Epistemic diffusion” is the claim that effects get progressively harder to predict with temporal distance, until after a certain point we’re completely clueless about the far future. The authors mention how “chaotic systems” magnify “small initial uncertainties” until the far future is essentially unpredictable no matter what present actions we perform.

But note that this is just one model of our epistemic situation. Other models might imply that we are merely uncertain, not completely clueless. We might have more reason to expect present population loss to result in a smaller rather than a larger long-term population, for example. We might expect that present economic growth is more likely to increase than to decrease long-term economic growth. We might expect that robust education and good-government norms are more likely to be good than bad for the future of humanity, while a global nuclear war would be the opposite. And so on. One can certainly imagine scenarios in which any one of these expectations turns out to be mistaken. But I take it that they would be surprising results.

Moreover, I take it that there is very little credibility to the opposite view, that we should regard the inverse of the above claims as disproportionately likely by default. So if you give some (higher-order) credence to views or models implying cluelessness, and some to views on which we can often reasonably expect commonsensically good things to be long-term good, then it seems the positive expectations could trivially win out. The possibility of cluelessness seemingly “drops out” as deliberatively inert, compared to any more epistemically committing views that we also take seriously. (Compare similar arguments for the deliberative irrelevance of moral nihilism.)

So again, rather than allowing the possibility of cluelessness to rule out high (expectational) stakes, it seems we should take the opposing possibility—of being “clued in” to high long-term stakes—to rule in the high stakes of reasonable longtermist efforts.

Next, the authors discuss the “optimism-pessimism dilemma” (inspired by Thorstad 2022). On “optimistic” models, we’ll survive by default so x-risk reduction is a waste of time and resources. On “pessimistic” models, we can’t permanently reduce the risks enough for the efforts to be worth much: avoiding one extinction event just saves us a few years until the next one strikes—far from the vast glorious future that longtermists imagine. It’s only in the sweet spot that x-risk reduction efforts deliver on their potential for “astronomical value”.

You can probably anticipate my response by now: we only need one credible model entailing extremely high stakes in order to establish high stakes in expectation. And “credible” here does not even require high credence.

Compare Thorstad’s classic argument:

I show that pessimism is unlikely to ground the overwhelming importance of existential risk mitigation unless it is coupled with an empirical hypothesis: the time of perils hypothesis. However, I argue, the time of perils hypothesis is probably false. I conclude that existential risk pessimism may tell against the overwhelming importance of existential risk mitigation.

His paper’s conclusion more modestly calls the arguments for the time of perils hypothesis “inconclusive”. But either way, the time of perils hypothesis can (and should) rationally shape our expected value judgments without needing to be conclusively established or even probable. Warranting some non-negligible credence would suffice. Because, again, even just a 1% chance of extremely high stakes establishes high stakes in expectation.

The way expected value works, high stakes can get ruled in by a single credible scenario (absent a counterbalancing case with values inverted), and only ruled out by establishing that there is no way to rule them in. To seek to establish low expected value via appeal to a single “best guess” model or scenario is tempting but ultimately misguided. To rule out high stakes, you need to establish that the most longtermist-friendly scenario or model is not just unlikely, but vanishingly so.^[3]

1. ^{^}

   In particular, I take it that there are many candidate interventions whose potential for vast positive value is not counterbalanced by a comparable risk of vast negative value. We can thus attribute vastly positive expected value—while we face immense uncertainty, we are not completely clueless about what sorts of things are apt to be good for the longterm future.

2. ^{^}

   This assumes a simple total axiology for valuing lives. Some may wish to give extra moral weight to avoiding outright unacceptable outcomes (like early extinction) compared to “luxury benefits” like increasing population further above a generally acceptable-seeming level.

3. ^{^}

   To be clear, people are perfectly entitled to assign vanishingly low credence to claims when they sincerely believe it warranted—I certainly don’t mean to rule this out as a matter of form. (It’s how I’m inclined to dismiss Pascalian muggers, after all.) I take the sorts of scenarios discussed above to be, despite their very uncertain and speculative nature, vastly more substantively credible than the claims of Pascalian muggers. But there’s certainly plenty of room for further substantive dispute about what credences are really warranted in all these cases.