Contra Thorstad on the Mathematics of Existential Risks

By Bentham's Bulldog @ 2025-12-07T19:51 (+7)

Crosspost of this article. 

David Thorstad is one of the more interesting critics of effective altruism. Unlike some, his objections are consistently thoughtful and interesting, and he’s against malaria, rather than against efforts to do something about it. Thorstad wrote a paper titled Three mistakes in the moral mathematics of existential risk, in which he argues that when one corrects for a few errors, the case for existential threat reduction becomes a lot shakeier. I disagree, and so I thought it would be worth responding to his paper.

The basic argument that Thorstad is addressing is pretty simple. The future could have a very large number of people living awesome lives (e.g. 10^52 according to one estimate). However, if we go extinct then it won’t have any people. Thus, reducing risks of extinction by even a small amount increases the number of well-off people in the far future by a staggering amount, swamping all other values in terms of utility.

1 Cumulative and background risk

The first error Thorstad discusses is neglect for cumulative risk. The standard explanation of why the future will have extremely huge numbers of people is that it could last a very long time. But it lasting a very long time means that there are many more opportunities for it to be destroyed.

In a billion years, there are a million 1,000 year times slices. So even if the odds of going extinct per thousand years were .001%, the odds we’d survive for a billion years would only be 0.004539765%. Actions that lower short-term existential risks by a little bit aren’t hugely significant because the odds of them influencing whether humans survive for a very long time are low.

Thorstad similarly claims that models of existential risk reduction don’t take into account background risk odds—the odds that we’d go extinct from something else if not this. If you think existential risks are significant this century, which you have to in order to think that it’s worth working on them, then probably they’re significant in other centuries, so we’re doomed anyways!

I think there are two crucial problems with this.

The first and biggest one: you should assign non-trivial credence to humans reaching a state where the odds of extinction per century are approximately zero. The odds are not trivial that if we get very advanced AI, we’ll basically eliminate any possibility of human extinction for billions of years. You shouldn’t just think of the average century-odds of extinction, but instead of whether there’s some chance that we’ll have the ability to survive for many centuries.

Let’s suppose that you think there’s only a 10% chance that reaching a state where existential risks are approximately zero is possible in principle. Let’s also say that you think that if an action stops an existential risk, there’s only a 10% chance that something else wouldn’t have killed us off before we reached the state where existential risks reach roughly zero (if such a state is possible). If so, these considerations lowers the value of existential risk reduction by a factor of about 100—it means existential threat reduction is about 100 times less good than it would be if averting an existential risk guaranteed we reached the period when existential risks hit approximately zero.

But the expected value of existential risk reduction is—if not infinite, which I think it clearly is in expectation—extremely massive. If you think the Bostrom number of 10^52 happy people has a .01% chance of being right, then you’ll get 10^48 expected future people if we don’t go extinct, meaning reducing odds of existential risks by 1/10^20 creates 10^28 extra lives. So even if we think the Thorstad math means that getting the odds of going extinct this century down 1% matters 100 times less, it still easily swamps short-term interventions in expectation.

Thorstad’s argument is part of a common pattern of arguments that I think go subtly wrong for similar reasons. These arguments argue against some specific very large quantity outweighing other stuff by suggesting that probabilities drop off in some way proportional to the size of the quantity (I’ll have an Appendix where I talk about this in more detail). In Thorstad’s case, he argues that though the future may be very large, the probability of getting such a future drops off proportionally to the amount of value it contains. But for that to be right, you must think the odds of a very large future are extremely low—that the odds we’ll get existential risks down to essentially zero is essentially zero—and that seems very hard to justify.

In other words, to get the mathematical model to cancel out the very large EV at stake, you have to have probability drop off proportional to value. But if you have any uncertainty about that model, then you’ll have some non-trivial credence that probability of reaching a future doesn’t drop off proportional to the value in that future. Then the EV stakes are very large in expectation.

A second reason this shouldn’t impact the core Longtermist argument too much: the basic argument can work civilization can’t last very long. Even if you thought that a space civilization would inevitably kill itself after 10,000 years, the case for existential risk reduction would still be significant, because plausibly an innovative space civilization could figure out ways of creating enormous value per year. There could be lots of happy digital minds, for example. We should have some non-trivial credence in it being possible to create absurd amount of welfare with very advanced future technology in a short amount of time.

Thus, even if you bought that existential risks will never get down to near zero, so humanity is doomed long-term, there is still the possibility of immense value before our untimely demise. You might place more value on, say, speeding up space development, but certainly Longterm considerations—where potential stakes are a lot larger than present stakes—will swamp present concerns.

This is especially clear given that the odds of bringing about infinite value in the far future are non-zero. This will, assuming Fanaticism—which Thorstad seems to grant for the paper, as he’s internally critiquing Longtermists—swamp everything else.

2 Population dynamics

In this section, Thorstad argues that assumptions behind existential risk reduction overestimate the probability that humanity will try to maximize the number of people. Standard demographic models generally assume future populations will decline, perhaps permanently. Fertility declines as countries develop—so why think the far future will have so many people?

This argument is vulnerable to a parallel criticism. Sure, maybe we won’t try to create as many people as possible. But if the odds are 1% that we do, then this only decreases the EV of existential risk reduction by two orders of magnitude. Next to numbers like 10^52, two orders of magnitude are nothing! Existential risk reduction still wins out easily.

For the record, I’m pretty skeptical that we will use space resources to maximize the number of happy minds (or maximize value in some other way, if there’s another better way to do it). But I don’t think it’s so outrageously improbable that it makes existential threat reduction anything other than massively important.

The considerations Thorstad presents are interesting and do present some ways that the models of existential threat reduction might be slightly optimistic. But they don’t change things by more than a few orders of magnitude, and as a result, existential risk reduction still comes out looking quite good.

3 10^52 is an extreme underestimate

If you previously thought that existential risk reduction just barely edged out competitors, then you might, after reading Thorstad’s piece, think it gets beaten by competitors. You might think this if you were going with lower-end estimates of the number of future people. If you think that the number of future people in best-case scenarios is going to be, say, 10^20, and have some other concerns about Longtermism, then maybe this should dissuade you.

But I think you shouldn’t have such views. In expectation, the number of future people is way more than 10^20. It is in fact way more than 10^52.

Now, it’s reasonably probably that even best case scenarios don’t have that many people. But there is some chance that the far future could contain stupidly large numbers of people. For instance, maybe we come up with some system that produces exponential growth with respect to happy minds relative to resources input. So, as you increase the amount of energy by some constant amount, you double the number of minds.

I wouldn’t bet on such a scenario, but it’s not impossible. And if the odds are 1 in a trillion of such a scenario, then this clearly gets expected value much higher than the 10^52 number. Such a scenario potentially opens up numbers of happy minds like 2^1 quadrillion. There’s also some chance we’ll discover ways to bring about infinite happy minds—if the odds of this are non-zero, the expected number of future happy minds is infinity.

Now, one could do a fermi estimate over the different scenarios, but any specific probabilities will be handwavy. The core intuition is that there’s some chance of bringing about incomprehensible quantities of value—next to which 10^52 happy minds looks like nothing. Because this incomprehensible quantity of value could, in principle, be basically arbitrarily large, this means that the far future has more than enough expected value for existential risk reduction to outweigh other stuff. Even a number like 10^500 future expected people would be an underestimate.

I want to make very clear: I am not saying that it is likely that we’ll get more than 10^52 future people. The point I’m making is about how numbers work; if there is some non-trivial chance of a positive value being astronomically more than N, then its expected value will almost certainly end up more than N. If you thought there was a 99% chance that you were picking a random number between 1 and 10, a .9% chance that you were picking a random number between 1 and 10 quadrillion, and a .1% chance you were picking a random number between 1 and Graham’s number (a ridiculously massive number) the expected numerical value would be way more than a quadrillion.

What are the odds that there could be way more value than would be experienced by 10^52 happy minds? I don’t know, maybe 1%. But that’s enough to get the job done.

Now, a lot of estimates put the number of future people there could be around 10^30. Even if you think that there’s a 1% chance this estimate is right, and Thorstad’s arguments both lower the expected value of existential risk reduction by two orders of magnitude, then you’ll think preventing present extinction adds, in expectation, 10^24 extra lives. This would mean reducing existential risks by one in a billion adds in expectation 10^15 lives. Pretty good!

4 Conclusion

As I said at the start, I think Thorstad is a thoughtful guy and his arguments are consistently interesting. But to defeat arguments for existential threat reduction, one needs to argue that existential risk reduction is many, many orders of magnitude less valuable than is typically thought. Thorstad’s arguments maybe, if we’re generous, get us a few orders of magnitude, but nowhere near enough to unseat the primacy of existential threat reduction. Longtermist considerations remain decisive. In other words, they commit the infamous “suggesting some value has some chance of being low in response to claims that it is large in expectation,” fallacy.

Appendix: Bad arguments with probabilities that drop off

There’s a class of arguments a bit like the one I discussed in section 2 that typically go wrong for similar reasons. Such arguments are intended to respond to claims that some quantity might be very large. They argue that though the quantity might be very large, the probability of getting that quantity drops off proportional to its size. Thus, things add up to normality. I’ll discuss two other examples here.

In his excellent book Approaching Infinity, Huemer comments on the St. Petersburg paradox. Suppose that someone offers to flip a coin until it comes up tails. You get 2^N utils where N is the number of coinflips. So, if there’s one coin flip you get two utils, if two you get four, if three you get eight, etc.

This scenario appears paradoxical because its expected utility is infinite. A 1/2 chance of 2 + a 1/4 chance of 4 + a 1/8 chance of 8…adds up to 1+1+1+1 forever. It’s weird that its expected utility is infinite, even though there’s a 100% chance its actual utility is finite. In addition, it’s pretty counterintuitive that you ought to give up any number of utils to play this game, even though it probably will only get you a few utils. The odds it gets you more than 8 utils is 1/8. How is that a game worth sacrificing everything for?

(If you want my solution to the alleged paradox, see section 12).

Here’s Huemer’s solution: he says that as the probabilities get more remote, the odds that they’ll actually pay you get lower and lower. It wouldn’t be terribly unlikely if they’d could give you 2 utils. It would be very unlikely if they could give you Graham’s number utils. As a result, Huemer says, the probability of payout drops faster than the payout itself. Thus, its expected value isn’t infinite.

Such solutions look elegant because another factor cancelling out the first factor is neat. But I don’t think they generally work. Among other things, this assumes that the probability that the person can instantiate any amount of utility is zero (a one in a million chance of getting the real payouts from the St. Petersburg game still has infinite expected utility). But surely it shouldn’t be zero? Maybe it’s really low, but zero? If something has a probability of zero, you can never update into thinking its true. Surely that can’t be right? And for the St. Petersburg game, we’re generally supposed to stipulate that you have some good reason to think they’re telling the truth.

There’s another argument that I think makes a similar error in the first paper providing a sustained and comprehensive defense of the self-indication assumption—woot🥳 woot🥳. The self-indication assumption (SIA), for those poor benighted unlettered souls who haven’t heard, is the idea that your existence is X times likelier if there are X times more people that you might be. So, for example, if a coin is flipped that creates one person if heads and ten people if tails, if you’re created from the coin flip, SIA says you should think at ten to one odds that the coin came up tails. The idea: because the coin coming up tails results in ten people being created, it makes it ten times likelier that you would be created.

Now, there’s an obvious objection to this called the presumptuous philosopher objection. The core idea: SIA implies that your existence gives you infinitely strong evidence that the universe is infinite. If a world with ten people makes your existence ten times likelier and a world with a hundred people makes your existence a hundred times likelier, then a world with infinite people makes your existence infinitely likelier. But it seems nutty that your existence gives you infinitely strong evidence that there are infinite people (for what’s wrong with this objection, see section 4.8).

In the first paper on SIA—a really excellent paper that I recommend everyone read—Ken Olum gave a response to this objection that I think is pretty clearly wrong. He suggested that even though your existence favors theories on which there are more people, because those theories posit more stuff, they’re more complicated. Thus, a bigger world makes your existence likelier, but it has a lower prior. These two cancel out and make it so that you shouldn’t ultimately be highly certain the world is enormous.

If this worked out, it would be very cool. But sadly, we cannot have nice things. I mean, first of all, this doesn’t seem like the right way to calculate probabilities. Does a theory where there’s 1,000 times more stuff have a prior that’s 1,000 times lower? Surely complexity can’t just be about the amount of stuff. And for this solution to work out, you need to have the probability drop off proportional to the increase in the number of people. But even if greater complexity lowers priors, why would a theory on which there’s twice as much stuff have exactly twice as low of a prior.

Things get really gnarly when it comes to infinity. On Olum’s view, it would have to be that the prior probability of an infinitely big universe is zero—but then you end up thinking that odds are decent thethat universe is infinite because it makes your existence infinitely likelier. But why would this be? It doesn’t seem like an infinitely big universe is infinitely intrinsically unlikely! Even if you lean towards views on which infinitely is impossible or has an infinitely low prior, you should also have some credence in views on which it has a higher prior. But any credence in those views makes its all things considered prior non-zero.

If you think there’s a 1/googol chance that the right theory of priors is one that assigns a 1/googol probability to there being infinite people, then your overall prior in there being infinite people will be 1/googol^2. But this is enough to make you certain that there are infinite people after the infinite SIA update.

In short, you should generally be suspicious of rebutting arguments for why some magnitude is very large by claiming that the probability drops off proportionally to the magnitude. Any uncertainty about this will generally be enough to ruin such arguments. And they tend to imply crazy stuff about infinity, like that the odds of the parameter being infinite is zero. This is very hard to believe.