Thanks for the comment, Fin! Strongly upvoted.

Imagine graphing out percentiles for your credence distribution over values the entire future can take.

I personally find it more intuitive to plot the PDF of the value of the future, instead of its cumulative distribution function as you did, but I like your graphs too!

The closest thing to a ‘classic’ story in my head looks like below: on which (i) the long-run future is basically biomodal, between ruin and near-best futures, and (ii) the main effect of extinction mitigation is to make near-best futures more likely.

I do not know whether the value of the future is bimodal. However, even if so, I would guess reducing the nearterm risk of human extinction only infinitesimaly increases the probability of astronomically valuable worlds. Below is an illustration. The normal and dashed lines are the PDFs of the value of the future before and after the intervention. I suppose the neutral worlds would be made slightly less likely (dashed line below the normal line), and the slightly negative and positive worlds would me made slightly more likely (dashed line above the normal line), but the astronomically negative and positive worlds would only be made infinitesimaly more likely (basically no difference between the normal and dashed line).

Are there any interventions whose estimates of (posterior) counterfactual impact, in terms of expected total hedonistic utility (not e.g. preventing the extinction of a random species), do not decay to 0 in at most a few centuries? From my perspective, their absence establishes a strong prior against persistent longterm effects.

I'll also add that, in the case of AI risk, I think that framing literal human extinction as the main test of whether the future will be good seems like a mistake, in particular because I think literal human extinction is much less likely than worlds where things go badly for other reasons.

I wonder whether the probability of things going badly is proportional to the nearterm risk of human extinction. In any case, I am not sure the size of the risk affects the possibility of some interventions having astronomical cost-effectiveness. I guess this mainly depends on how fast the (posterior) counterfactual impact decreases with the value of the future. If exponentially, based on my Fermi estimate in the post, the counterfactual impact linked to making astronomically valuable worlds more likely will arguably be negligible.

I would be curious to know how low would the risk of human extinction in the next 10 years (or other period) have to be for you to mostly:

• Donate to animal welfare interventions (i.e. donate at least 50 % of your annual donations to animal welfare interventions).
• Work on animal welfare (i.e. spend at least 50 % of your working time on animal welfare).

Thanks for the comment, Owen!

Honestly I don't really like "astronomically cost-effective" framings; I think they're misleading, because they imply too much equivalence with standard cost-effectiveness analysis, whereas if they're taken seriously then it's probably the case that many many actions have astronomical expected impact.

Agreed^[1].

On the intuition pump about human life expectancy:

• Suppose that you in fact avert a death that would have occurred in any period, however small
  • Then this does have a pretty big impact on life expectancy!
    • Unless you're deferring the death to mere moments later (like the fact that you were maybe about to die would be evidence that you were in a dangerous situation, and so maybe we care about getting to a non-dangerous situation

You kind of anticipated my reply with your last point. Once I condition on a given human dying in the next e.g. 1.80*10^-35 seconds (calculation in the post), I should expect there is an astronomically high cost of extending the life expectancy to the baseline e.g. 100 years (e.g. because it is super hard to avoid simulation shutdown or vacuum decay). With reasonable costs, I would only be able to increase it infinitesimaly (e.g. by moving a few kilometers away from the location from which the universal collapse wave comes).

• An toy example:
  • Suppose that we could exogenously introduce a new risk which has a 1% chance of immediately ending the universe, otherwise nothing happens
  • That definitely decreases the expected value of the future by 1% (assuming non-existence is zero)
  • Therefore, eliminating that risk definitely increases the expected value of the future by 1%
• If you think the expected value of the future is astronomical, then eliminating that risk would also have astronomical value

Agreed.

• But it seems to me like a wild assumption to say that all of the probability mass goes onto such ["slightly-more valuable worlds"] worlds

I know you italicised "all", but, just to clarify, I do not think all the probability mass would go to such worlds. Some non-negligible mass would go to astronomically valuable worlds, but I think it would be so small that the increase in the expected the value of these would be negligible. Do you have a preferred function describing the difference between the PDF of the value of the future after and before the intervention? It is unclear to me why assuming a decaying exponential, as I did in my calculations for illustration, would be wild.

• If you want to avoid the conclusion that avoiding nearterm extinction risk has astronomical value, you therefore need to take one of three branches:
  1. Claim that nearterm extinction risk is astronomically small
     • I think you're kind of separately making this claim, but that it's not the main point of this post?
  2. Claim that the situation is importantly disanalogous from the toy example I give above
     • Maybe you think this? But I don't understand what the mechanisms of difference would be
  3. Claim that the future does not have astronomical expected value
     • Honestly I think there's some plausibility to this line, but you seem not to be exploring it

I sort of take branch 2. I agree eliminating a 1 % instantaneous risk of the universe collapsing would have astronomical benefits, but I would say it would be basically impossible. I think the cost/difficulty of eliminating a risk tends to infinity as the duration of the risk tends to 0^[2], in which case the cost-effectiveness of mitigating the risk also goes to 0. In addition, eliminating a risk becomes harder as its potential impact increases, such that eliminating a risk which endargers the whole universe is astronomically harder than eliminating one that only endangers humans.

As the duration and scope of the risk decreases, postulating that the relative increase in the expected value of the future matches the (absolute) reduction in risk becomes increasingly less valid.

Describing a risk as binary as in your example^[3], it is more natural to assume that the probability mass which is taken out of the risk is e.g. loguniformly distributed across all the other worlds, thus meaningully increasing the chance of astronomically valuable worlds, and resulting in risk mitigation being astronomically cost-effective. However, describing a risk as continous (in time and scope), it is more natural to suppose that the change in its duration and impact will be continuous, and this does not obviously result in risk mitigation being astronomically cost-effective.

On 1, I guess the probability of humans going extinct over the next 10 years is 10^-7, so nowhere near as small as needed to result in non-astronomically large cost-effectiveness when multiplied by the expected value of the future. However, you are right this is not the main point of the post.

On 3, I guess you find a non-astronomical expected value of the future more plausible due to your guess for the nearterm risk of human extinction being orders of magnitude higher than mine.

1. ^{^}

   I say in the post that:

   Firstly, I do not think reducing the nearterm risk of human extinction being astronomically cost-effective implies it is astronomically more cost-effective than interventions not explicitly focussing on tail risk, like ones in global health and development and animal welfare.

2. ^{^}

   In general, any task (e.g. risk mitigation) can be made arbitrarily costly/hard by decreasing the time in which it has to be performed. At the limit, the task becomes impossible when there is no time to complete it.

3. ^{^}

   Either it is on or off. Either it affects the whole universe or nothing at all.

Do you have much reason to believe Claim 1 is true? I would've thought that's where most people's intuitions differ, though maybe not Vasco's specific crux.

Thanks for the comment, Dan.

Claim 4: Given claims 1-3, and that the "some" civilizations described in claims 1-3 are not vanishingly rare (enough to balance out the very high value), the expected value of averting a random extinction event for a technologically advanced civilization is astronomically high.

I think such civilisations are indeed vanishingly rare. The argument you are making is the classical type of argument I used to endorse, but no longer do. As the one Nick Bostrom makes in Existential Risk Prevention as Global Priority (see my post), it is scope sensitive to the astronomical expected value of the future, but scope insensitive to the infinitesimal increase in probability of astronomically valuable worlds, which results in an astronomical cost of moving probability mass from the least valuable worlds to astronomically valuable ones. Consequently, interventions reducing the nearterm risk of human extinction need not be astronomically cost-effective, and I currently do not think they are.

Thanks for engaging, Larks!

10^-35 is such a short period of time that basically nothing can happen during it - even a laser couldn't cut through your body that quickly.

Right, even in a vacuum, light takes 10^-9 s (= 0.3/(3*10^8)) to travel 30 cm.

To explicitly do the calculation, lets assume a handgun bullet hits someone at around ~250m/s, and decelerates somewhat, taking around 10^-3 seconds to pass through them. Assuming they were otherwise a normal person who didn't often get shot at, intervening to protect them for ~10^-3 seconds would give them about 50 years ~= 10^9 seconds of extra life, or 12 orders of magnitude of leverage.

I do not think your example is structurally analogous to mine:

• My point was that decreasing the risk of death over a tiny fraction of one's life expectancy does not extend life expectancy much.
• In your example, my understanding is that the life expectancy of the person about to be killed is 10^-3 s. So, for your example to be analogous to mine, your intervention would have to decrease the risk of death over a period astronomically shorter than 10^-3 s, in which case I would be super pessimistic about extending the life expectancy.

This example seems analogous to me because I believe that transformative AI basically is a one-time bullet and if we can catch it in our teeth we only need to do so once.

The mean person who is 10^-3 away from being killed, who has e.g. a bullet 25 cm (= 250*10^-3) away from the head if it is travelling at 250 m/s, presumably has a very short life expectancy. If one thinks humanity is in a similar situation with respecto to AI, then the expected value of the future is also arguably not astronomical, and therefore decreasing the nearterm risk of human extinction need not be astronomically cost-effective. Pushing the analogy to an extreme, decreasing deaths from shootings is not the most effective way to extend human life expectancy.

Thanks for the comment, Ariel!

From reading your post, your main claim seems to be: The expected value of the long-term future is similar whether it's controlled by humans, unaligned AGI, or another Earth-originating intelligent species.

I did not intend to make that claim, and I do not have strong views about it. My main claim is the 2nd bullet of the summary. Sorry for the lack of clarity. I appreciate I am not making a very clear/formal argument, although I think the effects I am pointing to are quite important. Namely, that the probability of increasing the value of the future by a given amount is not independent of that amount.

You also dispute that we're living in a time of perils, though that doesn't seem so cruxy, since your main claim above should be enough for your argument to go through either way. Still, your justification is that "I should be a priori very sceptical about claims that the expected value of the future will be significantly determined over the next few decades". There's a lot of literature (The Precipice, The Most Important Century, etc) which argues that we have enough evidence of this century's uniqueness to overcome this prior. I'd be curious about your take on that.

I do not think we are in a time of perils, in the sense I would say the annual risk of human extinction has generally been going down until now, although with some noise^[1]. A typical mammal species has a lifespan of 1 M years, which suggests an annual risk of going extinct of 10^-6. I  have estimated values much lower than that. 5.93*10^-12 for nuclear wars, 2.20*10^-14 for asteroids and comets, 3.38*10^-14 for supervolcanoes, a prior of 6.36*10^-14 for wars, and a prior of 4.35*10^-15 for terrorist attacks. My actual best guess for the risk of human extinction over the next 10 years is 10^-7, i.e. around 10^-8 per year. However, besides this still being lower than 10^-6, it is driven by the risk from advanced AI which I assume has some moral value (even now), so the situation would not be analogous.

(Separately, I think you had more to write after the sentence "Their conclusions seem to mostly follow from:" in your post's final section?)

Thanks for noting that! I have now added the bullets following that sentence, which were initially not imported (maybe they add a little bit of clarity to the post):

I cannot help notice arguments for reducing the nearterm risk of human extinction being astronomically cost-effective might share some similarities with (supposedly) logical arguments for the existence of God (e.g. Thomas Aquinas’ Five Ways), although they are different in many aspects too. Their conclusions seem to mostly follow from:

• Cognitive biases. In the case of the former, the following come to mind:
  • Authority bias. For example, in Existential Risk Prevention as Global Priority, Nick Bostrom interprets a reduction in (total/cumulative) existential risk as a relative increase in the expected value of the future, which is fine, but then deals with the former as being independent from the latter, which I would argue is misguided given the dependence between the value of the future and increase in its PDF. “The more technologically comprehensive estimate of 10^54 human brain-emulation subjective life-years (or 10^52 lives of ordinary length) makes the same point even more starkly. Even if we give this allegedly lower bound on the cumulative output potential of a technologically mature civilisation a mere 1 per cent chance of being correct, we find that the expected value of reducing existential risk by a mere one billionth of one billionth of one percentage point is worth a hundred billion times as much as a billion human lives”.
    • Nitpick. The maths just above is not right. Nick meant 10^21 (= 10^(52 - 2 - 2*9 - 2 - 9)) times as much just above, i.e. a thousand billion billion times, not a hundred billion times (10^11).
  • Binary bias. This can manifest in assuming the value of the future is not only binary, but also that interventions reducing the nearterm risk of human extinction mostly move probability mass from worlds with value close to 0 to ones which are astronomically valuable, as opposed to just slightly more valuable.
  • Scope neglect. I agree the expected value of the future is astronomical, but it is easy to overlook that the increase in the probability of the astronomically valuable worlds driving that expected value can be astronomically low too, thus making the increase in the expected value of the astronomically valuable worlds negligible (see my illustration above).
• Little use of empirical evidence and detailed quantitative models to catch the above biases. In the case of the former:
  • As far as I know, reductions in the nearterm risk of human extinction as well as its relationship with the relative increase in the expected value of the future are always directly guessed.

1. ^{^}

   Even from the start of World War 2 in 1939 to when nuclear warheads peaked in 1986, the fraction of people living in democracies increased 21.6 pp (= 0.156 + 0.183 - (0.0400 + 0.0833)).

Thanks for looking into the post, Toby!

I used to think along the lines you described, but I believe I was wrong.

The expected value contained >1 year in the future is:

p * 0 + (1-p) * U

One can simply consider these 2 outcomes, but I find it more informative to consider many potential futures, and how an intervention changes the probability of each of them. I posted a comment about how I think this would work out for a binary future. To illustrate, assume there are 101 possible futures with value U_i = 10^i - 1 (for i between 0 and 100), and that future 0 corresponds to human extinction in 2025 (U_0 = 0). The increase in expected values equals to Delta = sum_i dp_i*U_i, where dp_i is the variation in the probability of future i (sum_i dp_i = 0).

Decreasing the probability of human extinction over 2025 by 100 % does not imply Delta is astronomically large. For example, all the probability mass of future 0 could be moved to future 1 (dp_1 = -dp_0, and dp_i = 0 for i between 2 to 100). In this case, Delta = 9*|dp_0| would be at most 9 (dp_i <= 1), i.e. far from astronomically large.

So if U is astronomically big, and dp is not astronomically small, then the expected value of reducing nearterm extinction risk must be astronomically big as well.

For Delta to be astronomically large, non-negligible probability mass has to be moved from future 0 to astronomically valuable futures. dp = sum_(i >= 1) dp_i not being astronomically small is not enough for that. I think the easiest way to avoid human extinction in 2025 is postponing it to 2026, which would not make the future astronomically valuable.

The impact of nearterm extinction risk on future expected value doesn't need to be 'directly guessed', as I think you are suggesting? The relationship can be worked out precisely. It is given by the above formula (as long as you have an estimate of U to begin with).

I agree the impact of decreasing the nearterm risk of human extinction does not have to be directly guessed. I just meant it has traditionally been guessed. In any case, I think one had better use standard cost-effectiveness analyses.