Thank you for the post! I have a question: I wonder whether you think the trajectory to be shaped should be that of all sentient beings, instead of just humanity? It seems to me that you think that we ought to care about the wellbeing of all sentient beings. Why isn't this prinicple extraopolated when it comes to longtermism? 

For instance, from the quote below from the essay, it seems to me that your proposal's scope doesn't neccessarily include nonhuman animals. "For example, a permanent improvement to the wellbeing of animals on earth would behave like a gain (though it would require an adjustment to what v(⋅) is supposed to be representing)." 

Existential risk, and an alternative framework

One common issue with “existential risk” is that it’s so easy to conflate it with “extinction risk”. It seems that even you end up falling into this use of language. You say: “if there were 20 percentage points of near-term existential risk (so an 80 percent chance of survival)”. But human extinction is not necessary for something to be an existential risk, so 20 percentage points of near-term existential risk doesn’t entail an 80 percent chance of survival. (Human extinction may also not be sufficient for existential catastrophe either, depending on how one defines “humanity”))

Relatedly, “existential risk” blurs together two quite different ways of affecting the future. In your model: $V=\overline{v}\tau$. (That is: The value of humanity's future is the average value of humanity's future over time multiplied by the duration of humanity's future.)

This naturally lends itself to the idea that there are two main ways of improving the future: increasing $\overline{v}$ and increasing $\tau$.

In What We Owe The Future I refer to the latter as “ensuring civilisational survival” and the former as “effecting a positive trajectory change”. (We’ll need to do a bit of syncing up on terminology.)

I think it’s important to keep these separate, because there are plausible views on which affecting one of these is much more important than affecting the other.

Some views on which increasing $\overline{v}$ is more important:

• If the future is of zero or net-negative value
• If large drops in future population size are not of enormous importance (e.g. the average view, variable-value views)
• If nonhuman-originating civilisation would use the resources that we would use, and is similarly good

Some views on which increasing $\tau$ is more important:

• If there’s a “low” upper bound on value, which we expect almost all future civilisations to meet
• If think that moral convergence, conditional on survival, is very likely 

What’s more, changes to $\tau$ are plausible binary, but changes to  $\overline{v}$  are not.  Plausibly, most probability mass is on $\tau$ being small (we go extinct in the next thousand years) or very large (we survive for billions of years or more). But, assuming for simplicity that there’s a “best possible” and “worst possible” future,  $\overline{v}$  could take any value between 100% and -100%.  So focusing only on “drastic” changes, as the language of “existential risk” does, makes sense for changes to $\tau$, but not for changes to  $\overline{v}$ .

Enhancements

I felt like the paper gave enhancements short shrift. As you note, they are the intervention most plausibly competes with existential risk reduction, as they scale with $\overline{v}$ $\tau$ .

You say: “As with many of these idealised changes, they face the challenge of why this wouldn’t happen eventually, even without the current effort. I think this is a serious challenge for many proposed enhancements.”

I agree that this is a serious challenge, and that one should have more starting scepticism about the persistence of enhancements compared with extinction risk reduction.

But there is a compelling response as to why the improvements to v(.) don’t happen anyway: which is that future agents don’t want them to happen. Taking a simplified example: In one scenario, society is controlled by hedonists; in another scenario, society is controlled by preference-satisfactionists. But, let us assume, the hedonists do in fact produce more value. I don’t think we should necessarily expect the preference-satisfactionists to switch to being hedonists, if they don’t want to switch.

(Indeed, that’s the explanation of why AI risk is so worrying from a longterm perspective. Future AI agents might want something valueless, and choose not to promote what’s actually of value.)

So it seems to me that your argument only works if one assumes a fairly strong form of moral internalism, that future agents will work out the moral truth and then act on that basis.
 

Here's the EAG London talk that Toby gave on this topic (maybe link it in the post?).

Hi Toby,

Thanks so much for doing and sharing this! It’s a beautiful piece of work - characteristically clear and precise.

Remarkably, I didn’t know you’d been writing this, or had an essay coming out that volume! Especially given that I’d been doing some similar work, though with a different emphasis. 

I’ve got a number of thoughts, which I’ll break into different comments.

Humanity

Like the other commenter says, I feel worried that v(.) refers to the value of “humanity”. For similar reasons, I feel worried that existential risk is defined in terms of humanity’s potential.

One issue is that it’s vague what counts as “humanity”. Homo sapiens count, but what about:

• A species that Homo sapiens evolves into 
• “Uploaded” humans
• “Aligned” AI systems
• Non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes

I’m not sure where you draw the line, or if there is a principled place to draw the line.

A second issue is that “humanity” doesn’t include the value of:

• Earth-originating but nonhuman civilisations, for example if Homo sapiens go extinct, but some other species later evolves that has technological capability.
• Non-Earth-originating alien civilisation. 

And, depending on how “humanity” is defined, it may not include non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes.

I tried to think about how to incorporate this into your model, but ultimately I think it’s hard without it becoming quite unintuitive.

And I think these adjustments are potentially non-trivial. I think one could reasonably hold, for example, that the probability of a technologically-capable species evolving, if Homo sapiens goes extinct, is 90%, that non-Earth-originating alien civilisations settling the solar systems that we would ultimately settle is also 90%, and that such civilisations would have similar value to human-originating civilisation. 

(They also change how you should think about longterm impact. If alien civilisations will settle the Milky Way (etc) anyway, then preventing human extinction is actually about changing how interstellar resources are used, not whether they are used at all .)

And I think it means we miss out on some potentially important ways of improving the future. For example, consider scenarios where we fail on alignment. There is no “humanity”, but we can still make the future better or worse. A misaligned AI system that promotes suffering (or promotes something that involves a lot of suffering) is a lot worse than an AI system that promotes something valueless.  

It seems like it would have been worth discussing AI more explicitly, but maybe that's a discussion for a separate article?

How plausible is it that we can actually meaningfully advance or speed up progress through work we do now, other than through AI or deregulating AI, which is extremely risky (and could "bring forward the end of humanity" or worse)? When sufficiently advanced AI comes, the time to achieve any given milestone could be dramatically reduced, making our efforts ahead of the arrival of that AI, except to advance or take advantage of advanced AI, basically pointless.

I suppose we don't need this extra argument, if your model and arguments are correct.

Humanity

Like the other commenter says, I feel worried that v(.) refers to the value of “humanity”. For similar reasons, I feel worried that existential risk is defined in terms of humanity’s potential.

One issue is that it’s vague what counts as “humanity”. Homo sapiens count, but what about:

• A species that Homo sapiens evolves into 
• “Uploaded” humans
• “Aligned” AI systems
• Non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes

I’m not sure where you draw the line, or if there is a principled place to draw the line.

A second issue is that “humanity” doesn’t include the value of:

• Earth-originating but nonhuman civilisations, for example if Homo sapiens go extinct, but some other species later evolves that has technological capability.
• Non-Earth-originating alien civilisation. 

And, depending on how “humanity” is defined, it may not include non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes.

I tried to think about how to incorporate this into your model, but ultimately I think it’s hard without it becoming quite unintuitive.

And I think these adjustments are potentially non-trivial. I think one could reasonably hold, for example, that the probability of a technologically-capable species evolving, if Homo sapiens goes extinct, is 90%, that non-Earth-originating alien civilisations settling the solar systems that we would ultimately settle is also 90%, and that such civilisations would have similar value to human-originating civilisation. 

(They also change how you should think about longterm impact. If alien civilisations will settle the Milky Way (etc) anyway, then preventing human extinction is actually about changing how interstellar resources are used, not whether they are used at all .)

And I think it means we miss out on some potentially important ways of improving the future. For example, consider scenarios where we fail on alignment. There is no “humanity”, but we can still make the future better or worse. A misaligned AI system that promotes suffering (or promotes something that involves a lot of suffering) is a lot worse than an AI system that promotes something valueless.  

I've developed a clean mathematical framework in which possibilities like this can be made precise, the assumptions behind them can be clearly stated, and their value can be compared.

Sorry if I'm missing something (I've only skimmed the paper), but is the "mathematical framework" just the idea of integrating value over time?

I'm quite surprised to see this idea presented as new. Isn't this idea very obvious? Haven't we been thinking this way all along?

Like, how else could you possibly think of the value of the future of humanity? (The other mathematically simple option that comes to mind is to only value some end state and ignore all intermediate value, but that doesn't seem very compelling.)

Again, apologies if I'm missing something, which seems likely. Would appreciate anyone who can fill in the gaps for me if so!

Speed-ups

You write: “How plausible are speed-ups? The broad course of human history suggests that speed-ups are possible,” and, “though there is more scholarly debate about whether the industrial revolution would have ever happened had it not started in the way it did. And there are other smaller breakthroughs, such as the phonetic alphabet, that only occurred once and whose main effect may have been to speed up progress. So contingent speed-ups may be possible.”

This was the section of the paper I was most surprised / confused by. You seemed open to speed-ups, but it seems to me that a speed-up for the whole rest of the future is extremely hard to do. 

The more natural thought is that, at some point in time, we either hit a plateau, or hit some hard limit of how fast v(.) can grow (perhaps driven by cubic or quadratic growth as future people settle the stars). But if so, then what looks like a “speed up” is really an advancement.

I really don’t see what sort of action could result in a speed-up across the whole course of v(.), unless the future is short (e.g. 1000 years). 

Gains

“While the idea of a gain is simple — a permanent improvement in instantaneous value of a fixed size — it is not so clear how common they are.”

I agree that gains aren’t where the action is, when it comes to longterm impact. Nonetheless, here are some potential examples:

• Species loss / preservation of species
• Preservation of historical information
• Preservation of Earth’s ecosystem
• Preservation of areas of natural importance, like Olympus Mons

These plausibly have two sources of longterm value. The first is that future agents might have slightly better lives as a result: perhaps one in a billion future people are willing to pay the equivalent of $1 in order to be able to see a real-life panda, or to learn about the life and times of a historically interesting figure. This scales in future population size, so is probably an “enhancement” rather than a “gain” (though it depends a little on one’s population ethics).

The second is if these things have intrinsic value. If so, then perhaps they provide a fixed amount over value at any time. That really would be a gain. 

Another possible gain is preventing future wars that destroy resources. Suppose that, for example, there’s a war between two factions of future interstellar civilisation, and a solar system is destroyed as a result. That would be a loss. 

Advancements 

I broadly agree with the upshots you draw, but here are three points that make things a little more complicated:

Continued exponential growth

As you note: (i) if v(.) continues exponentially, then advancements can compete with existential risk reduction; (ii) such continued exponential growth seems very unlikely.

However, it seems above 0 probability that we could have continued exponential growth in v(.) forever, including at the end point (and perhaps even at a very fast rate, like doubling every year).  And, if so, then the total value of the future would be dramatically greater than if v(.) increases cubically and/or eventually plateaus. So, one might argue: this is where most of the expected value is. So advancements, in expectation, are competitive with existential risk reduction.

Now, I hate this argument: it seems like it's falling prey to “fanaticism” in the technical sense, letting our expected value calculations be driven by extremely small probabilities. 

But it at least shows that, when thinking about longterm impact, we need to make some tough judgment calls about which possibilities we should ignore on the grounds of driving “fanatical”-seeming conclusions, even while only considering only finite amounts of value. 
 

Aliens

Elsewhere, you note the loss of galaxies due to the expansion of the universe, which means that ~one five-billionth of the universe per year becomes inaccessible.

But if the “grabby aliens” model is correct, then that number is too low.  By my calculation, if we meet grabby alien civilisations in, for example, one billion years (which I think is about the median estimate from the grabby aliens model), then we “lose” approximately 1 millionth of accessible resources to alien civilisations every year. This is still very small, but three orders of magnitude higher than what we get by just looking at the expansion of the universe.

(Then there’s a hard and relevant question about the value of alien civilisation versus the value of human-originating civilisation.)
 

Length of advancements / delays

“An advancement of an entire year would be very difficult to achieve: it may require something comparable to the entire effort of all currently existing humans working for a year.”

This is true when considering “normal” economic trajectories. But I think there are some things we could do that could cause much greater advancements or delays. A few examples:

• Preventing the collapse of civilisation. If there were a catastrophe so severe that civilisation went back to pre-industrial or even pre-agricultural levels of technology, this could result in a delay, in expectation, of thousands of years.
• Preventing long-term stagnation. Rapid developments in AI are making this seem increasingly unlikely, but it’s possible at least that the world enters a period of long-term stagnation. Combined, potentially, with other catastrophes, this could lead to a delay of hundreds or thousands of years.
• The world institutes something like a Long Reflection, delaying space settlement by a thousand years.

Combining this with the “grabby aliens” point, there is potentially 0.1% of the value of the future that could be gained from preventing delays (1000 years * 1 millionth loss per year).  Still much lower than the loss of value from anthropogenic existential risks, but higher than from non-anthropogenic risks. It’s enough that I think it’s not really action-relevant, but so at the same time not totally negligible.

This is really cool, I'm curating. It seems super clear and crisp to me, which doesn't seem like it'd be easy to do.

The work is appearing as a chapter for the forthcoming book, Essays on Longtermism, but as of today, you can also read it online here.

Thank you for providing this reading for free, and sharing this post! Inside this online PDF, under the section "Gains", there's a sentence now looking like this:

"We can all such a change a gain"

I suppose you meant to say "call" instead of "all".

It is hard for pure advancements to compete with reducing existential risk as their value turns out not to scale with the duration of humanity's future. Advancements are competitive in outcomes where value increases exponentially up until the end time, but this isn't likely over the very long run.

What about under something like Tarsney (2022)'s cubic growth model of space colonization?

Great article, Toby!

It is hard for pure advancements to compete with reducing existential risk as their value turns out not to scale with the duration of humanity's future.

Relatedly, you say that (emphasis mine):

In that case ["if humanity’s instantaneous value spends a long time at a high plateau"], advancements are better roughly when δt/τ > (γp – 1) — when the percentage of the duration of humanity’s future that we advance is greater than the percentage by which our survival probability is increased. This makes it difficult for advancements to beat existential risk reduction in scenarios where humanity would have a long lifespan if only it could survive the near-term risks. For example, on a million-year lifespan (that of a typical species) a one-year advancement would be roughly as important as a one-in-a-million improvement in nearterm survival probability — but the latter seems much more achievable.

Is the intuition that such advancement could be achieved by doubling the value growth rate for 1 year, which is much harder than decreasing existential risk by 0.0001 %?

Flow vs fixed resources

In footnote 14 you say: “It has also been suggested (Sandberg et al 2016, Ord 2021) that the ultimate physical limits may be set by a civilisation that expands to secure resources but doesn’t use them to create value until much later on, when the energy can be used more efficiently. If so, one could tweak the framework to model this not as a flow of intrinsic value over time, but a flow of new resources which can eventually be used to create value.”

This feels to me that it would really be changing the framework considerably, rather than just a “tweak”. 

For example, consider a “speed up” with an endogenous end time. On the original model, this decreases total value (assuming the future is overall good). But if we’re talking about gaining a pot of fixed resources, speeding up progress forever doesn’t change total value.

This is great. I'm so glad this analysis has finally been done!

One quick idea: should 'speed-ups' be renamed 'accelerations'? I think I'd find that clearer personally, and would help to disambiguate it from earlier uses of 'speed-up' (e.g. in Nick's thesis).

Thanks Toby - so, so exciting to see this work progressing!

One quibble:

The value of advancements and speed-ups depends crucially on whether they also bring forward the end of humanity. When they do, they have negative value

...when the area under the graph is mostly above the horizontal axis?

Even if you assign a vanishingly small probability to future trajectories in which the cumulative value of humanity/sentientkind is below zero, I imagine many of the intended users of this framework will at least sometimes want to model the impact of interventions in worlds where the default trajectory is negative (e.g. when probing the improbable)?

Maybe this is another 'further development' consciously left to others and I don't know how much the adjustment would meaningfully change things anyway - I admit I've only skimmed the chapter! But I find it interesting that, for example, when you include the possibility of negative default trajectories, the more something looks like a 'speed-up with an endogenous end time' the less robustly bad it is (as you note), whereas the more it looks like a 'gain' the more robustly good it is .

Very clear piece!

In this framework, the value of our future is equal to the area under this curve and the value of altering our trajectory is equal to the area between the original curve and the altered curve.

You mentioned optimal planning in economics, and I've wondered whether an optimal control framework might be useful for this sort of analysis. I think the difference between optimal control and the trajectory-altering framework you describe is a bit deeper than the different typical domains. There's not just one decision to be made, but a nearly-continuous series of decisions extending through the future (a "policy"). Under uncertainty, the present expected value is the presently realized ("instantaneous") value plus the expectation taken over different futures of the expected value of those futures. Choosing a policy to maximize expected realized value is the control problem.

For the most part, you get the same results with slightly different interpretation. For example, rather than impose some τ, you get an effective lifetime that's mainly determined by "background" risk but also is allowed to vary based on policy. 

One thing that jumped out at me in a toy model is that while the value of reducing existential risk is mathematically the same as an "enhancement" (multiplicative), the time at which we expect to realize that extra value can be very different. In particular, an expected value maximizer may heavily backload the realization of value (even beyond the present expected survival time) if they can neglect present value to expand while reducing existential risk.

I suspect one could learn or make clearer a few other interesting things by following those lines.

Humanity

Like the other commenter says, I feel worried that v(.) refers to the value of “humanity”. For similar reasons, I feel worried that existential risk is defined in terms of humanity’s potential.

One issue is that it’s vague what counts as “humanity”. Homo sapiens count, but what about:

• A species that Homo sapiens evolves into 
• “Uploaded” humans
• “Aligned” AI systems
• Non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes

I’m not sure where you draw the line, or if there is a principled place to draw the line.

A second issue is that “humanity” doesn’t include the value of:

• Earth-originating but nonhuman civilisations, for example if Homo sapiens go extinct, but some other species later evolves that has technological capability.
• Non-Earth-originating alien civilisation. 

And, depending on how “humanity” is defined, it may not include non-aligned AI systems that nonetheless produce morally valuable or disvaluable outcomes.

I tried to think about how to incorporate this into your model, but ultimately I think it’s hard without it becoming quite unintuitive.

And I think these adjustments are potentially non-trivial. I think one could reasonably hold, for example, that the probability of a technologically-capable species evolving, if Homo sapiens goes extinct, is 90%, that non-Earth-originating alien civilisations settling the solar systems that we would ultimately settle is also 90%, and that such civilisations would have similar value to human-originating civilisation. 

(They also change how you should think about longterm impact. If alien civilisations will settle the Milky Way (etc) anyway, then preventing human extinction is actually about changing how interstellar resources are used, not whether they are used at all .)

And I think it means we miss out on some potentially important ways of improving the future. For example, consider scenarios where we fail on alignment. There is no “humanity”, but we can still make the future better or worse. A misaligned AI system that promotes suffering (or promotes something that involves a lot of suffering) is a lot worse than an AI system that promotes something valueless. 

When is the release date for the book? Around many copies are you planning on selling e.g. compared to WWOTF?

There seems to be a typo in this link (on my laptop I can't access your other links, not sure why).

On page 22 I think you're missing the word "in" in the sentence below (I have added it in bold):

"If the lasting benefit would have been achieved later in the default trajectory, then it is only temporary so not a true gain."

Hi Toby,

Do you have any plans to empirically validade your framework? You model the interventions as having effects which are maintained (advancements and gains) or increased (speed-ups and enhancements) across time. In contrast, as far as I can tell, the (posterior) counterfactual impact of interventions whose effects can be accurately measured, like ones in global health and development, decays to 0 as time goes by, and can be modelled as increasing the value of the world for a few years, decades or centuries at most.