The Epistemic Challenge to Longtermism (Tarsney, 2020)

In case anyone is interested, Rob Wiblin will be interviewing Tarsney on the 80,000 Hours podcast next week. Rob is accepting question suggestions on Facebook (I think you can submit questions to Rob on Twitter or by email too).

tl;dr: Tarsney's model updates me towards thinking reducing non-extinction existential risks should be a little less  of a priority than I previously thought. 

Here's a quote from Tarsney (which makes more sense after reading the rest of the paper):

The potentially enormous impact that the long-term rate of ENEs [events which nullify the intended effect of a longtermist intervention, e.g. a later extinction event] has on the expected value of longtermist interventions has implications for “intra-longtermist” prioritization: We have strong pro tanto reason to focus on bringing about states such that both they and their complements are highly stable, since it is these interventions whose effects are likely to persist for a very long time (and thus to affect our civilization when it is more widespread and resource-rich). This suggests, in particular, that interventions focused on reducing existential risk may have higher expected value than, say, interventions aimed at reforming institutions or changing social values: Intuitively, the intended effects of these interventions are relatively easy to undo, or to achieve at some later date even if we fail to achieve them now. So the long-term rate of ENEs (i.e., value of r) may be significantly higher for these interventions than for existential risk mitigation.

See also Greaves and MacAskill's concept of "attractor states".

This indeed seems like an interesting implication of Tarsney's model, and indeed updates me towards placing a bit less emphasis on reducing non-extinction existential risks - e.g., reducing the chance of lock-in of a bad governmental system or set of values. 

(I already considered this a lower priority from longtermists as a whole than reducing extinction risks. But I also thought that longtermists should prioritise investigating this potential priority more than they currently do. I still think that, but now with a bit lower confidence.)

---

That said, I also think Tarsney's phrasing is a bit misleading. He compares "interventions focused on reducing existential risk" to "interventions aimed at reforming institutions or changing social values". But interventions may be aimed at doing the latter as a means to doing the former; one could try to change institutions or social values with the primary goal of ultimately reducing existential risk (or extinction risk specifically). And Tarsney's model doesn't seem to push against those interventions relative to other means of reducing existential risk. 

I think Tarsney really wants to compare interventions aimed at reducing extinction risk to interventions ultimately aimed at changing aspects of the long-term future other than whether humanity goes extinct - e.g., again,  reducing the chance of lock-in of a bad governmental system or set of values. 

This highlights another way in which Tarsney's phrasing seems a bit misleading: existential risk itself already includes non-extinction existential risk. So I think Tarsney should use the term "extinction risk" here. 

tl;dr: Tarsney seems to me to understate the likelihood that accounting for non-human animals would substantially affect the case for longtermism.

Tarsney includes a helpful appendix listing the simplifications made in his model/paper, and the rationales for these simplifications. Here's a passage from that:

Simplification: The model ignores effects on the welfare of beings other than Homo sapiens and our “descendants”. 

Rationale: (1) The sign and magnitude of the effects of paradigmatic longtermist interventions on the welfare of non-human animals (or their far-future counterparts) are very unclear. (2) Dropping this simplification seems unlikely to change our quantitative results by more than 1–2 orders of magnitude (though this is far from obvious), and so unlikely to affect our qualitative conclusions.

I appreciate Tarsney's caveat that "this is far from obvious", and, given that caveat, I don't strongly disagree with this sentence. But it seems quite plausible to me[1] that considering those effects would strengthen or weaken the case for paradigmatic longtermist interventions by more than 1-2 orders of magnitude, or even that it would flip the sign of the expected value of those interventions. 

Relatedly, I also think that considering those effects should plausibly change which longtermist interventions we support (not just whether we support them vs non-longtermist interventions).

(I'm not sure how likely I see these things as, so maybe I actually agree with Tarsney that this "seems unlikely [but with that being far from obvious]".)

See also Non-Humans and the Long-Term Future.

[1] We could operationalise "it seems quite plausible to me that X" as something like "there's at least a 20% chance that I would think X if I spent another 100 hours of thinking about the topic".

On his estimate of the difference in probability we can achieve promoting one state over its complement, it's worth mentioning that this does not consider the possibility of doing more harm than good, e.g. AI safety work advancing AGI more than it aligns it, and with the very low (but in his view, extremely conservative) probabilities that he uses in his argument, the possibility of backfire effects outweighing them becomes more plausible.

Furthermore, it does not argue that we can effectively predict that any particular state is better than its complement, e.g. is extinction good or bad? How should we deal with moral uncertainty, especially around population ethics?

For these reasons, it may be difficult to justifiably identify robustly positive expected value longtermist interventions ahead of time, which the case for longtermism depends on. I mean this even with subjective probabilities, since such probabilities supporting longtermist interventions tend to be particularly poorly-informed (largely for absence of good evidence) and so seem more prone to biases and whims, e.g. wishful thinking and the non-rational particulars of people's brains and priors. This is just deep uncertainty and moral cluelessness.

For what it's worth, I don't think it makes much sense for this paper to address such issues in detail given its current length already, although they seem worth mentioning.

(Also, I read the paper a while ago, so maybe it did discuss these issues and I missed it.)

tl;dr: Tarsney writes "resources committed at earlier time should have greater impact, all else being equal". I think that this is misleading and an oversimplification. See Crucial questions about optimal timing of work and donations and other posts tagged Timing of Philanthropy. 

(But that claim was not necessary for any of Tarsney's arguments; he just gave it as one reason why the actual case for longtermism might be stronger than his deliberately conservative estimates suggest.)

Context and explanation:

A core part of Tarsney's model is - roughly speaking - the amount by which spending $1 million on mitigating existential risks changes the probability of being in the target state at a given time, relative to the probability that would occur if the short-termist intervention was used. This parameter is represented by p. The target state means something like "The accessible region of the Universe contains an intelligent civilization”.

Tarsney makes:

a lower-bound estimate [of p] based on the details of our working example, that is almost certainly far too pessimistic, but nevertheless informative.

The estimate proceeds in two stages: First, how much could humanity as a whole change the probability of [the target state at a particular time]
(i.e., roughly, the probability that we survive the next thousand years), relative to the status quo, if we committed all our collective time and resources solely to this objective for the next thousand years? “One percent” seems like a very safe lower bound here (remembering that we are dealing with epistemic probabilities rather than objective chances). Now, if we assume that each unit of time and resources makes the same marginal contribution to increasing the probability of [the target state at that time], we can calculate p simply by computing the fraction of humanity’s resources over the next thousand years that can be bought for 1 million, and multiplying it by 0.01. This yields p [roughly equal to 10 to the power of negative 14].

Tarsney writes that "This is an extremely conservative lower bound", and that "I think it would be justifiable to adjust p upward from this lower-bound estimate by a several-order-of-magnitude “fudge factor”, if we were so inclined" (though he doesn't do this for his paper). He gives two reasons for this. 

The first has to do with diminishing marginal returns and the fact that we'll by default spend far less than all our collective time and resources over the next 1000 years to reducing existential risk. Thus, spending an extra $1 million on the current margin will probably achieve far more than one would expect by "simply by computing the fraction of humanity’s resources over the next thousand years that can be bought for 1 million". This argument makes sense to me, and I do think it suggests Tarsney's estimate for p is a very conservative one (as he intends).

But then he writes:

Second, resources committed at [an] earlier time should have greater impact, all else being equal. (If nothing else, this is true because resources that might be committed to existential risk mitigation, say, 500 years from now can do nothing to prevent any of the existential catastrophes that might occur in the next 500 years, while resources committed today are potentially impactful any time in the next thousand years.)

It's definitely true that there are many reasons why resources committed at an earlier time could have a greater impact. And the reason Tarsney raises is a valid one; we could describe this as discounting for the possibility that the later use of resources would be "too late". This is an extreme example of how we might miss "windows of opportunity" if we wait too long.

But there are also many reasons why resources committed at a later time could have a greater impact. This is especially true if we don't count resources as committed to a problem when they're used in an investment-like way in order to generate more resources that can be committed later, but it's even true if we do count resources as already committed to a problem when they're "merely invested". 

In particular, it's possible that "leverage over the future" (or hingyness, pivotality, etc.) will increase in future. This could occur if:

• We know more in future about what we should do
• Longtermist priorities become more neglected in future
• Windows of opportunity that aren't currently open become open in future
  • E.g., there could be a future point at which important global governance institutions are being set up or policy frameworks for a currently unforeseen technology are being set

(For explanation and discussion of the above points, see here.) 

Of course, the opposite effects could also occur. My point is merely that "resources committed at [an] earlier time should have greater impact, all else being equal" seems to be either false or misleading.

(I think it'd be reasonable for Tarsney to merely claim that his all-things-considered view is that resources committed at an earlier time will in practice probably have a greater impact. But this more uncertain stance would then weaken the case for a several-order-of-magnitude upwards adjustment of p.)

A final quick thing that came to mind: I think that Ord's concept of existential security could be represented in Tarsney's models as the value of r  asymptotically decreasing towards 0 over time. I'd be interested to hear people's thoughts on whether that seems accurate and, if so:

• how accounting for various likelihoods of existential security could affect Tarsney's models conclusions
• how considering Tarsney's models could affect Ord's conclusions

(I haven't tried to think this through myself yet.)

I think it'd be interesting to run a sensitivity analysis on Tarsney's model(s), and to think about the value of information we'd get from further investigation of: 

• how likely the future is to resemble Tarsney's cubic growth model vs his steady model
• whether there are other models that are substantially likely, whether the model structures should be changed
• what the most reasonable distribution for each parameter is. 

It seems like the value of information from that might be very high, at least if we think we don't want to accept fanaticism. This is because Tarsney's paper suggests reasonable empirical views could either support the case for longtermism without requiring fanaticism or only support the case for longtermism if we accept fanaticism. So further research on these models, alternative models, and these parameters could perhaps give us a much better sense of how robust the case for longtermism is.

To some extent, this comment can be boiled down to something that was obvious already: "The case for longtermism seems plausible but uncertain, and whether it's true seems very decision-relevant, so maybe investigating whether it's true would be really valuable." But I think Tarsney's paper highlights specific points to look into, and that it would allow for (rough) quantitative estimates of the value of information to be gained by investigating each point. 

For a quick and non-quantitative example, it seems that the probability of interstellar settlement has a very large bearing on the results of the model, and it also seems like we should be quite uncertain about that probability.

Some caveats to that:

• I'm not sure how tractable these investigations would be.
  • But note that it could be useful just to become somewhat less uncertain than we currently are, even if we still remain quite uncertainty.
• Tarsney's models focus on a particular working example of a longtermist intervention/priority (increasing the chance that there's an intelligent civilization at any given time point). As discussed in other comments here, how good the success of that intervention would be depends on other things not modelled by Tarsney (essentially, what that civilization does with the accessible universe), and there are many other interventions/priorities we might focus on. 
  • So ideally we'd run the sensitivity analysis and value of information calculations on either a more general version of the models or on a set of models that collectively represent various major possible priorities.
• Tarsney's models make various ethical and decision-theoretic assumptions that are conducive to longtermism. An ideal version of the sensitivity analysis and value of information calculations might also allow for investigation of what happens if these assumptions are relaxed. 
  • But that might become unwieldy. 

There's also a talk. https://globalprioritiesinstitute.org/christian-tarsney-the-epistemic-challenge-to-longtermism/

When I reference work by GPI, I usually link to the page with both the talk and the pdf.

Just a nitpick

By the rules of the expected value game, the case for longtermism appears to survive the epistemic challenge with which we confronted it. But it has prevailed in a way that should make us slightly uneasy: by appealing to potentially-minuscule probabilities of astronomical quantities of value.

I think that this particular sentence is false or misleading. As Tarsney notes earlier and later, his model and parameter estimates[1]  suggests that the case for longtermism survives given either acceptance of fanaticism or plausible but non obvious empirical views. That is, on some plausible empirical views, longtermism doesn't require an appeal to minuscule probabilities of astronomical quantities of value.

(Tarsney's sentence may still be technically accurate, since he says potentially-minuscule. But it seems at least a bit misleading to me.)

[1] Along with certain ethical and decision-theoretic assumptions, e.g. total utilitarianism.

tl;dr: The paper ignores 2 factors that could strengthen the case for longtermism - namely, possible increases in how efficiently resources are used and in what extremes of experiences can be reached.

Tarsney writes:

The case for longtermism starts from the observation that the far future is very big. A bit more precisely, the far future of human-originating civilization holds vastly greater potential for value and disvalue than the near future. This is true for two reasons. The first is duration. On any natural way of drawing the boundary between the near and far futures (e.g., 1000 or 1 million years from the present), it is possible that our civilization will persist for a period orders of magnitude longer than the near future. For instance, even on the extremely conservative assumption that our civilization must die out when the increasing energy output of the Sun makes Earth too hot for complex life as we know it, we could still survive some 500 million years. Second is spatial extent and resource utilization. If our descendants eventually undertake a program of interstellar settlement, even at a small fraction of the speed of light, they could eventually settle a region of the Universe and utilize a pool of resources vastly greater than we can access today. Both these factors suggest that the far future has enormous potential for value or disvalue.

I essentially agree with all those points. Furthermore, given my current moral and empirical views, I think those factors are probably the main factors driving the case for longtermism. 

But I think there are at least two other factors that are relevant and that might substantially add to the case for longtermism. (Though it's possible that they add so little relative to the other factors that they won't really be decision-relevant.)

---

The first factor is possible increases in efficiency of resource usage. For a given quantity and type of matter or energy, future civilizations may be able to more efficiently convert that into moral value or disvalue than current civilization can. For example, if we can create simulated humans or animals (or artificial sentiences) that are morally relevant, these may be able to experience the same pleasures or pains we can with substantially less energy required. 

Thus, the factor by which total quantity of moral (dis)value in the long-term future is expected to be larger than that in the present + near-term future may be even larger than one would think if one considered only the duration, spatial extent, and resources used in the future. 

(Tarsney's term "resource utilization" might seem like it should capture this idea, but his description suggests that he has in mind only changes in how much resources we use, not changes in how efficiently we use them.)

---

The second factor is possible increases in the extremes of experience that can be reached. It seems plausible that future civilizations will be able to create experiences more extremely good or bad than experiences that we can create today or that are experienced in nature. If so, this might increase the importance of the long-term future, if either of the following things are true:

• Those experiences can be created relatively efficiently (e.g., just slightly less efficiently than substantially less extreme experiences)
• There is some moral reason why the extreme experiences matter disproportionately more than other experiences (i.e., if the moral significance of an experience increases superlinearly with the extremity of the experience, at least at some points of that "function")

I'd guess that this factor is much less important than the efficiency factor, but it seems very hard to say.

The same basic point might also apply to non-experience things that might be morally good or bad. (E.g., if art has intrinsic moral value, perhaps future civilization could create art that is more extremely good than current art.)

---

I've seen roughly those ideas idea discussed in various places before, though I can't remember precisely where. The concept of hedonium can be seen as a special case of the efficiency factor. 

Chapter 8 of The Precipice, on "Our Potential", is also relevant here. Ord splits that chapter into discussion of the future's potential duration, its potential scale, and its potential quality. I imagine that the points I raised above were covered in that chapter, but I can't remember for sure (I read the book a year ago, and foolishly enough I had not yet converted to using Anki as I read). 

---

I think it'd be interesting for someone to think about how Tarsney's models or parameter estimates could be tweaked to account for these factors, and maybe to see how much difference this makes (after plugging in some reasonable-seeming distributions for the parameters).

tl;dr: Tarsney slightly misrepresents an existential risk estimate.

Tarsney writes:

To my knowledge, the most pessimistic estimate of near-term existential risk in the academic literature belongs to Rees (2003), who gives a 0.5 probability that humanity will not survive the next century.

But what Rees actually writes is:

I think the odds are no better than fifty-fifty that our present civilisation on Earth will survive to the end of the present century.

(Here's one online source quoting Rees. I've seen the same quote elsewhere too.)

Whether "our present civilisation on Earth" survives is very different from whether humanity survives. I haven't read Rees' book, so I don't know what he intended that quote to mean, but I'd guess he'd include things like a major population collapse that lasts a few decades as "our present civilisation on Earth not surviving". Arguably, his forecast could even be seen as capturing the chance that we just very substantially change our political, cultural, and economic systems, in the same way as how Europe in the 1900s was arguably a "different civilisation" to Europe in the year 100CE.

Also, Rees doesn't give a 0.5 probability; he gives a probability no better than 0.5.

Thanks for posting this! Your linkpost actually got me to watch the talk for the first time, even though I was aware of this paper for a while.

I think some variant of the cubic growth model could be useful for figuring out whether trying to reduce x-risk is better than trying to make durable changes to the long-term "trajectory" of the social welfare curve. I spent some time a few months ago trying to address this by modeling the trajectory of humanity, so I appreciate this paper for proposing even a simpler toy model.

I have rough thoughts about how the utility from economic growth could be incorporated: Assume that each star system has a growth rate $g$ that the residents of that star system can influence (e.g. through policy). The economy of each star system tends to grow exponentially, but GDP per capita has logarithmic utility, so the utility of the star system $u_s\left(t\right)$ grows roughly linearly.

If the economy of each star system starts at a steady state, then grows exponentially at $g$ starting at time $t_0$, the time at which humanity arrives at the star system, we get $u_s\left(t\right)=u_0+max(0,g(t-t_0\left)\right)$. If the star system's GDP is capped at $\exp \left(u_{max}\right)$, then we get $u_s\left(t\right)=u_0+min(max(0,g(t-t_0)),u_{max})$.

To incorporate economic growth into the trajectory model used in the paper, we can replace $n(s\cdot (t-t_{\ell }\left)\right)$ with the cross-correlation of $u_s\left(t\right)$ and $n(s\cdot (t-t_{\ell }\left)\right)$ (this assumes that all star systems have the same growth rate). Since $u_s\left(t\right)$ is piecewise linear and $n(s\cdot (t-t_{\ell }\left)\right)$ is cubic, the cross-correlation is piecewise quintic (it's the integral of a cubic function times a linear function). My gut tells me that having a piecewise quintic term in the trajectory function instead of a cubic term isn't going to change much about the implications of the model.

Note: I realize that by using GDP per capita, I'm leaving out the population of each star system. This would result in multiplying $u_s\left(t\right)$ by a function that models the population over time, starting at time $t_0$.