Thanks a lot for sharing this denise. Here are some thoughts on your points. 

1. On your point about moral realism, I'm not sure how that can be doing much work in an argument against longtermism specifically, as opposed to all other possible moral views. Moral anti-realism implies that longtermism isn't true, but then it also implies that near-termism isn't true. The thought seems to be that there could only be an argument that would give you reason to change your mind if moral realism were true, but if that were true, there would be no point in discussing arguments for and against longtermism because they wouldn't have justificatory force.
2. Your argument suggests that you find a person-affecting form of utilitarianism most plausible. But to me we should not reach conclusions about ethics on the basis of what you find intuitively appealing without considering the main arguments for and against these positions. Person-affecting views have lots of very counter-intuitive implications and are actually quite hard to define.
3. I don't think it is true that the case for longtermism rests on the total view. As discussed in the Greaves and MacAskill paper, many theories imply longtermism.
4. Your view that humanity is not super-awesome seems to me compatible with longtermism. The 'not super-awesome' critique attacks a premise of one strand of longtermism which is especially focused on ensuring human survival. But other forms of longtermism do not rely on these premises. For example, if you don't think that humanity is super awesome, then focusing on values change looks a good bet, as does reducing s-risks.
5. I'm not sure your point that 'the future will not be better than today' hits the mark. More precisely, I think you want to say that 'today the world is net bad and the future will be as well'. It could be true that the future is not better than today but that the future is extremely good. In that case, reducing extinction risks would still have astronomical expected value.
6. Independently of point 5, again I don't think one needs to hold that the future is good for longtermism to be true. Suffering-focused people are longtermists but don't think that the future is good. Totalists could also think that the future is not good in expectation. But still even if the future is  bad in expectation, if the variance of possible realisable states of the future is high, that makes affecting the trajectory of the future extremely important.
7. On existential security, this is a good and underdiscussed point. I hadn't thought about this much until recently,  but after looking into it I became quite convinced that a period of existential security is very likely provided that we survive catastrophic risks and avoid value lock-in. My thoughts are not original and owe a lot to discussions with Carl Shulman, Max Daniel, Toby Ord, Anders Sandberg and others. One point is that biorisk and AI risk are transition risks not state risks. The coordination problems involved in solving them are so hard that once they are solved, they stay solved. To ensure AI safety, one has to ensure stability in coordination dynamics for millions of subjective years of strategic interplay between advanced AI systems. Once we can do that, then we have effectively solved all coordination problems. Solving biorisk is also very hard if you think there will be strongly democratised biotech. If you solve that and build digital minds to explore the galaxy, then you basically eliminate biorisk. If we go to the stars, then we at least avoid earth-bound GCRs. In short, we would have huge technological power and have solved the hardest coordination problems. If you buy limits to growth arguments, you might also think that tech progress will slow down, and so catastrophic risks will fall, as they are driven by tech progress. All of this suggests that conditional on survival of the time of perils, the probability that the future is extremely long is >>10%. So, the probability is not Pascalian.
8. You seem to suggest that we cannot influence the long-term future with the exception of what you call 'lock-in events' like extinction and permanent collapse, which are attractor states that could lock in a state of affairs in the next 100 years. I suppose another one would be AI-enabled permanent lock-in of bad values. But these are the main things that longtermists are working on, so I don't see how this could be a criticism of longtermism.
9. I'm don't think that the inference from 'I don't know how to influence the future' to 'donate to AMF' follows. If you buy these cluelessness arguments (I don't personally), then it seems like two obvious things to do would be to give later to prepare for when a potential lock in event rears its head. So you could invest in stocks, or you could grow the movement so that it is ready to deal with a lock in event. If you are very uncertain about how to affect the longterm future, but accept that the future is potentially extremely valuable, then this is the strongest argument ever for 'more research needed'. If there is a big asteroid heading our way but we currently feel very unsure about how to affect that, but there are some smart people who think they can stop the asteroid, the correct answer seems to me to be "let's put tonnes of resources into figuring out how to stop this asteroid" not "let's donate to AMF".

Hey, great post, I pretty much agree with all of this.

My caveat is: One aspect of longtermism is that the future should be big and long, because that's how we'll create the most moral value. But a slightly different perspective is that the future might be big and long, and so that's where the most moral value will be, even in expectation.

The more strongly you believe that humanity is not inherently super awesome, the more important that latter view seems to be. It's not "moral value" in the sense of positive utility, it's "moral value" in the sense of lives that can potentially be affected.

For example, you write:

I do not feel compelled by arguments that the future could be very long. I do not see how this is possible without at least soft totalitarianism, which brings its own risks of reducing the value of the future.

And I agree! But where you seem to be implying "the future will only be stable under totalitarianism, so it's not really worth fighting for", I would argue "the future will be stable under totalitarianism, so it's really important to fight totalitarianism in particular!" An overly simplistic way of thinking about this is that longtermism is (at least in public popular writing) mostly concerned with x-risk, but under your worldview, we ought to be much more concerned about s-risk. I completely agree with this conclusion, I just don't think it goes against longtermism, but that might come down to semantics.

FWIW my completely personal and highly speculative view is that EA orgs and EA leaders tend to talk too much about x-risk and not enough about s-risk, mostly because the former is more palatable, and is currently sufficient for advocating for s-risk relevant causes anyway. Or more concretely: It's pretty easy to imagine an asteroid hitting the planet, killing everyone, and eliminating the possibility of future humans. It's a lot wackier, more alienating and more bizarre to imagine an AI that not only destroys humanity, but permanently enslaves it in some kind of extended intergalactic torture chamber. So (again, totally guessing), many people have decided to just talk about x-risk, but use it as a way to advocate for getting talent and funding into AI Safety, which was the real goal anyway.

On a final note, if we take flavors of your view with varying degrees of extremity, we get, in order of strength of claim:

1. X-risk is less important than s-risk
2. We should be indifferent about x-risk, there's too much uncertainty both ethically and in terms of what the future will actually look like
3. The potential for s-risk is so bad that we should invite and actually trying to cause x-risk, unless s-risk reduction is really tractable
4. S-risks aside, humanity is just really net negative and we should invite x-risk no matter what (to be clear, I don't think you're making any of these claims yourself, but they're possible paths views similar to yours might lead to).

Some of these strike me as way too strong and unsubstantiated, but regardless of what we think object-level, it's not hard to think of reasons these views might be under-discussed. So I think what you're really getting at is something like, "does EA have the ability to productively discuss info-hazards". And the answer is that we probably wouldn't know if it did.

Thanks for this! Quick thoughts:

• Curious what you make of writings like these. I think they directly addresses your crux of whether there are long-lasting, negative lock-in scenarios on the horizon which we can avoid or shape.
  • Relatedly, you mention wanting to give the values of people who are suffering the most more weight. Those and related readings make what some find a good case for thinking that those who suffer most will be future generations--I imagine they'd wish more of their ancestors had been longtermists.
• I personally find arguments like these and these compelling for being tentatively optimistic about the value of the long-term future, despite sharing many of your intuitions in sections (2) and (3).
  • These assume that positive experiences/lives are at least of nontrivial value, relative to the disvalue of suffering. It's not fully clear to me from this post whether that matches your values.
• Since you mention you care about others' reflected values being implemented, it seems relevant that:
  • In the absence of a long, flourishing future, a wide range of values (not just happiness) would go for a very long time unfulfilled: freedom/autonomy, relationships (friendship/family/love), art/beauty/expression, truth/discovery, the continuation of tradition/ancestors' efforts, etc.
  • Most people seem to value the creation of happiness and happy people quite a bit, relative to the prevention of suffering. This is suggested by e.g. a survey and the fact that most adults have (or say they want) children.
    • (Maybe these aren't their reflected values, but they're potentially decent proxies for reflected values.)

-

Additional thoughts:

I do not see how this is possible without at least soft totalitarianism, which brings its own risks of reducing the value of the future.

I think the word "totalitarianism" is pulling too much weight here. I'm sympathetic to something like "existential security requires a great combination of preventative capabilities and civilizational resilience." I don't see why that must involve anything as nasty as totalitarianism. As one alternative, advances in automation might allow for decentralized, narrow, and transparent forms of surveillance--preventing harmful actions without leaving room for misuse of data (which I'd guess is our usual main concern about mass surveillance).

(Calling something "soft totalitarianism" also feels like a bit odd, like calling something "mild extremism." Totalitarianism has historically been horrible in large part because it's been so far from being soft/moderate, so sticking the connotations of totalitarianism onto soft/moderate futures may mislead us into underestimating their value.)

I also have traditional Pascal’s mugging type concerns for prioritizing the potentially small probability of a very large civilisation.

I don't see how traditional Pascal's mugging type concerns are applicable here. As I understand them, those apply to using expected value reasoning with very low (subjective) probabilities. But surely "humanity will last with at least our current population for as long as the average mammalian species" (which implies our future is vast) is a far more plausible claim than "I'm a magical mugger from the seventh dimension"?

Thanks for this post I am always interested to hear why people are sceptical of longtermism. 

If I were to try to summarise your view briefly (which is helpful for my response) I would say:

1. You have person-affecting tendencies which make you unconcerned with reducing extinction risks
2. You are suffering-focused
3. You don’t think humanity is very good now nor that it is likely to be in the future under a sort of ‘business as usual’ path, which makes you unenthusiastic about making the future long or big
4. You don’t think the future will be long (unless we have totalitarianism) which reduces the scope for doing good by focusing on the future
5. You’re sceptical there are lock-in scenarios we can affect within the next few decades, and don’t think there is much point of trying to affect them beyond this time horizon

I’m going to accept 1, 2 as your personal values and I won’t try to shift you on them. I don’t massively disagree on point 3.

I’m not sure I completely agree on point 4 but I can perhaps accept it as a reasonable view, with a caveat. Even if the future isn’t very long in expectation, surely it is kind of long in expectation? Like probably more than a few hundred years? If this is the case, might it be better to be some sort of “medium-termist” as opposed to a “traditional neartermist”. For example, might it be better to tackle climate change than to give out malarial bednets? I’m not sure if the answer is yes, but it’s something to think about.

Also, as has been mentioned, if we can only have long futures under totalitarianism, which would be terrible, might we want to reduce risks of totalitarianism?

Moving onto point 5 and lock-in scenarios. Firstly I do realise that the constellation of your views means that the only type of x-risk you are likely to care about is s-risks, so I will focus on lock in events that involve vast amounts of suffering. With that in mind, why aren’t you interested in something like AI alignment? Misaligned AI could lock-in  vast amounts of suffering. We could also create loads of digital sentience that suffers vastly. And all this could happen this century. We can’t be sure of course, but it does seems reasonable to worry about this given how high the stakes are and the uncertainty over timelines. Do you not agree? There may also be other s-risks that may have potential lock-ins in the nearish future but I’d have to read more.

My final question, still on point 5, is why don’t you think we can affect probabilities of lock-in events that may happen beyond the next few decades? What about growing the Effective Altruism/longtermist community, or saving/investing money for the future, or improving values? These are all things that many EAs think can be credible longtermist interventions and could reasonably affect chances of lock-in (including of the s-risk kind) beyond the next few decades as they essentially increase the number of thoughtful/good people in the future or the amount of resources such people have at their disposal. Do you disagree?

"If there have been any intentional impacts for more than a few hundred years out"

There have been a number of stabilizing religious institutions which were built for exactly this purpose, both Jewish, and Christian. They intended to maintain the faiths of members and peace between them, and have been somewhere between very and incredibly successful in doing so, albeit imperfectly. Similarly, Temple-era Judaism seems to have managed a fairly stable system for several hundred years, including rebuilding the Temple after its destruction.  We also have the example of Chinese dynasties and at least several European monarchies which intended to plan for centuries, and were successful in doing so. 

But given the timeline of "more than a few hundred years out," I'm not sure there are many other things which could possibly qualify. On a slightly shorter timescale, there are many, many more examples. The US government seems like one example - an intentionally built system which lasted for centuries and spawned imitators which were also largely successful. But on larger and smaller scales, we've seen 200+ year planning be useful in many, many cases, where it occurred. 

The question of what portion of such plans worked out is a different one, and a harder one to answer, but it's obviously a minority. I'm also unsure whether there are meaningful differentiators between cases where it did and didn't work, but it's a really good question, and one that I'd love to see work on.

Thank you everyone for the many responses! I will address one point which came up in multiple comments here as a top-level comment, and otherwise respond to comments.

Regarding the length of the long-term future: My main concern here is that it seems really hard to reach existential security (i.e. extinction risks falling to smaller and smaller levels), especially given that extinction risks have been rising in recent decades. If we do not reach existential security the future population is much smaller accordingly and gets less weight in my considerations. I take concerns around extinction risks seriously - but they are an argument against longtermism, not in favour of it. It just seems really weird to me to jump from 'extinction risks are rising so much, we must prioritize them!' to 'there is lots of value in the long-term future'. The latter is only true if we manage to get rid of those extinction risks.

The line about totalitarianism is not central for me. Oops. Clearly should not have opened the section with a reference to it.

I think even with totalitarianism reaching existential security is really hard - the world would need to be permanently locked into a totalitarian state.

I recommend reading this shortform discussion on reaching existential security.

Something that stood out to me in that discussion (in a comment by Paul Christiano: "Stepping back, I think the key object-level questions are something like "Is there any way to build a civilization that is very stable?" and "Will people try?" It seems to me you should have a fairly high probability on "yes" to both questions.")

as well as Toby's EAG Reconnect AMA is how much of the belief that we can reach existential security might be based on a higher level of baseline optimism than I have about humanity.

I am largely sympathetic to the main thrust of your argument (borrowing from your own title: I am probably a negative utilitarian), but I have 2 disagreements that ultimately lead me to a very different conclusion on longtermism and global priorities:

1. Why do you assume we cannot effect the future further than 100 years? There are numerous examples of humans doing just that: in science (inventing the wheel, electricity or gunpowder), government (the US constitution), religion (the Buddhist Pali cannon, the Bible, the Quran), philosophy (utilitarianism), and so on. One can even argue that the works of Shakespeare have had an effect on people for hundreds of years.
2. Though humanity is not inherently awesome, it does not inherently suck either. Humans have the potential to do amazing things, for good or evil. If we can build a world with a lot less war and crime and a lot more collaboration and generosity, isn't it worth a try? In Parfit's beautiful words: "Life can be wonderful as well as terrible, and we shall increasingly have the power to make life good. Since human history may be only just beginning, we can expect that future humans, or supra-humans, may achieve some great goods that we cannot now even imagine. In Nietzsche’s words, there has never been such a new dawn and clear horizon, and such an open sea ... Some of our successors might live lives and create worlds that, though failing to justify past suffering, would give us all, including some of those who have suffered, reasons to be glad that the Universe exists.”

For similar moral views (asymmetric, but not negative utilitarian), this paper might be of interest:

Teruji Thomas, "The Asymmetry, Uncertainty, and the Long Term" (also on the EA Forum). See especially section 6 (maybe after watching the talk, instead of reading the paper, since the paper gets pretty technical).

This is a link collection for content relevant to my post published since, for ease of reference.

Focusing on the empirical arguments to prioritise x-risks instead of philosophical ones (which I could not be more supportive of):

1. Carl Shulman’s 80,000hours podcast on the common sense case for existential risk

2. Scott Alexander writing about the terms long-termism and existential risks

On the definition of existential risk (as I find Bostrom’s definition dubious):

3. Linch asking how existential risk should be defined

4. Based on this comment thread in a different question by Linch

5. Zoe’s paper which also has other stuff I have not yet read in full

How GCBRs could remain a solved problem, thereby getting us closer to existential security:

6. A blogpost by Carl which cross-posted to the EA Forum later than it was published on the blog

Thanks for this clear write-up in an important discussion :) 

I'm not sure where exactly my own views lie, but let me engage with some of your points with the hope of clarifying my own views (and hopefully also help you or other readers).

You say that care more about the preference of people than about total wellbeing, and that it'd change your mind if it turns out that people today prefer longtermist causes. 

What do you think about the preferences of future people? You seem to take the "rather make people happy than to make happy people" point of view on population ethics, but future preferences extend beyond their preference to exist. Since you also aren't interested in a world where trillions of people watch Netflix all day, I take it that you don't take their preferences as that important. 

That said, you clearly do care about the shape of the future of humanity. Whether people have freedom, whether people suffer, whether they are morally righteous, etc. In fact, you seem to be pretty pessimistic about humanity's future in those aspects. Also, it seems like you aren't interested in transhumanist futures - at least, not how they are usually depicted. 

Some thoughts on that. But first, please let me know if (where) I was off in any of the above. Sorry if I've misinterpreted your views.

1. I think that the length of the long-term future might be a strong double-crux here. If you'd expect the future to be mostly devoid of value, or even not many orders of magnitude more than the near future, then I'd find it very hard to justify working on longtermist causes (mostly due to traceability). Instead of addressing that, I'll just respond to your other points conditional on there being a likely long-term future with lots of valuable life.
2. I feel some uneasiness about not considering future people's preferences as mostly equal to people alive today. I think that the way I feel about it is somewhat like child-rearing: I'd want some sort of a balance between directing my children's future to become "better people" and give them the freedom to make their own choices and binge on Netflix. Furthermore, I can already predict many of their preferences for which I can make some preparation (say, save up on money or buy an apartment in child-friendly areas). Another analogy here is that of colonialism, where one entity acts to shape the future of another (weaker) entity. Overall, I feel like we have a lot of responsibility for future people and we should take care not to enforce our own worldview too much. 
3. Very relevant is the question of whether moral growth is possible (or even expected). I'm not sure of my own views here, but I definitely think that improving moral progress could be potentially a very important cause. 
4. I think that some sort of a transhumanist future is inevitable. It's hard for me to imagine economic/intellectual progress completely stopping or slowing down drastically forever without any major catastrophe, and it's hard for me to imagine non-transhumanist futures with consistent exponential growth. Holden Karnofsky makes this case here in his recent The Most Important Century series. 
5. Now, since you seem to disvalue transhumanist futures, I think this might be where our opinions might differ the most but maybe most malleable. I can imagine many potential futures where sentient beings are living in abundance and having meaningful lives. I don't think that paperclip-maximizers and ruthless dictatorships are the most likely futures (although, I do think that these kinds of futures are important risks). For one thing, our values aren't that weird. But other than that, a likely scenario is that of gradual moral change, rather than locking-in to some malign set of random values. I think that some discussions of Utopias are very relevant here, but they may be misleading.  This is something I want to think more about, as I'm easily biased into believing weird futuristic scenarios.

(My views are suffering-focused and I'm not committed to longtermism, although I'm exploring s-risks slowly, mostly passively.)

I do not feel compelled by arguments that the future could be very long. I do not see how this is possible without at least soft totalitarianism, which brings its own risks of reducing the value of the future.

Do you mean you expect all of our descendants to be wiped out, with none left? What range would you give for your probability of extinction (or unrecoverable collapse) each year?

If we colonize space and continue to expand (which doesn't seem extraordinarily unlikely), the probabilities of extinction in distant colonies become less and less correlated, and the probability of all colonies being wiped out with none left to continue expanding would  decrease over time. Maybe this doesn't happen fast enough, in your view?

Thanks for the post. Here are some comments (I am confident there is considerable overlap with the other comments, but I have not read them):

• What was done well:
  • Willingness to challenge EA ideas in order to better understand them and improve them.
  • Points to possibly neglected topics in long-termism (e.g. mitigation of very bad outcomes).
  • Sections “What would convince me otherwise”.
  • Good arguments for why it is uncertain whether the long-term future will be good/bad. 
• What could be improved:
  • “While creating happy people is valuable, I view it as much less valuable than making sure people are not in misery. Therefore I am not extremely concerned about the lost potential from extinction risks”.
    • What about x-risks which do not involve extinction? For example, decreasing s-risk would decrease the likelihood of a future with large “misery”.
  • Sections “I do not think humanity is inherently super awesome” and “I am unsure whether the future will be better than today”.
    • Longtermism only requires that most of the expected value of our actions is in the future. It does not rely on predictions about how good the future will be.
  • Section “The length of the long-term future”.
    • Similarly, given the uncertainty about the length of the long-term future (Toby Ord guesses in The Precipice there is a “one in two chance that humanity avoids every existential catastrophe and eventually fulfils its potential”), most of the expected value should concern the long-term.
    • Explicit expected value calculations could overestimate the importance of the long-term. However, a more accurate Bayesian approach could still favour the long-term as long as the prior is not unreasonably narrow.
  • Section “The ability to influence the long-term future”.
    • The concept of s-risk could be mentioned here.

A lot to say on this but keeping short on time I make some quick points.

1. Humanity is "not super awesome" in comparison to what alternative? From my POV I am grateful at all that humanity has some measure of empathy, charity and compassion. I can imagine much worse and to humanity's credit we live in a very hostile universe. Have we come to forget how cruel nature could be?
2. Are we taking for granted that being "good" or altruist is not a natural state? It requires a lot of foresight, it is learned, it is instilled through generations of evolution which I believe to still be in the process of being selected.
3. It is hard to be "good". It is easier to be less bad but generally we're most likely doing a lot of bad without realizing it most of the time. I don't find it so easy to point fingers at others for not being so moral. Their life is an entire universe I have yet to discover. I don't know them and their burdens. I am grateful that I am in a position to think of morals and ethics at all. Charity is a privilege.
4. Perhaps I believe that generally humans are good and that they behave badly because they are lacking something which has prevented their growth into become a better human. I think our world in data really highlights the trend of what humanity does as it becomes more efficient/wealthy.
5. Could humanity become totalitarian and cruel? Absolutely. Which is why EA and our fight is so important. Perhaps I believe in humanity because I must.
6. Lastly I can't help but feel a sense of irony at people who are living in the modern age in one of the developed countries thinking humanity is bad while benefiting from all the labor of our ancestors. We are so privileged today standing on their shoulders. All the while talking about how bad it is.