Existential Risk and Economic Growth

I think Carl Shulman makes some persuasive criticisms of this research here :

My main issue with the paper is that it treats existential risk policy as the result of a global collective utility-maximizing decision based on people's tradeoffs between consumption and danger. But that is assuming away approximately all of the problem.

If we extend that framework to determine how much society would spend on detonating nuclear bombs in war, the amount would be zero and there would be no nuclear arsenals. The world would have undertaken adequate investments in surveillance, PPE, research, and other capacities in response to data about previous coronaviruses such as SARS to stop COVID-19 in its tracks. Renewable energy research funding would be vastly higher than it is today, as would AI technical safety. As advanced AI developments brought AI catstrophic risks closer, there would be no competitive pressures to take risks with global externalities in development either by firms or nation-states.

Externalities massively reduce the returns to risk reduction, with even the largest nation-states being only a small fraction of the world, individual politicians much more concerned with their term of office and individual careers than national-level outcomes, and individual voters and donors constituting only a minute share of the affected parties. And conflict and bargaining problems are entirely responsible for war and military spending, central to the failure to overcome externalities with global climate policy, and core to the threat of AI accident catastrophe.

If those things were solved, and the risk-reward tradeoffs well understood, then we're quite clearly in a world where we can have very low existential risk and high consumption. But if they're not solved, the level of consumption is not key: spending on war and dangerous tech that risks global catastrophe can be motivated by the fear of competitive disadvantage/local catastrophe (e.g. being conquered) no matter how high consumption levels are.

I agree with Carl; I feel like other commenters are taking this research as a strong update, as opposed to a simple model which I'm glad someone's worked through the details of but which we probably shouldn't use to influence our beliefs very much.

I read the paper (skipping almost all the math) and Philip Trammell's blog post. I'm not sure I understood the paper, and in any case I'm pretty confused about the topic of how growth influences x-risk, so I want to ask you a bunch of questions:

1. Why do the time axes in many of the graphs span hundreds of years? In discussions about AI x-risk, I mostly see something like 20-100 years as the relevant timescale in which to act (i.e. by the end of that period, we will either go extinct or else build an aligned AGI and reach a technological singularity). Looking at Figure 7, if we only look ahead 100 years, it seems like the risk of extinction actually goes up in the accelerated growth scenario.

2. What do you think of Wei Dai's argument that safe AGI is harder to build than unsafe AGI and we are currently putting less effort into the former, so slower growth gives us more time to do something about AI x-risk (i.e. slower growth is better)?

3. What do you think of Eliezer Yudkowsky's argument that work for building an unsafe AGI parallelizes better than work for building a safe AGI, and that unsafe AGI benefits more in expectation from having more computing power than safe AGI, both of which imply that slower growth is better from an AI x-risk viewpoint?

4. What do you think of Nick Bostrom's urn analogy for technological developments? It seems like in the analogy, faster growth just means pulling out the balls at a faster rate without affecting the probability of pulling out a black ball. In other words, we hit the same amount of risk but everything just happens sooner (i.e. growth is neutral).

5. Looking at Figure 7, my "story" for why faster growth lowers the probability of extinction is this: The richer people are, the less they value marginal consumption, so the more they value safety (relative to consumption). Faster growth gets us sooner to the point where people are rich and value safety. So faster growth effectively gives society less time in which to mess things up (however, I'm confused about why this happens; see the next point). Does this sound right? If not, I'm wondering if you could give a similar intuitive story.

6. I am confused why the height of the hazard rate in Figure 7 does not increase in the accelerated growth case. I think equation (7) for ${\delta }_t$ might be the cause of this, but I'm not sure. My own intuition says accelerated growth not only condenses along the time axis, but also stretches along the vertical axis (so that the area under the curve is mostly unaffected).

   As an extreme case, suppose growth halted for 1000 years. It seems like in your model, the graph for hazard rate would be constant at some fixed level, accumulating extinction probability during that time. But my intuition says the hazard rate would first drop near zero and then stay constant, because there are no new dangerous technologies being invented. At the opposite extreme, suppose we suddenly get a huge boost in growth and effectively reach "the end of growth" (near period 1800 in Figure 7) in an instant. Your model seems to say that the graph would compress so much that we almost certainly never go extinct, but my intuition says we do experience a lot of risk for extinction. Is my interpretation of your model correct, and if so, could you explain why the height of the hazard rate graph does not increase?

   This reminds me of the question of whether it is better to walk or run in the rain (keeping distance traveled constant). We can imagine a modification where the raindrops are motionless in the air.

I thought this was one of the most exciting pieces of research I've seen in the last few years. It also makes me really eager to see GPI hopefully making more work in a similar vein happen.

[Disclaimer: I co-organized the summer research fellowship, as part of which Leopold worked on this research, though I didn't supervise him.]

I think this is an extremely impressive piece of work in economics proper not to mention a substantial contribution to longtermism research. Nice going.

I think this research into x-risk & economic growth is a good contribution to patient longtermism.  I also think that integrating thoughts on economic growth more deeply into EA holds a lot of promise -- maybe models like this one could someday form a kind of "medium-termist" bridge between different cause areas, creating a common prioritization framework.  For both of these reasons I think this post is worth of inclusion in the decadal review.

The question of whether to be for or against economic growth in general is perhaps not the number-one most pressing dilemma in EA (since everyone agrees that differential technology development into x-risk-reducing areas is very important), but it is surely up there, since it's such a big-picture question that affects so many decisions.  Other than X-risk concerns, economic growth obviously looks attractive -- both in the developing world where it's a great way to make the world's poorest people more prosperous, or in the first world where the causes championed by "progress studies" promise to create a prosperous and more dynamic society where people can live better lives.  But of course, by longtermist lights, how fast we get to the future is less important than making sure we get there at all.  So, in the end, what to do about influencing economic growth?  Leopold's work is probably just a starting point for this huge and perhaps unanswerable set of questions.  But it's a good start -- I'm not sure if I want more economic growth or not, but I definitely want more posts like these tackling the question.

For the decadal review, rather than the literal text of this post (which merely refers to the pdf) or the comprehensive 100-page pdf itself, I'd suggest including Leopold's "Works in Progress" article summarizing his research.

Neat paper. One reservation I have (aside from whether x-risk depends on aggregate consumption or on tech/innovation, which has already been brought up) is the assumption of the world allocating resources optimally (if impatiently). I don't know if mere underinvestment in safety would overturn the basic takeaways here, but my worry is more that a world with competing nation-states or other actors could have competitive dynamics that really change things.

Thanks very much for writing this, I found it really interesting. I like the way you follow the formalism with many examples.

I have a very simple question, probably due to my misunderstanding - looking at your simulations, you have the fraction of workers and scientists working on consumption going asymptotically to zero, but the terminal growth rate of consumption is positive. Is this a result of consumption economies of scale growing fast enough to offset the decline in worker fraction?

This sounds really cool. Will have to read properly later. How would you recommend a time pressured reader to go through this? Are you planning a summary?

This post was awarded an EA Forum Prize; see the prize announcement for more details.

My notes on what I liked about the post, from the announcement:

While Leopold’s paper was written away from the Forum, his taking the time to publish it and ask for feedback made him eligible for the Prize; it was also good to see his detailed replies to questions from other users.

Meanwhile, the paper itself is well-formatted (in the standard style of many economics papers) and seems easy to follow. I’d expect an economist, or someone else familiar with the field’s mathematical background, to be able to track Leopold’s points and, if they disagree with anything, to be able to pin down where his argument goes awry.

Note: Since the Prize committee won’t always have domain expertise for posts on technical topics, we generally care more about structure, organization, and clarity than whether we think an author’s conclusion is correct. A really good Forum post is one that explains itself point-by-point, such that critics and supporters alike can engage more effectively with the author.

I'm getting site not secure errors on all 4 browsers for the draft. Could you please make it more accessible?

I think that existential risk is still something that most governments aren't taking seriously. If major world governments had a model that contained a substantial probability of doom, there would be a Lot more funding. Look at the sort of funding anything and everything that might possibly help that happened in the cold war. I see this not taking it seriously as being caused by a mix of human psychology, and historical coincidence. I would not expect it to apply to all civilizations.