How many EA 2021 $s would you trade off against a 0.01% chance of existential catastrophe?

Here are my very fragile thoughts as of 2021/11/27:

Speaking for myself, I feel pretty bullish and comfortable saying that we should fund interventions that we have resilient estimates of reducing x-risk ~0.01% at a cost of ~$100M. 

I think for time-sensitive grants of an otherwise similar nature, I'd also lean optimistic about grants costing ~$300M/0.01% of xrisk, but if it's not time-sensitive I'd like to see more estimates and research done first. 

For work where we currently estimate ~$1B/0.01% of xrisk, I'd weakly lean against funding them right now, but think research and seed funding into those interventions is warranted, both for value of information reasons and also because we may have reasonable probability on more money flowing into the movement in the future, suggesting we can lower the bar for future funding. I especially believe this for interventions that are relatively clear-cut and the arguments are simple, or otherwise may be especially attractive to non(currently)-EA billionaires and governments, as we may have future access to non-aligned (or semi-aligned) sources of funding, with either additional research or early-stage infrastructure work that can then leverage more funding.

For work where we currently estimate >>$10B/0.01% of xrisk, I'd be against funding them and weakly lean against deep research into them, with important caveats including a) value of information and b) extremely high scale and/or clarity. For example, the "100T guaranteed plan to solve AI risk" seems like a research project worth doing and laying the foundations of, but not a project worth us directly funding, and I'd be bearish on EA researchers spending significant time on doing research for interventions at similar levels of cost-effectiveness for significantly smaller risks.

One update that has probably not propagated enough for me is a (fairly recent, to me) belief that longtermist EA has a much higher stock of human capital than financial capital. To the extent this is true, we may expect a lot of money to flow in in the future. So as long as we're not institutionally liquidity constrained, we/I may be systematically underestimating how many $s we should be willing to trade off against existential risk.

With 7 billion people alive today, and our current most cost effective interventions for saving a life in the ballpark of $5000 USD, then with perfect knowledge we should probably be willing to spend up to ~$3.5 billion USD to reduce the risk of extinction or similar this century by 0.01 percentage points even if you're not all that concerned about future people. Is my math right?

(Of course, we don't have perfect knowledge and there are other reasons why people might prefer to invest less.)

My very rough gut estimate says something like $1B. Probably more on the margin, just because we seem to have plenty of money and are more constrained on other problems. I think we are currently funding projects that are definitely more cost-effective than that, but I see very little hope in scaling them up drastically while preserving their cost-effectiveness, and so spending $1B on 0.01% seems pretty reasonable to me.

I thought it would be interesting to answer this using a wrong method (writing the bottom line first). That is, I made an assumption about what the cost-effectiveness of grants aimed at reducing existential risk is, then calculated how much it would cost to reduce existential risk by one basis point if that cost-effectiveness was correct.

Specifically, I assumed that Ben Todd’s expectation that Long-Term Future Fund grants are 10-100x as cost-effective as Global Health and Development Fund grants is correct:

I would donate to the Long Term Future Fund over the global health fund, and would expect it to be perhaps 10-100x more cost-effective (and donating to global health is already very good).

I made the further assumptions that:

• Global Health and Development Fund (GH) grants are as cost-effective as GiveWell’s top charities in expectation
• Global Health grants are 8x as cost effective as GiveDirectly
• GiveWell’s estimate that “donating $4,250 to a charity that’s 8x GiveDirectly is as good as saving the life of a child under five” (Ben Todd’s words) is correct.
• All existential catastrophes are extinction events.
• Long-Term Future Fund grants do good via reducing the probability of an extinction event in the next 100 years.
• The only good counted towards Long-Term Future Fund grant cost-effectiveness is the human lives directly saved from dying in the extinction event in the next century. (The value of preserving the potential for future generations is ignored, as is the value of saving lives from any non-extinction level global catastrophic risk reduction.)
• Saving the lives of people then-alive from dying in an extinction event in the next 100 years is as valuable on average as saving the life of a child under five.
• If an extinction event happens in the next century, 8 billion people will die.

From these assumptions, it follows that:

• Donating to the LTFF is 10-100x as cost-effective as donating to the GHDF and 80-800x as cost-effective as donating to GiveDirectly.
• Donating $4,250 to the LTFF does as much good as saving the lives of 10-100 people from dying in an extinction event in the next 100 years.
• Reducing the risk of human extinction in the next century by 0.01% is equivalent to saving the lives of 800,000 people.
• It costs $34M-$340M (i.e. ~$100M) (donated to the LTFF) to reduce the probability of extinction in the next 100 years by 0.01% (one basis point). (Math: $4,250 * 800,000/100 to $4,250 * 800,000/10.)

To reiterate, this is not the right way to figure out how much it costs to reduce existential risk.

I worked backwards from Ben Todd’s statement of his expectations about the cost-effectiveness of grants to the LTFF, making the assumption that his cost-effectiveness estimate is correct, and further assuming that those grants are valuable only due to their effect on saving lives from dying in extinction events in the next century.

In reality, those grants may also help people from dying in non-extinction level global catastrophic risks, increasing the estimate of the cost to reduce extinction risk by one basis point. Further, there is also a lot of value in how preventing extinction preserves the potential for having future generations. If Ben Todd’s 10-100x expectation included this preserving-future-generations value, then that too would increase the estimate of the cost to reduce extinction risk by one basis point.

Just to complement Khorton's answer: With a discount rate of $d$ [1], and a steady-state population of N, and a willingness to pay of $X, the total value of the future is $N\ast \$X/d$, so the willingness to pay for 0.01% of it would be $0.01\ast N\ast \$X/d$

This discount rate might be because you care about future people less, or because you expect a $d$% of pretty much unavoidable existential risk going forward.

Some reference values

• $N={10}^{10}$ (10 billion), $X=\${10}^4$, $r=0.03$ means that willingness to pay for 0.01% risk reduction should be $0.0001\ast {10}^{10}\ast \${10}^4/0.03=333\ast {10}^9$, i.e., $333 billion
• $N=7\ast {10}^9$ (7 billion), $X=\$5\ast {10}^3$, $r=0.05$ means that willingness to pay for 0.01% risk reduction should be $0.0001\ast 7\ast {10}^9\ast 5\ast {10}^3/0.05=70\ast {10}^9$  i.e., $70 billion.

I notice that from the perspective of a central world planner, my willingness to pay would be much higher (because my intrinsic discount rate is closer to ~0%). Taking $d=0.0001$

• $N={10}^{10}$ (10 billion), $X=\${10}^4$, $r=0.0001$ means that willingness to pay for 0.01% risk reduction should be $0.0001\ast {10}^{10}\ast \${10}^4/0.0001=100\ast {10}^{12}$, i.e., $100 trillion 

To do:

• The above might be the right way to model willingness to pay from 0.02% risk per year to 0.01% risk per year. But with, e.g,. 3% remaining per year, willingness to pay is lower, because over the long-run we all die sooner.
  • E.g., reducing risk from 0.02% per year to 0.01% per year is much more valuable that reducing risk from 50.1% to 50%.

[1]: Where you value the $i$-th year in the steady-state at $(1-d{)}^i$ of the value of the first year. If you don't value future people, the discount rate $d$ might be close to $1$, if you do value them, it might be close to $0$. 

NunoSempere @ 2021-11-28T19:15 (+4)

Here is a Guesstimate which calculates this in terms of a one-off 0.01% existential risk reduction over a century.

NunoSempere @ 2021-11-28T19:02 (+4)

Here is a Guesstimate model which addresses the item on the to-do list. Note that in this guesstimate I'm talking about a -0.01% yearly reduction.

robirahman @ 2021-11-30T18:48 (+3)

The steady-state population assumption is my biggest objection here. Everything you've written is correct yet I think that one premise is so unrealistic as to render this somewhat unhelpful as a model. (And as always, NPV of the eternal future varies a crazy amount even within a small range of reasonable discount rates, as your numbers show.)

NunoSempere @ 2021-12-01T12:27 (+2)

For what it's worth,  I don't disagree with you, though I do think that the steady state is a lower bound of value, not an upper bound.

NunoSempere @ 2021-11-28T20:05 (+3)

Thinking more about this, these are more of an upper bound, which don't bind because you can probably buy a 0.01% risk reduction per year much cheaper. So the parameter to estimate would be more like 'what are the other cheaper interventions'

In Ajeya Cotra's interview with 80,000 Hours, she says:

This estimate is roughly $200 trillion per world saved, in expectation. So, it’s actually like billions of dollars for some small fraction of the world saved, and dividing that out gets you to $200 trillion per world saved

This suggests a funding bar of 200T/10,000 world~= 20B for every 0.01% of existential risk. 

These numbers seem very far from my own estimates for what marginal $s can do or the (AFAICT) apparent revealed preferences of Open Phil's philantropic spending, so I find myself very confused about the discrepancy. 

I think reducing x-risk is by far the most cost-effective thing we can do, and in an adequate world all our efforts would be flowing into preventing x-risk. 
The utility of 0.01% x-risk reduction is many magnitudes greater than the global GDP, and even if you don't care at all about future people, you should still be willing to pay a lot more than currently is paid for 0.01% x-risk reduction, as Korthon's answer suggests.

But of course, we should not be willing to trade so much money for that x-risk reduction, because we can invest the money more efficiently to reduce x-risk even more. 
So when we make the quite reasonable assumption that reducing x-risk is much more effective than doing anything else, the amount of money we should be willing to trade should only depend on how much x-risk we could otherwise reduce through spending that amount of money.

To find the answer to that, I think it is easier to consider the following question:

How much more likely is an x-risk event in the next 100 years if EA looses X dollars?

When you find the X that causes a difference in x-risk of 0.01%, the X is obviously the answer to the original question.

I only consider x-risk events in the next 100 years, because I think it is extremely hard to estimate how likely x-risk more than 100 years into the future is.

Consider (for simplicity) that EA currently has 50B$.
Now answer the following questions:
How much more likely is an x-risk event in the next 100 years if EA looses 50B$?
How much more likely is an x-risk event in the next 100 years if EA looses 0$?
How much more likely is an x-risk event in the next 100 years if EA looses 20B$?
How much more likely is an x-risk event in the next 100 years if EA looses 10B$?
How much more likely is an x-risk event in the next 100 years if EA looses 5B$?
How much more likely is an x-risk event in the next 100 years if EA looses 2B$?

Consider answering those questions for yourself before scrolling down and looking at my estimated answers for those questions, which may be quite wrong. Would be interesting if you also comment your estimates.

The x-risk from EA loosing 0$ to 2B$ should increase approximately linearly, so if $x$ is the x-risk if EA looses 0$ and $y$ is the x-risk if EA looses 2B$, you should be willing to pay $d=\frac{0,01\%}{y-x}2B\$$ for a 0.01% x-risk reduction.

(Long sidenote: I think that if EA looses money right now, it does not significantly affect the likelihood of x-risk more than 100 years from now. So if you want to get your answer for the "real" x-risk reduction, and you estimate a $z\%$ chance of an x-risk event that happens strictly after 100 years, you should multiply your answer by $1/(1-z\%)$ to get the amount of money you would be willing to spend for real x-risk reduction. However, I think it may even make more sense to talk about x-risk as the risk of an x-risk event that happens in the reasonably soon future (i.e. 100-5000 years), instead of thinking about the extremely long-term x-risk, because there may be a lot we cannot foresee yet and we cannot really influence that anyways, in my opinion.)

Ok, so here are my numbers to the questions above (in that order):
17%,10%,12%,10.8%,10.35%,10.13%

So I would pay $\frac{0.01\%}{0.13\%}2B\$=154M\$$ for a 0.01% x-risk reduction.

Note that I do think that there are even more effective ways to reduce x-risk, and in fact I suspect most things longtermist EA is currently funding have a higher expected x-risk reduction than 0.01% per 154M$. I just don't think that it is likely that the 50 billionth 2021 dollar EA spends has a much higher effectiveness than 0.01% per 154M$, so I think we should grant everything that has a higher expected effectiveness.

I hope we will be able to afford to spend many more future dollars to reduce x-risk by 0.01%.

I'd guess that quite-well-directed marginal funding can buy a basis point for something like $50M (for example, I’d expect to be able to buy a basis point by putting that money toward a combination of alignment research, AI governance research, and meta-stuff like movement-building around AI). Accounting for not all longtermist funding being so well-directed gives something like $100M of longtermism funding per basis point, or substantially more if we're talking about all-of-EA funding (insofar as non-longtermism funding buys quite little X-risk reduction).

But on reflection, I think that's too high. I arrived at $50M by asking myself "what would I feel pretty comfortable saying could buy a basis point." Considering a reversal test, I would absolutely not take the marginal $100M out of [OpenPhil's longtermism budget / longtermist organizations] to buy one basis point. Reframing as "what amount would I not feel great trading for a basis point in either direction," I instinctively go down to more like $25M of quite-well-directed funding or $50M of real-world longtermism funding. EA could afford to lose $50M of longtermism funding, but it would hurt. The Long-Term Future Fund has spent less than $6M in its history, for example. Unlike Linch, I would be quite sad about trading $100M for a single measly basis point — $100M (granted reasonably well) would make a bigger difference, I think.

I suspect others may initially come up with estimates too high due to similarly framing the question as "what would I feel pretty comfortable saying could buy a basis point," like I originally did. If your answer is $X, I encourage you to make sure that you would take away $X of longtermist funding from the world to buy a single basis point.

Point of clarification: Is this like the world had a 3% chance of ending and now the world has a 2.99% of ending? For what length of time? Are we assuming perfect knowledge?

Nitpick: Assuming that for every positive state there is an equally negative state is not enough to think that the maximally bad state is only -100% of the expected value of the future, it could be much worse than that..

We have to be a little careful when the probability of existential catastrophe is not near zero, and especially when it is near one. For example, if the probability that unaligned AI will kill us (assuming no other existential catastrophe occurs first) is 90%, then preventing an independent 0.1% probability of climate catastrophe isn't worth 10 basis points — it's worth 1. The simulation argument may raise a similar issue.

I think we can get around this issue by reframing as as "change in survival probability from baseline." Then reducing climate risk from 0.1% to 0 is about a 0.1% improvement, and reducing AI risk from 90% to 89% is about a 10% improvement, no matter what other risks there are or how likely we are to survive overall (assuming the risks are independent).

I think most of the variance of estimates may come from the high variance in estimations of how big x-risk is. (Ok, a lot of the variance here comes from different people using different methods to estimate the answer to the question, but assuming people all would use one method, I expect a lot of variance coming from this.)
Some people may say there is a 50% probability of x-risk this century, and some may say 2%, which causes the amount of money they would be willing to spend to be quite different.
But because in both cases x-risk reduction is still (by far) the most effective thing you can do, it may make sense to ask how much you would pay for a 0.01% reduction of the current x-risk. (i.e. from 3% to 2.9997% x-risk.)
I think this would cause more agreement, because it is probably easier to estimate for example how much some grant decreases AI risk in respect to the overall AI risk, than to expect how high the overall AI risk is.
That way, however, the question might be slightly more confusing, and I do think we should also make progress on better estimating the overall probability of an existential catastrophe occurring in the next couple of centuries, but I think the question I suggest might still be the better way to estimate what we want to know.

Suppose there are $N$ people and a baseline existential risk $r$. There's an intervention that reduces risk by $\delta \times 100\%$ (ie., not percentage points). 

Outcome with no intervention: $rN$ people die.
Outcome with intervention: $(1-\delta )rN$ people die.

Difference between outcomes: $\delta rN$. So we should be willing to pay up to $\delta r\cdot u\left(N\right)$ for the intervention, where $u\left(N\right)$ is the dollar value of $N$ lives.

[Extension: account for time periods, with discounting for exogenous risks.]

I think this approach makes more sense than starting by assigning $X to 0.01% risk reductions, and then looking at the cost of available interventions.

I'm curious about potential methodological approaches to answering this question:

1. Arrive at a possible lower bound for the value of averting x-risk by thinking about how much one is willing to pay to save present people, like in Khorton's answer.
2. Arrive at a possible lower bound by thinking about how much is willing to pay for current and discounted future people
3. Thinking about what EA is currently paying for similar risk reductions, and arguing that one should be willing to pay at least as much for future risk-reduction opportunities
   • I'm unsure about this, but I think this is most of what's going on with Linch's intuitions.

Overall, I agree that this question is important, but current approaches don't really convince me. 

My intuition about what would convince me would be some really hardcore and robust modeling coming out of e.g., GPI taking into account both increased resources over time and increased risk. Right now the closest published thing that exists might be Existential risk and growth and Existential Risk and Exogenous Growth—but this is inadequate for our purposes because it considers stuff at the global rather than at the movement level—and the closest unpublished thing that exists are some models I've heard about that I hope will get published soon.

From the perspective of a grant-maker, thinking about reduction in absolute basis points makes sense of course, but for comparing numbers between people, relative risk reduction might be more useful?

E.g. if one person thinks AI risk is 50% and another thinks it's 10%, it seems to me the most natural way for them to speak about funding opportunities is to say it reduces total AI risk by X% relatively speaking.

Talking about absolute risk reduction compresses these two numbers into one, which is more compact, but makes it harder to see where disagreements come from.

It's a minor point, but with estimates of total existential risk sometimes more than an order of magnitude apart from each other, it actually gets quite important I think.

Also, given astronomical waste arguments etc., I'd expect most longtermists would not switch away from longtermism once absolute risk reduction gets an order of magnitude smaller per dollar.

Edited to add: Having said that, I wanna add that I'm really glad this question was asked! I agree that it's in some sense the key metric to aim for and it makes sense to discuss it!

If asteroid risk is 1/1,000,000, how are you thinking about a 0.01% reduction? Do you mean 0.01pp = 1/10,000, in which case you're reducing asteroid risk to 0? Or reducing it by 0.01% of the given risk, in which case the amount of reduction varies across risk categories?

The definition of basis point seems to indicate the former.