I liked this post and would like to see more of people thinking for themselves about cause prioritization and doing BOTECs.

Some scattered thoughts below, also in the spirit of draft amnesty.

I had a little trouble understanding your calculations/logic, so I'm going to write them out in sentence form: GiveWell's current giving recommendations correspond to spending about $0.50 to save an additional person an additional year of life.  A 10% chance of extinction from misaligned AI means that postponing misaligned AI by a year gets us 10%*current population number of person days, or about 10 million. If we take GiveWell's willingness to spend and extrapolate it to the scenario of postponing misaligned AI, we get that GiveWell might be willing to spend $500 million to postpone misaligned AI by a day. 

I think it's important that these are different domains, and the number of people who would be just as happy to see their donation buy a bednet as lobby for tech regulation (assuming similar EV) is unfortunately small. Many donors care about much more than some cause-neutral how-much-good their donation does. e.g., for instance I see to care (I'm confiused) that some of my donations help extremely poor people.

You point out that maybe regulation doesn't work, but there's the broader problem which is that we don't have shovel ready projects that can turn $500 million into postponing misaligned AI by a day; I suspect there are many interventions which can do this for much cheaper, but they are not interventions which can just absorb money and save lived like many global health charities can (perhaps they need projects to be founded and take years to develop). 

The above problems point to another important idea: The GiveWell bar is able to be where it is because of what projects to improve the world can actually be funded by GiveWell dollars — not because of some fact about the value of postponing a life by a day. You might think about the GiveWell bar as, the cheapest scalable ways to save lives in the global health and development space can provide an additional day of life for $0.50. If you ask individuals in the US how much they would pay to extend their own life or that of a loved one by a day, you will get numbers much higher than this; if you look at spending on healthcare in the developed world my guess is that it is very normal to spend thousands of dollars to postpone a life by a day. GiveWell's bar for funding would be higher if there were other great opportunities for saving lives salably for cheap (at least in global health and development). 

An abstraction I notice I'm using is thinking about $0.50/person/day as the current market price. However, this is not an efficient market, for a number of reasons. This post draws the parallel of "hey look, at that price we should be willing to spend $0.5b on postponing misaligned AI by a day". However, if we actually had many opportunities to spend $0.5b on postponing misaligned AI by a day, the funding bar would increase, because there isn't enough money in the cause-neutral altruism bucket. 

Some implications: cause-neutral donors who put above negligible probability on existential risks from AI will probably get much more bang for their buck trying to reduce existential risks or buy time, at least contingent on there being projects that can absorb money in that space. More importantly, those working on reducing AI x-risk have a lot of work to do in terms of closing the cost-effectiveness gap between themselves and global health. By closing the gap I mean getting enough projects to exist in the space such that we can consistently take in more money and turn it into x-risk reduction or buying time. 

If you haven't read Astronomical Waste, you might like it. 

Hoo boy, I think we can get a paradox out of this:

Givewell is looking at the loss of potential life for the people that currently exist. Save 30000 lives that live for an average of 50 years each, and that’s 30000*50 =1.5 million years worth of potential lives saved. 

Suppose the population stayed constant at 10^10 people forever, with average 50 years left. 

In the event of a human extinction, the number of potential life years lost if the apocalypse happened today would be 50*10^10. 

Now what happens if we survive another hundred years, and the extinction happens then? 

Well, there are still 10^10 people averaging 50 years left, so the number of potential life years lost as a result of the apocalypse is… 50*10^10.  

So it seems like if you go by the givewell calculations, delaying the apocalypse makes no difference at all! In fact, if the population were to increase, then the amount of potential life lost in the extinction would be greater, so arguably delaying would make it worse.  

Ride the current way of AI skepticism by people worried about it being racist, or being replaced and left unemployed. To lobby for significantly more government involvement, to slow down progress (like the FDA in medicine).

I agree! In recent days, I've been soundboarding an idea of mine: 

Idea: AI Generated Content (AIGC) Policy Consultancy

Current Gaps:
1. Policy around services provided by AIGC is probably not gonna be good within the next decade, despite the speed with which AI will begin automating tasks and industries. See: social media, crypto policy.

2. AI Safety community currently struggles with presenting strong, compelling value propositions or near-term inroads into policymaking circles. This is consistent with other x-risk topics. See: climate and pandemic risk.

Proposition: EA community gathers law and tech people together to formulate and AIGC policy framework. Will require ~10 tech/law people which is quite feasible as an EA project.

Benefits:

1. Formulating AIGC policy will establish credibility and political capital to tackle alignment problems

2. AIGC is the most publicly understandable way to present AI risk to the public, allowing AIS to reach mainstream appeal

3. Playing into EA’s core competencies of overanalysing problems

4. Likely high first mover advantage, where if EA can set the tone for AI policy discourse, it will mitigate people believing misconceptions about AI as a new tech, which of course benefits AIS in the long run

Further Thoughts

Coming from a climate advocate background, I think this is the least low-probability way for EA to engage the public and policymakers on AIS. It seeks to answer “How to we get politicians to take EA’s AIS stances seriously”

I find that some AIS people I've talked to don't immediately see the value of this idea. However, my context is that having been a climate advocate, I learned of an incredibly long history of scientists' input being ignored simply because the public and policymakers did not prioritise the value of climate risk work.

It was ultimately engaging, predominantly youth, advocacy that mobilised institutional resources and demand to the level required. I highly suspect this will hold true for AI Safety, and I hope this time, the x-risk community doesn't make the same mistake of undervaluing external support. So this plan is meant to provide a value proposition for AI Safety that non-AIS people understand better.

So far, I haven't been able to make much progress on this idea. Problem being that I am neither in the law field nor technical AIS field (something I hope to work on next year), so if it happens, I essentially need to find someone else to spearhead it.

Anyway, I posted this idea publicly because I've procrastinating on developing it for ~1 week, so I figured it was better to send it out into the ether and see if anyone feels inspired, rather than just let it sit in my Drafts. Do reach out if you or anyone you know might be interested!

[deleted]

[Edit: this post has been updated, and this comment applies substantially less now. See this thread for details. ]

As a longtermist, I think this post is bad and harmful. I strongly dislike this framing, and I think it's very unhealthy for an altruistic community. 

First, I think the fermi estimate here is  not good, principally for a lack of any discounting and for failing to try to  incorporate the objections raised in the post into the actual estimate.  But'll leave the specifics of the back-of-the envelope  estimate aside in favor of putting emphasis on what think is the most harmful thing. 

Pitting X-risks against other ways of making the world better (1) is extremely unlikely to convince anyone to work on x-risk who isn't already doing so, (2) hedges on very unlikely risky scenarios without incorporating principles involving discounting, epistemic humility, or moral uncertainty, (3)  is certain to alienate people and  is the kind of thing that makes enemies--not friends--which reduces the credibility and sociopolitical capital of longtermism, and (4) is very disrespectful toward real people in the real world who suffer from real, large problems that GiveWell charities try to address. 

I would encourage deleting this post. 

Hoo boy, I think we can get a paradox out of this:

Givewell is looking at the loss of potential life for the people that currently exist. Save 30000 lives that live for an average of 50 years each, and that’s 30000*50 =1.5 million years worth of potential lives saved. 

Suppose the population stayed constant at 10^10 people forever, with average 50 years left. 

In the event of a human extinction, the number of potential life years lost if the apocalypse happened today would be 50*10^10. 

Now what happens if we survive another hundred years, and the extinction happens then? 

Well, there are still 10^10 people averaging 50 years left, so the number of potential life years lost as a result of the apocalypse is… 50*10^10.  

So it seems like if you go by the givewell calculations, delaying the apocalypse makes no difference at all! In fact, if the population were to increase, then the amount of potential life lost in the extinction would be greater, so arguably delaying would make it worse.