Should we be spending no less on alternate foods than AI now?

By Denkenberger🔸 @ 2017-10-29T23:28 (+38)

Summary: As part of a Centre for Effective Altruism (CEA) grant, I have estimated the cost effectiveness of preparing for agricultural catastrophes such as nuclear winter. This largely involves planning and research and development of alternate foods (roughly those not dependent on sunlight such as mushrooms, natural gas digesting bacteria, and extracting food from leaves). Sun-blocking catastrophes could cause the collapse of civilization, and there are a number of reasons why humanity might not recover. Not recovering from the collapse of civilization is one form of existential (X) risk because humanity would not fulfill its potential. I have developed a model that uses Monte Carlo (probabilistic) sampling to estimate uncertain results using open source software (Guesstimate) that incorporates an earlier model of artificial general intelligence safety (hereafter AI) cost-effectiveness. With a number of assumptions unfavorable to alternate foods, spending approximately $100 million on alternate foods has a similar cost effectiveness to AI safety. Because the agricultural catastrophes could happen immediately and because existing expertise relevant to alternate foods could be co-opted by charitable giving, it is likely optimal to spend most of this money in the next few years. I continue to believe that AI is extremely important, and do not advocate a reduction in AI funding. I think that this alternate foods funding gap could be filled by large and small EA donors with additional capacity, and possibly donors who are concerned about X risk but find claims about AI implausible. The bigger picture is that even more funding is justified for both AI and alternate foods even from the perspective of the present generation, let alone future generations. Having alternate foods as a top priority would be a significant realignment of focus in the X risk community, so I invite more feedback and discussion (including playing with the model).¹

Disclaimer/Acknowledgements: I would like to acknowledge CEA for funding the EA grant to perform research on solutions to agricultural catastrophes, Ozzie Gooen for developing Guesstimate, Oxford Prioritisation Project for the AI model, and Joshua Pearce, Alexey Turchin, Michael Dickens, Owen Cotton-Barratt, Finan Adamson, Anders Sandberg, Allen Hundley, and Anthony Barrett for reviewing content. Opinions are my own and this is not the official position of CEA, the Global Catastrophic Risk Institute nor the Alliance to Feed the Earth in Disasters (ALLFED).

Introduction

The greatest catastrophic threat to global agriculture is full-scale nuclear war between US and Russia, with corresponding burning of cities and blocking of the sun for 5-10 years. The obvious intervention is prevention of nuclear war, which would be the best outcome. However, it is not neglected, as it has been worked on for many decades and is currently funded at billions of dollars per year quality adjusted. The next most obvious solution is storing food, which is far too expensive (~tens of trillions of dollars) to have competitive cost effectiveness (and it would take many years so it would not protect us right away, and it would exacerbate current malnutrition). I have posted before about getting prepared for alternate foods (roughly those not dependent on sunlight that exploit biomass or fossil fuels). This could save expected lives in the present generation for $0.20 to $400 for only 10% global agricultural shortfalls like the year without a summer in 1816 caused by a volcanic eruption, and would be even more cost effective if sun blocking scenarios were considered. Of course alternate foods would not save the lives of those people directly impacted by the nuclear weapons, which is potentially hundreds of millions. But since about 6 billion people would die with our current ~half a year of food storage if the sun were blocked for 5 years, alternate foods would solve ~90% of the problem. Current awareness of alternate foods is relatively low: about 700,000 people globally have heard about the concept based on impression counters for the ~10 articles, podcasts, and presentations for which there were data including Science (out of more than 100 media mentions). Also, many of the technologies need to be better developed. Planning, research and development are three interventions, which could dramatically increase the probability of success of feeding everyone, each costing in the tens of millions of dollars. This post analyzes the cost effectiveness of alternate foods from an X risk perspective. It is generally thought to be very unlikely that the agricultural catastrophes such as nuclear war with the burning of cities (nuclear winter), super volcanic eruption, or a large asteroid/comet impact would directly cause human extinction.² However, there is significant probability that by blocking the sun for about 5 years, these catastrophes could cause the collapse of civilization. One definition of the collapse of civilization involves short-term focus, collapse of long distance trade, widespread conflict, and loss of government (Coates, 2009). Not recovering from the collapse of civilization is one form of X risk because humanity would not fulfill its potential. Reasons that civilization might not recover include: Easily accessible fossil fuels and minerals are exhausted, we might not have the stable climate of last 10,000 years, we might lose trust or IQ permanently because of the trauma and genetic selection of the catastrophe, an endemic disease could prevent high human population density, and a permanent loss of grains (e.g. from an engineered crop disease that affects the entire grass family) could preclude high human population density. If the loss of civilization persists long enough, a natural catastrophe could cause the extinction of humanity, such as a super volcanic eruption or an asteroid/comet impact.

AI has been a top priority in the X risk community. EAs have been very important in raising awareness and funding for this cause. I seek to compare the cost effectiveness of alternate foods with AI to see if alternate foods should also be a top priority. The Guesstimate model for AI cost-effectiveness was developed by the Oxford Prioritisation Project (which uses input from Owen Cotton-Barrett’s and Daniel Dewey’s model). I use a subset of this model, because I do not try to quantify the value of the far future. I do not discuss the assumptions in the AI model here. Another possible AI model to compare to would be Michael Dickens’, but this is future work.

Model of alternate foods

I implemented the model in Guesstimate here. Be aware that with the large uncertainties, when you load the model, some values could be different by a factor of two (which I have noted in the model and shown as grey in the tables). Ozzie Gooen (the developer of Guesstimate) is concerned about runtime and Chrome compatibility with increasing the sample size, so I may need to move back to Analytica for publication repeatability. But for now, getting the order of magnitude right is all that is needed.

I am interested in further feedback on the assumptions. Most of the numbers are closely analogous to numbers in the published literature. Therefore, I will focus here on the numbers with less support here. Table 1 has the input variables for the sun-blocking scenarios.³

Table 1. Input variables for the sun-blocking scenarios

For the probability of full-scale nuclear war, Barrett 2013 analyzes only accidental nuclear war with a fault tree looking at close calls. Many fear that with the current leaders of Russia and the United States, an intentional strike has a significant probability, so I am being conservative (unfavorable to alternate foods) by ignoring this. Also, this does not include super volcano or asteroid/comet risk, but they are relatively small. For the probability of agricultural collapse given full-scale nuclear war, many have assumed this is near 100%. I have done some conservative modeling in (Denkenberger 2017) that produced roughly 20% probability. I have added some variance around this. For the probability of collapse of civilization given the collapse of agriculture, this is based on ~10 workshop participants at EA Global 2016 in San Francisco, and ~10 at the International Disaster Risk Reduction conference in Davos, Switzerland in 2016. Most thought civilization would not survive the loss of agriculture for ~5 years. For the probability of not recovering civilization, this is based on an estimate of an oral poll of ~20 participants taken at "Existential Risk to Humanity" in Gothenburg, Sweden in 2017. I have reduced the median somewhat and increased the variation. I used the same source for the probability of loss of civilization given 10% agricultural shortfall in Table 2. For the probability of saving civilization, I use the probability from the papers of feeding everyone.⁴ It is much easier to save civilization than feed everyone, so this is quite conservative.

A number of catastrophic events could cause a roughly 10% global agricultural shortfall, including a medium-sized asteroid/comet impact, a large but not super volcanic eruption (like the one that caused the year without a summer in 1816), regional nuclear war (for example, India-Pakistan), abrupt regional climate change (10°C in a decade, which has happened in the past multiple times), complete global loss of bees as pollinators, a super crop pest or pathogen, and coincident extreme weather, resulting in multiple breadbasket failures. Though it would be technically straightforward to reduce food consumption by 10% by making less food go to waste, animals, and biofuels, the prices would go so high that the poor may not be able to afford food. We found an expected 500 million lives lost in such a catastrophe. There could also be extreme global climate change of >5°C that happens over a century (so slow in comparison to “abrupt” climate change). This could make conventional agriculture impossible in the tropics, which would be a larger than 10% agricultural impact, but it would occur over ~1 century, so the impact might be similar to the abrupt 10% shortfalls. Other events would not directly affect food production, but still could have similar impacts on human nutrition. Some of these include a conventional world war or pandemic that disrupts global food trade, and causes famine in food-importing countries. Though significantly less likely, it is possible that these catastrophes could result in instability and full scale nuclear war, possibly collapsing civilization. The preparation for 10% agricultural shortfalls would be very similar as to agricultural collapse, especially because the former could lead to the latter. In the Table 2, I list the variables for this scenario. The probability per year of 10% agricultural shortfall leaves many risks unquantified, so it is conservative. The other variables are the same as the sun blocking case.

Table 2. Input variables for the 10% global agricultural shortfalls

Results

Table 3 shows the mean and probability bounds of the output variables.

Table 3. Risks and cost effectiveness for the average of spending $100 million (grey have lower repeatability).

Discussion

The AI model produced a probability of averting existential catastrophe per dollar of 4E-13 to 4E-11 with a mean (expectation) of 8E-12. This is significantly smaller variance than alternate foods, which was the opposite of what I expected. It could be that the general population would estimate AI as more uncertain than nuclear war (e.g. by giving significant weight to the impossibility of AGI) or that even with experts, the AI model should have greater uncertainty (I am not modifying the AI model). The expected cost effectivenesses of alternate foods and AI are similar. In expectation, alternate foods from very small funding to $100 million is about three times as cost effective as AI at the margin (see Table 4).

In order to justify the full $100 million for alternate foods, the marginal cost effectiveness of alternate foods would need to be competitive with the marginal cost effectiveness of AI. When I looked at the marginal cost effectiveness of each of the three interventions of planning, research, and development, I found little declines in cost effectiveness. However, different amounts of money could be spent in each of these categories, so I would expect some declining cost-effectiveness. Returns to donations may be logarithmic fairly generally, which means the marginal cost effectiveness is just one divided by the cumulative money spent. In this case, the marginal cost effectiveness on the last dollar for alternate foods would be about ⅙ the average cost-effectiveness (see the bottom of the Guesstimate model). Then the expected cost-effectiveness of alternate foods on the last dollar would be about half as cost-effective as marginal AI (see Table 4). If we move to funding alternate foods at the margin right now, we need an estimate of the cumulative money spent on alternate foods. Under $1 million equivalent (mostly volunteer time) has been spent so far directly on this effort, nearly all by the Alliance to Feed the Earth in Disasters (ALLFED) (disclaimer, which I cofounded).⁵ The cost-effectiveness of the marginal dollar now is about 20 times greater than average of $100 million assuming logarithmic returns. Then the expected cost effectiveness of marginal dollar now for alternate foods would be nearly two orders of magnitude greater than AI (see Table 4). So there is an even stronger case for a small amount of money now.  

But it is not required for alternate foods to be more cost effective than AI in order to fund alternate foods on a large scale. Funding of X risk in the EA community goes to other causes, notably an engineered pandemic. It is future work to perform a detailed comparison of cost effectiveness with engineered pandemic. Here is a paper on biosecurity with significantly lower cost effectiveness than for AI and alternate foods, but the authors were being very conservative.

Table 4. Mean cost effectiveness (probability of averting existential catastrophe per dollar) and ratios (grey is extrapolation from detailed estimates)

There are additional sources of conservatism for alternate foods. Being prepared for agricultural catastrophes might protect against unknown risks, meaning the cost-effectiveness would be even higher. Also, 80,000 Hours estimates that global climate change of >5℃ would reduce the future potential of humanity by ~20% through a risk of extinction, worse values, international conflict or social breakdown and a failure to recover. My model currently has the reduction in future human potential at only ~0.13% given these types of "10%" global agricultural shortfalls (losing civilization and not recovering). Using their numbers would increase overall cost effectiveness of alternate foods by an order of magnitude just from changing the 10% shortfall numbers. Adding 80,000 Hours estimate of the possibility of worse values to the sun blocking scenarios meaning a 30% reduction in future potential of humanity would ~triple overall cost effectiveness again, meaning ~40 times as cost effective as my model.

Steelmanning the opposition to funding alternate foods:

The Open Philanthropy Project (OPP) is funding a detailed investigation of the impact of nuclear war. Shouldn't we wait until those results before funding alternate foods? The biggest source of uncertainty in the model here is the chance of nuclear war. The OPP investigation will examine which nuclear detonation scenarios are plausible, but they are not planning on assigning quantitative probabilities to these scenarios. Furthermore, every year we wait to get prepared with alternate foods, we expose ourselves to an ~0.01% additional chance of existential catastrophe. In addition, even if the probability of losing civilization turns out to be zero from nuclear war, spending $100 million on alternate foods would still be competitive with AI from a long term future perspective. Also, this spending is already highly justified for the present generation even not including nuclear winter risk, so it is likely a no-regrets policy (at least with some uncertainty about what to value or if flow through effects to the far future of saving lives now are significant). Of course AI safety would save expected lives in the present generation, but it would be 1-2 orders of magnitude less than alternate foods (see below). This is because an AI catastrophe is likely to kill everyone, while agricultural catastrophes can kill many people without causing an existential catastrophe.

Another steelman is that the estimate of cost effectiveness of AI is too low. Some think that the total existential risk associated with developing highly capable AI systems, bearing in mind all of the work on safety that will be done, is higher than the current 95th percentile of 7% chance.⁷ This could reduce the optimal amount of funding for alternate foods if they have to be competitive with AI, but it would be very unlikely to eliminate additional funding for alternate foods, and alternate foods do not necessarily have to be competitive with AI for it to be optimal to fund them.

A further steelman is that cost effectiveness estimates tend to worsen over time, as GiveWell found for global poverty interventions. This could apply both to AI and alternate foods, though alternate foods are newer, so one might expect that it would apply more to alternate foods. However, given my conservatism, I would expect the estimate of the cost effectiveness of alternate foods to rise over time. Indeed, this has been the case for me over the last few years as I have discovered more catastrophes that alternate foods could ameliorate. In my experience, one has to be conservative to pass peer review at a mainstream journal like the biosecurity paper (though admittedly, not all of my inputs here have been peer reviewed).

Less detailed view

The importance, tractability, neglectedness (ITN) framework is useful for screening cause areas. There is debate about using the ITN framework for interventions as well as risks. Indeed, alternate foods can only potentially solve 90% of the mortality (and perhaps similar for X risk), but this is within the uncertainty of the analysis. The larger concern is that if we allocate money across all the interventions for a risk, less money should be spent on an individual intervention than on the entire cause area. One could look at the categories of interventions for nuclear winter, which might be prevent war, eliminate/reduce nuclear weapons, prevent nuclear winter given nuclear war, and adapt to nuclear winter. I have argued that adaptation should focus on alternate foods, so perhaps alternate foods should have ⅕ the funding of the cause. But since 80,000 Hours has listed about 10 interventions, let’s say alternate foods should have 1/10 the funding a priori. This means spending $100 million on alternate foods should be equivalent in cost effectiveness as spending $1 billion on the nuclear winter cause area.

The less detailed view could look at just importance and neglectedness. According to these models, AI poses a 4% expected risk of existential catastrophe this century (bearing in mind all of the work on safety that will be done), and agricultural catastrophes pose a 1.6% expected risk (of which alternate funds can only address about 90%). With equal tractability and the same level of funding for the cause areas, one would expect that AI would be about three times as cost effective as alternate foods because of the greater importance of AI. Table 5 shows different levels of funding and assumes logarithmic returns to investment (see the bottom of the Guesstimate model). For the alternate foods cost effectiveness for average of $100 million (from $10 million to $1 billion equivalent for nuclear winter cause area) this less detailed view predicts that alternate foods should be 5 times as cost-effective as AI marginal at $3 billion. Since the result in Table 3 was 3 times as cost effective as AI, this implies similar tractability of AI to alternate foods. This was surprising to me, because I expected alternate foods to be something like an order of magnitude more tractable than AI because there are clear ways to make progress in alternate foods and alternate foods are not talent constrained. This could indicate that all those ways I am being conservative with alternate foods really add up. A big one is that I assume the interventions are only valuable for about 20 years instead of a century like for AI (though it is true that if AGI comes soon, alternate foods will be moot). If the conservatism is removed, it could be that alternate foods at the $100 million spending level are significantly more cost-effective than I am claiming and therefore significantly more cost effective than AI at the $3 billion spending level.

This less detailed view can also be used to estimate the marginal cost effectiveness of the one hundred millionth dollar and right now assuming logarithmic returns.  It is about 5 times less cost-effective for the $100 million marginal to alternate foods than for $100 million of funding on average (see bottom of Guesstimate model). Since this is $1 billion equivalent for nuclear winter cause area, this yields the same cost-effectiveness as AI (see Table 5). It is about 20 times as cost-effective for the marginal dollar now ($1 million for alternate foods or $10 million equivalent to the nuclear winter cause area) than for $100 million of funding on average. This yields 100 times as cost-effective as AI (see Table 5). This is roughly consistent with my detailed estimates of the cost effectiveness of the marginal dollar now.

Table 5. Less detailed view cost effectiveness (only importance and neglectedness, assuming equal tractability) (grey because of extrapolations)

Interestingly, if we assume that an existential catastrophe with AI means the loss of 9 billion humans, the cost effectiveness of AI now is $5-$900 per expected life saved (very bottom of Guesstimate model). This is not nearly as cost-effective as alternate foods, but it is significantly lower cost than GiveWell estimates for global health interventions: $900-$7,000. Since AI appears to be underfunded from the present generation perspective, it would be extremely underfunded when taking into account future generations. If this were corrected, then in order to have similar cost-effectiveness with alternate foods, more funding for alternate foods would be justified. Indeed, in order to fund alternate foods just from a current generation perspective at a level of similar cost-effectiveness to global poverty interventions, billions of dollars of alternate food funding would be justified. Much more funding would be justified if valuing future generations. It is kind of depressing that, while we in the X risks community are generally motivated by future generations, we cannot even get work on these risks funded at a level that would be justified by the present generation, a point made in the book Catastrophe: Risk and Response.

Timing of funding

If one agrees that alternate foods should be in the EA budget for X risks, the next question is how to allocate funding to the different causes over time. For AI, there are arguments both for funding now and funding later. For alternate foods, since most of the catastrophes could happen right away, there is significantly greater urgency to fund alternate foods now. Furthermore, it is relatively more effective to scale up the funding quickly because we can, through requests for proposals, co-opt relevant expertise that already exists (e.g. in the different foods, such as biofuel experts who know how to turn fiber into sugar). Since I have not monetized the value of the far future, I cannot use traditional cost-effectiveness metrics such as the benefit to cost ratio, net present value, payback time, and return on investment. However, in the case of saving expected lives in the present generation, the return on investment was from 100% to 5,000,000% per year. This suggests that the $100 million or so for alternate foods should be mostly spent in the next few years to optimally reduce X risk (a smaller amount would maintain preparedness in the future). Since AI safety funding is now about $10 million per year, this would mean more funding for alternate foods than AI in the near term. I think that this alternate foods funding gap could be filled by large and small EA donors with additional capacity. Also, donors who are concerned about X-risk (or just present generations) but find claims about AI implausible could contribute (but I don’t want to let those concerned about AI off the hook!). The formal optimization including other causes such as asteroid deflection and terrestrial/space refuges to repopulate the earth is future work and is related to work at GCRI including value of information and integrated assessment. Other ways of contributing to the alternate foods effort than donating will be the subject of a future post.

Sensitivity analysis

The greatest uncertainty is the probability of nuclear war. I performed a sensitivity analysis on this for the present generation here. Basically, you can just scale up and down the cost-effectiveness numbers by the probability of nuclear war that you feel is most accurate relative to the current expectation of ~1% per year.

Notes:

1 You can change numbers in viewing model to see how outputs change, but they will not save. If you want to save, you can make a copy of the model. Click View, visible to show arrows. Mouse over cells to see comments. Click on the cell to see the equation.

2 Though there were concerns that full scale nuclear war would kill everyone with radioactivity, it turns out that most of the radioactivity is rained out within a few days. One possible mechanism for extinction would be that the hunter gatherers would die out because they do not have food storage. And people in developed countries would have food storage, but might not be able to figure out how to go back to being hunter gatherers.

3 Most distributions are lognormal, but some are beta to avoid greater than 100% probability. Lognormal results in the median being the geometric mean of the ends (multiply the 5th and 95th percentiles and take the square root) (as they say in statistics, the means justify the ends :) ). Note that the mean is generally a lot higher than the median.

4 In the 10% shortfall, alternate foods have some chance of preventing worse outcomes like full scale nuclear war, but then even if full scale nuclear war occurs, alternate foods reduce the chance of losing civilization.

5 Of course a very large amount of money has been spent on trying to prevent nuclear war. More relevant, money has been spent developing alternate foods for other reasons, such as mushrooms and natural gas digesting bacteria. This could easily be tens of millions of dollars that would have needed to be spent for catastrophe preparation. So this would be relevant for the marginal $100 million. However, there are very high value interventions we would do first, like figuring out how to exploit mass/social media in a catastrophe to get the right people to know about alternative foods. Though the alternative foods would not work as well as with $100 million of R&D, just having the leaders of countries know about them and implement them in their own countries without trade could still significantly increase the chance of retaining civilization. The cost of these first interventions would be very low, so it would be very high cost effectiveness.

6 Open AI already has ~$1 billion, so I estimate that $3 billion will be committed to AI. This is roughly consistent with the expectation of number of researchers in the AI model. This should also include quality weighted volunteer time, as I have done in the case of alternate foods.

7 Broadening these uncertainty bounds from the current 1.3% to 7% would also partially address the surprising result that the overall uncertainty of AI is lower than alternate foods.