CE: Rigorously prioritizing the top health security (biosecurity) ideas

By CE, weeatquince @ 2023-06-30T13:25 (+69)

Every year at Charity Entrepreneurship (CE), we try to find the best interventions for impact-focused charities to launch through our Incubation Program. Our research in 2022 focused on health security (which includes biosecurity) and on large-scale global health. 

This post outlines:

• An introduction to health security at CE. What CE means by health security exactly, some of CE’s views on preventing catastrophic and existential risks, and why we picked “health security” as a topic. 
• Our prioritization of the top health security interventions. Why rigorous prioritization of ideas is needed to be effective in the world (especially for longtermist cause areas), and CE’s approach to this within health security.
• Our results. The long list of nearly 200 ideas, our top six shortlisted ideas, and our final recommendations.

If you’d like to learn how to conduct similar prioritization research, and how to evaluate the results that come out of starting high-impact interventions, we have just launched our Research Training Program that you can apply to by July 17, 2023.

Introduction to health security at CE

What we mean by health security

We took a fairly broad interpretation of health security. Our initial definition was: anything that minimizes the danger of acute public health events.

This included several broad categories:

• Pandemic prevention
• Pandemic preparedness
• General risk preparedness for acute public health events
• Antimicrobial resistance
• Health system strengthening
• Other / meta ideas (e.g., better biosecurity forecasting)

This included looking at both natural pandemics and engineered pandemics (which could be due to accidental or deliberate misuse of biotech).

How much does CE value preventing catastrophic/existential risks? 

This may seem like a bit of a cop-out answer, but there are a wide range of views on this topic among CE staff. There is no clear overarching organizational view or policy. This section gives a rough sense of how “CE staff” might tend to think about these topics, but probably doesn’t accurately reflect any individual staff member’s stance.

In general, CE’s staff tend to apply a very high level of empirical rigor to decision making. Staff tend to trust empirical evidence (e.g., RCTs, M&E of existing charities, base-rate forecasting, etc.) above other kinds of evidence, particularly valuing such evidence above theoretical reasoned arguments. That said, staff tend to accept that making good decisions requires robust/cluster thinking and look for cases where many kinds of evidence align, including theoretical reasoning. Along the same lines, staff are likely to think that doing good is really difficult and having some ongoing measurable evidence of impact is probably required.

In general, CE staff believe that preventing extinction is a worthwhile endeavor. However, given the above, staff are likely to be skeptical about:

• The ability to know what the biggest future risks are, especially where risk estimates rely on reasoned speculation about future technologies.
• The success of any organization that doesn’t have a clear path to measure the impact of its activities.

We took all these views into consideration and chose to focus on health security, including biorisks. This focus allows us (and future CE charities) to explore risk areas that have at least some chance of being globally catastrophic, but also have  clear historical precedents and evidence of less extreme catastrophes. Additionally, we expected to find a number of options within biorisk that could be impactful  in addressing catastrophic biorisks while also demonstrating health impacts in the short run, meaning a new charity should be able to track and demonstrate impact.

In short, we were hoping to find interventions in this sweet spot:

Aiming for this sweet spot led to a bit of a mixed approach. For example:

• In our quantitative modeling of the cost-effectiveness of different interventions, we considered lives saved over the next 50 years. This is not to say that we did not care about lives post-50 years, just that this 50-year metric appeared to be the best proxy for our decision making. Given that pandemic risks are likely to be heavy-tailed, we should expect a reasonable correlation between interventions that save many lives and interventions that prevent extreme risks.
• In our theoretical work mapping out the various theories of change, we mostly focused on understanding the plausible theories of change relevant to avoiding globally catastrophic biological risks. So if an intervention looked to be robustly good on multiple of these theories of change, we were more excited by it.

In practice, we probably aimed a bit to the left of this ‘sweet spot’, tending to favor more near-termist interventions. In the end (as discussed below), we somewhat missed this sweet spot in our final recommendations, one of which fell off to the right and one off to the left.

Going forward, I expect it is unlikely that CE will focus on another cause area that crosses into existential risk prevention in the near future. I think we will want to see how these charities fare first.

Prioritizing the top health security interventions

Prioritization matters

There are many ways to do good in the world. Some of those paths to impact are likely to be much better than others. Ben Todd explores this in this excellent post. He concludes that “the best of all interventions in an area are about 10 times more effective than the mean, and perhaps as much as 100 times” and “using data to identify the best interventions can perhaps boost your impact in the area by 3–10 times compared to picking randomly”.

CE expects successful charities to run for many years. We believe that putting in the time upfront to choose good ideas can significantly magnify their impact, perhaps by an order of magnitude. When CE considers a new cause area, we typically list 200-400 ideas and carry out a six-month research process to pick the very best idea we can from that list.

To date, a number of lists have been made of high-impact biosecurity interventions (for example, here and here), but it is very unclear how these have been prioritized. Across the nonprofit sector, thousands of people work on health security and thousands more on preventing catastrophic risks more generally. Yet as far as I know (please correct me if I am wrong), the research outlined in this post represents the first time anyone has systematically carried out and published an exercise prioritizing the most important interventions and technologies for preventing health security events, biorisks, or any existential risk.

I, therefore, want to showcase it a bit as an example of what could be done, and discuss some of the lessons learned.

How we prioritized in a data-poor environment

The typical CE research round begins by compiling a list of hundreds of ideas. We then gradually narrow the list down through a series of research rounds, each of which examine the best options from the previous round in greater depth. 

Health security was a hard area to research. On the whole, it is a high-uncertainty and data-poor cause area – much more so than global health, and somewhat more so than animal welfare (the two areas that CE researches the most).

Here are three things we did to mitigate this:

1. As with all our research, we looked to address uncertainty by using a robust/cluster thinking approach. This means we looked for places where different sources of evidence all point in the same direction. Although concrete data on the impact of different interventions was poor, there were many sources of evidence that we could, and did, pull upon. These included: some global health data, data about past pandemics, CE’s institutional knowledge about what makes a good charity, collated expert opinions, case studies of similar projects, theoretical reasoning, cost-effectiveness analyses, forecasts, and so on.
2. We worked a bit more closely with more experts. We sought the opinion of experts, both from inside and outside of the effective altruism community, and from different sectors (government, multilateral organizations, NGO, and research). We asked for inputs at more stages of our research than in past research rounds, and spoke to more experts overall. We are extremely grateful for the views and input of everyone who helped with our research.
3. We did more theoretical work. CE’s standard research process is very bottom-up, listing lots of ideas and narrowing on the best ones. We did a bit more top-down work to supplement this, where we broadly mapped out the health security space (or various sub-parts of this space) and different theories of change in the space, and looked for the gaps and ideas that cut across multiple theories of change. Some examples of this work can be found on the slides at: 2022 Health Security  _ Addressing pandemics, theories of change.

I think prioritizing in a data-poor environment is achievable

In the aforementioned post on prioritization, Ben Todd suggests that the benefits of prioritization likely require being “in an at least somewhat data-rich area”. So, did we find this to be the case?

I came away from this research with a sense that we managed to prioritize fairly successfully, despite the data scarcity. Overall, I am a little more confident in our prioritization of different intervention types (where CE has institutional expertise) than our prioritization of key pandemic-addressing technologies. Of course, my sense of success is not strong evidence of the value of prioritization, and certainly not enough for me to be able to put a precise estimate on the expected increase in value due to having carried out a prioritization exercise. However, it does make me optimistic about the value of other prioritization exercises. 

The results of our research

Our long list

You can see our long lists at: 2022 Bio _ Health Security _ Summary of ideas and research progress tracker

There is a list of about 150 intervention ideas on the ‘IDEA LIST’ tab. There is also a list of about 30 ideas for new technologies (broken up further into direct research interventions and advocacy interventions for various technologies) on the ‘TECH IDEAS’ tab.

Each idea was evaluated with a score, indicating the likelihood of recommendation. It is important to exercise caution when interpreting these scores, as there may be some artifacts that affect their accuracy. For instance, some ideas may have lower scores than warranted because we already know someone is working on starting that idea.

A number of columns of notes and details have been redacted from this spreadsheet, or were carried out on other documents. If you need access to more of our research, do get in touch.

Some ideas on this list were contributed by people and organizations outside of Charity Entrepreneurship. We have not credited every idea, but are highly grateful to those who added ideas and supported our research at all stages.

Our shortlist and the final ideas

You can see our tabulated summary of our top six ideas here:

2022 Health Security _ Decision-making spreadsheet _ Version to share / make public 

The final ideas we looked into and recommended for launching:

1. Policy advocacy for subscription models to drive the production of new antimicrobials
2. Advocating for academic guidelines to limit dual-use research of concern

An idea we were very excited about, but were not sure that CE is best positioned to get it off the ground, was:

3. R&D for high-priority biotech: PoC tests (and PPE)

Other ideas that came high up on our prioritisation exercise and for which we wrote deep dive reports but ultimately did not recommend for launching were: 

4. Advocacy for investment for high-priority biosecurity technologies

5. Policy advocacy to reduce deforestation to prevent zoonotic pandemics

6. Modeling the drivers of antimicrobial resistance

Additionally, there were areas of research which emerged from this list that straddled the border between health system strengthening for health security purposes and general global health interventions. The one that lead to a full research report was:

7. Water chlorination in LMICs

We have already written about our recommended ideas here: CE: Announcing our 2023 Charity Ideas. Apply now! - EA Forum.

We will be publishing more posts, including information about our other ideas, in the coming weeks.

Missing the ‘sweet spot’

When choosing charity ideas, we want to reduce the number of failure modes. This means that although we may pick a radical area for a charity to work in (e.g., shrimp welfare), or a radical approach to driving change (e.g., mass media), we are unlikely to pick both. 

As discussed above, we aimed for interventions that hit a ‘sweet spot’ between global catastrophic risk prevention and more indisputable health impacts.

Ultimately, we ended up picking top charity ideas that fell a little outside this sweet spot in both directions. One of our top ideas was justified more by its effect on preventing globally catastrophic biorisks (radical cause area), although hopefully in a way that ongoing impact measurement and feedback is possible (less radical approach) and the other idea is justified more by the effect it has on near-term health outcomes (less radical area), although with a fairly long theory of change requiring global policy changes (radical approach). 

Thanks

We wish to express our deep gratitude to the research team for their remarkable effort and dedication in conducting this research. In particular, we would like to thank Akhil Bansal, Emma Buckland, Lin Bowker-Lonnecker, Melanie Basnak and Moritz Von Knebel – all of whom helped research this cause area.

We are also grateful to the many others who have contributed to this research in various ways, including the experts who provided their expertise and guidance in specific areas. Each of these individuals is recognized and thanked in the report specifically related to their contributions. This research would not have been possible without the generous support of all of these individuals, and we are deeply appreciative of their contributions.