Critical Review of Modelling Strategies against Pandemics

By Jérémy Andréoletti🔸 @ 2024-10-07T11:18 (+26)

Context: This project was carried out as part of the Biosecurity Fundamentals “Pandemics Course”, by BlueDot Impact, and won the runner-up prize in the 'Novel research' category. I warmly thank the organisers, facilitators and other cohort participants and encourage anyone interested in biosecurity to attend this course.

Epistemic status: As a PhD student in macroevolutionary modelling, I am confident of my expertise in the modelling aspects, and epidemiological modelling specifically to a lesser extent. I’m more uncertain about the policy implications, in particular the extent to which these models are or would actually be used by policymakers. The various sources presented in this post aim to fill this gap, and I look forward to feedback from people with more hands-on experience in the comments.

How to read this post: 1) Quickly browse the TL;DR; 2.a) if you just want some more detail, read the Executive Summary, 2.b) if you want to delve into specific modelling interventions, read the Introduction, Inclusion and evaluation criteria, then proceed to the relevant sections.

Thanks: I am very grateful to @JoshuaBlake, @Rory Greig, @Alix Pham, Emil @Iftekhar, @Al-Hussein Saqr and @Vincent Niger AE for their detailed comments on the manuscript and excellent suggestions, as well as to Will, Bea, Aminu, Charbel-Raphaël, Ava, Élisa, William, Moeen and Nadia for their helpful feedback at various stages of the project!

TL;DR

• Overview: A Theory of Change (ToC) is a roadmap that outlines how a specific intervention is expected to lead to desired outcomes, mapping out the intermediary causal relationships. This review critically examines the theories of change of various interventions that use modelling tools — mathematical or computational frameworks for understanding disease dynamics — to prevent, contain or mitigate catastrophic pandemic scenarios.
• Methodology: interventions and ToCs were collected and synthesised from various biosecurity sources, then assessed based on decision relevance to catastrophic pandemics (including robustness, relevance, pathogen-agnosticism and empirical evidence), tractability, neglectedness, and dual-use risks.
• Most Promising Modelling Interventions:
  • Assessing pathogen properties: Crucial for early pandemic response by rapidly estimating key properties like transmissibility and virulence.
  • Estimating the pandemic potential: Essential for determining a pathogen's potential to cause a pandemic and credibly raising the alarm when necessary.
• Promising Modelling Interventions:
  • Understanding factors of pathogen (re-)emergence: Provides insights for prevention but may have limited relevance for worst-case scenarios; some dual-use risks exist.
  • Simulating realistic preparedness scenarios: Useful for stress-testing response strategies to diverse pandemic scenarios, including catastrophic ones.
  • Detecting (stealth) outbreaks early: Important for triggering mitigation measures, especially in stealth pandemics, though likely too delayed for containment
  • Forecasting joint health and economic impacts of policies: Aids decision-making by balancing pandemic response strategies in real-time, less relevant in extreme scenarios.
• Less Promising Modelling Intervention:
  • Assessing countermeasure effectiveness in real-time: Useful for long-term adjustment of responses, less critical in the early stages of catastrophic pandemics.
• Key recommendations:
  • Prioritise preparedness and early response models
  • Focus on catastrophic scenarios: Ensure models are robust enough to handle scenarios like "wildfire" (very rapid disruption) or "stealth" (initially asymptomatic spread) pandemics.
  • Invest in model development between pandemics: Prepare analytical pipelines ready for immediate use in various scenarios.
  • Enhance modeller-policymaker collaboration: Establish dedicated teams to bridge the gap between modelling efforts and policy application.
• Key challenges, open questions and areas to investigate further are detailed in the conclusion.

Executive summary

Context

The COVID-19 pandemic highlighted the devastating impacts that pandemics can have on global health, economies, and societal stability. It also underscored the need for better preparedness to handle future pandemics, especially those that may lead to even more catastrophic scenarios. To this end, we must leverage all the tools that can help predict, prevent, and mitigate pandemics more effectively. In this report, we will focus on modelling tools — mathematical or computational frameworks for understanding disease dynamics — in that they represent a key ingredient for data-driven decision-making before and during pandemics.

A Theory of Change (ToC) is a roadmap that outlines how a specific intervention is expected to lead to desired outcomes, mapping out the intermediary causal relationships. This report provides a critical review of various modelling interventions and their theories of change targeted at pandemic preparedness, prevention, and response. The objective is to identify which types of models are most effective in different stages of a pandemic and how these models can be integrated into policy decisions to enhance global health security. Major challenges include the scarcity of data early in a pandemic and the currently imperfect connection between model development and public health decision-making.

Interventions and Theories of Change

Seven relevant interventions and their associated ToCs are examined — the selection process is detailed in the full report. Each intervention starts at one of three stages of pandemic response : preparedness, early response or late response (see diagram below). Details on each intervention are given in the next section.

Evaluation

This report compiles a range of modelling approaches, and proposes a qualitative evaluation through a structured assessment framework. All interventions were analysed based on the following criteria.

Each assessment is supported by a range of sources, such as reports from international or national health organisations, modelling papers or expert blog posts. References and excerpts can be found in the respective section of each intervention.

Then the modelling interventions are classified into one of 3 categories based on their final score as follows:

• Most Promising Interventions ⭐⭐⭐ (at least two thirds of the maximum possible score)
  • [4] Assessing pathogen properties: Crucial for early pandemic response by rapidly estimating key characteristics like transmissibility and virulence properties.
  • [5] Estimating the pandemic potential: Essential for determining a pathogen's potential to cause a pandemic and credibly raising the alarm when necessary.
• Promising Interventions ⭐⭐ (at least half of the maximum possible score)
  • [1] Understanding factors of pathogen (re-)emergence: Provides insights for prevention but may have limited relevance for worst-case scenarios; some dual-use risks exist.
  • [2] Simulating realistic preparedness scenarios: Useful for stress-testing response strategies to diverse, including catastrophic, pandemic scenarios.
  • [3] Detecting (stealth) outbreaks early: Important for triggering mitigation measures, especially in stealth pandemics, though likely too delayed for containment
  • [6] Forecasting joint health and economic impacts of policies: Aids decision-making by balancing pandemic response strategies in real-time, less relevant in extreme scenarios.
• Less Promising Intervention ⭐ (at least a third of the maximum possible score)
  • [7] Assessing countermeasure effectiveness in real time: Useful for long-term adjustment of responses, less critical in the early stages of catastrophic pandemics.

Key Recommendations

I have identified two major challenges and formulated recommendations to mitigate these problems, based on the evaluation and on interventions suggested in the examined sources:

Challenge - Modelling Under Uncertainty: A recurring theme in this review is the difficulty of making informed decisions in the early stages of a pandemic, where data is scarce, and uncertainty is high.

• Recommendation 1 - Focus on Preparedness and Early Response Models: Prioritise developing models that may either prevent emergence or enable rapid early response to pandemics. These models should aim to integrate all available data sources to improve early detection and swift assessment of emerging threats.
• Recommendation 2 - Focus on Catastrophic Scenarios: Future research and modelling efforts should prioritise scenarios with the highest stakes - in particular “wildfire” pandemics and “stealth” pandemics - ensuring that our preparedness, prevention and response strategies are robust enough to handle even the most extreme pandemic threats, which are currently neglected.
• Recommendation 3 - Invest in Model Development Between Pandemics: Continuously refining models during inter-pandemic periods is much more effective and robust than having to adjust ill-adapted models in an emergency, as the COVID-19 experience has taught us. Analytical pipelines pre-configured for various scenarios and pathogen families should be developed to ensure immediate, actionable insights in real-time when new threats arise.

Challenge - Integration into Policy: One of the most significant challenges is the need for closer integration between modelling efforts and policymaking. This is the first Initiative recommended in the report Strengthening pandemic preparedness and response through integrated modelling by the WHO, OECD and World Bank. Real-time collaboration between modellers and decision-makers, as demonstrated by initiatives like the MIDAS network, can ensure that the right questions are asked and that the models used are relevant and actionable.

• Recommendation 4 - Enhance Collaboration Between Modellers and Decision-Makers: Establishing dedicated expert teams within governments that work closely with academic institutions and modelling groups facilitates dialogue and better integration of modelling into pandemic response strategies. In this way, modellers gain a deeper understanding of the constraints faced by decision-makers, and decision-makers develop a clearer understanding of the models' capabilities and limitations.

Conclusion

Pandemics are inevitable, but their catastrophic impacts are not. By identifying and leveraging effective interventions, including modelling tools, we can build a more resilient global health infrastructure that is better prepared to prevent, detect, and respond to future pandemics.

Full report — Introduction

The threat of catastrophic pandemics, particularly those that could emerge unnoticed or spread rapidly, has been a central concern in EA discussions. Despite the potential scale of this risk, it's not enough to assume that all biosecurity interventions are inherently valuable or cost-effective. In fact, many proposed strategies may not justify the investment required, especially when compared to more established alternatives in other cause areas — a concern already raised by Joshua Blake on his blog.

Why this post?

I am personally considering a career shift towards epidemiological modelling, with the goal of contributing to the prevention or mitigation of catastrophic pandemics. However, I feel the need for more rigorous justification before dedicating a significant portion of my time and energy to this field.

Only, it's hard to assess the effectiveness of any intervention if we don't first know what it's supposed to accomplish, and how! This post aims to do just that by reviewing key modelling interventions and their theories of change (ToCs) targeted at preventing catastrophic pandemics. I'll provide a qualitative evaluation of 7 modelling interventions and their ToCs, drawing from various sources, case studies, and my own estimates. This review is intended as a first step toward more comprehensive evaluations, ensuring that investments in modelling for pandemic preparedness and response are truly justified and impactful.

Modelling tools for preventing and responding to pandemics

When it comes to preventing and responding to pandemics, various modelling tools have been employed, each with its own strengths and applications. Here are some definitions that may be useful for the rest of the post.

In this project, I primarily focus on epidemiological models, excluding other computational and AI tools that have been developed to model biological systems, and are used to design vaccines and therapeutics (see the 2022 CEPI report Delivering Pandemic Vaccines in 100 Days).

Inclusion and evaluation criteria

To ensure that the interventions and theories of change included in this assessment are relevant, they were selected based on criteria designed to filter out approaches that may lack a clear connection to public health outcomes or fail to focus on large-scale catastrophic pandemics.

Properties of included interventions and theories of change

1. Clear connection from a modelling intervention to a public health impact:
   Each ToC must link the proposed modelling intervention to tangible public health outcomes, leaving no gap in the expected path to impact.
2. Relevance for large-scale catastrophic pandemics:
   The ToCs should specifically target scenarios that involve widespread, high-impact pandemics.
3. At least one source (implicitly or explicitly) promoting it:
   Interventions and ToCs must be backed by at least one credible source within the broader biosecurity or epidemiological community that advocates for their use in the context of pandemic prevention, preparedness or response.

Evaluation factors

To my knowledge, the report by Founders Pledge summarised in Box 1 provides the best qualitative framework to compare biosecurity interventions, although they haven’t published a comprehensive and systematic assessment of ToCs in the field.

In order to make my qualitative evaluation more transparent, I used the following four criteria:

• Decision-relevance for catastrophic pandemics:
  • Robustness of the ToC: A strong ToC should demonstrate clear pathways for influencing pandemic outcomes.
  • Relevance for worst-case scenarios: The ToC should be applicable to scenarios where the stakes are highest (Gopal et al. | Geneva Center for Security Policy | 2023 | Securing Civilisation Against Catastrophic Pandemics):
    • “Wildfire” pandemics = “highly lethal and transmissible enough to infect most essential workers [which] leads to the breakdown of essential services”
    • “Stealth” pandemics = “a rapidly spreading virus with a long incubation period analogous to HIV infects most of humankind”
    • I won’t focus on existential biorisks, pending more robust arguments for their plausibility (David Thorstad | Reflective altruism | "Exaggerating the risks" series - Biorisk Archives)
  • Pathogen-agnosticism: The intervention and ToC should ideally be applicable across different types of pathogens, rather than being tailored to a specific one.
  • Empirical evidence from real-world applications: ToCs backed by real-world examples or successful applications in past pandemics carry more weight.
• Neglectedness:
  • Personal estimate of the attention the intervention has received.
  • Number of Google Scholar results for associated keywords. See the results here.
• Tractability:
  • Personal estimate of the counterfactual progress per additional researcher on the topic, including an estimate of whether the field is saturated.
• Dual-use risk:
  • Personal estimate of the misuse potential: Some modelling tools and results might have risks if they are repurposed for harmful uses.

 These criteria were employed to build a weighted factor model (i.e. a weighted sum over individual scores for each criterion) in this evaluation sheet: Evaluation - ToCs for modelling in epidemiology.

NB: I am more confident in my evaluations of the decision-relevance for the catastrophic scenarios, which are each supported by several sources, than in my evaluations of neglectedness, tractability and the dual-use risk, which are less well-grounded. This is why I give more weight in the first criterion in the factored model. If you have relevant expertise and disagree with any of the evaluations, please share it in the comments.

At the end, each intervention is classified into one of 3 categories based on its final score as follows:

•       = less than a third of the maximum possible score
• ⭐ = at least a third of the maximum possible score
• ⭐⭐ = at least half of the maximum possible score
• ⭐⭐⭐ = at least two thirds of the maximum possible score

The ⚠️ sign indicates a potential substantial dual-use risk.

This framework sets a structured approach for evaluating interventions, selecting the most relevant and safe ones for further exploration and investment.

Modelling interventions, by chronological order

The interventions were collected through a literature review of national and international reports, EA forum posts and some scientific articles, on the topic of modelling against pandemics, with a preference for documents focused on prioritisation, evidence of connections to decision-making or prevention of catastrophic scenarios. Theories of change are sometimes explicitly extracted from one of the sources (referenced in the corresponding section), and otherwise proposed de novo to concisely synthesise the underlying rationale.

Overview

The diagram outlines three potential health outcomes that can be achieved through effective modelling interventions. The first — and most ideal — outcome is the prevention of a potential emerging outbreak, when proactive preparedness measures avert the onset of an epidemic before it even emerges. The second outcome is the suppression of a local outbreak, achieved through the early detection of outbreaks and the rapid implementation of containment strategies, preventing the spread of the virus beyond a localised area. Finally, if the epidemic has failed to be prevented or contained, the last positive outcome is the mitigation of the overall impact of a pandemic, where a various efforts — encompassing prevention, early response, optimised interventions, and adaptive policies — informed by modelling, could decrease the pandemic's overall severity.

We will investigate each of these in more detail.

Preparedness Interventions

[1] Understanding factors of pathogen (re-)emergence ⭐⭐⚠️

Description: This ToC outlines a strategic pathway that emphasises preventive interventions. It begins with understanding the factors contributing to pathogen emergence, which informs public and private investment in preventive measures aimed at reducing the likelihood of an outbreak. If a pathogen does emerge, some preventive measures (e.g. genetic surveillance, broad-spectrum preventive vaccines, indoor air hygiene) may make it more likely to have early containment and mitigation measures ready to be deployed.

Examples of such models:

• Models integrating climate change data to predict zoonotic infection emergence.
  • Rupasinghe et al. | Acta Tropica | 2022 | Climate change and zoonoses: A review of the current status, knowledge gaps, and future trends - ScienceDirect
• Models identifying regions at high risk of pathogen spillover through the combination of geospatial analysis of pathogen spread with socio-economic data.
  • WHO Health Emergencies Programme | Report | 2024 | Research prioritization for pandemic and epidemic intelligence: technical brief | Their 7th research priority (out of 23) is “Explore analytical techniques and modelling methodologies to enhance the understanding of pathogen emergence and re-emergence”

  • “The (re-)emergence of infectious disease pathogens is influenced by a combination of biological factors and a complex set of climate variations, human behaviour, socio-economic conditions, geographical distribution, and population mobility. [...] Advanced analytical techniques and modelling methodologies that can handle the diversity and scale of these data sets and incorporate interdisciplinary approaches such as geospatial analysis, help identify patterns and correlations within the data. These models not only help to predict potential outbreaks, but also provide important insights for preventive measures.”

  • Muylaert et al. | Nature Communications | 2023 | Using drivers and transmission pathways to identify SARS-like coronavirus spillover risk hotspots
• Models evaluating the risk of accidental pathogen release from high-containment (BSL-3/BSL-4) labs.
  • R. Tieman, P. Oliboni, S. Campos | Preprint | 2024 | Assessing Global Pandemic Risks from Emerging Infectious Diseases and High Containment Laboratory Leaks: A Country Level Spatial Network SIR Model Analysis
• Models identifying priority pathogen families that are most likely to pose emerging threats based on taxonomic and ecological factors.
  • WHO R&D Blue Print team | Report | 2024 | Pathogens prioritization: a scientific framework for epidemic and pandemic research preparedness | This report argues that identifying pathogen families helps both prevention and response to new threats:

  • “By focusing on entire pathogen families and Priority and Prototype pathogens, the strategy aims to create generalizable knowledge and tools that can be rapidly adapted to emerging threats. [...] Implementing these key research actions will significantly enhance the world's ability to detect, prevent, and respond to potential pandemic threats.”

    Priority Families contain at least one Priority Pathogen, defined based on the following criterion: “Existing knowledge of pattern of transmission, virulence, and access to MCMs [Medical Counter-Measures] suggests a Pathogen can cause a PHEIC (or a pandemic)”.

Conclusion: Models of pathogen (re-)emergence are valuable for guiding preventive measures against natural pandemics, with high pathogen-agnosticism and tractability. However, their relevance for worst-case scenarios and effectiveness in practice remains to be established, plus there is some dual-use risk. I would tentatively recommend focusing on models of accidental lab release.

[2] Simulating realistic preparedness scenarios ⭐⭐⚠️

Description: This ToC focuses on the use of (new or existing) simulation models for pandemic preparedness. These models can vary key epidemic and response parameters and simulate a range of realistic pandemic scenarios, including catastrophic ones, with the goal to inform the design of pandemic preparedness and response strategies. Notably, simulations give the possibility for decision-makers to engage in wargaming exercises, stress-testing their strategies against diverse threats. This approach aims to help governments be better prepared to deploy early containment measures, optimised response policies, and ultimately reduce the overall impact of a pandemic. On a broader level, simulation models can help as well to assess which other modelling or non-modelling interventions should be prioritised, and they are also highly relevant to other interventions (see n°5 and n°6).

Examples of such models:

• Simulation models designed to enhance realism by integrating the various processes at play.
  • Peiyu Xu et al. | Preprint | 2024 | e3SIM: A novel epidemiological-ecological-evolutionary simulation framework for genomic epidemiology | An example of a recent simulation model aiming at coupling diverse processes to capture the interplay of pathogen ecology and evolution with epidemiological spread. The authors also provide a graphical interface designed to make the tool accessible for diverse users. Such increasingly realistic simulation models can be used for various applications:

  • “In genomic epidemiology, [simulations] provide a controlled platform to understand the complex relationships between disease transmission, pathogen evolution, and environmental factors, as well as for evaluating intervention strategies and predicting future outbreaks. Simulations also facilitate the development and testing of hypotheses, algorithms, and models, particularly when real-world data with known ground truth are unavailable, as is often the case in epidemiological, ecological, and evolutionary studies. High-quality synthetic datasets are especially crucial for deep learning models, which are increasingly used in genomic epidemiology, as these models require extensive, accurate data with known ground truth to effectively learn and predict epidemic dynamics.”

• Simulation models of epidemic scenarios for “wargaming” exercises, including catastrophic pandemics.
  • Rory Greig | EA Forum | 2022 | Simulation models could help prepare for the next pandemic | Simulation models enable both to raise the alarm before and at the beginning of a pandemic, and to design and practise response strategies for various scenarios in advance.

  • “[modelling] can help illuminate the distribution of possible outcomes. This can make the true potential impact and cost of such disasters much more visible and salient ahead of time. [...] Models can act as a focal point for coordination and group decision making.”

  • “Simulations can be used for “wargaming” exercises, where decision makers can train and practice by running through simulated pandemic scenarios. [...] Properties of the model can be varied to produce different scenarios, such as infectiousness of the virus, place of geographical origin, the speed of vaccine development and rollout, and the evolution of new variants. We could even simulate scenarios that have never occurred before, such as the deliberate release of a bio-engineered pathogen, or a nightmare scenario where more than one novel pathogen is released at once.”

  • UK Health Security Agency (UKSHA) Corporate report | UKHSA strategic plan 2023 to 2026: executive summary | The UKSHA includes among its strategic priorities to “strengthened UK preparedness for future pandemics by planning and undertaking contingency exercises for the range of health threats, transmission routes and scenarios.”
• Simulation models exploring the comparative effects of response policies to various epidemic scenarios.
  • WHO, OECD, The World Bank | 2024 | Strengthening pandemic preparedness and response through integrated modelling | The 3rd Initiative proposed in this report is the following: Match the policy questions of interest to the appropriate integrated models. They argue that such models can provide pathogen-agnostic insights:

  • “building base-modelling structures during interepidemic times based on key policy questions of interest. [...] Even though pathogens can differ substantially from each other, and one epidemic may not be representative of others, some principles and mechanisms are common across all epidemics”

    They further recommend “building capabilities to determine how to match the appropriate models with the right policy questions” and “encouraging the refinement of integrated models to ensure they can adapt to policy priorities”, warning that:

  • “When the Multi-Model Comparison Collaboration reviewed COVID-19 dynamic epidemiological models, they found that existing models are not designed to answer all COVID-19-related questions decision-makers may have”

    Empirical case study: The G20 Joint Finance and Health Task Force for pandemic prevention, preparedness, and response – In 2021 the task force used integrated modelling to assess pandemic impacts from several respiratory pathogen profiles, informed by past epidemics. They explored trade-offs in health, social, and economic outcomes for different response strategies to future pandemics.

Conclusion: Simulating realistic scenarios is robust and relevant for pandemic preparedness, particularly in generating realistic and catastrophic scenarios. Its high pathogen-agnosticism and tractability make it a valuable tool. However, we should make sure that these models do meet the needs of decision-makers, and probably focus on making a better use of existing – potentially unsophisticated – models rather than prioritising the development of new ones.

Early response Interventions

[3] Detecting (stealth) outbreaks early ⭐⭐

Description: This ToC emphasises the importance of early detection in managing potential pandemics, particularly those caused by stealth pathogens that spread widely before symptoms become apparent. Early detection models, such as those analysing wastewater or social media data, aim to identify signs of an outbreak before it escalates. Once detected, these models enable governments to deploy early containment measures, such as sanitation efforts or localised lockdowns, which can suppress the outbreak at its source. If unsuccessful, these actions can still delay the global spread and/or lead to earlier mitigation measures, leading to a reduced overall impact of the pandemic.

Examples of such models:

• Models detecting early-warning signals of a stealth outbreak from environmental genetic data:

  • Nucleic Acid Observatory (collaboration between SecureBio and MIT’s Sculpting Evolution group): “the NAO’s fundamental mission is to ensure that humanity can reliably detect stealth pandemics as soon as possible”.

    Regarding early detection methods:

  • “the challenge is to identify which [biosurveillance approaches] are the most sensitive, reliable, and cost-effective for pathogen-agnostic early warning. Through epidemiological modeling, data analysis, and experimentation, we are characterizing and comparing the merits of different approaches, with a special focus on municipal and airplane wastewater.”

    Will Bradshaw and Simon Grimm | NAO | 2024 | Comparing sampling strategies for early detection of stealth biothreats | This report makes clear that their ToC for early detection is not to achieve suppression, but to trigger earlier mitigation measures.

  • “[we are] pessimistic that a stealth threat will be detected early enough to achieve suppression, and optimistic that even much later detection (e.g. at 1/1000 or even 1/100 cumulative incidence) can provide major benefits compared to no detection. [...] For example, by enabling widespread deployment of infection-mitigation measures (e.g. PPE or isolation) to protect uninfected individuals, alongside development of countermeasures that might otherwise be delayed by many months or years.”

    In addition, they highlight the need to evaluate early detection methods based on their “capacity to trigger an effective response to prevent or mitigate a severe pandemic”. This means both providing enough information to enable adequate response and provide a “credible signal that can persuade relevant authorities to take such actions”.

  • @Conrad K. | EA Forum | 2021 | Three Reasons Early Detection Interventions Are Not Obviously Cost-Effective — EA Forum | The article gives three reasons why early detection of pandemics might not be worth the cost.
    In summary, for a wildfire pandemic scenario, the post argues that an early detection system would probably not bring much more additional time to respond, and would not currently trigger useful measures anyway.

  • “the efficacy of early detection is heavily dependent on the ability to quickly trigger an epidemiological response”

    “early detection systems may only provide a lead time on the order of days to weeks compared to "naive detection" from symptomatic spread”, especially “for highly transmissible pathogens”

    “the cost-effectiveness of early detection is highly dependent on the feasibility and efficacy of post-detection containment measures”. Robust “pre-established response plans and thresholds” need to be developed first

    For a stealth pandemic scenario, they are more optimistic that “the prima facie case for early detection as an essential tool to mitigate GCBRs is very strong”.

• Models integrating diverse sources of data for biosurveillance systems:
  • WHO Health Emergencies Programme | Report | 2024 | Research prioritization for pandemic and epidemic intelligence: technical brief | Their 2nd research priority (out of 23) is “Explore methods to improve detection, verification, and notification of epidemics and pandemics data through multidisciplinary data integration”. They clearly explain their ToC:

  • “This integration helps detect health threats earlier, assess their potential impact more reliably and initiate faster communication with health authorities and the public. Overall, it can lead to more effective coordination of containment strategies, resource allocation and ultimately a reduction in the spread and severity of infectious disease outbreaks on a global scale.”

    Several of their priority research axes additionally focus on the potential use of AI to assist outbreak detection, and on the importance of careful data collection, sharing and analysis.

  • UK Health Security Agency (UKSHA) | Corporate report | UKHSA strategic plan 2023 to 2026: executive summary | Example of a national health agency supporting the development of early detection systems:

  • “UKHSA undertakes horizon scanning and systematic surveillance to detect emerging hazards and provides analysis of threats for the NHS (such as specialist diagnostics, epidemiology, genomic sequencing, and laboratory testing) as well as guidance and advice. [...] UKHSA’s timely responses to notifications of infectious disease cases and outbreaks minimises onward spread, reducing the number of cases and potential impact on the NHS and keeping wards and emergency departments operating.”

    Note that most of these developments focus on data acquisition and single case monitoring rather than modelling.

  • Empirical case study: Taiwan - Hao-Yuan Cheng, Ding-Ping Liu | Journal of the Formosan Medical Association | 2024 | Early Prompt Response to COVID-19 in Taiwan: Comprehensive surveillance, decisive border control, and information technology support | Taiwan is a good example of an early surveillance system prompting an effective response, though the specific role of models remains to be clarified:

  • “the early response not only successfully blocked the first wave of imported cases, but also slowed down subsequent large local outbreaks. [...] The experience of Taiwan's prompt and comprehensive response at the early stage may contribute to the preparedness for the next disease X outbreak.”

Conclusion: The theory of change of early-detection models appears less robust in several respects, particularly for wildfire pandemics or if the aim is to locally contain the outbreak. The main value seems to reside in the detection of stealth pandemics in order to trigger early mitigation measures. Following one of Conrad Kunadu’s recommendations, it therefore appears essential to “enhance the strength and credibility of early detection signals to improve public compliance and political will for rapid response”.

[4] Assessing pathogen properties ⭐⭐⭐

Description: This ToC focuses on the early assessment of pathogen transmission and virulence properties shortly after a pathogen is detected. Accurate and rapid estimation of these properties is critical for enabling health institutions to perform risk assessments and declare a public health emergency if needed. This, in turn, triggers early containment and mitigation measures by governments, which aim to suppress the outbreak at the local level before it can escalate into a pandemic, or reduce the overall impact of the pandemic. These parameters are also key to calibrate other models used to estimating the pandemic potential (n°5) or forecasting the impacts of policies (n°6).

Examples of such models:

• Models estimating transmissibility and severity

  • WHO R&D Blue Print team | Report | 2024 | Pathogens prioritization: a scientific framework for epidemic and pandemic research preparedness | Figure 2 of this report presents the three key categories of “Evidence elements considered to assess a pathogen’s potential to cause a PHEIC [Public health emergency of international concern] or pandemic”:

    • Transmission: “Reservoir of infection, main mode of transmission, efficiency of transmission. asymptomatic/pre-symptomatic/symptomatic spread, natural protective immunity. geographic distribution, risk of mutation affecting transmissivity, impact of climate change”
    • Virulence: “Case fatality without treatment, severe symptoms or complications, severe sequelae, high-risk populations, risk of mutations that will impact virulence”
    • Medical Counter-Measures: “Availability, effectiveness and accessibility of vaccines, treatments, and diagnostic tools; stages of clinical development or licensure.”

    Modelling tools are essential to infer pathogen parameters in the first two categories.

  • U.S. Center for Disease Control and Prevention (CDC) | Pandemic Severity Assessment Framework (PSAF) and associated paper: Reed et al. | Emerging Infectious Diseases | 2013 | Novel Framework for Assessing Epidemiologic Effects of Influenza Epidemics and Pandemics | The PSAF is a tool for evaluating the potential impact of influenza pandemics through a two-step process: an initial rapid assessment during the early stages of the outbreak, focusing on transmissibility and clinical severity, followed by a more refined assessment as additional data become available. Unfortunately, this framework is apparently rarely used in practice.

  • 

  • Marc Lipsitch | Bio(un)ethical podcast | 2024 | How to Ethically Prevent the Next Pandemic | In this episode, epidemiology professor Marc Lipsitch points out that emergency declarations generally come too late for local containment, because once a pathogen has spread widely in a region, there are too many small local outbreaks. To have any chance of recognising an epidemic early enough, the property that is essential to establish is whether there is confirmation of transmission between humans.
• Several research networks coordinate to develop epidemiological inference models.
  • Epiverse-TRACE - “Founded in 2021, Epiverse-TRACE enables distributed data analysis to power pandemic response. From the start, all the tools we develop are open, collaborative, and intended for real-world impact. Over 100 people contributed to 33 software releases since”. | Some of their tools allow to estimate severity (e.g. cfr: Estimate Disease Severity and Case Ascertainment) or quantify transmission (e.g. serofoi: Estimates the Force-of-Infection of a Given Pathogen from Population Based Seroprevalence Studies, superspreading: Estimate Individual-Level Variation in Transmission).

  • 

    Data.org | 2023 | Epiverse-TRACE Summit Report | Epiverse-TRACE is part of the wider data.org’s Epiverse initiative, which promotes not only the development of these models, but also the connection with users, in particular decision makers and multilateral organisations (they partnered with the WHO’s Hub for Pandemic and Epidemic Intelligence).

  • R Epidemics Consortium | “The R Epidemics Consortium (RECON) is an international not-for-profit, non-governmental organisation gathering experts in data science, modelling methodology, public health, and software development to create the next generation of analytics tools for informing the response to disease outbreaks, health emergencies and humanitarian crises, using the R software and other free, open-source resources” | This consortium also provides tools for estimating pathogen properties (e.g. EpiEstim - Quantifying transmissibility throughout an epidemic from incidence time series, earlyR - Estimation of infectiousness in the early stage of an outbreak, Outbreaker2 - Modular framework for outbreak reconstruction, branchr - R Estimation from Cluster Sizes)

Conclusion: Assessing pathogen properties is robust and essential in an early pandemic response to characterise new pathogens in order to inform containment (with a low chance of success) and mitigation strategies. Although these models are pathogen-agnostic and are supported by empirical evidence, they are not widely neglected. This suggests a need to focus on specific research projects that hold more promise, such as tailoring inference models to catastrophic pandemic scenarios, or preparing pipelines for various pathogen families to provide credible transmission and virulence estimates much earlier.

[5] Estimating the pandemic potential ⭐⭐⭐

Description: This ToC centres on the rapid estimation of a pathogen's pandemic potential, which is critical for raising the alarm and guiding early response strategies. The key goal is to quickly assess how likely a newly emerged pathogen is to cause widespread, severe outbreaks, and whether it could escalate into a pandemic. Health institutions can then perform risk assessments under uncertainty that inform the decision to declare a public health emergency. This early estimation helps governments and organisations deploy containment and mitigation measures in a timely manner, ideally suppressing the outbreak locally and reducing its overall impact. For instance, the 100 Days Mission aims to fast-track global pandemic response by preparing the deployment of effective diagnostics, therapeutics and vaccines within 100 days of a WHO-declared Public Health Emergency of International Concern (PHEIC).

Examples of such models:

• Simulation models of pandemic outcomes (e.g. potential death toll) for health emergency assessment.

  • Rory Greig | EA Forum | 2022 | Simulation models could help prepare for the next pandemic | One of the uses of simulation models defended in this post is to forecast potential pandemic scenarios, in particular the death toll, given that: “In my experience output from a simulation can be very persuasive for nontechnical decision makers, if it is presented in the right way.”

    As an empirical case study,

  • “ The famous (or infamous) Imperial College model from Neil Ferguson et. al. was credited as one of the factors causing the UK government to change course and institute a lockdown in March 2020, by predicting 500,000 deaths in the UK if no action was taken”

    [More detail available in Nature’s Special report: The simulations driving the world’s response to COVID-19.]

  • UK’s National Health Service (NHS) | 2022 | NHS Emergency Preparedness Resilience and Response Framework | This report gives us useful information regarding the criteria that are used to raise the level of national emergency at the beginning of an outbreak.

    A Major Incident is defined by “The Cabinet Office, and the Joint Emergency Services Interoperability Principles (JESIP) [...] as an event or situation with a range of serious consequences that require special arrangements to be implemented by one or more emergency responder agency”.
    In an epidemiological context, this includes scenarios of “Rapid onset – develops quickly, and usually with immediate effects, thereby limiting the time available to consider response options (in contrast to rising tide) e.g. a serious transport accident, explosion or series of smaller incidents.” and “Rising tide – a developing infectious disease epidemic or a capacity/staffing crisis or industrial action”.

    Escalation from level 3 (Regional Response) to level 4 (National Response) can be decided if hospital capacity is critically overwhelmed or at risk, essential NHS functions are disrupted, a major incident affects multiple regions, or military aid is needed. Then, the Department of Health and Social Care would be informed if national coordination is required, central government emergency powers are invoked, or a major incident has national/international implications.

• Models estimating the pandemic potential from transmission and virulence parameters, such as asymptomatic transmission:
  • Prof. Joshua S. Weitz | Hopkins Press | 2024 | Asymptomatic | Asymptomatic and presymptomatic transmission (i.e. when individuals can become infectious while showing only mild or no symptoms) is increasingly being singled out as a major factor in pandemic potential. For example, this partly explains why the SARS epidemic was contained, but the SARS-Cov-2 epidemic was not.
    Some of Joshua S. Weitz’s modelling work on this topic: The time scale of asymptomatic transmission affects estimates of epidemic potential in the COVID-19 outbreak Epidemics (2020), Intermediate levels of asymptomatic transmission can lead to the highest epidemic fatalities PNAS Nexus (2023).
  • Conrad Kunadu | EA Forum | 2021 | Three Reasons Early Detection Interventions Are Not Obviously Cost-Effective — EA Forum | Conrad Kunadu points out that at the beginning of the Covid pandemic, the limiting factor for declaring an international health emergency was not late detection, but the lack of confidence in the pandemic potential of the outbreak.

  • “Much of [the 3-weeks WHO emergency declaration] delay reflects the usual epidemiological complexities of predicting whether an outbreak will lead to an epidemic or pandemic; the usual complexities of motivating political institutions to act quickly, and time spent on further testing to ensure an appropriate level of confidence.”

  • Joshua Blake | Deconfusion device (personal blog) | 2023 | Cost-effective pandemic preparedness | Among several recommendations in this post, Joshua Blake also argues that reducing our uncertainty on the pandemic potential as quickly as possible is key to adequate policy response.

  • “Another avenue to pursue is to improve our ability to calibrate responses early in a pandemic. The large data and model uncertainty means that true estimates of our uncertainty are extremely large in an outbreak’s early phase. Yet, the quicker we can characterise the likely severity of an outbreak the more quickly we can respond appropriately. Numerous academic groups are exploring ways to enhance our response. Incremental progress across areas is likely our best hope.”

Conclusion: Estimating pandemic potential seems highly relevant and robust for raising the alarm and managing early responses to potential pandemics. Models play a critical role in quickly showcasing the potential (catastrophic) impacts of a new pathogen, thereby convincing decision-makers to act quickly. More work could be done to push the limits of modelling under strong uncertainty early in a pandemic when access to data is limiting, and integrate these models to decision-making processes.

Late response Interventions

[6] Forecasting joint health and economic impacts of policies ⭐⭐

Description: This ToC focuses on real-time and medium-term forecasting to guide the deployment of optimised response policies during pandemics. By using integrated models, policymakers can evaluate the trade-offs inherent in different intervention strategies, such as lockdowns, vaccination campaigns, or economic support measures, to minimise the overall impact of the pandemic on both public health and the economy. This intervention is similar to intervention 2 (Simulating realistic preparedness scenarios), but focuses on improving the policy response during a pandemic, whereas the latter focuses on preparedness.

Examples of such models:

• Integrated health and economic models.
  • @Rémi T & @simeon_c | EA Forum | 2021 | How can economists best contribute to pandemic prevention and preparedness?

  • “Many economists have recently developed macroeconomic models that integrate covid transmission dynamics (McAdams, 2021). They used these models to make recommendations regarding lockdowns, contact tracing and vaccine policies. This type of models could help make better decisions during future pandemics (Berger et al., 2021) - especially if they were ready to be used before the next outbreak.”

  • WHO, OECD, The World Bank | 2024| Strengthening pandemic preparedness and response through integrated modelling | This very comprehensive report details the various uses of integrated health and economic models, in particular during a pandemic: “to refine policy questions, predict health and economic outcomes of alternative response options, and to evaluate preferred strategies”.

    More precisely, they are of great value for:

    • Capturing key basic factors that drive differential impacts across populations,
    • Formalizing ways to weigh the benefits and costs of various policies
    • Designing and optimizing epidemic or pandemic control policies
    • Evaluating crisis-related economic policies (such as income support, subsidies, cash and in-kind transfers)
    • Understanding which policy options may be more robust in the face of uncertainty

    The report further highlights the importance of gathering data to populate, calibrate and validate these integrated models during the pandemic.

    • Empirical case study: Approaches to calibrate the Tekanelo integrated model in South Africa – South African researchers combined epidemiological and macroeconomic models to estimate COVID-19 and vaccination impacts but faced challenges due to a lack of local data on contact patterns and public health measures. They used international age contact matrices and assumptions based on earlier productivity changes to create realistic scenarios, and modelled the impact of future vaccinations amid emerging variants.

    They present application examples of these integrated models:

    • Empirical case study: Modelling the COVID-19 pandemic at the Bank of Italy - The Bank of Italy collaborated with public health experts to model COVID-19's impact, overcoming interdisciplinary challenges to create a flexible epidemiological model. This model was improved over time, incorporating regional data and adaptive restrictions and enhanced economic forecasting in order to better compare alternative policy mechanisms and counterfactual scenarios.
    • Empirical case study: Integrated modelling to inform pandemic control strategies in Norway – The Norwegian government formed an expert committee combining economists and epidemiologists to create an integrated model for evaluating COVID-19 policies. This model, refined over four assessments, allowed cost-benefit analyses of public health measures like quarantines and vaccinations, estimating impacts on GDP and quality-adjusted life years.

    However, they conclude by pointing out the current lack of incorporation of these integrated models into policy-making in most countries.

  • Empirical case study: The UK Royal Society DELVE Initiative (Data Evaluation and Learning for Viral Epidemics) - 2020 report on the Economic Aspects of the COVID-19 Crisis in the UK: “We suggest methodologies including how economic models can incorporate insights from epidemiology; we review evidence about pandemic economic impacts; we suggest tools and methods that will be useful in monitoring the economy as it attempts to recover; and we suggest data required for conducting economic analysis.”
• Real-time forecasting models.
  • UK Health Security Agency (UKSHA) Corporate report | UKHSA strategic plan 2023 to 2026: executive summary

  • “UKHSA detects, tracks, analyses and interprets data, and develops forecasts on threats to health. Data underpins our ability to make policy and operational decisions which are grounded in evidence. [...] The insight from our data is recognised internationally and is used to support global evidence-based action to tackle health threats.”

  • The US MIDAS Network (Models of Infectious Disease Agent Study)

  • “Initially formed in 2003 the MIDAS network has allowed infectious disease modellers throughout the United States to connect with each other as well as with decision-makers at the federal level. [...] The MIDAS network has assisted with the response to nearly every major infectious threat to the United States since its formation, including the threats of avian influenza, the spread of methicillin-resistant Staphylococcus aureus, the Zika outbreak and the COVID-19 pandemic”

    The MIDAS Catalogue presents the projects and tools developed by the network, including related to pandemic forecasting:

    • Aggregating statistical models and human judgment - “This project aims to provide public health officials metaforecasts - a combination of probabilistic predictions from computational models, subject matter experts, and trained forecasters - of the COVID-19 outbreak, and an expert consensus of the most effective interventions to prevent the spread of SARS-CoV-2 and impact of COVID-19. It also contains software used to aggregate predictions.”
    • COVID Scenario Pipeline - “A flexible scenario modeling pipeline that tailors models for decision makers seeking to compare projections of epidemic trajectories and healthcare impacts from multiple intervention scenarios in different locations. It projects epidemic trajectories and healthcare impacts under different suites of interventions in order to aid in scenario planning. The model is generic enough to be applied to different spatial scales given shapefiles, population data, and COVID-19 confirmed case data.”

    • COVID-19 Scenario Modeling Hub | They created a website which provides regular real-time forecasts and longer-term projections of the COVID-19 epidemiological situation, conditional on various policy interventions.
       

      

      For example, in the latest Scenario Modeling Hub Round at the time of writing (n°18), they forecast that “Vaccination of high-risk individuals is projected to prevent over 76,000 hospitalizations and 7,000 deaths”.

  • WHO Health Emergencies Programme | Report | 2024 | Research prioritization for pandemic and epidemic intelligence: technical brief | Their 13th research priority (out of 23) is “Develop methods and frameworks for real-time infectious disease forecasting, ensuring the rapid recognition, validation, and scaling of statistical models for pandemic and epidemic intelligence”. They point out that:

  • “Accurate forecasting models help public health officials and governments make data-driven decisions, regarding resource deployment, implementation of containment measures and planning of healthcare needs. Frameworks for real-time forecasting are designed and refined to quickly adapt to emerging data, recognise patterns indicative of outbreaks, and validate the reliability of these models in various scenarios.”

Conclusion: This Theory of Change is relevant for managing pandemics by forecasting the joint health and economic impacts of various policy options, albeit potentially less for catastrophic scenarios. Continued refinement and integration into real-time decision-making processes would further enhance their effectiveness.

[7] Assessing countermeasure effectiveness in real time ⭐

Description: This ToC focuses on the ongoing assessment of the effectiveness of public health countermeasures during a pandemic. The key aim is to continuously evaluate and adapt these interventions to ensure they remain effective as the pandemic evolves.

Examples of such models:

• Inference models assessing the effectiveness of Non-Pharmaceutical Interventions (NPIs), such as masks, travel restrictions, lockdowns, etc.
  • Brauner et al. | Sciences | 2020 | Inferring the effectiveness of government interventions against COVID-19 | This article analyses the effectiveness of various government interventions during the COVID-19 pandemic, estimating the relative impact of several measures in controlling the virus spread.
  • WHO Health Emergencies Programme | Report | 2024 | Research prioritization for pandemic and epidemic intelligence: technical brief | Their 3rd research priority (out of 23) is “Identify rapid evaluation methods to assess the effectiveness of surveillance efforts and public health interventions during epidemics and pandemics”.
• Inference models assessing the effectiveness of Medical Counter-Measures (MCMs), such as vaccines, drugs, etc.
  • UK Health Security Agency (UKSHA) Corporate report | UKHSA strategic plan 2023 to 2026: executive summary | The UKHSA lists among its capacities the role of : “optimising the effectiveness of the UK vaccination programme, with new vaccines introduced safely and effectively, through the use of evaluation, modelling and expert public health advice”
  • The Epiverse-TRACE initiative provides an example of such modelling tools, vaccineff: Estimate Vaccine Effectiveness Based on Different Study Designs.

Conclusion: Assessing the effectiveness of countermeasures is relevant for managing pandemics effectively in the long term by ensuring that public health interventions are continuously refined, although it may prove less useful for catastrophic scenarios. Promising avenues could include exploring the possibility of providing good estimates in real time, or focusing entirely on using these models retrospectively for preparedness rather than response.

Conclusion

This review explored various modelling interventions and how they can be leveraged to prevent or mitigate the impacts of pandemics. Each intervention was evaluated based on its decision-relevance (i.e. the robustness, relevance to catastrophic scenarios, pathogen-agnosticism and empirical evidence of its theory of change), as well as its tractability, neglectedness, and potential dual-use risks.

Key Insights

Preparedness: Modelling interventions that inform prevention measures, e.g. through predicting the risk of zoonotic or lab-leak outbreaks, offer valuable insights, even though their cost-effectiveness and relevance to worst-case scenarios should be better established. Simulation models, in turn, are crucial for preparing for a wide range of potential pandemics, including catastrophic ones.

Early Response: The importance of early detection and rapid assessment of pathogen properties cannot be overstated. Quickly providing credible signals of pandemic potential to policymakers might be particularly effective in this regard. However, these models might prove less relevant against highly transmissible outbreaks, which may render local containment efforts moot.

Late Response: Forecasting the joint health and economic impacts of policies, as well as assessing the effectiveness of countermeasures in real time are important for managing pandemics as they unfold. On the other hand, actions need to be taken very early, even preventatively, to avoid catastrophic pandemic scenarios, which undermines the relevance of these late-response interventions.

Challenges and Recommendations

Challenge - Modelling Under Uncertainty: A recurring theme in this review is the difficulty of making informed decisions in the early stages of a pandemic, where data is scarce, and uncertainty is high

• Recommendation 1 - Focus on Preparedness and Early Response Models: Prioritise developing models that may either prevent emergence or enable rapid early response to pandemics. These models should aim to integrate all available data sources to improve early detection and swift assessment of emerging threats.
• Recommendation 2 - Focus on Catastrophic Scenarios: Future research and modelling efforts should prioritise scenarios with the highest stakes - in particular “wildfire” pandemics” and “stealth” pandemics - ensuring that our preparedness, prevention and response strategies are robust enough to handle even the most extreme pandemic threats, which are currently neglected.
• Recommendation 3 - Invest in Model Development Between Pandemics: Continuously refining models during inter-pandemic periods is much more effective and robust than having to adjust ill-adapted models in an emergency, as the COVID-19 experience has taught us. Analytical pipelines pre-configured for various scenarios and pathogen families should be developed to ensure immediate, actionable insights in real-time when new threats arise.

Challenge - Integration into Policy: One of the most significant challenges is the need for closer integration between modelling efforts and policymaking. This is the first Initiative recommended in the report Strengthening pandemic preparedness and response through integrated modelling by WHO, OECD and The World Bank. Real-time collaboration between modellers and decision-makers, as demonstrated by initiatives like the MIDAS network, can ensure that the right questions are asked and that the models used are relevant and actionable.

• Recommendation 4 - Enhance Collaboration Between Modellers and Decision-Makers: Establishing dedicated expert teams within governments that work closely with academic institutions and modelling groups can facilitate transparent dialogue and better integration of modelling into pandemic response strategies. Having teams on site allow modellers to gain a deep understanding of the specific challenges and constraints faced by decision-makers, and allow decision-makers to develop a clearer understanding of the models' capabilities and limitations.

Open Questions and Areas for Further Investigation

• Precise Timing of Pandemic Scenarios: Understanding the expected timeline of various pandemic scenarios (in particular wildfire and stealth pandemics) is crucial. Specifically, how much time typically elapses between the emergence of a pathogen and its uncontainable spread, and between global spread and when the damage becomes irreversible? This knowledge would strongly inform the timing and relevance of different biosecurity interventions.
• Track Record of Containing Highly Transmissible Outbreaks: How successful have past efforts been in locally containing highly transmissible outbreaks? Sources in this post suggest it may often be too late by the time containment efforts are initiated. I was pointed towards the book Crisis Averted by Caitlin Rivers on this topic.
• Inventory of Key Policy Questions: It would be useful to work with policymakers to inventory in advance the key policy questions that need to be addressed during a pandemic. Otherwise, there is a risk of working on modelling questions that are irrelevant, too theoretical, not applicable to the context of interest, or failing to capture critical aspects of the system.
• Retrospective Calibration of Early SARS-CoV-2 Estimates: Reviewing the retrospective accuracy, calibration and timing of early SARS-CoV-2 transmission and virulence parameter estimates, and how they impacted early policy decisions, would help understand how well current models perform in the early stages of an outbreak.
• Comparison with alternative non-modelling interventions: The value of an intervention should not be examined on its own, but in comparison with its best alternative. Whenever we want to achieve a goal through modelling, we should check whether we could achieve it more effectively without it. This post looks at this question at the level of modelling interventions, but it should then be considered at the level of the whole biosecurity landscape.

In conclusion, while significant progress has been made in the field of pandemic modelling, notably during the COVID-19 pandemic, there is still much work to be done to ensure that these tools are as effective and impactful as possible. In addition, I call for similar work to be carried out for other interventions and ToCs in biosecurity, and in a more systematic way, combining for example the EA approach of Founders Pledge’s Global Catastrophic Biological Risks report with the systematicity of WHO’s Pathogens prioritization: a scientific framework for epidemic and pandemic research preparedness report.