How much (more) data do we need to claim extreme cost-effectiveness?

By Niek Versteegde, founder GOAL 3 @ 2024-10-01T12:36 (+26)

TLDR:  GOAL 3’s IMPALA patient monitoring system has shown extremely promising results, with initial cost-effectiveness estimates ranging from $1.61 to $3.35 per life-year saved. This is based on a 42% reduction in pediatric mortality in a Malawian hospital over 18 months and a 27% reduction in neonatal mortality in a Tanzania hospital over 12 months. Supported by a $260,000 grant from Founders Pledge and its community, we are now conducting studies to further assess  While these  results are encouraging, we seek feedback from the community to ensure the robustness of our methodology and to explore what is needed for broader support

Epistemic status: Experienced but open to further validation.

"As a medical doctor with 14 years of global health experience and five years working on the IMPALA solution, I am confident in the real-world results we’ve seen so far. However, I recognize that our current data, while promising, is still developing, and further research and input from the community are essential to confirm and strengthen these findings while balancing it against the urgency of the problem." 

 

Dear EA Forum Members,

We need your crowd wisdom to help us evaluate our IMPALA solution. Our initial results are incredibly positive compared to existing interventions, but we want to ensure we haven’t overlooked key factors or introduced biases in our methodology.

Here’s where we’d love your feedback:

  1. Are our reasoning and methods sound? Have we missed any major issues or biases?
  2. How does our current evidence resonate with you? What else would be needed to gain your trust and the larger community?
  3. (How) should we highlight less obvious benefits, such as reducing workloads and societal impacts?
  4. How can we best engage the EA community to accelerate our mission?

After five years of hard work, I may be biased, but we’re results-driven and eager to learn. Please share your thoughts and feedback to help us take IMPALA to the next level.

Part 1: Why GOAL 3

In 2015, I worked as a tropical doctor in Tanzania and had an experience that changed my life. A premature boy was named after me (Niek) after I helped him survive sepsis and asphyxia. It was a fantastic experience that became even more special when I saw how this small Niek played with my one-year-old son, Guus, two years later. 

Unfortunately, I also saw many children who were not as lucky. Treatment was available, but we could not recognize and treat the condition in time due to permanent staff shortages and a lack of adequate equipment. These deaths were unnecessary, which was the reason for starting GOAL 3: a social enterprise aiming to create a scalable, sustainable solution to help achieve more with scarce resources.

Part 2: Theory of change: facilitating a sustainable transition from reactive to proactive care. 

In many low-resource healthcare settings, reactive care dominates, resulting from severe understaffing and a lack of appropriate medical technology and/or information systems. Health systems typically respond to crises and acute conditions only after they’ve worsened, leading to late-stage interventions that are more costly and complex and often result in poor outcomes. This creates a vicious cycle: health workers are overwhelmed by emergencies, making it challenging to allocate time and resources for proactive care. The strain on the system leads to inefficiencies, higher costs, and worsening health outcomes—a cycle that’s hard to break due to limited resources.

IMPALA is a patient monitoring solution that continuously measures vital signs and brings this information together in an intuitive tablet application, enabling a shift from reactive to proactive care. Through real-time data collection, intuitive risk assessment, and alarms that signal deterioration, health workers can perform timely interventions, preventing crises before they happen and reducing the need for emergency interventions.

This shift reduces healthcare providers' workloads, as fewer emergencies mean less time spent on complex, high-stress cases. IMPALA also automates routine tasks and streamlines workflows, allowing health workers to focus on direct patient care. With shorter hospital stays and fewer severe cases, the overall workload decreases, and patient and health facility costs decrease.

By breaking the reactive care cycle, IMPALA improves patient outcomes, lowers costs, and reduces the strain on healthcare workers. It is a pragmatic, scalable solution that helps health systems do more with limited resources, aligning with GOAL 3’s mission to work smarter and achieve better results.

 

IMPALA MonitorIMPALA Tablet

Part 3: Results from 2 pilots over 12 and 18 months: Can this really be true?

After multiple studies with IMPALA, the Ministry of Health of Malawi authorized GOAL 3 to implement IMPALA outside a research setting. In early 2023, the system was installed in the pediatric and neonatal wards in St Luke’s Hospital in collaboration with AMREF Malawi (the largest Africa-based NGO). A second pilot, involving local governments, started in Sengerema Hospital in Tanzania in July 2023, together with the Ifakara Health Institute.

The implementation of the IMPALA System involved a comprehensive approach. After an initial assessment of each facility, a tailor-made 1-day training on the use of the system and emergency and critical care was provided to all nurses, doctors, and ward attendants. After this, the system was installed in the neonatal and pediatric wards. Using a before-and-after approach, the impact on mortality and admission time of the system was monitored over time using aggregated routine patient outcome data, stored monitor data, and regular supervisory visits.

Impact on mortality

In St. Luke’s Hospital, the implementation of the IMPALA System led to a 42% relative reduction in mortality over 18 months (Feb ‘23 - Aug ’24), decreasing from 3.3% to 1.9% (p < 0.05) with 1146 admissions in the 12 months before and 1828 admissions in the 18 months after implementation. The data from the neonatal care setting could not be compared between the two periods because there was a significant change in the population due to a concurrent intervention.

In Sengerema Hospital the 12 months after implementation (Jul ’23 -Jun ‘24), neonatal mortality decreased by 27%, from 12.7% to 9.3% (p < 0.05) compared to the 12 months prior to implementation (Jul ’22 - Jun ‘23). This mortality reduction occurred while both patient admissions and in-referrals of babies born outside the hospital somewhat increased in the post-IMPALA period. This indicates that a larger and potentially more severely ill patient group was served, and therefore, the observed mortality reduction may be considered a conservative estimate of effect.

Part 4: Cost-effectiveness estimates

Pediatric care setting

In St. Luke’s Hospital, 6 monitors were implemented in the pediatric ward for 18 months. Using the monthly total cost of ownership price of $50$, the total costs for this period were $5,400.

A total of 1829 patients were admitted during this period, and with 42% fewer deaths (1.40% absolute reduction), an estimated 26 pediatric deaths were prevented during the post-implementation period. Therefore, the cost of saving one life was $208 or $3.35 per life-year (assuming a remaining life expectancy of 62 years).

Neonatal care setting

In Sengerema Hospital, 9 monitors were implemented for 12 months. Using the total monthly cost of ownership of $50$, the total costs in this period were $5.400. 

A total of 1519 patients were admitted during this period, and with 27% fewer deaths (3.38% absolute reduction), an estimated 51 neonatal deaths were prevented during the post-implementation period. The cost of saving one life was, therefore, $105 per averted death, or $1.61 per life-year (assuming a remaining life expectancy of 65 years). 

In comparison to other standard-of-care interventions for preventable pediatric deaths, such as facility-based management of pneumonia (3.5$ per life-year saved), Vitamin-A suppletion (7.1$ per life-year saved), or kangaroo mother care ($20.1 per life-year saved), this initial assessment shows great potential for cost-effective impact at scale. This does not yet take into account other observed benefits, such as a reduced workload and admission time, which could translate into cost-savings for the patient and health system. 

 

Part 5: In God we trust; all others must bring data.

But how much do we need?

While the evidence so far is extremely promising, we acknowledge its limitations. Our methodology could be improved with more direct comparisons, and we haven't yet had the resources to dive into individual-level data. Thanks to the support of Founders Pledge, we’ve launched two new studies in Malawi and Rwanda that aim to address these gaps. These studies will assess the impact on health workers, track usage data from IMPALA, and measure long-term effects on costs, outcomes, and cost-effectiveness.

Balancing Data with Urgency

We know how crucial it is to ensure IMPALA consistently delivers results across different settings. If we fail to reduce workloads or mortality, not only will we have failed in our mission, but we’ll also have spilled precious resources. That's why we are committed to being data-driven, constantly learning, adapting, and improving—because success isn’t just about if it works, but how to make it work effectively. This will be different in different settings, so implementation and evaluation are the backbone of our solution. 

At the same time, I often feel frustrated by the lack of urgency. I believe we’ve found a revolutionary solution that could have a massive global impact, yet we’re still waiting for more data. When is it sufficient to take the next leap? While we focus on the costs of failure, we must also consider the cost of maintaining the status quo. 

I would be delighted to hear some of your reflections on determining the right balance and what additional steps we could or should take.

What do you think? 

For those who are interested, we included the link to the Cost-effectiveness assessment - a detailed analysis providing more background to our current assessment

 

FAQs

1.Where can I find out more about the calculation of cost-effectiveness?

2. Can I learn more about the IMPALA System itself?

3. How can I contribute to the GOAL 3 organization?

https://www.goal3.org/foundation 

Ian Turner @ 2024-10-01T21:23 (+7)

As a donor to global health and development causes, what I would be looking for are:

I haven’t spent any time picking apart this particular program but one thing I see missing here is any skepticism about the results. Usually when it comes to charitable interventions, the headline number overstates things by a lot, because when you dig into it, it turns out there are various confounding or offsetting considerations that lower one’s confidence. What are some reasons that the intervention might actually not accomplish anything at all (or even be counterproductive), that would be consistent with these results so far?

You might also ask GiveWell what it would take to get their attention.

Niek Versteegde, founder GOAL 3 @ 2024-10-02T07:45 (+1)

Hi Ian, thank you for your thoughtful feedback. We fully agree that independent evaluations and randomized controlled trials (RCTs) are an important tools for understanding impact. However, we must also acknowledge that RCTs come with significant costs, long timelines, and often produce results that don’t fully translate into real-world settings. Our experience has shown that the complexity of low-resource healthcare environments requires adaptable, on-the-ground solutions that are continuously tested and refined in real-time.

That’s why, at GOAL 3, continuous evaluation and learning are embedded into our approach, aimed at Real World Evidence. We feel that waiting for a one-off trial to tell us whether IMPALA works is not appropriate. Instead, we are committed to assessing our effectiveness at every site in the real-world, following up over time, and ensuring year-on-year improvement. This approach allows us to make iterative and localized changes based on the needs of health workers and patients, ensuring we create a sustainable impact that lasts. Our goal is to see a measurable improvement not just immediately after implementation but as a consistent part of the health system's development. 

Of course we do want this to be evaluated independently, to also assess unanticipated negative and/or positive impact from the intervention. We know we are biased and we understand that we are not the best evaluators. That is why we leverage the Founders Pledge grant to ensure independent research institutes can do this evaluation across different settings. This will strenthen the evidence on effectiveness across different settings with a larger poule (7) hospitals, over time (>1year follow-up) while also assessing differences. 

On a personal note, while I understand and endorse the rigorous processes that donors and evaluators require, I sometimes feel frustrated by the lack of urgency. The slowness of decision-making in global health is an enormous barrier to innovation. Many life-saving solutions are delayed because innovators and startups simply don’t have the time or resources to wait for trial results. During COVID-19, we saw how innovations were fast-tracked to address an urgent crisis. It feels unfair that the 5 million pediatric deaths every year are not considered urgent enough to prioritize similarly fast solutions. Basically the risk of unanticipated consequences should be balanced against the potential positive impact and use that to navigate decisions about 

Ultimately I  agree with the process and I should not complain after receiving a grant from Founders Pledge to support us in building the evidence base that is needed. Its just that I am frustrated because it feels like a continuous fight to move things forward or even survive, while deep in my heart I know we are right and that every delay will hinder us from preventing deaths that are unneeded.

Ian Turner @ 2024-10-02T17:39 (+3)

Thanks Niek for your thoughts.

Regarding this:

On a personal note, while I understand and endorse the rigorous processes that donors and evaluators require, I sometimes feel frustrated by the lack of urgency.

I don't mean to speak for anyone but myself, but I sense that there has been an increase in skepticism in recent decades, as interventions that seemed promising and which "we know work" turn out not to have any measurable effect, or even to have negative effects, when carefully studied. My prior, at least, regarding any new intervention that I hear about is that any effect is probably illusory or minimal.

It's not that there is no urgency about the problems in the world — those are indeed urgent and demanding — but rather, our resources are so limited, and good interventions so hard to find, that we should probably dedicate only a small amount of our funding to more speculative projects, so that we can build evidence over time and not waste funding on projects that turn out not to work. As an example of what we want to avoid doing, look at what happened with microfinance.

Of course the situation would be different if we had already fully funded well-studied projects like bednets, deworming, Vitamin A, etc.; but sadly that's not the world we actually inhabit.

Niek Versteegde, founder GOAL 3 @ 2024-10-02T19:57 (+1)

I guess then it's just my title: In God we trust, but all others must bring data. It just feels frustrating sometimes, also because people actually don't say they don't believe you. So if you don't call me a liar, then why don't you help or support me?

I would really like it if a more systematic way of identifying, testing and scaling successful innovations was available.

My post are definitely not intended to complain by the way. I am blessed with a great and supportive team and have great partners as well. It's a genuine interest to understand, learn and improve.

Niek Versteegde, founder GOAL 3 @ 2024-10-02T07:53 (+1)

PS. over the next days I will try to add a bit more background about the solution and our evidence, where we think it needs to be strenthened. I was a bit reluctant to put too much information in, because I thougt people might not read it. 

Niek Versteegde, founder GOAL 3 @ 2024-10-01T13:30 (+3)

Apologies all, I accidentally posted this without the content at first. Still getting used to the platform.

tobycrisford 🔸 @ 2024-11-12T07:52 (+1)

A question jumped out at me when reading these results. I should caveat this by emphasizing that I am very much not an expert in this kind of evaluation and this question may be naive.

Is there any seasonal effect on mortality in Malawi? If so, is it ok for the pre-intervention period to be 12-months while the post-intervention period is 18-months?

Niek Versteegde, founder GOAL 3 @ 2024-12-05T20:00 (+5)

Hi Toby, 

Apologies for a belated response. You are correct that there are quite large seasonal influences. So we matched everything with the respective months in the same period to compensate for it. 2 weeks ago we also got the individual patient level data. This data shows that the post-intervention group was actually on average in a worse condition than the pre-intervention group. If this is taking into account the mortality reduction is close to 50%. 

Regards,

Niek