The Best Health Systems Strengthening Interventions barely qualify as HSS - My take on RP's report

By NickLaing @ 2025-04-28T16:07 (+51)

TLDR: RP's best interventions barely qualify as Health Systems Strengthening - they focus directly on the Health worker and their implementation. Not only these, but almost all HSS interventions are measureable and should be measured! Plus some light disagreements with RP.

First a huge thanks to @Open Philanthropy and @Rethink Priorities for this report - I’m not sure any serious attempt at a cost-effectiveness analysis of Health Systems Strengthening (HSS) interventions has been done before - and they’ve done a great job. I won’t spend much time affirming all the good stuff in the report (please read at least the summary before reading this) rather I’ll double down on things I think are important or that I disagree with.

HSS is not neglected

“Health Systems Strengthening” has been a loud clarion call in the public health world for over 30 years now. There can be an almost religious fervor that the only way to make sustainable, permanent improvements in health outcomes is to improve the health system from the top down. We often hear the classic arguments that we should not put “band aids” on problems by funding vertical programs like mosquito nets and deworming, rather we should fix the “root” of the problem at the heart of the system itself.

Over the last 20 years, somewhere in the range of 100 Billion dollars might have been spent under the broader bucket of HSS^[1] - hardly a neglected area. Despite this general non-neglectedness, I strongly agree with RP though that are likely to be interventions within the broader HSS bucket, which are relatively neglected and very cost-effective.

RP’s best Interventions are barely HSS and focus on Health Workers

Interventions RP found most promising (IMCI, quality improvement, CHWs) are right on the edge of “Health Systems Strengthening”. They are direct interventions that improve the number, or quality of frontline health workers. A health economics professor told me yesterday that they wouldn't consider RP's top interventions of IMCI, CHW expansion and community scorecards to qualify as HSS. It's also interesting and not noted in the report that these interventions are similar to what @CE orgs like Ansh (Close support supervision) or AMI (Med Supply chains) or @Lafiya Nigeria (CHW expansion) are already doing.

Improving the fundamental output unit of the health system - a health worker treating patients makes intuitive sense. If Health workers (CHWs) are at a facility (Scorecards) implement evidence based treatment guidelines (IMCI) and are supervised to follow those guidelines then patients will consistently receive high quality treatment and less people will die. Below I’ve cooked up a model for the essentials of a health system, which despite being painfully basic can help make sense of why these interventions might work better than the higher level ones RP considered potentially less impactful.

Too often Health Systems Strengthening focus on interventions one or more steps removed from the fundamentals, the core problem. They focus on largely useless but nice-sounding Interventions like “improving funding structures”, "high level governance", and health worker trainings that don’t solve the core problems in Low income countries. If there are no health workers or medications at a health facility then any other interventions are a waste of time and resources until those fundamental problems are fixed.

Cost-Effectiveness of HSS interventions can and should be measured

The RP recommended interventions are all measurable and indeed measured. I disagree though that most HSS interventions are too hard to measure. I don’t understand this argument from the article “Moreover, many studies do not evaluate HSS interventions comprehensively, but only focus on a few narrow outcomes. Thus, we expect that narrow, outcomes-focused studies would tend to underestimate the overall effects of an intervention.”

Yes we should look for a narrow outcome. There is usually one primary outcome in health interventions, which is healthier, happier people (measured in DALYs/WELLBYs). We often have to estimate this effect through a proxy of quality/quantity of care as mortality/morbidity is hard to measure directly. So yes the outcome is narrow. A HSS intervention might be complex, but the outcome measure should be narrow. I don’t understand how a “narrow, outcome focused study would tend to underestimate the overall effects of an intervention”

Most Health Systems interventions can be measured by RCTs. Governance, financing, and supply chain interventions can be randomised at state or district level, while service delivery and workforce interventions can be randomised at facility level^[2]. For all of these, treatment quantity and quality can be used as an outcome measure - often quite straightforward in government systems with decent information systems. I challenge you to give me the intervention and I’ll give you a plausible RCT^[3].

In my eyes the bigger issue is that many HSS advocates believe that interventions don’t need to be tested because HSS interventions are by nature better than vertical ones. Or sometimes they don’t have a background in RCTs or quantitative analysis so are skeptical of these methods. Also the kind of people who are drawn to HSS are people are more interested in complex systems than numbers. None of your 5 named experts seem to have a serious background in quantitative analysis. I think there are great HSS interventions and vertical ones - lets continue tomeasure them and see.

Context matters - Country >>> Intervention

RP looks at countries where health systems and outcomes have greatly improved, and conclude correctly that “large share of health improvements in positive-outlier countries were likely driven by non-health system factors (e.g., economic growth, political stabilization, improved education”. I just want to emphasize a similar point that particularly with complex HSS interventions, many systemic factors to line up for the intervention to work. We can distribute mosquito nets successfully even in fairly dysfunctional countries, but if we want to introduce a complex health insurance system, the country should be in a great state before we even try.

A development practitioner once quipped to me “You could do any intervention in Rwanda right now and it would probably work great”. NGOs have clamoured to work in Rwanda because their programs were likely to actually work there, even when it might fail in Uganda or Nigeria. Although I don’t know that much about the country, Bangladesh Kerala and Thailand seem to be similar with many development interventions working well across all sectors.

When a country (like Rwanda or Bangladesh)
1. Has better institutions
2. Cracks down on corruption
3. Collaborates with NGOs throughout their government system

With these in place, Healthcare and other interventions are likely to succeed. We can look to success stories of countries for weak guidance, but I don’t think we should consider one intervention working in these countries a strong data point that it will work in another. As a historical example Microfinance initiatives fell into a trap trying to replicate success in Bangladesh across Africa, with more failure than success.

IMCI and other guidelines have enormous potential

“IMCI” is at its core just a guideline for diagnosing and treating kids correctly (see diagram above) and a great one at that. When there has been a successful “IMCI” implementation, in simple terms that means that nurses who were diagnosing and treating patients badly, now follow guidelines which make sure they treat them well. Guideline based healthcare is a hugely under-resourced and under-researched area. I would almost go as far as to say that well implemented guidelines are the only way to achieve “correct diagnosis and treatment” in many low-income countries.

As we task-shift and expect nurses and community health workers to manage patients, flowchart based guidelines are the best way of simplifying and clarifying the new, more difficult work we expect these lower level health workers to perform. We do however need carrots and sticks for health workers to make guidelines work, so the worse the health system is in general, the harder guidelines like IMCI are to implement and maintain.

Light Disagreements

Community Health workers are often not cost-effective

I agree that CHWs are an essential, scalable part of many LMIC health systems. CHWs however are touted to be very cost-effective when they often aren’t. This statement made my eyes pop a little. “Many studies have found CHW programs to be cost-effective by various metrics. In Mozambique, the annual cost per beneficiary was just $47.12”.

50 dollars is a HUGE cost to serve just one beneficiary. In 2010 Mozambique’s health spend per capita was around $30 dollars. This means that to pay for a CHW to provide basic care for one patient, Mozambique would spend more than its per capita allocation, an expense that doesn’t make sense for a low income government.

By nature, CHWs are relatively expensive health providers because they treat few patients per month. They can’t treat large numbers because they only treat a handful of conditions in young children and can’t treat most of the disease burden. They also operate in small geographical areas. Although data is hard to find, cost per patient treated by orgs like LastMile Health, Muso and Living Goods seems to vary between about $5 and $12 which is more expensive than patients treated at static government facilities, or at our OneDay Health Centers ($2 per patient).

To further emphasise (from RP's appendix)

“Per $10,000 invested, CHAs delivered over a thousand home visits, 370 child cases of malaria, pneumonia and diarrhea treated, 65 pregnancy home visits, 331 malnutrition screenings for children under five, and provision of family planning access to 117 women.”

These numbers might look good at first, but to use the example I know best, our OneDay Health Centers are set up for $4,000, and within just a year treat 1500 patients on average, including about 400 kids with malaria, pneumonia and diarrhea. After that they continue to treat that many patients year on year at a lower cost.

LMH is far more well known and better funded than Living Goods

“LMH is perhaps a bit less well-known compared to a related CHW program called Living Goods (which has already received several Open Philanthropy grants)”.

Last Mile Health are the most well funded, well known and celebrated CHW organisation in the world - they won both the TED prize and the multi-million dollar Skoll award in 2017. They are far better known in the public health world than Living Goods. Their co-founder and Head Raj Panjabi is on the board of the Skoll foundation and has worked in the President’s office.

Transitioning to government ownership is a risky bet - that might sometimes be worth it

“We think this (funding LMH programs) represents an example of a catalytic grant opportunity for Open Philanthropy: to help fund and establish a public health program until it is transitioned to national government ownership.”

This comment here is perhaps naive - transition to government ownership is indeed the cliche path to scale (and dream) for many of us in the social enterprise world, but it rarely happens.

The idea that Open Philanthropy could pick a country, and fund community health workers until the government takes over sounds great but has a very low chance of success. I’m not saying its necessarily a bad bet, but it would be vhigh risk. There are already hundreds of millions of dollars pouring in to fund attempts to transition CHWs to government ownership. Why would OpenPhil be better at this than the other funders?

I do agree though that LastMile Health are one of the best orgs around at making this pathway happen and they have at least partially succeeded with it in a handful of countries. They have done better at government adoption than most organisations

Supply Chains interventions have largely failed.

I agree that there is potential with supply chain interventions, but think the overall record of supply chain interventions are poorer than RP paints. The failures may at least match the successes. Chemonics spent 10 Billion Dollars of USAID money on trying to improve supply chains and on review achieved very little. Also I’m skeptical of RidersForHealth. My feeling is they are now solving a problem that no longer exists at scale. To their credit they were part of solving an important problem 30 years ago when there was no good way of getting medications to facilities, but now that’s pretty straightforward through an abundance of private motorcycle taxis. I don’t understand how an NGO in this area can now add much value outside of a few niche cases.

I do agree though that there could be some excellent opportunities. After all, having medicines at the health facility is the no. 1 necessity before we even think about anything else. On this front I think Access to Medicines initiative have a logical approach to improving family planning commodities, although they are still a very new org.

Again I really like the RP report and would love to comment more, but don't have time! Keen for discussion and comments as always!

^{^}
Mostly from Deep research analysis - broadly matching with RP’s estimate of 7 billion last year
^{^}
Service delivery interventions usually do have RCTs backing them up
^{^}
Yes there are a few cases you can catch me out on here, but the vast majority are measurable

jenny_kudymowa @ 2025-05-06T09:32 (+11)

Nick, on behalf of our team at Rethink Priorities, I’d like to thank you for engaging with our work and sharing your insights. We really appreciate it! You raise some excellent points, and I’d like to respond to them in turn.

HSS is not neglected:

I’d agree that looking at total dollar spending on HSS doesn’t by itself give a clear sense of neglectedness. While HSS has certainly been central to the discourse, I’d argue it has often been more rhetoric than reality. Much of the ~$100B labeled as “HSS” may in practice have gone toward narrow or superficial fixes, at least that’s something I’ve seen noted repeatedly in the literature. In some yet-unpublished follow-up work, we’ve also tried to relate those spending figures to the share of the health burden attributable to weak health systems, though doing so is quite tricky and comes with considerable uncertainty.

RP’s best Interventions are barely HSS and focus on Health Workers:

That’s a good point, and I’d clarify that we don’t necessarily think interventions like IMCI, CHW expansion, or community scorecards are the best or most impactful HSS interventions overall. Rather, they are the ones where decent evidence on cost-effectiveness was most readily available, which made them rise to the top in our initial analysis. There may well be higher-level or more systemic interventions that could deliver larger or more sustained health gains, but they often lack the kind of robust evaluation data we need to assess them confidently. In that sense, the ranking reflects the current evidence base more than an absolute judgment about where the biggest opportunities lie.
I share your impression that some nice-sounding or fuzzy interventions can be hard to see as meaningfully improving things. That said, governance is one area I strongly believe in (see also what I wrote on governance further below). While its impact can be difficult to capture from the bottom up when looking only at individual impact evaluations, it becomes more visible from the top down when examining which factors plausibly contributed most to meaningful health systems strengthening in a country over time. This is also something I found quite convincing in the work of Exemplars in Global Health and the Good Health at Low Cost study.

Cost-Effectiveness of HSS interventions can and should be measured:

Regarding the “narrow outcomes,” I’d be thrilled if more HSS evaluations reported actual health outcomes (or direct effects on household finances), but as you’ve noted, that’s not always feasible or practical. In the meantime, we can focus on intermediate outputs, but I see two main concerns. First, these outputs are often so far upstream that linking them to health impact requires a lot of guesswork and assumptions. Second, they often capture only a small slice of the system, even though the reform affected much more, making it hard to understand the broader effects. For example, an evaluation might report that a procurement reform reduced drug prices but provide no data on whether availability or stockouts improved, even though that was a key reform goal. And when prices change, the downstream and general equilibrium effects can be complex and hard to anticipate. I’m often surprised by how little is measured around big, system-wide reforms, given how much they can shift across the system.
I’m very sympathetic to the idea that RCTs could be used more often for HSS interventions, and I think they are technically feasible in many cases. However, I do believe RCTs in the context of HSS tend to be more complicated and expensive, especially when multiple interventions are rolled out simultaneously, timelines are long, and there are potential spillover or general equilibrium effects, typically more so than in vertical programs. I also suspect it’s often a public goods problem: the benefits of generating generalizable evidence mostly accrue to others, while the costs and burdens of implementing the reform in a way that allows an RCT can largely fall on the implementing government. At least, that was my impression when I was involved in evaluating different reforms in Ethiopia, where the government seemed more keen on rolling things out quickly and evenly rather than on randomizing across districts and making some people wait longer for the reform.

Context matters - Country >>> Intervention:

I strongly agree with you that the country context often matters more than the specific intervention. I think many of the classical success stories, like Bangladesh, Rwanda, or Ethiopia, are ultimately stories of strong governance and sustained political commitment. That commitment manifested in a wide range of ways, from institutional reforms to close collaboration with NGOs. In those contexts, my impression is often that any specific intervention mattered less than the broader enabling environment.
I also agree with your point about not overinterpreting success in high-functioning countries. As you noted, interventions like microfinance didn’t translate well from Bangladesh to other settings, and I think the same caution is warranted for health systems. That said, I still see a lot of value in studying what worked and where, especially to understand the combination of factors that enabled success in specific contexts.

IMCI and other guidelines have enormous potential:

What you wrote makes a lot of sense to me and aligns with my general intuition about the importance of guideline-based healthcare. It’s not an area I’ve looked into closely myself, so I really appreciate the insight.

Community Health workers are often not cost-effective:

I agree with you that community health workers (CHWs) are unlikely to be among the most cost-effective interventions. We noted in the report that “CHW interventions are likely to require large amounts of ongoing financial support, and do not offer a clear path toward highly cost-effective or leveraged interventions”, which was meant to reflect this view.
That said, if we look beyond the immediate, short-run effects of CHWs (such as patients treated), their longer-term contributions may be quite substantial. In Bangladesh, for example, CHWs played a transformative role not only in delivering services but also in shifting norms around family planning, increasing female education, and building trust in the formal healthcare system, all of which helped increase demand for institutional care over time. Those broader effects may not show up in short-term cost-effectiveness metrics, but I believe they are likely substantial (though I can’t point to concrete numbers). Exemplars in Global Health illustrate this nicely in their case studies, e.g. here and here.

LMH is far more well known and better funded than Living Goods:

We only had time to look at LMH at a very superficial level, so you might be right about that.

Transitioning to government ownership is a risky bet - that might sometimes be worth it:

I really appreciate this insight from someone experienced on the ground. To be clear, we have not spent any time coming up with a probability of success for this pathway, and I agree that assuming a smooth transition to government ownership would be naive. At best, I think of this as a high-risk, potentially high-reward opportunity, and one that might only make sense if there’s a particularly strong implementing partner (like LMH), unusually strong government buy-in, and a clear reason to believe philanthropic support would be catalytic rather than duplicative. But it’s certainly not a “default” safe bet.

Supply Chains interventions have largely failed:

We actually did some (unpublished) follow-up research that focused more heavily on supply chain interventions, and it left us more cautious as well. The evidence for large-scale structural reforms seems quite mixed, and we struggled to find clear, well-documented success stories. The USAID-Chemonics project you mentioned illustrates this challenge well, though it’s hard to disentangle whether the issue stemmed primarily from the inherent complexity of supply chain reform or from shortcomings in the program’s design and execution.
Regarding the need for last-mile delivery interventions, you probably have a better sense of the situation on the ground than we do. That said, some recent publications make me think that the problem isn’t quite solved yet. For example, a recent last-mile delivery intervention for COVID-19 vaccines in Sierra Leone increased vaccination rates by 26 percentage points within 48-72 hours (Meriggi et al., 2024). Moreover, the Gates Foundation still seems pretty active in supporting last-mile delivery interventions (e.g., here and here), which suggests that they still see meaningful gaps to address. We haven’t reviewed RidersForHealth in detail, so I can’t speak to their specific model.

Again, thanks a lot for engaging with our report!

JoshuaBlake @ 2025-05-01T07:04 (+6)

Governance, financing, and supply chain interventions can be randomised at state or district level

While I agree this is true in theory, is it practical? I imagine the size needed to power such a study is prohibitive except for the largest organisations, the answer still wouldn't be definitive (eg due to generalisation concerns), and there would be lots of measurement issues (eg residents in one district crossing to another district with better funded healthcare).

If you tell me I'm wrong, I'd definitely bow to your experience and knowledge in this field but this isn't obviously true.

NickLaing @ 2025-05-01T09:58 (+6)

I don't have great experience and knowledge here AT ALL as a caveat. Never "bow" to anything I say, my takes are often more on the "loose" than "rock solid" end of things :D.

I think if we can randomise things like socrecard studies and IMCI across hundreds of health facilities (done a number of times), then I don't see why we can't do the same with a supply chain interventions or governance interventions. The community Health worker movement has done some impressive large scale RCTs like this one. Perhaps 1-3 million dollars could make these studies happen without too much trouble. Give 10 randomised districts the governance/ intervention and 10 not, then just see if healthcare outputs improve. I actually think its easier than many other types of studies because

1. I think its good enough to measure outcomes in terms of facility level outputsd, so we don't necessarily need community level morbidity/mortality data
2. Outcome measures (no. of patients treated, correct diagnosis) would be super easy and not expensive to measure compared with other studies. In many cases routinely collected DHIS data should be enough to answer the primary outcome question so we don't even necessarily need much expense on data collection (a big study cost)

I would say from an RCT perspective if people crossed to another district because healthcare was getting that much better, that would be a strong sign that the intervention is working insanely well. If it was a financing type intervention, then making it close to cost-neutral between the intervention and control group. People are NOT very mobile in places like Uganda at least. Where I work in rural places Transport is often (if not usually) the biggest healthcare cost people incur.

I think the biggest reasons these studies haven't happened more (there are some) are less practicality and more...

1. Most governance, financing and supply chain interventions are funded by bilateral aid not philanthropy, so they don't usually think about rigorous testing. Instead they often spend quite large amounts of money on "baseline and endline surveys" which I think are often like pouring money down a sink. Or to be slightly more generous perhaps a direct cash transfer to the richest 1% of people in low income countries ;)

2. The kind of people who are into HSS interventions are often pretty religious about HSS being the "only" way to go and are also often RCT averse, so are less likely to commit time and resources to an RCT as part of their intervention.

Dee Tomic @ 2025-04-30T00:38 (+3)

I appreciate the call for more scrutiny of cost-effectiveness in HSS. Given the EA community’s focus on measurable impact, do you think there's room to shift some methodological attention toward capturing system-level resilience, which is harder to quantify but critical for long-term outcomes (especially during shocks like pandemics or conflicts)? For example, Rwanda’s long-term investment in community health workers and data systems didn’t show immediate returns, but it was credited with enabling a rapid and coordinated pandemic response – suggesting that some HSS benefits may only become visible during moments of acute stress.

NickLaing @ 2025-04-30T10:00 (+6)

Thanksd @Dee Tomic. I'm interested in what you mean by "resilience" exactly, I'm guessing you mean a Health system which continues to perform at a high standard when put under some stress? I think this is unfortunately really hard to test - how can we know what to attribute any reslience to? Its an important part of a health system, but hard to test. Community Health workers though should show other major effects which are testable besides reslience.

if you're talking the covid pandemic, then the narrative around a strong "Pandemic response" in East Africa was largely a myth - at least measured by outocomes. Covid spread was hardly prevented at all by the seemingly harsh control measures taken. Most people in East Africa caught Delta within 3 months of it being in the country, and most Omicron within 2 months. The epi curves are insane, they are straight out of a textbook showing uncontrolled spread of a disease. While the virus ripped through the population It was just good fortune that the population was young and didn't have many cardiovascular comorbidities, so while most people cught covid the mortality rate was super low. We wrote a small piece on this situation in Uganda which is interesting (the epi curves are nuts), and I remember when I checked at the time the Epi curves were almost exactly the same in Rwanda and Kenya.

https://www.researchgate.net/publication/378869273_Does_epidemiological_evidence_support_the_success_story_of_Uganda's_response_to_COVID-19

I'm also interested that you think Rwanda's investment in community health workers didn't show immediate returns. This is an area I dont know about, but I was under the impression that community healthworkers did help the country reduce malaria mortality and improve access to care overall in a relatively short time after they were implemented. Again I'm not an expert at all here

You're right that the effect of data systems are hard to measure in an RCT, that's certainly one of the exceptions to the rule.

Dee Tomic @ 2025-05-01T00:24 (+11)

Hi Nick, I really appreciate the thoughtful and detailed reply. You're absolutely right that "resilience" is a broad and slippery concept, and I agree that attributing outcomes to specific components like CHWs is tricky, especially in crises with many moving parts.

By resilience, I meant something like maintained or recoverable delivery of essential services under stress – not necessarily stopping viral spread, but preserving routine care, adapting quickly (e.g., home-based care), and using real-time data for coordination. I take your point about the COVID epi curves – and agree that the mortality patterns owe much to demographic structure – but I’d argue that Rwanda’s capacity to maintain service delivery (e.g., immunisations, maternal health) and implement decentralised home-based care was in part enabled by their CHW and data infrastructure. That doesn’t negate the limitations of their COVID containment, but points to other dimensions of system functioning.

And yes, you're absolutely right that Rwanda saw early gains from its CHW programs in areas like malaria and maternal health – I didn’t mean to suggest otherwise, only that some system-wide or crisis-response benefits can be delayed or harder to isolate. I’ll have a read of your Uganda piece – thanks for sharing!

NickLaing @ 2025-05-01T06:51 (+5)

Great insights there, hope to hear from you more on the forum!

Yes thanks to much I understand better now. I agree that Rwanda's structure is resilient and enabled them to maintain service delivery during the pandemic. Uganda was terrible in this front, for a number of reasons. During the early lockdowns people died because of poor access - this is an article about a kid dying after we couldn't refer them from one of our OneDay health centers

https://www.google.com/amp/s/www.aljazeera.com/amp/features/2020/4/21/children-women-casualties-of-ugandas-coronavirus-transport-ban

I think resilience is important but I'm struggling to see how we can test for it in solid ways. In this case I think we have to really on case studies like Rwanda during covid like you say. As a side note like I mentioned in the article, over a 10 year period almost every development thing in Rwanda seemed to work well and bore fruit, and I consider them a success story more than necessarily a model for other countries. Takes a pretty savage dictator to stay on that tight path!

I would hope that many of the same interventions that bring resilience also bring more immediate benefits that would be measurable, but you're right that won't always be the case.