What would make us scale or stop? NOVAH's pre-commitment before seeing the RCT results

By I.J.J., AlexisAt @ 2026-08-11T11:51 (+95)

TL;DR

NOVAH (No Violence At Home) was incubated by Charity Entrepreneurship (now Ambitious Impact) in 2024 to test a promising idea: preventing intimate partner violence through edutainment, in our case a serialised radio drama. Over the past two years we have produced and aired two seasons in Rwanda.

We are currently evaluating our second season through a randomized controlled trial with 2,400 couples in Rwanda in partnership with Innovations for Poverty Action (IPA). Preliminary results are expected in October 2026. The end-line analysis is run by IPA and our academic partners, not by NOVAH.

Edutainment to prevent intimate partner violence is promising, but it is still young as a field. We do not want to spend donor money scaling it unless the evidence shows it works at least as well as the strongest existing approaches. The RCT is what will tell us that. Therefore, before seeing those results, we want to make our decision rule public:

We will scale our programming only if the RCT finds a statistically significant relative reduction of at least 5% in physical or sexual intimate partner violence prevalence. How quickly we proceed will depend on the size and certainty of the effect.

Our current plan is:

We are sharing this now to anchor and commit as an organization to only continue if our programming meets our threshold. 

Note: AI was used to help writing and polishing this post, but ideas are ours.

Background on NOVAH’s cause area and intervention

Globally, 1 in 4 women are beaten or sexually assaulted by their partner in their lifetime, and 1 in 4 each year in Rwanda.

NOVAH was founded in 2024 to test whether radio edutainment could prevent domestic violence (“intimate partner violence”, IPV) cost-effectively. At that time, and still today, the intervention appeared particularly promising but faced a relatively thin literature.

Our radio drama, Twubakane (“Let’s build together”) follows couples dealing with recognizable challenges around money, alcohol, communication, gender roles and conflict. Rather than telling listeners what to do, the programme shows characters making mistakes, experiencing consequences and gradually changing their behaviour with the help of other role model characters.

In early 2025, our first six-episode season aired regionally and reached an estimated 30,000 listeners in Rwanda. 

The initial findings were encouraging. In qualitative interviews, listeners frequently described better communication, more joint decision-making and healthier relationships. A pilot study with 200 couples found a suggested six-percentage-point decline in women’s reports of physical or sexual violence—equivalent to a 40% relative reduction—as well as improvements in relationship quality and women’s participation in financial decisions. However, the study was small and not rigorously designed, so it could not provide definitive evidence of impact. We treated its findings as a reason to run a better test, not as proof that the intervention works. 

We decided to run a rigorous study, with the idea that positive or negative results could be particularly helpful both for us and the ecosystem. Thanks to the help of many partners and supporters, we were able to do a large-scale randomized controlled trial (RCT), in a naturalistic setting.

How we are evaluating our impact

We have now completed a second season of 12 episodes, which we recently aired regionally as the implementation of the RCT. We expect preliminary RCT results in October 2026. We deliberately held back national broadcasts until endline data collection was complete, so that control villages were not exposed to the programme before we measured them. Unless the results indicate that the programme is causing harm, we plan to air nationally shortly after the end-line. 

Although it is not the key objective of the post, we make a short detour to explain the RCT’s design, as we know mass media interventions are particularly hard to evaluate - feel free to skip this and move to the next section. 

The evaluation includes ~2,500 couples (5,000 participants), living across ~100 villages in Northern Rwanda. We distributed one solar-powered radio for each household and trained 2 youth volunteers per village to support the programme and encourage couples to listen each week, on top of small monthly incentives for couples to listen. Randomization took place at the village level: in half of the villages, couples are incentivised to listen to our program on a regional radio station, and others are encouraged to listen to an unrelated radio program airing on the country’s most popular radio station at the same time on Monday evening.

The evaluation is being conducted with Innovations for Poverty Action (IPA) and academic research partners. It measures a large range of outcomes and impact metrics from our theory of change, in particular physical and sexual IPV, subjective wellbeing, mental health, communication quality, gender attitudes, and alcohol use. The analysis plan is being preregistered publicly.

Our pre-commitment

Our central rule is:

We will proceed with our programming only if the RCT finds a statistically significant relative reduction of 5% or more in physical or sexual IPV.

For example, if IPV prevalence in the comparison group were 20%, a 5% relative reduction would correspond to prevalence of 19% in the intervention group. 

Note: The exact metric is any self-reported physical or sexual IPV in any of the two recall periods (last 4 months, last month). This bar is not a prediction of what the RCT will find. We will only consider statistically significant effects (depending on IPV’s baseline, the RCT may not be powered to detect a 5% relative reduction, as it has been designed to detect a 2pp effect). 

Even if IPV is reduced by more than 5%, if any safety concern arises (e.g. a negative effect for depression or a reduction in one form of violence alongside an increase in another) and/or we discover serious implementation problems or evidence of harm, we would not proceed with airing the show nationally.

Below, we first discuss our plans depending on results and this pre-commitment more broadly.

What happens under each result?

Effect below 5%, or not statistically significant: pause and explore

We would pause our expansion plans for a few months, and conduct exploratory research. We would then make one of two decisions:

1. There is a specific and credible explanation for the weak result, and an adaptation is sufficiently promising to justify another test; or
2. We shut down NOVAH (as this intervention) and openly share what we learned.

Effect of 5%–20%: continue as planned

We would implement our central programme for 2027: producing a new season in Rwanda and in Burundi, and exploring a third country of operations. We would continue developing processes and methodologies for others to replicate our work from 2028 onward. The planned budget for 2027 is approximately 550k$.

Effect above 20%: accelerate

Results would land in the "exceptionally good" category and we would aim to accelerate, by working in an additional country already in 2027 (for an approximately 750k$ budget).

Why did you choose a 5% reduction?

Our currently projected cost-effectiveness is ~$60 per IPV-year prevented, but we have set ourselves a bar of $200 per IPV-year prevented. This bar reflects the estimates from the current most cost-effective interventions in the field (*). 

(*) According to TLYCS, the implementation of SASA! by CEDOVIP achieves $150 but doesn’t take into account overhead organization costs. Another community mobilization program finds $194.

A 5% relative reduction in IPV, under our current assumptions, would make NOVAH meet the bar of $200-per-IPV-year averted. Under our current assumptions:

Is everything driven by this RCT?

These RCT results are only the first decision gate. A media intervention needs to both work and reach people cheaply. 

The RCT tells us whether Twubakane changes outcomes among the population offered and encouraged to listen - which seems the hardest challenge. 

It does not tell us how many people will choose to listen during an ordinary national broadcast. Our current model assumes 7% active listenership during a first national broadcast - a hypothesis we will test in the Q4 2026 airing and measure in a survey planned for January 2027. The combination of the two pieces should lead to a robust CEA.

Low listenership would be a different kind of problem from a null result. If the January 2027 survey shows listenership well below 7%, we are reasonably confident we can improve this: the main levers (station and time slot, co-broadcast partnerships, promotion, and adapting the show for a national rather than a regional audience) are ones we control. We would treat weak reach as something to fix rather than a reason to stop. A null or negative effect in the RCT would, for us, suggest that something more fundamental is not working in our intervention.

Why make this public?

Once an organisation has hired staff, developed partnerships, raised money and spent several years building a programme, there are strong reasons to keep going, and we do not think we are immune to that.

We are hopeful about the results (especially when reading listeners’ feedback) but we want to stay true to the commitment we made when starting: we want to design and scale an intervention that is more cost-effective than the current best alternative. Also, we do not want to use valuable donor money to scale interventions that don’t work.

How you can hold us to this

A pre-commitment is only worth something if someone outside the organisation can check it. Three things make ours checkable.

• We have pre-defined in advance exactly the metric that will be used.
• The estimate is produced by IPA and our academic partners, who are external to NOVAH. We do not compute the number we are judged against.
• We will publish the results on this Forum.

Should more non-profits do this?

We are also curious whether more evidence-focused NGOs should make similar public commitments, or whether the disadvantages outweigh the benefits.

There are pros (avoiding bias, being transparent, staying evidence-based) and cons (false precision from a numerical threshold, strategic optimization, etc.) for such communications. Also, pre-commitments are easier for direct work organizations than for research, policy, or ecosystem building ones.

We plan to share more on that after we get the results.

Final words

When the preliminary results become available, we will come back to this post and explain what they mean for NOVAH, including if the conclusion is that we should stop.

If you find this approach useful and would like to back it, we would welcome conditional support. Our conditional support plan for funders sets out what a commitment at each threshold would look like. It lets a funder commit now on the condition that the evidence clears the bar.

If you have any feedback on our approach, please get in touch.

Learn more and get in touch

• Read our newsletter and subscribe to our quarterly posts (also on LinkedIn)
  • Note: if the effect is 5% or higher, in ~October 2026, we would hire a Head of Research, responsible for the research parts of our Iterative Content Production process in Rwanda and in Burundi - subscribe to our newsletter so you don't miss the job ad!
• Volunteer
• Contact: ivy@novah.ngo and alexis@novah.ngo

Appendix

Why do you use IPV-year instead of DALYs in your cost-effectiveness?

We believe this metric best reflects our work, although this choice has been the result of tradeoffs between different options (see table). We also use WELLBYs and DALYs as complementary indicators.

NOVAH’s cost-effectiveness indicators and the tradeoffs they offer:

(*) Self-reported by women, but this follows all standards in the domain

(**) We work with the Happier Lives Institute to ensure we follow best practices in this domain - we are very grateful to them for their support and time

(***) We have recently updated our CEA, working with external researchers and revising assumptions. This led us to lower our estimate of the DALY burden caused by a year of IPV and we think the current version is more conservative and credible.