What you can do to help stop violence against women and girls

By Akhil @ 2023-01-12T11:51 (+333)

Introduction

I previously wrote an entry for the Open Philanthropy Cause Exploration Prize on why preventing violence against women and girls is a global priority. For an introduction to the area, I have written a brief summary below.  In this post, I will extend that work, diving deeper into the literature and the landscape of organisations in the field, as well as creating a cost-effectiveness model for some of the most promising preventative interventions. Based on this, I will offer some concrete recommendations that different stakeholders should take - from individuals looking to donate, to funders, to charity evaluators and incubators.

The key recommendations I make, in order of importance, are: 

1. Support community-based interventions that seek to ​​shift harmful gender norms and reduce violence- they have a high quality of evidence, and cost $180/DALY (disability adjusted life years) or $150 for a woman to live a year free from violence. Two particularly promising organisations are CEDOVIP and Raising Voices.
2. Fund and conduct a well-designed randomised control trial of radio or television dramas to shift harmful gender norms and reduce violence- they have a startling cost-effectiveness of $13/DALY or $11 for a woman to live a year free from violence, but currently lacks a well-established evidence base.
3. Fund organisations undertaking economic programs supporting women (e.g. microfinancing, cash transfers, village savings and loans association) to add on social empowerment programs focused on reducing violence; they have a cost-effectiveness of $180/DALY, or $145 for a woman to live a year free from violence.
4. Found new charities focused on community-based interventions that seek to ​​shift harmful gender norms and reduce violence, particularly in neglected geographies and populations
5. Consider supporting self-defence training programs - although analysis suggests that self-defence training may not be as cost-effective as other interventions for VAWG, it is nevertheless a relatively cost-effective (at $260/DALY or $215 for a woman to live a year free from violence) and potentially scalable intervention. No Means No Worldwide and Ujamaa appear to be two organisations scaling this intervention well.
6. Fund academic research to understand what types of culture change programs targeting boys and men are most effective at reducing violence against women and girls

You can find a 2 page summary of my initial post and this post here

Why VAWG is an important cause area

• Nearly one third of women and girls aged 15 years of age or older have experienced either physical or sexual intimate partner violence (IPV) or non-partner sexual violence globally, with 13% (10–16%) experiencing it in 2018 alone (Sardinha et al 2022). It is one of the leading burdens of disease globally, responsible for 8.5 million DALYs and 68 500 deaths annually. 
• In several countries, violence against women is in the top 3-5 leading causes of death for young women aged between 15 and 29 (Mendoza et al 2018).
• VAWG has wide-ranging effects on women’s physical, sexual and mental health- in fact, it is responsible for 11% of the DALY burden of depressive disorders and 14% of the DALY burden of HIV in women (IHME 2019)
• Globally, the rates of VAWG are both alarmingly high and have increased over the last 30 years, despite gains in other areas of women’s health, such as maternal care (Think Global Health).
• In 2016, the global economic cost of violence against women was estimated by the UN to be US$1.5 trillion, equivalent to approximately 2% of the global GDP (UN Women 2016).
• Although there are many groups working to stop VAWG, it is fairly neglected relative to the scale of harm that it causes (Ellsberg et al 2015). In addition, a lot of work may not be prioritising interventions that have the greatest impact (What Works, 2019).
• Please see my original post for more details

Summary of interventions

In my initial post, I argued that interventions that seek to prevent violence from occurring in the first place are the most likely to be cost-effective (as opposed to interventions that support women who have already experienced violence or which treat the effects of violence, which undoubtedly are also very important).  Here, I evaluate some of the most promising types of primary prevention interventions for VAWG^[1], evaluating them and establishing some concrete recommendations. 

The main metrics that were considered were the quality of evidence for the intervention (a subjective assessment based on a non-exhaustive literature search), and the cost-effectiveness, which is reported in both cost per disability adjusted life year (DALY)^[2] and cost  for a woman to live a year free from violence. For more detail, the full spreadsheet model can be found here. It should be noted that the estimates of the cost for a woman to live a year free from violence is directly taken from the literature on the benefits and costs of interventions, whilst the estimates for cost/DALY are based on a more speculative model of the economic and health harms of IPV.

 Of course, these cost-effectiveness calculations cannot capture many of the greatest harms of violence- its infringement on women’s rights, and many of the broader societal level effects that violence against women and girls has.  Despite this, these models are useful in comparing interventions in this field to other areas. In that light, and for reference, Givewell, a charity focused on recommending the most cost-effective global health charities, has a ‘bar’ of $125/DALY to consider and potentially recommend charities.

For in-depth detail on how the cost-effectiveness analysis model was created, please see Supplementary Material 1.

Details by interventions

Community activist social empowerment

Social empowerment programs to reduce VAWG broadly aim to shift harmful attitudes, roles and social norms in relation to VAWG. This encompassess programs in many different settings (e.g. workshops at schools, hospitals, in the community)  with different content (e.g. didactic, self-determined) and of differing lengths (e.g. one-off sessions, 6 month programs, longer interventions)

One of the most promising types of social empowerment interventions  is community-level activist programs. This approach is built on the premise that community-led and community-level work can enable sustained change on VAWG prevention at a population level (Michau et al, 2015). These programs are characteristically:

• Based in the community, rather than a specific setting like a school or hospital 
• Train and utilise local community members who act as activists, rather than didactic instructors – Community activists are supported by manuals and other materials to enable structured or guided engagement with men and women in their community
• Involves community activists increasing local knowledge about VAWG and positively changing social and gender norms, directly supporting survivors or engaging couples with known problems with IPV, providing counselling and support to couples. They often work with local religious and traditional leaders and state actors, such as the police, health and social services, to strengthen their responses to survivors and influence their attitudes and practices in their work. 
• Are carried out over a medium to long-term time frame (loosely defined as more than 12 month program duration).

Although many other program types were identified, they were ultimately not thoroughly investigated in this report as it seemed unlikely that they would be as promising as community activist programs, which have the strongest evidence base and effect size. 

A non-exhaustive literature search of all social empowerment programs was conducted, through which community activist programs were selected for. (the details of all programs identified, their results, and their reason for inclusion exclusion from this analysis, can be found here). 

From this, there are several studies evaluating community activist social empowerment programs- Abramsky et al (2014), Dunkle et al (2020), Leight et al (2020), Wagman et al (2015), Ogum Alangea et al (2020), Le Roux et al (2020), Chatterji et al (2020). They differ in the specific community activist model they used, including SASA!,^[3] Unite for a Better Life (UBL),^[4] and rural response systems (RRS).^[5] Based on these studies , community activist based programs have an established evidence base, and do lead to significant reductions in violence. In estimating the cost-effectiveness of this intervention,^[6] community activist based social empowerment programs were found to have a cost-effectiveness of $180/DALY, or $150 for a woman to live a year free from violence.

In addition to the strong evidence base and cost-effectiveness of the intervention, it also appears that the intervention is relatively scaleable. In particular, SASA! has created an activist kit, and works with regional partners and NGOs to implement the program; two particularly promising organisations, CEDOVIP and Raising Voices, are providing technical assistance to other NGOs and governments and themselves scaling up SASA! programs (e.g. Namy et al 2019). They seem particularly promising organisations to support.^[7] 

In conjunction with supporting existing organisations in scaling up community activist programs, and given the relative dearth of organisations using this approach, it also seems prudent to found new charities in the area. Given that in any given year, approximately 200 million women are affected by violence, it is almost certainly the case that the amount of organisations working in the space is neglected. A brief geographic assessment of particularly neglected and tractbale geographies where new organisations was carried out (more details on the methodology and the results can be found in Supplementary Material 2); based on this, it seems like particularly promising areas to start new charities might include:

• Central/southern Africa- Zambia, Angola, Zimbabwe, Tanzania and Botswana
• Asia Pacific region- East Timor, Fiji, Papua New Guinea
• Asia- India
• Other- Kuwait

Self-defence programs

Another common intervention is self-defence programs, most of which are based on the evidence-based IMPower program. They typically involve 10-15 hours of  physical self-defence, as well as many areas of non-physical defence and training, such as verbal de-escalation and negotiation, assertive communication and personal awareness;^[8] they are conducted over several weeks to months. They are most commonly targeted towards school-aged girls (and often boys as well).

There are several cohort studies and RCTs of IMPower - Sinclair et al (2013), Sarnquist et al (2014), Baiocchi et al (2017), Decker et al 2018, Edwards et al 2020, Senn et al (2013)-  from which the estimate of the effectiveness of the intervention were derived, as well as several other studies for the persistence of the effects-  Senn et al 2017, Hollander et al 2014. Although more details on these are described in the cost-effectiveness model, it seems that these programs may  reduce the prevalence of violence by approximately 45%, with an annual effect persistence of 20-30%. However, moderate to high risk of bias in the included studies, possible methodological flaws in the way questions were asked, as well as my priors led me to discount the effectiveness of this intervention slightly more than other interventions.

In the model, self-defence programs have a cost-effectiveness of approximately $260/DALY, or $215 for a woman to live a year free from violence. This is reasonably cost-effective, but perhaps not as cost-effective compared to some of the interventions that are presented in this report. However, one comparative advantage is that there seem to be NGOs with a proven track record that could scale IMPower programs. Such NGOs include No Means No Worldwide (NMNW),^[9] which has reached over 2 million school aged children in 2021 and Ujamaa, which has significantly scaled up in schools in Kenya. Although I did not conduct a comprehensive literature search, it seems that there are a number of other organisations who also teach self-defence, some of which use IMPower; however, there was less data on them, and a lot of them appear to be local or opportunistic, such as this self-defence program in a refugee camp and this organisation in Lucknow, India.

Mass media campaigns

Mass media campaigns are a promising intervention carried out by governments and organisations around the world to effect behavioural and attitudinal change. They can operate via various media including radio, TV and the internet. They are a potentially cost-effective and scalable intervention that have a growing evidence base in a number of different areas, from HIV prevention to family planning (Banerjee et al., 2019b, Glennerster 2021 et al 2021, Sarassat et al 2018, Family Empowerment Media, Development Media International); for more information, the following cause area report from Founder’s Pledge is a good introduction.

There are several  mass media campaigns that have been undertaken for the purpose of reducing VAWG- broadly, they have two goals; to reduce violence from occurring, and by making victims of violence more aware of support services available to them. I focused on studies aiming to prevent violence from occurring. 

I conducted a non-exhaustive search of the literature in this area; studies vary in the media format and their target audience (Lapsansky and Chaterjee 2013, Paterson et al 2000, Arias 2018, Khosla et al 2013, Simons and Khan 2019). Indeed, one observational, correlative study even found that the introduction of cable television in India reduced the acceptability and rate of VAWG (Jensen and Oster 2009). It seems like the most cost-effective is likely to be serialised drama radio programs targeting both women and men,^[10] on which there is one RCT Green et al (2020). This study found a significant increase in willingness of women to seek help in the event of IPV, and a 25% relative reduction in rates of violence experience (as reported by women). This study seems significantly higher than studies in other mass media campaigns for sexual behaviour and cohort studies in the space (Synder et al (2004), Jesmin and Amin (2017), Wray et al (2004)), which generally show a relative reduction of 3-5%.

In estimating the cost-effectiveness of this intervention, a relatively conservative estimate of a 7% relative reduction and moderately strong discounts for internal and external validity were applied. In spite of this, mass media campaigns appear to be a very cost-effective intervention, with a cost/DALY of $13 or $11 for a woman to live a year free from violence.

There are a few important caveats that I want to mention here:

• The evidence base here is less strong than for other interventions
• The type and implementation of the mass media campaign is likely to be fairly instrumental in its acceptability and impact (Walker 2021)
• Although one of the appeals of mass media campaigns is their scalability, I feel less confident, due to social and cultural differences in the circumstances within which violence occurs and the political attitudes towards gender roles, that the same type of media campaigns in one setting  would work in another (as opposed to mass media campaigns for typical global health interventions, like vitamin supplementation or vaccinations). However, I I think to a large extent,  appropriate co-design of campaigns with local stakeholders (Stanley et al 2017), including script writing of the media, could mitigate this to a significant extent

Overall, and as discussed above, mass media campaigns could be a promising intervention in reducing VAWG; however, there is insufficient evidence at this time to understand how much effect it has, and how persistent its effects are. I would recommend funding an experimental study, ideally a cluster randomised RCT, which investigated the effects of  a drama-style radio campaign on the incidence of violence against women and girls. Ideally, the study would have a relatively large sample size and have a follow up duration of at least 24 months. This study could be undertaken by an organisation that is already carrying out mass media campaigns, such as Family Empowerment Media or Development Media International). 

It is relevant to note that Innovations for Poverty Action (IPA) are currently running an RCT of a radio campaign in Tanzania with Uzikwasa (details here), but on balance it seems like more evidence here would be quite useful in determining the effectiveness and cost-effectiveness of this intervention.  However, it does also seem plausible that  the value of an RCT might be relatively low (especially given their high cost and the difficulty of running a well designed mass media RCT), and that just directly funding an organisation to run mass media campaigns might be a reasonable option.

Social empowerment added on to economic initiatives for women

There are a number of financial empowerment interventions focused on supporting women- from microfinancing loans for women, savings and lending groups (VSLA) to unconditional and conditional cash transfers. By themselves, these economic interventions may reduce IPV (Del Campo et al 2022); however, there is increasing interest in the impact that adding on a structured, social empowerment intervention alongside the financial empowerment program has on rates of VAWG .  

This are several RCTs looking at this topic- Pronyk et al (2006), Kapiga et al (2019), Green et al (2015), Gupta et al (2013), Halim et al (2019), Ismayilova et al (2018), Bulte and Lenslnk (2019) and Iyengar and Ferrari (2016).^[11] These social empowerment programs were quite heterogeneous in their structure, quality and strength of implementation, but mostly were short term (2-6 month), workshop-based gender training focused. Most of these studies found a modest reduction in VAWG in their intervention arm (finance + social empowerment) compared to their control arm (financial intervention only). 

To estimate the cost-effectiveness, the effect of economic initiatives on IPV was first ascertained, taking the effect size from a recent meta-analysis (Del Campo et al 2022), which was then accounted for in the baseline prevalence of IPV in the study population. Then, through analysis of the literature on the additional relative reduction of VAWG from adding on a social empowerment program to an economic initiative, the estimated cost-effectiveness was $180/DALY, or $145 for a woman to live a year free from violence.

Culture change programs targeting men

There were a number of programs that aimed to change gender norms, attitudes and behaviours of boys and men only- Hosssain et al (2014),Christofides et al (2020), Valiant et al (2020), Pulerwitz et al (2015), Miller et al (2012), Miller et al (2014), Kalichman et al (2009), Jewkes et al (2015). Overall:

• The evidence from the studies is conflicting and inconclusive.
• The programs varied in who they targeted (bystanders, school-aged boys, men who had previously been violent) and their duration. It may be that certain groups of men are more responsive to culture change programs  (Gibbs et al 2020).
• Overall, although working with men and boys may be a useful approach and is likely an important priority, at this stage, it is somewhat unclear which groups of boys and men, and what types of interventions, are most effective

What can you do?

Overall, this in-depth investigation has updated me positively on the importance of funders, including Open Philanthropy, investing in VAWG. In light of that:

• If you are an average punter: Engage with this post and the field more broadly, consider making personal donations to the organisations identified in this post- CEDOVIP,  Raising Voices, as well as No Means No Worldwide and Ujamaa^[12]
• If you are a funder: Consider funding these organisations and studies recommended above within your existing programs, or having a new program focused on VAWG
• If you are a charity evaluator: Conisdering adding charities focusing on VAWG, especially those identified here, on your list of recommended charities.
• If you are an entrepreneur: Consider starting a charity in one of the areas of need identified within the report

Acknowledgements

 I would like to acknowledge the following individuals, who provided feedback on an earlier draft of this report: Ilona Arih, Jack Rafferty, Sarah Hough, Leonie Falk, Ben Williamson, Jessica Leight, Vicky Cox, Sam Hilton, Rosie Bettle, Sanjay Joshi, Michael Townsend. All mistakes are my own.

Supplementary Material

Supplementary Material 1- Cost-effectiveness model

There were three steps in creating this cost-effectiveness model:

1. Assess program benefits using a standardised formula
2. Quantify the health and economic impact of interventions to reduce VAWG, per woman affected by violence
3. Assess program costs using a standardised approach

1.Program benefits

The second step was to look at different types of interventions and collected several data points on benefit- 

1. Prevalence rate reduction as a result of the intervention (looking at years free from IPV as the primary outcome)
2. Persistence of the effects beyond the program (if any)- do the effects of the program last for more than one year
3. Spillover effects (if any)- Do the benefits of the program extend to anyone not actively involved or studied in an intervention
4. An internal and external validity adjustment, which was a subjective assessment made by the author based on the quality of the evidence for a given intervention

These estimates, based on the existing literature in the area, are likely within a factor of 2-3 of accuracy.

2. Quantify the impact of interventions to reduce VAWG

The outcomes measured in studies and organisations working on VAWG differ significantly; however, in most of the literature, intervention outcomes are reported in terms of prevalence of years free from intimate partner violence (IPV). As a result, the first step was to quantify the health and economic benefits of a year free from IPV, such that the impact of interventions in this space could be cross-compatible with other cause areas and interventions. This quantification is inherently a difficult task that will reduce the harms of VAWG; despite this, it is important that such calculations are attempted to be able to compare interventions in this area to other global health interventions. However, we should certainly take these estimates with a grain of salt, and with an acknowledgement that they don't capture a lot of the nuances and  complexities of this area. 

Based on the quality of evidence, and particularly causal evidence, I looked at the health and income effects of VAWG. 

Health impact
For the health impact (see this sheet) I used three different approaches to estimate the DALY-burden per woman affected by a year of IPV. I then gave a weighted average of these three estimates, giving more weight to estimates I thought were more accurate. Below are the respective methods and reasons behind the weights:

1. IHME Data 
   1. Description: The IHME data on the prevalence and DALY burden of IPV was used to estimate the burden/individual per year affected by IPV. The data source only included the relative risk of anxiety and depression, interpersonal/physical violence and HIV in its calculation of the DALY burden of IPV, and my prior was that it would therefore be a very low approximation of the burden of IPV
   2. Estimates 0.06 DALYs per year of IPV. 
   3. I assigned a weighting of 40% to this because, although it is likely to be an underestimate, it is from a highly reputed source.
2. Ferrari et al 2022 estimate
   1. Description: Estimate from recent economic evaluation study of different IPV interventions
   2. Estimates 1.60 DALYs per year of IPV
   3. I assigned a weighting of 20% because the methods used to estimate this burden were not completely clear, and this method has not been validated.
3. Personal calculation of relative risks
   1. Description: A personal calculation of the relative risk of different health conditions for women who have experienced IPV versus the general population. This was then used to generate an estimate of the total health, DALY burden attributable to IPV.
   2. Estimates 1.13 DALYs per year of PIV 
   3. I assigned a weighting of  of 40% to this because, although it seems the most comprehensive of the three approaches, it has not been validated.

These estimates showed significant variance of  factor of 25, which suggests that any such estimate has a confidence interval of at least one order of magnitude. My impression is that there are a number of longer-term health outcomes and benefits that are not captured here, that an estimate closer to the personal calculation of relative risk or Ferrari et al 2022 estimate are most accurate.

Economic impact

For the economic impact (see this sheet), the effects of IPV on productivity, workforce participation and healthcare and legal costs incurred were included. A conservative estimate assumed that these effects persisted for one year after women were affected by violence, and the economic harm in that second  was half that of the first year. The Open Philanthropy methodology and moral weights to convert income effects to DALYs was then used.

These estimates, which are fairly conservative, are likely the lower bound of an estimate that has a one order of magnitude confidence interval. This is because these estimates take a conservative estimate of female earning and how much it might be impacted by violence, and  modest estimate of long-term impact of violence.

3. Cost

The third step was to estimate the cost; for this, estimates from NGOs or academic studies that had published cost data for that intervention type were used. If appropriate. Initially, a scaling adjustment was added in the event that it was anticipated that an intervention might become cheaper/more expensive as it scaled up. This adjustment was not applied due to insufficient data to determine.

Supplementary Material 2-  Geographic Assessment

In order to consider which geographies and contexts it might be promising to found new charities working on interventions for VAWG, especially community activist interventions, a weighted factor model to assess potentially promising countries was created.  This involved considering several factors that might make a country more or less promising to work in, as summarised in the table below:

Based on this, the 10 most promising countries were:

• Central/southern Africa- Zambia, Angola, Zimbabwe, Tanzania and Botswana
• Asia Pacific region- East Timor, Fiji, Papua New Guinea
• Asia- India
• Other- Kuwait

1. ^{^}

    The following report, produced as a result of the UK DFID’s What Works program, explores additional preventative interventions for VAWG, such as working in antenatal/postnatal settings, interventions targeting sex workers, and others. I have focused on the most evidence-based and promising interventions, which is a subset of those. 

2. ^{^}

    Result presented as cost/DALY, but includes both health and income gains converted using Open Philanthropy’s most recent moral weights.

3. ^{^}

     SASA! Is a community mobilisation program developed by Raising Voices; the activist kit which the program is based on has four stages - Start Phase, where community volunteers are recruited and trained, awareness phase- which comprises tools to engage the community and raise awareness of men’s power over women, support phase- guides communities in how to offer support to one another, and action phase- suggestions and tools on how to engage more people from more circles of influence and reach to change broader community norms (more here)

4. ^{^}

    Unite for a Better Life (UBL) is a gender-transformative, participatory intervention delivered to men, women and couples in Ethiopia in the context of the coffee ceremony, a traditional forum for community-based discussion. The programme aims to reduce physical and sexual IPV and HIV risk behaviours as well as promote healthier, more equitable relationships (Sharma et al 2020)

5. ^{^}

    Developed by the Gender Centre,  community based action teams are selected by community members and the Gender Centre. They work together to undertake community sensitization and awareness raising, and work to change norms in the community, such as at community festivals, weddings, schools, religious groups and others. There are many similarities between this and the SASA! Program (Addo-Lartey et al 2019)

6. ^{^}

    It is important to note that the community activists are unpaid volunteers, which accounts for some of the low cost of this intervention. This may lead to some concerns about the ethics and long-term sustainability of this intervention and model, but my impression is that this is relatively scaleable and that the ethical and practical constraints of a volunteer led program for such an intervention are relatively minor and can be mitigated.

7. ^{^}

    I have reached out to a few of these organisations in the last week, to get a better understanding of their work and their organisation, as well as their plans to scale and room for more funding. Of course, further and in-depth evaluation would be yet to be carried out, but directionally I am supportive of these.

8. ^{^}

    It is unclear to me from the literature or from my work in the space whether the physical self-defence or the non-physical defence and training has more impact. My intuition is that many of the assertive communication and verbal de-escalation training may be where a significant amount of the impact comes from. This would be interesting to further investigate, as it might affect what specific type of intervention we fund, and because it might be easier to scale a program that has more of one component than the other.

9. ^{^}

    NMNW has undergone a shallow investigation by both Founder’s Pledge and SoGive (publication forthcoming). For some data e.g. on the cost of the intervention per participant, I benefited from their notes. Other than that, all the analysis here is independent.

10. ^{^}

     My prior was that mass media campaigns that were more substantive were more likely to be effective than campaigns with less ‘detail’ e.g. poster campaigns, television advertisements

11. ^{^}

     For scope sensitivity, studies that looked at social empowerment initiatives for women at their workplace or in a set employment were not included; such as Adia et al (2019), Naved et al (2021). Several studies also had an economic livelihood and social empowerment training program, rather than a specific financial support (e.g. Gibbs et al (2019), Gibbs et al (2020))- these were also excluded.

12. ^{^}

     I have not yet spoken to these organisations, or done a deeper analysis of the organisations themselves yet, but would recommend prioritising them for further investigation. However, based on the external facing information they have on their work, they look like promising bets.

Sanjay @ 2023-01-14T16:15 (+30)

Great that you've looked into this Akhil! Speaking as someone with a wife and daughter (and a mother, and other female family members, and female friends...) this is close to my heart.

A key problem with all of these is how to assess effectiveness. IPV typically occurs behind closed doors, which makes it hard to know what's really happening.

Largely because of these considerations, I predict that on further analysis, I will probably be less positive than you. 

While this sounds consistent with a generalised GiveWellian sceptical prior, I say this with some sadness, because I would very much like reducing VAWG to be a high impact cause area.

Also, thank you for asking me for comments before publishing.

---

My main reason for being more pessimistic than you is that your internal and external validity adjustments seem very generous:

Source: your model

For brevity, I'll focus on Community based social empowerment, since it's the one you're most positive about.

• You have adjustments of 95% internal validity (aka replicability) adjustment, and 90% external validity (aka generalisability) adjustment^[1]. I'd consider these numbers to be high (i.e. more prone to lead to generous cost-effectiveness evaluations)^[2]. 
• Your model's 95% internal validity adjustment is the same internal validity adjustment that GiveWell uses for bednets. For comparison... 
• ... malaria nets do merit a 95% internal validity adjustment. We have seen plenty of positive evidence for the effectiveness of bednets, and I'm told that there is so much evidence that it's difficult to get ethics approval for more RCTs because ethics boards argue that it's unethical to do studies with controls on something that is such a robustly proven intervention.
• ... cash transfers do merit a 95% internal validity adjustment. They are a robustly effective way of reducing poverty.
• ... Community Based Social Empowerment does not merit a 95% internal validity adjustment, in my view. Gathering this sort of evidence from surveys is very difficult, and I'd be surprised if the protocols are robust enough to give us the same confidence we have about the effect of malaria nets on mortality (deaths are relatively easy to count).
• I also suspect the external validity adjustment is too generous. The intervention relies heavily on cultural context; several GiveWell external adjustments are high too, but human bodies are pretty consistent from one place to the next, whereas cultures vary a lot with geography.

Therefore I predict that:

• in 90% of worlds where I (or someone from SoGive) sat down and reviewed this carefully, we would have validity adjustments lower than yours (i.e. lower than 95% and 90%).
• in 50% of worlds where I (or someone from SoGive) sat down and reviewed this carefully, we would have validity adjustments substantially lower than yours (i.e. lower than 50%).
• In summary, I think there's a 75% chance that we conclude with a >2x worse cost-effectiveness than you, and a 25% chance of a greater than >4x worse cost-effectiveness than you for Community Based Social Empowerment.
• This would be unlikely to be at the levels of cost-effectiveness where we would deem the intervention high impact.

I haven't thought enough about the other interventions apart from Self-defence (IMPower, which has been done by No Means No). As Matt has alluded to, SoGive has done some work on this topic, and received some information which is not in the public domain. I can't say too much about this, but I can discuss privately and guide you to the relevant researchers. SoGive's plans are to press for permission to publish on this, and finalise within the next few months.

---

For clarity, I've alluded to SoGive in this comment, but this is not an official SoGive comment. Content written in a SoGive capacity has to gone through a certain level of review which has not happened here, so this is written in a personal capacity.

 

1. ^{^}

   For those less familiar with these models, they are applied in a straightforward, intuitive way. It's roughly equivalent to (Step 1) Calculate the benefit assuming full trust in the evidence; (Step 2) Multiply the benefit by the validity adjustments; (Step 3) divide by costs.

2. ^{^}

   For those who want access to data to help them form their own view on whether these adjustment are high are not: In SoGive, we have pulled together a spreadsheet with GiveWell's internal and external validity adjustments (we're supposed to also add in SoGive's own adjustments at the bottom, not just GiveWell's, but have been less diligent at doing that). It's meant to be a (not-rigorously vetted) internal resource, but I'm sharing it here in case it helps. It's also probably a couple of years out of date now, but I'd from memory I don't think there are changes material enough to matter in the last couple of years.

Akhil @ 2023-01-14T16:44 (+9)

Hey Sanjay, thanks for your comment

1. Internal validity-I think it is important to bear in mind there are a number of high quality RCT's conducted with low quality of bias- given that, although this might not feel like it has as strong an evidence as bednets or vaccines, it does have a strong evidence base. You can see the cRCTs here-  Abramsky et al (2014), Dunkle et al (2020), Leight et al (2020), Wagman et al (2015), Ogum Alangea et al (2020), Le Roux et al (2020), Chatterji et al (2020).
2. On surveys- To summarise, the concern you raise is that the programs aim to reduce social acceptability of VAWG, and mostly use local interviewers to assess incidence of violence, which may introduce social acceptability bias. Although this seems plausible, interviews were trained, conducted privately and used validated questionnaires. To quote one paper

The study was conducted in accordance with WHO guidelines for the safe and ethical collection of data on violence against women [[24]]. These guidelines seek to minimize reporting biases and risk of harm to both respondents and interviewers. At both baseline and follow-up, interviewers received at least three weeks of training on the ethical and methodological issues surrounding the conduct of a survey relating to IPV and HIV, as well as ongoing support during the course of the survey. Interviewers were all from the local area, and interviewed respondents of the same sex as themselves. Interviews were conducted in private settings, in Luganda or English, and were concluded by providing information on additional support services in the area. At baseline, interviewers conducting the baseline survey were blinded as to the allocation of the intervention. It was not, however, possible to keep follow-up interviewers blinded.
 

As a result, I think this risk of bias is quite low. Also, I think that inherently, any impact evaluation of interventions in this space would require surveys.

3. External validity- A  valid concern. I have two comments: (1)There are a number of studies in different settings which show positive results, suggesting external validity.  (2) Although cultural and social drivers of violence vary, the intervention is co-designed with community and quite locally tailored, which mitigates some of the concern around external validity.

4. Meta-comment- I think that some of my estimates of the persistence of effects were quite conservative, which may counterbalance slightly smaller discounts for external and internal validity

Sanjay @ 2023-01-15T18:42 (+5)

Re item 4, it's fair to note that I haven't checked how conservative you've been on other assumptions, so if I did a replication of your work and it ended up being similar, then I agree that could be a reason.