Family Empowerment Media: track record, cost-effectiveness, and main uncertainties

By Rethink Priorities, Greer Gosnell, Melanie Basnak🔸 @ 2023-06-12T09:36 (+98)

This is a linkpost to https://rethinkpriorities.org/publications/family-empowerment-media

Editorial note and disclaimer

This report assesses the nonprofit Family Empowerment Media (FEM). The project was commissioned and supported by a donor who wishes to remain anonymous. It was conducted in February 2023 over two weeks.

This report attempts to estimate the cost-effectiveness of FEM, mainly relying on two existing cost-effectiveness analyses (from GiveWell and Founders Pledge). We were asked to look into the cost-effectiveness of the organization in an unbiased way, and we started the research without a prior for whether this organization would be cost-effective or not. In this report, we express cost-effectiveness as a function of the cost-effectiveness of cash transfers (like the original models mentioned above). We did not analyze the cost-effectiveness of any other family planning interventions, so we cannot and do not make any claims about how FEM compares to other family planning interventions.

In addition to looking at the cost-effectiveness of FEM’s program, we describe their track record to date, and try to determine the main uncertainties about the program and its (cost-)effectiveness.

We relied on four expert interviews to attempt to answer some of the questions: We interviewed Anna Christina Thorsheim (Executive Director and cofounder of FEM), Andrew Martin (Senior Research Analyst at GiveWell), Rosie Bettle (Applied Researcher at Founders Pledge), and Dr. Mukhtar Muhammad (medically trained expert in social and behavior change communication with a historical focus on family planning programming).

We have tried to flag major sources of uncertainty in the report and are open to revising our views as more information becomes available.

We haven’t attempted to assess the value of different population ethics views, or how those would affect the (cost-)effectiveness of FEM’s work. We believe that that is a highly complex topic that would take more time than the short period we had to conduct this research. Work on this would benefit from the Worldview Investigations Team at Rethink Priorities, which could explore family planning topics in the future.

We would like to mention that Melanie Basnak previously worked at Charity Entrepreneurship (CE), the nonprofit organization that incubated FEM. Melanie did not overlap with Anna Christina Thorsheim during her time at CE and believes she oversaw this research in an unbiased manner.

Executive summary

In a randomized controlled trial in Burkina Faso from 2016-2018, a family planning mass media campaign led by the nonprofit Development Media International (DMI) led to a 5.9 percentage point (20%) increase in the modern contraceptive prevalence rate (mCPR), leading the study authors to estimate a cost effectiveness of $7.70 per couple-year protection, albeit with uncertainty. The nonprofit Family Empowerment Media (FEM) ran a pilot family planning mass media campaign in Kano, Nigeria, and they observed a 6 percentage point (75%) increase in mCPR using less rigorous pre- and post-pilot data comparisons. GiveWell conducted a cost-effectiveness analysis (CEA) of DMI’s intervention, finding it to be 1.1-5.4x the cost-effectiveness of cash transfers, depending on the context. FEM altered the model to pertain to its pilot intervention in Kano, finding that their intervention was ~27x the cost-effectiveness of cash transfers. Founders Pledge conducted a separate CEA, finding a cost-effectiveness of ~22x that of cash transfers, complete with a full (unpublished) report.

We used the original models as a starting point to conduct our own cost-effectiveness estimation, tweaking certain values. In most cases, our adjustments would tend to increase these existing cost-effectiveness estimates, so we are confident that the cost-effectiveness of FEM’s intervention beats that of cash transfers, and we believe that their cost-effectiveness might be higher than GiveWell’s and Founders Pledge’s estimates. However, we would like to see several remaining uncertainties resolved before we would feel confident putting a bound on cost-effectiveness, and we discuss these uncertainties in some depth. For instance, we are highly uncertain regarding the persistence and magnitude of the treatment effect, as well as the nature and magnitude of spillover costs (e.g., to governments) and benefits (e.g., for women’s education, child health, animal welfare, and climate outcomes) at scale. We hope that FEM’s research will take these uncertainties into account, and we believe that further data exploration and conversations with experts could additionally help to reduce them.

Click here for the full version of this report on the Rethink Priorities website.

Contributions and acknowledgments

Greer Gosnell and Melanie Basnak were the main authors of this report. Melanie Basnak reviewed and supervised this report. Thanks to James Hu for assisting with the citations. Thanks to Adam Papineau for copyediting and to Rachel Norman for assistance with publishing the report online. Further thanks to Anna Christina Thorsheim (FEM); Andrew Martin (GiveWell); Rosie Bettle (Founders Pledge); and Dr. Mukhtar Muhammad for taking the time to speak with us.

If you are interested in Rethink Priorities' work, please consider subscribing to our newsletter. You can explore our completed public work here.

Devon Fritz @ 2023-06-12T10:58 (+26)

This is really great to hear!

Can I ask roughly how much it costs to commission such a report? I'd love to see more investigations into different organizations to get more robust data on how impactful they are.

Melanie Basnak @ 2023-06-15T17:19 (+10)

Thanks for asking Devon!

On the margin, our commissioned reports cost ~$7,000-$30,000 depending on the scope (this particular report being on the lower end of that spectrum). That being said, we don't take in every commissioned work that comes our way and only have the capacity to take on a few projects at this price because: (1) we have longer term agreements with a couple of organizations for which we do a lot of work, (2) we want our team to have the capacity to pursue independent projects we think are promising.

That being said, you're always welcome to reach out if there are specific projects you are interested in. We always welcome new ideas and look forward to connecting with new individuals/organizations :)

Ariel Simnegar @ 2023-06-12T20:44 (+21)

The sign of the effect of FEM seems to rely crucially on a very high credence in the person-affecting view, where the interests of future people are not considered.

In Kano, Anambra, and Ondo, FEM prevents one maternal death by preventing 281, 268, and 249 unintended pregnancies respectively. Even if only ~40% of these unintended pregnancies would have counterfactually been carried to term (due to abortion, replacement, and other factors), that still means preventing one maternal death prevents the creation of ~100 human beings. In other words, FEM's intervention prevents ~100x as much human life experience as it creates by averting a maternal death. If one desires to maximize expected choice-worthiness under moral uncertainty, assuming the value of human experience is independent of the person-affecting view, one must be ~99% confident that the person-affecting view is true for FEM to be net positive.

However, many EAs, especially longtermists, argue that the person-affecting view is unlikely to be true. For example, Will MacAskill spends most of Chapter 8 of What We Owe The Future arguing that "all proposed defences of the intuition of neutrality [i.e. person-affecting view] suffer from devastating objections". Toby Ord writes in The Precipice p. 263 that "Any plausible account of population ethics will involve…making sacrifices on behalf of merely possible people."

If there's a significant probability that the person-affecting view may be false, then FEM's effect could in reality be up to 100x as negative as its effect on mothers is positive.

Even if one rejects the person-affecting view, but supports FEM for its (definitely positive) effects on farmed animals, they should then be sure to not support lifesaving charities like AMF, which have the opposite effect on farmed animals. They should also find FEM saving mothers' lives to be an unfortunate side-effect of FEM's intervention, because saving mothers' lives is bad for the farmed animals the mothers eat.

Also, more of the farmed animals helped by reducing the human population don't exist yet and will be created in the future. So it's curious that one would account for the interests of farmed animals that don't exist yet, but ignore the interests of human beings that don't exist yet. (To be fair, there are views like the procreation asymmetry which could justify this.)

On the whole, whether or not there's a significant probability that the person-affecting view may be false seems to be a crucial consideration for the sign of the effect of family planning charities such as FEM and MHI. I'd be interested in how Rethink Priorities would approach incorporating moral uncertainty regarding the person-affecting view into its report on FEM.

Edit: Added "assuming the value of human experience is independent of the person-affecting view" for precision, as MichaelStJules pointed out.

Mart_Korz @ 2023-06-13T20:47 (+10)

I am just coming from a What We Owe the Future reading group - thanks for reminding me of the gap between my moral untuitions and total utilitarianism!

One reason why I am not convinded by your argument is that I am not sure that the additional lifes lived due to the unintended pregnancies are globally net-positive:

on the one hand, it does seem quite likely that their lives will be subjectively worth living (the majority of people agrees with this statement and it does not seem to me that these lives would be too different) and that they would have net-positive relationships in the future.
but on the other hand, given a level of human technology, there is some finite number of people on earth which is optimal form a total utility standpoint. And given the current state of biodiversity loss, soil erosion and global warming, it does not seem obvious that humanity is below that number^[1]
as a third part, given that these are unintended pregnancies, it does seem likely that there are resource limitations which would lead to hardships if a person is born. We would need to know a lot about the life situation and social support structures of the potential parents if we wanted to estimate how significant this effect is, but it could easily be non-trivial.
edited to add and remove: the number of 100 pregnancies averted does not correspond to 100 fewer children being born in the end. A significant part of the pregnancies would only be shifted in time. I would be surprised if the true number is larger than 10 and expect it to be lower than this. My reasoning here is that the total number of children each set of parents is going to have will hardly be reduced by 100x from access to contraception. If this number started at 10 children and is reduced to a single child, we have a reduction that corresponds to 10 fewer births per death averted. And stated like this, even the number 10 seems quite high (sorry, there were a few confusions in this argument)

This being said, the main reason why I am emotionally unconvinced by the argument you give is probably that I am on some level unable to contemplate "failing to have children" as something that is morally bad. My intuitions have somewhat cought up with the arguments that giving happy lives the opportunity to exist is a great thing, but they do not agree to the sign-flipped case for now. Probably, a part of this is that I do not trust myself (or others) to actually reason clearly on this topic and this just feels like "do not go there" emotionally.

It also does not seem obvious that we are above that number. Especially when trying to include topics like wild animal suffering. At least I feel confident that human population isn't off from the optimum by a huge factor. ↩︎

Melanie Basnak @ 2023-06-15T17:12 (+8)

Thanks for your comment Ariel. We haven’t attempted to assess the value of different population ethics views, or how those would affect the (cost-)effectiveness of FEM’s work. We believe that that is a highly complex topic that would take more time than the short period we had to conduct this research. Work on this would benefit from the Worldview Investigations Team at Rethink Priorities, which could explore family planning topics in the future. I'm sorry we neglected to add that to the editorial note and disclaimer. I will edit it to reflect this.

MichaelStJules @ 2023-06-13T22:05 (+7)

FWIW, basically the same argument would also undermine almost all global health work and other neartermist work. Why work on saving hundreds or thousands or even millions of lives when you can reduce the probability of extinction and marginally increase the probability of 10^50 (or whatever) happy conscious beings coming into existence?

The difference is mostly a matter of degree: in extinction prevention compared to family planning, we have much smaller probabilities for the high payoff possibility (of preventing extinction+total view) and much larger payoffs conditional on the payoff.

I don't think it makes sense to single out family planning in particular with this kind of argument.

Ariel Simnegar @ 2023-06-13T22:41 (+5)

I think there's a big difference between strong longtermism (the argument you state) and my comment's argument that FEM's intervention is net negative.

My comment argues that while FEM's intentions are well-meaning, their intervention may be net negative because it prevents people from experiencing lives they would have been glad to have lived. For my comment's argument to be plausible, all one needs to believe is that the loves and friendships future people may have is a positive good. Yes, my comment appeals to longtermism's endorsement of this view, but its claims and requirements are far more modest than those of strong longtermism.

There is no double standard or singling out here. I think global health work is good, and support funding for it on the margin. I believe the same about animal welfare, and about longtermism. Yes, some interventions are more cost-effective than others, and I think broadly similar arguments (e.g. even if you think animals don't matter, a small chance that they do matter should be enough to prioritize animal welfare over global health due to animal welfare's scale and neglectedness) do indeed go through.

If you provided me another example of a neartermist intervention which prevents people from experiencing lives they would have been glad to have lived, I would make the same argument against it as in my earlier comment. It could be family planning, or it could be something else (e.g. advocacy of a one-child policy, perhaps for environmentalist purposes).

I'm also quite sympathetic to the pure philosophical case for strong longtermism, though I have some caveats in practice. So yes, I don't think your statement of strong longtermism is unreasonable.

MichaelStJules @ 2023-06-14T01:07 (+5)

tl;dr: I think your strong argument based on MEC depends on pretty controversial assumptions, and your more modest argument doesn't imply we shouldn't support family planning at all in our portfolio, all-things-considered.

Your original argument does depend on MEC and (roughly) risk neutral EV maximization for the total view, or else high credence (>50%?) in moral views according to which it's good to make more happy people. You were multiplying the number of lives prevented by the credence in totalism. The standard argument for strong longtermism does essentially the same.

Similar arguments don't go through on most other popular approaches to moral uncertainty without high credence in the good of making more happy people. I'd guess most EA interventions are net negative according to some view that isn't totally implausible, and this isn't enough to stop us from pursuing them. Some approaches to moral uncertainty, like the property rights approach, could support family planning even if it's (very) net negative according to a credal supermajority of views. (Maybe that counts against the approach, though!)

Your more modest (non-MEC) argument about recognizing goods (love, friendship) impersonally, on the other hand, would not be very persuasive to most people who endorse person-affecting views. It might be the most basic standard objection to them, and pretty close to just the direct assertion that PAVs are false. It also wouldn't even follow that we shouldn't support family planning, if we consider moral uncertainty and assign some credence to person-affecting views. That would depend on the specifics.

If the difference between family planning work and the next best opportunities are small enough for welfare maximizers with person-affecting views, those without person-affecting views can pay those with PAVs for the difference to prevent family planning work. Or maybe those with PAVs avoid family planning work to cooperate with other EAs, but I don't think other EAs are very against family planning, and cooperation with non-EAs might actually support family planning instead (perhaps depending on the kind, maybe less so abortion specifically).

I agree that we can distinguish between net negative (compared to doing nothing or some other default) and net positive but worse than something else, but the result is the same under consequentialism and MEC (if the argument succeeds): you shouldn't support either family planning or neartermist work generally, because there are better options, namely doing nothing (for family planning), or extinction risk reduction (for both). Again under other approaches to moral uncertainty, it isn't obvious that family planning shouldn't be supported at all.

(This is ignoring some movement building effects of neartermist work and probably some other indirect effects. Under strong longtermism, maybe GiveWell did a lot of good by funneling people towards AI safety, building general expertise or buying reputation.)

Guy Raveh @ 2023-06-13T06:24 (+7)

This kind of conclusion is a great example of why a totalist utilitarian view is absurd.

MichaelStJules @ 2023-06-12T21:30 (+5)

(Edited slightly for accuracy/precision and grammar.)

MEC=maximizing expected choiceworthiness, and PAV=person-affecting view.

~99% doesn’t follow from MEC alone. You need MEC+specific intertheoretic comparisons where individual welfare under PAVs has similar or lower absolute moral value as individual welfare under the total view (or similar enough views). And you need to ground such intertheoretic comparisons. There may be a case for it, but some versions of the PAVs will probably ground value quite differently and in basically incompatible ways from any total view, so these comparisons wouldn't be justified between those versions and the total view. You'd have to use another approach to moral uncertainty (possibly along with MEC+intertheoretic comparisons in more limited cases), and the other approaches wouldn't generally be nearly as sensitive to the foregone welfare of the children not born.

I'd also guess wide person-affecting views (prioritize quality and longevity, not population size) and asymmetric person-affecting views are more popular than person-affecting neutrality, and would probably be endorsed under further reflection by most people initially attracted to neutrality over neutrality. These views still endorse making people happy over making happy people.

That being said, once you include the effects on a group of farmed animals, you should also probably include the effects on all (including wild) animals with similar or greater average moral weight and probability of moral patienthood, at least if you care about outcomes somewhat regardless of active contribution, e.g. if you're a utilitarian of any kind. This complicates things further.

Ariel Simnegar @ 2023-06-12T21:50 (+7)

Thanks for these caveats! I largely agree, but they seem to only have a modest impact on the 99% claim.

Regarding intertheoretic comparison, my prior is that a person-affecting view (PAV) should have little to no effect on one's valuation of welfare. I don't really see why PAV vs non-PAV would radically disagree on how important it is to help others. In this case, the disagreement would indeed have to be radical--even if for some reason, PAV caused someone to 10x their valuation of welfare, they'd still have to be 90% certain PAV was true for FEM to be positive.

For PAVs where value is grounded quite differently, I don't have an informed prior on just how different the PAV's grounding of value may be. If there are highly supported PAVs where welfare is clearly valued far greater than non-PAV, then that would update the 99% claim. However, I don't know of any such PAV, nor of any non-PAV where welfare is valued far greater than PAV (which would have the opposite effect).

Your second consideration makes sense, and might result in a modest dampening effect on the 99% number, if the increase in mothers' standard of living due to FEM's intervention is highly weighed.

Couldn't agree more on the farmed and wild animal effects :) I won't pretend to have any degree of certainty about how it all shakes out.

MichaelStJules @ 2023-06-12T23:04 (+3)

It's less about valuing individual welfare at a greater rate under PAVs (although that could happen in principle) and more about grounding value in ways that don't allow intertheoretic comparsions at all with total views, or just refusing to attempt such intertheortic comparisons altogether, or refusing to apply MEC using them. It could be like trying to compare temperature and weight, which seems absurd because they measure very different things. Even if the targets are at least superficially similar, like welfare in both cases, the units could still be incompatible, with no justifiable common scale or conversion rate between them.

A person-affecting view could ground value by using a total view-compatible welfare scale and then just restricting its use in a person-affecting way, and that would be a good candidate for a common scale with the total view and so intertheoretic comparsions under MEC in the obvious way: valuing welfare an existing individual's welfare identically across the views. However, it's not clear that this is the only plausible or preferred way to ground person-affecting views.

Stepping back, your argument depends on high confidence in multiple controversial assumptions:

the use of MEC at all (possibly alongside other approaches, rather than any other approaches to moral uncertainty not involving MEC, like a moral parliament or a property rights approach, which tend to be more generally applicable including to non-quantitative views, less fanatical, and, in my view, more fair),
the use of MEC with intertheoretic comparisons at all (possibly alongside other normalization approaches, rather than other normalization approaches for MEC without intertheoretic comparisons),
for almost each plausible grounding of a plausible PAV, the existence and use of a specific common scale for intertheoretic comparisons with some grounding of a total view (or similar) under MEC,
MEC with the given intertheoretic comparisons from 3 generally disapproving of family planning.

Your second consideration makes sense, and might result in a modest dampening effect on the 99% number, if the increase in mothers' standard of living due to FEM's intervention is highly weighed.

Ah, I meant to point this out because your quotes from MacAskill and Ord are critical of neutrality, and I don't expect neutrality to be very representative of those holding person-affecting views or who would otherwise support family planning for person-affecting reasons. It could be a strawman.

Ariel Simnegar @ 2023-06-13T00:45 (+3)

Your statements about PAV make sense. I typically think about PAV as you wrote:

A person-affecting view could ground value by using a total view-compatible welfare scale and then just restricting its use in a person-affecting way

But there could be other conceptions. Somewhat tangentially, I'm deeply suspicious of views which don't allow comparison to other views, which I see as a handwave to avoid having to engage critically with alternative perspectives.

If I'm talking to a person who doesn't care about animals, and I try to persuade them using moral uncertainty, and they say "no, but one human is worth infinity animals, so I can just ignore whatever magnitude of animal suffering you throw at me", and they're unwilling to actually quantify their scales and critically discuss what could change their mind, that's evidence that they're engaging in motivated reasoning.

As a result, I hold very low credence in views which don't admit some approach to intertheoretic comparison. I haven't spent much time thinking about which approach to resolving moral uncertainty is the best, but MEC has always seemed to me to be a clear default, as with maximizing EV in everyday decisionmaking. As with maximizing EV, MEC can also be fairly accused of fanaticism, which is a legitimate concern.

On neutrality, I've always considered the intuition of neutrality to be approximately lumpable with PAV, so please let me know if I'm just wrong there. From what I recall, Chapter 8 of What we Owe the Future argues strenuously against both the intuition of neutrality and PAV, and when I was reading it, I didn't detect much of a difference between MacAskill's treatment of the two.

MichaelStJules @ 2023-06-13T08:51 (+3)

I think there are legitimate possibilities for infinities and value lexicality, though (for me personally, extremely intense suffering seems like it could matter infinitely more), and MEC with intertheoretic comparisons would just mean infinity-chasing fanaticism.^[1] It can be a race to the bottom to less plausible views, because you can have infinities that lexically dominate other infinities, with a lexicographic order. You're stuck with at least one of the following:

infinity-chasing fanaticism (with MEC with intertheoretic comparisons),
ruling out these views with certainty,
ruling out the intertheoretic comparisons,
not using MEC.

The full MEC argument in response to a view X on which humans matter infinitely more than nonhuman animals, allowing lexicographic orders, is not very intuitive. There are (at least) two possible groups of views to compare X to:

Y. Humans and nonhuman animals both matter only finitely.

Y'. Humans and nonhuman animals both matter infinitely, an infinite "amplification" of Y.

(Also Z. Humans matter finitely, and some nonhuman animals matter infinitely.)

When you take expected values/choiceworthiness over X, Y and Y' (and Z), you will get that Y is effectively ignored, and you end up with X and Y' (and Z) deciding everything, and the interests of nonhuman animals wouldn't be lexically dominated. We can amplify X infinitely, too, and then do the same to Y', just shifting along the lexicographic order to higher infinities. And we can keep shifting lexicographically further and further. Then, the actual reason nonhuman animals' interests aren't lexically dominated, if they're not, will be because of exotic implausible views where nonhuman animals matter infinitely, to some high infinity. Even if it's the right answer, that doesn't seem like the right way to get to it.

If you don't allow lexical amplifications, then you have to rule out one of Y or Y'. Or maybe you only allow certain lexical amplifications.

For another critique of MEC's handling of infinities, see A dilemma for Maximize Expected Choiceworthiness (MEC), and the comments.

I think the intuition of neutrality is sometimes just called "the person-affecting restriction", and any view satisfying it is a person-affecting view, but there are other person-affecting views (like asymmetric ones, wide ones). I consider it to be one among many person-affecting views.

^{^}
Although you can also "amplify" any nonlexical view into a lexical one, by basically multiplying everything by infinity, e.g. shifting everything under a lexicographic order.

Ariel Simnegar @ 2023-06-13T15:40 (+1)

This is a good critique of MEC. Thanks for spelling it out, as I've never critically engaged with it before. At a high level, these arguments seem very similar to reductios of fanaticism in utilitarianism generally, such as the thought experiment of a 51% chance of double utility versus 49% chance of zero utility, and Pascal's mugging.

I could play the game with the "humans matter infinitely more than animals" person by saying "well, in my philosophical theory, humans matter the same as in yours, but animals are on the same lexicographic position as humans". Of course, they could then say, "no, my lexicographic position of humanity is one degree greater than yours", and so on.

This reminds me of Gödel's Incompleteness Theorem, where you can't just fix your axiomatization of mathematics by adding the Gödel statement to the list of axioms, because then a new Gödel statement pops into existence. Even if you include an axiom schema where all of the Gödel statements get added to the list of axioms, a new kind of Gödel statement pops into existence. There's no getting around the incompleteness result, because the incompleteness result comes from the power of the axiomatization of mathematics, not from some weakness which can be filled. Similarly, MEC can be said to be a "powerful" system for reconciling moral uncertainty, because it can incorporate all moral views in some way, but that also allows views to be created which "exploit" MEC in a way that other reconciliations aren't (as) susceptible to.

Jeffrey Kursonis @ 2023-06-12T21:06 (+3)

Really great to hear especially because they are using actors recording "radio drama's" telling stories relevant to their audience, ie. using art thoughtfully to influence. It's interesting to note in the cultures they broadcast within radio is the preeminent media people listen to. It's great to see the power of creative media and how it can be measured for impact. Every single EA org whether a charity or a research group or CEA could use more creative media to amplify the effects of their work. That's kind of a base understood thing outside of EA, but in this utilitarian bubble, it hasn't quite yet made it's mark..."wasteful aesthetics" is the false meme around here. Thanks to FEM for showing the reality.