Evidence of effectiveness and transparency of a few effective giving organisations

By Vasco Grilo🔸 @ 2023-07-01T08:10 (+60)

Summary

• Effective giving can be quite impactful.
• I estimated the factual non-marginal multipliers until 2021 of Ayuda Efectiva (Spain), Doebem (Brazil), Effektiv Spenden (Germany), and Giving What We Can (GWWC), i.e. how much donations they moved per dollar spent. Those of Ayuda Efectiva (1.34) and Doebem (5.53) are much lower than those of Effektiv Spenden (61.2) and GWWC (135).
• However, the results might differ accounting for future donations (received after 2021, but caused until then), counterfactuals, diminishing marginal returns, cost-effectiveness of caused donations, and indirect impacts of effective giving. Furthermore, the organisations were at different levels of maturity. Consequently, my estimates for the factual non-marginal multipliers are not directly comparable, and I do not know which of the 4 organisations are more effective at the margin.
• I did not find any proper cost-effectiveness analyses of Ayuda Efectiva, Doebem or Effektiv Spenden. I encourage these and other effective giving organisations as well as their funders (namely, Open Philanthropy) to do and publish cost-effectiveness analyses of their work (ideally including the indirect impacts of effective giving), as GWWC has done.

Introduction

Effective giving can be quite impactful:

• Supporting with 0.399 $/year the corporate campaigns for chicken welfare of The Humane League might be enough to neutralise the suffering of factory-farmed animals caused by a random person^[1]. This estimate can easily be off by a factor of 10, but illustrates that the (financial and non-financial) costs/savings of switching to a fully plant-based diet may well be much higher^[2].
• Helen Keller International’s vitamin A supplementation program has a cost-effectiveness of 3.5 k$ per life saved, i.e. one can save 13.8 lives (= 48.3/3.5) for the average transaction price of new cars in the United States in April 2023 of 48.3 k$.

So there are good reasons for giving effectively and significantly to become a cultural norm. This is a primary goal of effective giving organisations, and I have estimated the factual non-marginal multiplier of a few of them to get a sense of whether they are accomplishing it effectively. To clarify:

• A factual non-marginal multiplier of x means the effective giving organisation moved x $ of donations (hopefully to effective organisations) for each dollar it spent.
• A counterfactual non-marginal multiplier of y means the effective giving organisation caused y $ of donations for each dollar it spent.
• A counterfactual marginal multiplier of z means the effective giving organisation would have caused z $ of donations for each additional dollar it had spent.

y < x because effective giving organisations do not cause all the donations they move^[3], and z < y owing to diminishing marginal returns. The effective giving organisations is underfunded if z > 1, as long as the counterfactual marginal multiplier includes all relevant effects.

I was curious about Ayuda Efectiva and Doebem because their results could be more generalisable to Portugal (where I am from). I looked into Effektiv Spenden owing to it being regarded as a successful example of effective giving, and included GWWC as a major reference in this space.

Methods

I calculated the factual non-marginal multipliers from the ratio between donations received to be directed towards effective organisations and costs^[4]. I neglected future donations, and did not account for the opportunity cost of workers and volunteers. The greater the future donations, the greater my underestimation of the factual multipliers. The greater the opportunity cost, the greater my overestimation of the factual non-marginal multipliers^[5]. I converted all values to 2021-USD, using exchange rates from OECD, and inflation data from The World Bank^[6].

I did not study the counterfactual donations, i.e. how much donations would have been made in the absence of the effective giving organisation. As an example of why this matters, it might be the case that the early donations of new effective giving organisations are mostly caused by GWWC or effective altruism, with only a small fraction being due to new donors or increased donations (as a consequence of tax benefits) from people who were already effective givers.

I have not adjusted the amount of donations for their effectiveness^[7]. I have also not modelled the indirect impacts of effective giving:

• Advocating for effective altruism.
• Improving effective altruism culture.
• Indirectly moving money to effective charities.

However, I believe the above may be the major driver for the impact of effective giving organisations.

My calculations are in this Sheet (see tab “TOC”).

Results

Ayuda Efectiva

Doebem

Effektiv Spenden

Giving What We Can

Discussion

Factual non-marginal multipliers

The factual non-marginal multiplier until 2021 of Ayuda Efectiva (1.34) and Doebem (5.53) are much lower than those of Effektiv Spenden (61.2) and GWWC (135). However, the results might differ accounting for future donations (received after 2021, but caused until then), counterfactuals, diminishing marginal returns, cost-effectiveness of caused donations, and indirect impacts of effective giving. Furthermore, the organisations were at different levels of maturity in 2021:

• Ayuda Efectiva spent 273 k$ until then, and professionalised in 2020.
• Doebem spent 6.32 k$ until then, and professionalised in 2023.
• Effektiv Spenden spent 429 k$ until then, and professionalised in 2019.
• GWWC spent 1.50 M$ until then, and professionalised in 2012.

Consequently, my estimates for the factual non-marginal multipliers are not directly comparable, and I do not know which of the 4 organisations are more effective at the margin. Nevertheless, the counterfactual marginal multipliers adjusted for cost-effectiveness and indirect impacts should ideally be equal. In other words, donating to any effective giving organisation should be similarly effective taking into account all effects.

Cost-effectiveness

The donations until now of Ayuda Efectiva, Doebem and Effektiv Spenden went almost exclusively to organisations working on global health and development, animal welfare and climate change^[15]. The cost-effectiveness of these areas is quite unclear to me^[16] (see also section 4.2 of Mogensen 2019). Additionally, 81.0 % (= 207107.47/255640.20) of Doebem’s donations have gone to Brazilian organisations, which are arguably closer to neutral than GiveWell’s top charities (for better or worse^[17]). Focussing on donations to local organisations could make sense if Brazilian donors would hardly be persuaded to do otherwise.

In contrast, 11 % and 15 % of GWWC’s pledge and non-pledge donations went towards the area of creating a better future, which I think is much more effective. I do not think this corresponds to an extreme position. For example, Benjamin Todd “would donate [in November 2021] to the Long Term Future Fund over the global health fund, and would expect it to be perhaps 10-100x more cost-effective (and donating to global health is already very good)”. 80,000 Hours thinks global health is “important and underinvested in”, but it is not one of the 18 areas which are part of its list of the most pressing problems.

Evidence of effectiveness and transparency

I did not find any proper cost-effectiveness analyses of Ayuda Efectiva, Doebem or Effektiv Spenden. Ayuda Efectiva has an impact page, but it does not include any information about giving multipliers. Sebastian Schwiecker and Anne Schulze suggest Effektiv Spenden's non-marginal counterfactual multiplier is “between 11 and 91” based on other models, but these do not study the counterfactuality of Effektiv Spenden's donations. GWWC says Founders Pledge “has conducted an extensive evaluation highlighting its [Effektiv Spenden's] cost-effectiveness as part of its work on Giving Multipliers”, but I only found one unrelated occurrence of “Effektiv Spenden” in its report^[18]. Sjir Hoeijmakers commented Founders Pledge has an interval evaluation on Effektiv Spenden.

The funders of effective giving organisations have published little/no information about their cost-effectiveness. Here is Open Philanthropy’s write-up recommending a grant of 2.18 M$ to Effektiv Spenden in November 2022:

Open Philanthropy recommended a grant of €2,180,000 (approximately $2,120,000 at the time of conversion) to Effektiv Spenden for general support. Effektiv Spenden works to communicate ideas related to effective altruism and effective giving to German-speaking audiences.

This falls within our focus area of growing and empowering the EA community.

Open Philanthropy has similar write-ups recommending grants of 225 k$ to Ayuda Efectiva in February 2023, and 2.36 M$ to GWWC in the same month. Founders Pledge does not have a write-up for the grant of 179 k$ they made to Effektiv Spenden in January 2023.

I encourage Ayuda Efectiva, Doebem, Effektiv Spenden and other effective giving organisations as well as their funders to do and publish cost-effectiveness analyses of their work (ideally including the indirect impacts of effective giving), as GWWC has done.

We [GWWC, but the points hold more broadly] have several reasons for doing this:

• To provide potential donors with information about our past cost-effectiveness.
• To hold ourselves accountable and ensure that our activities are providing enough value to others.
• To determine which of our activities are most successful, so we can make more informed strategic decisions about where we should focus our efforts.
• To provide an example impact evaluation framework which other effective giving organisations can draw from for their own evaluations.

One could argue cost-effectiveness analyses of past work are not informative for effective giving organisations which only recently professionalised. Nonetheless, forecasting future costs and donations (and thus cost-effectiveness) could still be valuable in those cases.

Questions for reflection

Some questions to think about:

• Does the factual non-marginal multiplier correlate well with the counterfactual marginal multiplier?
• Is the direct impact of effective giving organisation a good proxy of their overall impact?
• If yes is a reasonable answer to the questions above, how high does the factual non-marginal multiplier have to be for the counterfactual marginal multiplier to exceed 1?
• Should effective giving organisations do and publish cost-effectiveness analysis? How early and often?
• Is GWWC well positioned to assess the cost-effectiveness of other effective giving organisations (even if the onus is arguably on their funders), given its experience running their own, and plans to evaluate evaluators?
• Should funders provide more information about their grants, at least for large ones?

Acknowledgements

I had the initial idea for this post chatting with José Oliveira. Thanks to Sebastian Schwiecker (Effektiv Spenden), José, Luan Paciencia (Doebem), Joan Montoya (Ayuda Efectiva), and Francisco Martins for feedback on the draft^[19]. Thanks to Michael Townsend and Luke Freeman for feedback on GWWC’s costs. Thanks to Sjir Hoeijmakers (GWWC) for clarifying that Founders Pledge conducted an internal evaluation of Effektiv Spenden. Thanks also to Luan for meeting with me to present Doebem’s history and future plans^[20].

1. ^{^}

    Search for “0.399” here. I like to present numbers with 3 significant digits in order not to propagate rounding errors, not because they have low uncertainty.

2. ^{^}

    From Faunalytics’ post on Springmann 2021:

   In high-income and upper-middle-income countries, flexitarian, vegetarian, and vegan diets were less expensive than the typical diet (the reduction ranged from 12 to 34%). In these countries, the pescatarian diet was about as expensive. In contrast, in lower-middle-income and low-income countries, all alternative diets were more expensive than the typical diet (the increase ranged from 18 to 45%).

3. ^{^}

    For context, GWWC estimates it only caused 26 % of its pledge donations from 2020 to 2022 (which would correspond to y/x = 0.26).

4. ^{^}

    Note the donations received to be directed towards effective organisations are classified as costs in the financial reports of effective giving organisations. The costs I present in this post, and use to compute the factual marginal multiplier exclude such donations.

5. ^{^}

    To illustrate, if 2 effective giving organisations A and B moved the same amount of donations, and had the same costs, they would have the same factual non-marginal multiplier. Nonetheless, if the workers and volunteers of A could do something almost as impactful in the absence of A, whereas the workers and volunteers of B would not do something nearly as impactful in the absence of B, A would be less effective than B. In other words, although A and B moved the same amount of donations, the existence of B would be more important than that of A. In its last impact evaluation, GWWC accounted for the opportunity cost of workers, but not volunteers. “While the time of our volunteers is valuable, we suspect the opportunity cost here is cancelled out by the fact that there are significant positive externalities to volunteering. For example, several volunteers have since gone on to do other valuable work that they may only have been able to do due to their earlier volunteering”.

6. ^{^}

    Data from The World Bank only goes up to 2021, so I used in2013dollars for 2022 (as GWWC estimated their costs for 2020 and 2021 in 2022-USD).

7. ^{^}

    If 2 effective giving organisations A and B had the same counterfactual marginal multiplier, but those caused by A went to organisations 2 times as effective as those caused by B, donating to organisation A would be 2 times as effective as to B.

8. ^{^}

    Sum of the values of points 8 (personnel costs) and 9 (other costs) of “CUENTA DE RESULTADOS MODELO ABREVIADO”.

9. ^{^}

    Taken from the figure here.

10. ^{^}

     Ayuda Efectiva was founded in 2019, but only started operating in June 2020.

11. ^{^}

     Provided by Luan Paciencia, Doebem’s research director.

12. ^{^}

     Taken from the blogposts linked in the last sentence of the page.

13. ^{^}

     From GWWC’s previous impact report, “Giving What We Can’s total monetary costs up to the end of 2014 were £208,000. Up to the middle of 2012, Giving What We Can was run entirely by volunteers”. So I set the costs from 2009 to 2011 to 0, considered the cost from 2012 to 2014 to be 208 k 2013-£, and allocated 20 % to 2012, 40 % to 2013, and 40 % to 2014. “[Michael Townsend would] guess it's best to assume something like 2015's budget was 67% of 2014's, and 2016's was 33% of 2014's, but really that's a pure guess [which nevertheless “closely matches” Luke Freeman’s “best guesses”] and even a bit of desk research to estimate the FTE in those times would lead to better informed guesses”. I supposed the costs in 2015 and 2016 to be 2/3 and 1/3 that of 2014. In addition, “[according to Michael, GWWC’s] best-guess is that the costs from 2017-2019 were ~$250,000 USD if you include the EA Funds donation platform as part of GWWC (if you don't, I think the costs might be closer to ~$50,000 USD per year)”. I assumed a cost of 83.3 k 2018-USD/year (= 250/3) from 2017 to 2019. I took the costs for 2020 and 2021 from GWWC’s last impact report.

14. ^{^}

     Donations calculated in the context of my impact assessment of GWWC.

15. ^{^}

     The exception is Effektiv Spenden’s fund to preserve the future fund, which made 2 grants totalling 215 k€ in 2022.

16. ^{^}

     I believe there are cost-effective interventions in these areas, namely ones which emphasise learning more, and keeping options open. Nevertheless, I think these are mostly not being targeted.

17. ^{^}

     Because I am unsure about whether GiveWell’s top charities are beneficial or harmful (see links in the 2nd sentence of this paragraph).

18. ^{^}

     “Effektiv Spenden, The Life You Can Save, and Effective Altruism Australia are examples of this [“straightforward fundraising”] model”.

19. ^{^}

     Names ordered by descending relevance of contributions.

20. ^{^}

     I have not described such history and future plans here, but Doebem intends to post about them in the future.

21. ^{^}

     Taken from the blogposts linked in the last sentence of the page.

22. ^{^}

     Provided by Luan Paciencia, Doebem’s research director.