Our Process for Identifying Top Charities

By GiveWell @ 2016-11-01T23:30 (+2)

This is a linkpost to https://www.givewell.org/how-we-work/process

This page describes the process we use to identify our top charities.

Introduction

GiveWell's mission is to find outstanding charities and to publish the full details of our analysis to help donors decide where to give. Our focus is on finding and reviewing the most outstanding charities possible rather than completing an in-depth investigation for each organization we consider.

We identify top charities by assessing them along four criteria: Their evidence of effectiveness, cost-effectiveness, transparency, and room for more funding. Learn more about our criteria here.

This page discusses our process for identifying and researching GiveWell’s top charities, as well as some of the context behind what our process has been in the past and how it has evolved to its current state.

Eligibility

In order to be eligible for "top charity" status, a charity must be explicitly focused on one or more of our priority programs—such as distributing insecticide-treated nets to prevent malaria, treating children for parasites, or direct cash transfers—or conduct compelling evaluation of its own program, such as the randomized controlled trials conducted by Development Media International and Living Goods. These programs represent, according to our research, the most evidence-backed approaches to helping the global poor.

Why these programs?

GiveWell has done multiple in-depth investigations looking to identify charities whose activities can be strongly connected—via empirical evidence—to improved life outcomes.

In 2007 (our first year), we conducted an open grant application process and reviewed 59 applications in global health and development.
In 2009:
- We reviewed the websites of over 300 charitable organizations aiming to find ones that either (a) implemented priority programs (as we defined them at the time) or (b) published, on their websites, meaningful evaluations of their programs. Details of this process are available here.
- We conducted a grant application process for organizations running economic empowerment programs in the developing world (details here).
In 2011, we repeated the above process and broadened our search to flag organizations based on additional characteristics (details here).
In 2012, we searched for organizations working on immunization, nutrition and other global health programs. Details of this search here.
In 2014, we (a) solicited applications from several organizations running priority programs and (b) evaluated two organizations that had conducted randomized controlled trials of their own programs. Details here.
In 2015 and 2016, we sought to expand top charity room for more funding and consider alternatives to our top charities by inviting other groups that work on deworming, net distributions, and micronutrient fortification to apply. This led to adding Sightsavers' deworming program, the END Fund's deworming program, Project Healthy Children, and Food Fortification Initiative to our lists in 2016. We also did very preliminary investigations into a large number of interventions and prioritized a few for further work. This led to interim intervention reports on seasonal malaria chemoprevention (SMC), among others, and recommending Malaria Consortium for its work on SMC. We also sought out new programs and charities through our experimental work. Details here and here.
Throughout, we have compiled a set of "intervention reports" which assess the evidence for the programs we have considered; all the programs we have considered are listed here.

Based on these investigations, we have concluded that:

Highly rigorous evidence connecting aid activities to improved life outcomes (for example, "distributing insecticide-treated nets reduces the burden of malaria and saves lives") is found in academic literature, and not (in any cases that we've seen) in internal self-evaluations by charities. Self-evaluations may provide an important part of the picture, helping to assess whether a charity is carrying out activities as intended, but rigorous evidence that a program has the intended effect on life outcomes is found in academia. For more on how we assess evidence, see our 2012 series on how we assess studies: Our Principles for Assessing Evidence, How We Evaluate a Study, Surveying the Research on a Topic.
Even within academic literature, many categories of intervention have insufficient rigorous evidence to determine their effectiveness. See our 2009 write-ups on developing-world education and economic empowerment.
Interventions targeting the global poor are likely to accomplish substantially more good (per dollar spent) than interventions targeting the developed-world poor. (More on this in the next section.)

Why focus on the global poor?

Having done substantial research on both developing-world aid and programs focused in the U.S., we've narrowed our focus to the former, for the following reasons:

The U.S. poor are wealthy by developing-world standards. All the empirical analysis we've come across, as well as the qualitative observations we've made on site visits, support the conclusion that "poverty" has very different meanings in the developed world and the U.S., and that even the lowest-income people in the U.S. have (generally speaking) far greater material wealth and living standards than the developing-world poor. More at our 2009 blog post on this topic.
We haven't found any U.S. poverty-targeting intervention that compares favorably to our international priority programs, in terms of (a) quality, robustness, and generalizability of evidence; (b) cost-effectiveness, i.e., "bang for the buck" in terms of lives saved or improved per dollar spent. For more, see our write-ups on priority programs and U.S. programs.

We believe the second point is largely explained by the first: the developing-world poor struggle to meet basic needs and to access products and services with well-established benefits, to a greater extent than the U.S. poor.

Finding eligible charities

In past years, we have conducted extensive searches for charities that focus on our priority programs. The details of our processes and findings follow:

We describe our work to find charities in more recent years in our annual review blog posts, which are linked from this page.

Our full list of eligible charities — charities that focus on evidence-backed programs serving the global poor — is available here.

We believe this list to be near-comprehensive, but if you know of organizations we are missing, we encourage you to submit them for our consideration.

Examining charities

We invite eligible charities to participate in our intensive evaluation process, which aims to deeply and critically question the case for the charity's impact, and lay out what we see as the strengths and weaknesses publicly.

Key questions

We look for top charities that meet our four criteria:

Evidence of effectiveness
Cost-effectiveness
Room for more funding
Transparency

Our process thus focuses on answering the following questions:

What do they do?

We aim for a comprehensive understanding of the charity's budget and the nature of its value-added. Large-scale aid projects often require coordination between many actors (including governments), so we aim to understand the charity's specific role thoroughly.

Does it work? (Evidence of effectiveness)

Because eligible charities focus on priority programs, we generally begin our investigation with at least a partial understanding of the evidence base behind what they do. When looking at a potential top charity, we generally greatly deepen our investigation of this evidence base, examining not just the strengths and weaknesses of key studies but also considerations regarding how likely their results are to generalize to larger-scale programs. (See, for example, our write-ups on insecticide-treated nets, deworming and cash transfers.)

In addition, we attempt to examine the specifics of the charity in question, including data from internal reports and interviews with charity representatives, to assess whether its program is being carried out with high quality (and, in some cases, whether the charity is adding substantial value to what the program would look like without its presence).

What do you get for your dollar? (Cost-effectiveness)

We try to quantify the impact of the program the charity executes, as it is executed (as opposed to in academic studies). We quantify impact using a variety of different measures, including "cost per life saved" (when applicable) and "financial benefits to recipients per dollar spent by donors" (when applicable). These estimates involve substantial judgment calls, and we publish the details of our analysis, spell out our major assumptions and allow readers to fill in their own. The most recent iteration of this analysis is available on our cost-effectiveness page.

We recognize that our cost-effectiveness estimates have major limitations, and we do not advise taking them literally. (More.) The purpose of these estimates is to:

Identify differences in cost-effectiveness that are large and robust enough to affect our rankings.
Thoroughly think through the activities in question, and raise critical questions about them.

How will additional funding impact the organization? (Room for more funding)

Because we expect GiveWell’s recommendation to result in increased funding for a charity, we want to know what the impact of additional funding will be.

We work to understand the impact of marginal dollars on a charity's activities and impact, because we are looking for charities that can use more funding productively — not just charities that have had strong impact in the past. More than once in the past, we have withdrawn our recommendation of a charity because it had raised as much funding as we felt it could productively absorb.

More on room for more funding.

How transparent are they?

We aim to publish the full details of our analysis for donors, and a charity’s willingness to share information with us and the public so we can assess its work is an important part of our process. More on the importance of transparency below.

Our process for answering key questions

Most of our investigation generally consists of conversations with charity representatives and review of their documents, which we make public (as sources for our charity reviews) to the extent we can.

In addition, we often speak to other current and potential funders of the charity in question, and we typically conduct at least one site visit to see the charity's work in the field.

Consistent with our commitment to transparency, we share as much as we can from these investigations — notes from conversations, notes and photos from site visits, original documents and details of our analysis — via our charity reviews.

Our process aims to create as complete a picture as possible of the answers to our key questions, and we use this picture to (a) decide whether to recommend a charity; (b) decide how to rank it relative to our other top charities; (c) create a basis for follow-up (discussed below), which allows us to learn more and improve our understanding over time.

Following up

We follow up intensively with our top charities over time, and consider this one of the major arguments in favor of supporting such charities. Because our recommendation directs substantial donations to a charity, top charities are generally willing to engage substantively with us and help us deepen our understanding of their activities and progress over time.

Crucially, we believe—and make clear to our top charities that we believe—in sharing both positive and negative developments, and we have written extensively in the past about unanticipated struggles faced by top charities. See, for example, our series of updates on VillageReach.

Pros and cons of our top charities as giving opportunities

We think the principal advantages of our current top charities are that:

They represent the best opportunities we're aware of to help low-income people with relatively high confidence and relatively short time horizons. If you're looking to give this year and you don't know where to start, we'd strongly recommend supporting our top charities.
Due to the emphasis on thorough vetting, transparency, and following up, our top charities represent excellent learning opportunities, and we feel that one of the most desirable outcomes of giving is learning more that will inform later giving. Supporting our top charities helps GiveWell demonstrate impact and improves our ability to learn, and we are dedicated to sharing what we learn publicly.

Some counter-considerations:

We have strict criteria about the sorts of charities we recommend. These criteria are partly about achieving maximum impact, but partly about having recommendations that others can fairly easily be confident in.
Seeking strong evidence and a straightforward, documented case for impact can be in tension with maximizing impact, as argued at this post by the Open Philanthropy Project. (The Open Philanthropy Project was incubated at GiveWell and looks for giving opportunities that can be longer term, harder to assess, and harder to explain. It does not have official recommendations for individual donors.)
Thus, we think there may be many giving opportunities that are better than our top charities but don't meet our criteria and/or are not known to us.
Even though we believe our top charities are backed by strong evidence, none of our recommendations are a 'sure thing' and a few carry a significant risk of failing to do much good.

Many, but not all, staff support GiveWell's top charities with their personal giving.