How GiveWell's Research is Evolving

By GiveWell @ 2019-02-09T01:17 (+47)

This is a linkpost to https://blog.givewell.org/2019/02/07/how-givewells-research-is-evolving/

To date, most of GiveWell’s research capacity has focused on finding the most impactful programs among those whose results can be rigorously measured. This work has led us to recommend, and direct several hundred million dollars to, charities improving health, saving lives, and increasing income in low-income countries.

One of the most important reasons we have focused on programs where robust measurement is possible is because this approach largely does not rely on subject-matter expertise. When Holden and I started GiveWell, neither of us had any experience in philanthropy, so we looked for charities that we could evaluate through data and evidence that we could analyze, to make recommendations that we could fully explain. This led us to focus on organizations that had impacts that were relatively easy to measure.

The output of this process is reflected in our current top charities and the programs they run, which are analyzed in our intervention reports.

GiveWell has now been doing research to find the best giving opportunities in global health and development for 11 years, and we plan to increase the scope of giving opportunities we consider. We plan to expand our research team and scope in order to determine whether there are giving opportunities in global health and development that are more cost-effective than those we have identified to date.

We expect this expansion of our work to take us in a number of new directions, some of which we have begun to explore over the past few years. We have considered, in a few cases, the impact our top and standout charities have through providing technical assistance (for example, Deworm the World and Project Healthy Children), supported work to change government policies through our Incubation Grants program (for example, grants to the Centre for Pesticide Suicide Prevention and Innovation in Government Initiative), and begun to explore areas like tobacco policy and lead paint elimination.

Over the next several years, we plan to consider everything that we believe could be among the most cost-effective (broadly defined) giving opportunities in global health and development. This includes more comprehensively reviewing direct interventions in sectors where impacts are more difficult to measure, investigating opportunities to influence government policy, as well as other areas.

Making progress in areas where it is harder to determine causality will be challenging. In my opinion, we are excellent evaluators of empirical research, but we have yet to demonstrate the ability to make good judgments about giving opportunities when less empirical information is available. Our values, intellectual framework, culture, and the quality of our staff make me optimistic about our chances, but all of us at GiveWell recognize the difficulty of the project we are embarking on.

Our staff does not currently have the capacity or the capabilities to make enough progress in this direction, so we are planning to significantly increase the size of our staff. We have a research team of ten people, and we are planning to more than double in size over the next three years. We are planning to add some junior staff but are primarily aiming to hire people with relevant experience who can contribute as researchers and/or managers on our team.

GiveWell’s top charities list is not going to change dramatically in the near future, and it may always include the charities we recommend today. Our top charities achieve outstanding, cost-effective results, and we believe they are some of the best giving opportunities in global health and development. We expect to conclude that many of the opportunities we consider in areas that are new for us are less cost-effective than those we currently recommend, but we also think it is possible that we will identify some opportunities that are much more cost-effective. We believe it is worth a major effort to find out.

What areas will we look into?

As with any exploration into a new area, we expect the specifics of the work we will undertake to shift as we learn more. Below we discuss two major areas of work we are embarking on and building our team for currently. In the long term, we are open to considering making grants or recommendations in all areas of global health and development. We have not yet comprehensively considered what those areas might be, but they could include (for example) research and development, or social entrepreneurship.

Using reasoned judgment and less robust evidence to come to conclusions about additional direct-delivery interventions

In the past, we have often asked, “does this intervention meet our criteria?” rather than “what is our best guess about how promising this intervention is relative to our top charities?” Our intervention report on education is a good example of asking the question, “does this meet our criteria?” It reviews all randomized controlled trials of education programs that measure long-term outcomes, but it does not attempt to reach a bottom line about how cost-effective education in developing countries is.

We plan to more deeply explore how we can reach conclusions about how areas such as nutrition, agriculture, education, reproductive health, surgical interventions, mental health, and non-communicable diseases compare to our current top charities.

Investigating opportunities to improve government spending and influence government policy

Note from Aaron: This post was cross-posted from GiveWell's website. To see their full table of examples in its original formatting, please see the original post.

Some of the areas we will consider exploring to leverage government resources and affect government policy are:

How will our analysis change? How will it be the same?

Writing up and publishing the details of the reasoning behind the recommendations we make is a core part of GiveWell. We will remain fully transparent about our research.

Judgment calls that are not easily grounded in empirical data have long been a part of GiveWell’s research. For example, we make difficult, decision-relevant judgment calls about moral weights, interpreting conflicting evidence about deworming, and estimating the crowding-out and crowding-in effects of our donations on other actors (what we call leverage and funging).

As we move into areas where measuring outcomes and attributing causal impact is more difficult, we expect subjective judgments to play a larger role in our decision making. For examples of the approach we have taken to date, see our writeup of our recent recommendation for a grant to the Innovation in Government Initiative, a grantmaking entity within the Abdul Latif Jameel Poverty Action Lab (J-PAL) or our page evaluating phase I of our 2016 grant to Results for Development (R4D). While writing about such judgments will be a challenge of this work, we are fully committed to sharing what has led us to our decisions, with only limited exceptions due to confidential or sensitive information.

What does this mean for staffing and organizational growth?

We need to grow our team to achieve our goals. Repeatedly this past year, we had to make the difficult choice to not take on a research project or investigate a grant opportunity that seemed promising because we did not have the capacity.

We are planning to roughly double our research team over the next few years, primarily by adding researchers who have experience and/or an academic background in global health and development. We are looking to add both individual contributors and research managers to the team. We expect that the people we hire in the next few years will play a critical role in shaping GiveWell’s future research agenda and will be some of the leaders of GiveWell in the future.

For more information about the research roles we’re hiring for, see our jobs.

Raemon @ 2019-02-11T00:48 (+17)

I'm curious how this relates to OpenPhil (I'd been bucketing "OpenPhil as the research team that does harder-to-quantify/justify stuff, and Givewell as the team that does... not that")

CatherineHollander @ 2019-02-15T17:13 (+10)

Hi Raemon,

I work at GiveWell; thanks for your question. There are a few key differences with the Open Philanthropy Project:

  1. Approach. While this work is still new and we're unsure exactly how we'll approach it, we continue to see our core activity as intensive critical assessment of specific giving opportunities to maximize the return of the funds we direct. By contrast, the core activity of the Open Philanthropy Project is intensive cause selection followed by intensive selection of Program Officers to lead grantmaking in those causes, and heavy reliance on the judgment of those Program Officers to identify hits.

    In the post above, we aim to highlight how our approach to conducting intensive critical assessment of giving opportunities (our research process) has changed. GiveWell traditionally relied on empirical, quantitative evidence (often randomized controlled trials and academic papers) as the primary input into this assessment. In the last few years, we began to expand the scope of the information we rely in our assessments. Our recent write-ups of the case for a grant to the Innovation in Government Initiative and our evaluation of Phase 1 of our grant to Results for Development are examples of how we have assessed new types of information in recent years.
  2. Transparency. Transparency remains a core GiveWell value. We continue to be committed to publishing the full details of our research and recommendations as we evolve our research process so that our donors and others who rely on our recommendations can follow our logic and evaluate our evidence to make their own determinations about how to allocate their charitable giving. The Open Philanthropy Project’s mission is to give as effectively as it can and share its findings openly so that anyone can build on its work, but it does not aim for comprehensive information sharing such that the reasoning behind every decision can be understood and critiqued. The Open Philanthropy Project describes its approach to transparency here.
  3. Focus area. GiveWell is focused on improving as much as possible the lives of humans today, and we focus on global health and development to do so. Even as our approach to research changes, we do not currently expect to look to areas focused explicitly on the long-term future or animal welfare, which are among the areas in which the Open Philanthropy Project makes grants. (The Open Philanthropy Project also supports global health and development, such as by supporting GiveWell's top charities and GiveWell Incubation Grants.)