Neil Buddy Shah: What animal advocates can learn from medicine and global development

By EA Global @ 2020-01-21T15:00 (+36)

This is a linkpost to https://www.youtube.com/watch?v=sMuP9OldVHc&list=PLwp9xeoX5p8MqGMKBZK7kO8dTysnJ-Pzq&index=6

Despite recent advancements, research on animal welfare is still underdeveloped compared to research in fields like medicine or development economics. In this talk, Neil Buddy Shah, the co-founder and CEO of IDinsight, draws on lessons from those fields to propose that a coordinated, multi-year research agenda can accelerate improvements in animal welfare.

A transcript of Neil’s talk, which we have lightly edited for clarity, is below. You may also watch it on YouTube or read it on effectivealtruism.org.

The Talk

I want to start with one big caveat: I am a doctor and development economist by training who works full-time in global development — and I am categorically not an animal welfare expert. However, I do think there's a tremendous amount that the animal welfare movement can learn from advancements in medicine and in social policy, both domestically and in international development.



What I'm hoping to do in this talk is:

1. Opine on a set of hypotheses about what those learnings might be and their implications for the animal welfare movement.

2. Perhaps more importantly, get ideas and critical feedback from those of you who do work in the animal welfare movement about which of these hypotheses may have legs and should be explored further — and which of them are the result of my own naivete.

I think we all start from a very similar proposition, which is that we're concerned with how we can most effectively improve the welfare of billions of animals.



There are a huge number of potential interventions we have at our disposal in order to effect change — everything from voluntary corporate campaigns, to regulatory and policy reforms, to individual behavior-change interventions.

My proposition is that, just as in medicine and global development, data and evidence have an absolutely crucial role for effective altruists and others who wish to determine how to do the most good possible.



I think that there are some very tactical things that we can take from those other fields as the field of animal welfare matures.



First, I think it's obvious to most people that animal welfare shares a huge number of common challenges with medicine, public health, and development economics. We're equally concerned with questions around how we can most effectively improve lives — whether those are the lives of humans or non-human animals — and which interventions are most cost-effective. And how do we know whether something works or is cost-effective? How do we design interventions and measure their effectiveness, both for individual behavior-change interventions and broader, systemic reforms?

A lot of other people have recognized these similarities. In fact, there's been a blossoming of a number of excellent animal welfare research organizations, such as Animal Charity Evaluators, the Sentience Institute, Faunalytics, Humane League Labs, and, of course, the Open Philanthropy Project.



But despite all of this progress in recent years, one of the things I want to argue — and I'm happy to get pushback on this — is that the animal welfare research infrastructure and ecosystem is incredibly immature compared to what has developed over decades in social policy, medicine, and public health.

This really complex diagram [see following slide] shows just how sophisticated the medical space is when it comes to the production of evidence, the synthesis of entire bodies of evidence, translating and disseminating that synthesis for practitioners, and then enforcing the use of that evidence by everyday doctors who are not necessarily well-versed in how to produce or use it.



You have a robust ecosystem of organizations that are specialized in ensuring incredibly high quality at each of the nodes representing the production and use of evidence. In the middle, there's a shared set of values and norms around what constitutes good evidence — and what doesn't.

I want to argue that there’s an analog to animal welfare. Some of the core components of that are:

1. There are long-term, coherent, and coordinated research agendas in medicine and in global development, such that individual papers and research objectives create a lot more impact together than they would otherwise have on their own. One of the risks that we run within the animal welfare movement is we have a bunch of separate actors running individual trials that people aren't aggregating into a larger picture.

2. There are clear evidentiary standards. People across the sector can refer to a common vocabulary in order to argue about what works and what doesn't, and what the quality of evidence is. I think that is still more immature in the animal welfare movement than it is in these other fields.

3. There are protective measures to prevent data mining and other publication biases through pre-registration and pre-analysis plans in these other fields, which could be worth adopting in the animal welfare space.

4. Finally, even if you have exceptional evidence creation, you need mechanisms to ensure that funding flows based on that evidence. Both medicine and global development have created results-based financing mechanisms to ensure that this happens. And while those mechanisms aren’t yet widespread, they can still teach us important lessons.

Based on this review, I want to propose two initial ideas for the animal welfare movement. And again, I am very open to pushback from those of you who work full-time in this space.



The first idea is to create the equivalent of the National Institutes of Health (NIH) for animal welfare. And the second is to start actively exploring results-based financing for animal welfare nonprofits.

An overarching national research body

We'll take each one in turn. The first idea, which proposes an overarching national body to organize research and funding for animal welfare, has two sub-proposals.



The first is that we need to start creating — as an entire ecosystem, not individual organizations — a long-term, coordinated research agenda that is based on what's called the “value-of-information (VOI) approach” in medicine and public health. And the second is to draft and adopt sector-wide evidence-grading standards. Both of these are going to be very difficult to implement because they require ecosystem-wide buy-in and collaboration between a number of different organizations. But if we're able to do that the same way that medicine has, I think you'll see disproportionate and nonlinear returns.

Why do we need a coordinated research agenda? Aren’t we already doing pretty well with all of these individual organizations, which I mentioned previously? I think the short answer is that there is an incredibly complex theory of change to get to the end outcomes of fewer animals farmed and farmed animals suffering less. And there's a whole host of interventions that can get us to those ultimate goals — for example, dietary change interventions and funding for meat alternatives. And then there are the individual behavior-change processes that have to happen once those technologies actually exist. Other interventions include public advocacy, regulatory and legislative reform, voluntary corporate campaigns, and humane farming advocacy.



At every single one of these nodes, in this complex theory of change, there are huge unanswered questions about which is actually the most cost-effective. And despite the blossoming of funding in this space (the Open Philanthropy Project has given over $90 million to date for animal welfare, which is incredible), there are still finite resources.

Which of these nodes we focus on should be informed by some hypothesis of where we're going to get the most bang for our buck.



If you look at what exists to date, there's a lot of nascent but high-quality evidence generation in the animal welfare movement, from work done by Animal Charity Evaluators, the Sentience Institute, the Open Philanthropy Project and a host of other organizations. But what's lacking is a systematic look at an overarching theory of change that's tied to the literature and evidence. Also, there is neither a standard and accepted review of how to evaluate the effectiveness of individual papers, nor a way to aggregate individual papers to form a holistic view (e.g., on the effect of leafleting on behavior change).

Then, if you look across the slide, what’s missing is the benefit from answering one question in that theory of change: What is the cost? Is it $10 million to answer one of those nodes, or is it something cheaper? We’re also missing a prioritization system that flows from that cost-benefit calculation — i.e., the value of getting more data on each of those particular nodes. And finally, is there a user-friendly presentation for NGOs that are funding organizations? It would be helpful to have something for them to easily reference, without having to be statistics experts, so that they can determine the top five things to fund based on the evidence, as well as see whether there is room for more funding.

This is no knock against the exceptional organizations that have done this work, but what we see is that no organization in the animal welfare movement has set out to create a long-term, cohesive, and coordinated research agenda. That's what will be required in order to achieve our long-term goals.

So, how do we get to that VOI analysis in order to determine where we should be putting our finite resources?



In medicine and global development, you essentially move from a theory of change, to individual research topics, to calculating the costs and benefits of answering individual research topics. Then, you arrive at a prioritized research agenda. This is a highly quantitative exercise in medicine and in global development.



You quantify the amount that a decision-maker — whether that's the Good Food Institute or the Open Philanthropy Project — would pay for a piece of evidence to make a decision.

Unfortunately, in the animal welfare movement, we're at such an early stage in the generation of evidence that we can't quantify it. However, we can use the same process in order to prioritize research topics. And as the literature gets more mature, we can start to quantify the returns we would expect from investing in different research topics.

There are five key components to the VOI approach:




1. What are your priors? What evidence exists now about the impact of an intervention — say, the effect of leafleting on behavior change and reducing meat consumption?

2. What is the variance on your prior? Do you have high confidence in your estimate or low confidence?

3. How much do you expect to spend on this? Would something like leafleting consume a huge amount of resources for the movement — yes or no?

4. How much would it cost to answer that question and update those priors that you have?

5. Is there actually room for more funding?

Based on these five components, we can arrive at a set of research priorities.



Again, this isn't quantified in the same way that it would be in medicine or health. And this is purely illustrative. There are still going to be differences of opinion among people in the community. But it makes explicit the assumptions behind why some of these initiatives should receive more research funding than others. It allows for some prioritization.

That's the first part of the proposal: that we need to create a long-term research agenda. I think if you look at the theory of change, a lot of people will say, “Hey, we're getting a huge bang for our buck from voluntary corporate campaigns, so let's double down on that.” But the point is that, over time, if we want to get to a dramatic reduction in the number of animals farmed, we need to start attacking other nodes in that theory of change. And to attack those nodes cost-effectively in five or 10 years, you need to start generating the research today about what's going to be most effective at those nodes.

The second part of the “NIH for animal welfare” proposal is that we need to institute evidence-grading standards to avoid the mistakes made in medicine and global development.



Some of you may be familiar with the fact that there's been a huge replication crisis in psychology and other social sciences. People have argued that most published research findings are false, and that there is an incredible waste of resources around poor medical research. We want to create and adopt the standards that have been used in medicine and global development to curb some of those ills in the animal welfare movement while the movement is still young. But we need to adapt them to the particularities of the animal welfare movement.



First, as with medicine and global development, standards of evidence quality should withstand academic scrutiny from other fields. Second, they have to be practitioner-focused, so that someone leading an animal welfare NGO or funding organization doesn't need to be an econometrician to be able to see the grading of a particular intervention and determine whether it is a good use of their money. Third, this evidence-rating system, as with medicine, needs to convey both the size of the impact of an intervention and its epistemological uncertainty. In other words, how certain are you that your estimate is correct?

I think the animal welfare movement needs to adopt a version of evidence-based medicine, which has a really simple rubric for all clinicians in order to determine whether to provide aspirin to a heart attack patient, order an x-ray for someone who has a presumed sprained ankle, or make some other decision. It's broken down into just two criteria.



First, using a rating of strong or weak, what is the potential magnitude of the effect? And second, how confident are you in your estimate of the evidence’s quality — do you have a low, medium, or high level of confidence in it? All interventions are based on these two criteria. It’s a very simple rubric. And while a lot of this rubric from medicine wouldn't apply to the animal welfare movement, you could adapt something similar to fit its particularities.



For example, instead of focusing on the inconsistency-in-publication bias, which might be less of a concern in the animal welfare movement, you might want to rate the movement’s literature on whether there is a robust measure of the decrease in people’s consumption of meat. A lot of studies rely on self-reported consumption of meat, which is very unreliable, versus grocery stores’ data on meat purchases. There are definitely significant alterations you would need to make to these criteria. But again, with a holistic, collaborative effort among different researchers in the animal welfare movement, you could arrive at something similar to what the NIH and the field of medicine have created.

This will not give you crystal clarity on what to do. Subjective judgment will always be involved. But it does allow for judgments that are made explicitly and with transparency around the underlying factors that led to a certain decision. There still might be debates on whether you should do leafleting, versus online ads, versus corporate campaigns. But having sector-wide, evidence-grading standards allows people to debate specific and important underlying factors and reasons.

Results-based financing mechanisms

The second proposal I’ve taken from medicine and global development is this: High-quality evidence about what works is simply not enough [to spur action]. There are a ton of great studies highlighting cost-effective interventions and organizations in the effective altruism community. GiveWell, J-PAL, and IDinsight can all attest to the fact that there is not enough uptake and scaling of programs that have proven to be effective.



Part of the reason for that is there are not sufficient incentives for organizations to deliver against results. Introducing results-based financing in the animal welfare movement has the potential to incentivize the creation of new solutions to intractable problems. For instance, a lot of people in the movement have said, “It's too hard to change individual behavior. We just need to wait until clean meat gets so good and so cheap that everyone's going to transition by themselves.” But the reality is if you incentivize it with financial returns and payment based on results, you might get more innovative solutions. We've seen analogs within global development. Second, it ensures that philanthropic dollars go toward programs that work.

There are a lot of different ways to adapt results-based financing mechanisms to the animal welfare movement. Some of our colleagues in the global development space at Instiglio have categorized those mechanisms into three broad buckets:




1. Performance-based contracts between a foundation or a government and an implementing organization. For these, a foundation or a government pairs off with an implementing organization, and the implementing organization gets paid based on the extent to which they achieve predetermined results, as judged by an independent evaluator.

2. Private-sector investors that shoulder the risk. There's a more complex mechanism that de-risks interventions for philanthropists. Private-sector investors put up risk capital. If the program works, they get a positive return on their investment. If it fails, then they lose that money. This allows a foundation to only pay for results. [Here's a real-world example of this form of financing.]

3. Prizes to incentivize competition. There are prizes like the XPRIZE for adult literacy. Basically, a huge pot of money incentivizes people to solve a particular problem — and then rewards whoever ultimately succeeds with a payout.

These are early-stage hypotheses, but my proposition is that the animal welfare movement can learn a lot from more mature social justice causes like medicine, public health, and global development. I think it makes sense for us to come together as a community to start evaluating which of those can be adapted to potentially accelerate animal welfare improvements for billions of non-human animals.

Thank you.

Nathan Labenz [Moderator]: One question occurs to me right off the bat. Since we're all humans, it seems like there's a more standard unit of account that can be put into place in medicine, whether it's quality-adjusted life years or something else like that. The conceptual problem I see is in trying to do something similar for different animals, would they count the same, or would they count differently? And do you think that this problem is amenable to the framework that you're proposing?

Neil: Great question. I think it’s a fundamental one, but not something that I talked about explicitly here. I do imagine some kind of “NIH for the animal welfare movement” doing some basic research along those lines. So, there are large unanswered or semi-answered questions around how to value the lives of different types of animals. And there's a lot of basic science research that needs to be done around better understanding different types of animals’ ability to feel pain and pleasure.

I imagine that if you establish an institute with clear evidentiary standards, you could start to lay the foundation for an equivalent of disability-adjusted life years (DALYS) in the animal welfare space. I know there are some researchers trying to do something like that. But the big benefit of a program like this is it doesn’t rely on individual researchers’ subjective judgment. It enforces across-the-board quality standards.

Nathan: Here’s a question from the audience. Animal advocacy researchers are studying how to influence the economic behavior of people in communities. By contrast, medical researchers are studying the biological effects of medical treatments in individuals. How relevant do you think that difference is?

Neil: I think that is a difference, but medical researchers are also studying the effects of interventions on populations in public health. And a lot of those are really analogous to interventions proposed by animal welfare movement. So much of public health and global development is about behavior change.

For example, how do you get people to test themselves for malaria or HIV/AIDS? And once they've tested positive, how do you get them to adhere to the program and the treatment in the same way you want to be able to test?

[In the animal welfare movement], you want to be able to test the effectiveness of interventions to get people who have expressed the desire to reduce their meat consumption to actually do it. So much of that is about individual behavior change and policy or regulatory reform. And those two things are quite analogous to large parts of both the medical and the development economics literature, even if there are certain segments of the medical literature that are more physiological.

I'd encourage anyone who works in animal welfare full-time to please contact me. We've been working with Lewis Bollard at the Open Philanthropy Project to see if any of these ideas have legs and are salient for the animal welfare movement. I’m really keen to get feedback from folks who are working on this 24/7.