Rethink Priorities Impact Survey

By Neil_Dullaghan🔹 @ 2019-11-21T19:20 (+54)

Summary

Two of our founding values at Rethink Priorities are transparency and impact assessment. Here we present the results of our stated intention to annually run a formal survey to discover if one of our target audiences, decision-makers and donors in the areas which we investigate, has read our work and if it has influenced their decision-making. Due to the small sample of 47 respondents, the disproportionate importance of some of these respondents, and the ability to highlight comments from only those who opted to share their responses publicly, the precise results should not be taken too seriously. It is also very important to note that this is just one of the many ways we are assessing our impact. Nevertheless, we will present the overall results and the results by cause area.

We found that a quarter of those who responded had read more than half of our work and the vast majority of our work was judged to be useful by at least 50% of those who read and rated it. 57% of respondents stated that our work had changed their beliefs. Our work did not influence any of these key players to make hiring decisions or start new charities. Instead, the majority of changes reported were in 22 unspecified ways (which we would be eager to find out more about in follow-up work), 13 instances of inspiring research in a topic, and influencing 7 donations.

Our work on the EA survey and invertebrate sentience was the most widely read and likely to have changed beliefs, while our work on corporate campaigns was the biggest influencer in changing actions. The majority of our work was judged to be useful, especially so for our work on fish stocking and our nuclear weapons series. Our work on PriorityWiki, RC Grants, and the cost-effectiveness of RC Forward were the least read and did not appear to be rated as particularly useful or influential by those who did read them.

Sample and survey

We ran a formal survey on SurveyMonkey of possible influencers, focused exclusively on funders and key decision-makers in our research areas. We sent the survey to 110 people and 47 responded.[1] We roughly estimate ~67 people we sent this to focus on animals and ~57 on longtermism (with some overlap). 10 respondents allowed us to release their responses publicly, 4 providing their names. We asked respondents to select which areas of our work they had heard of from a list of 13 options, including one placebo option of climate change. We then asked how useful they found this work, if it changed their beliefs, and if it changed their actions. The relevant question list is linked to in the appendix.

How widely read is our work

We regularly publish our research on both our blog and the EA Forum and disseminate it directly to key decision makers and influencers who we think it would be of value to. 44 of 47 respondents stated that they had heard of at least 1 of the 13 options of our work offered. 34 respondents (~77% of those who answered this question) selected the EA survey and 34 selected invertebrate sentience (not the same 34 respondents for both). A quarter of respondents had heard of 7 or more of the options and 75% had heard of at least 3-4 of the options. 6% of all respondents had not heard of any but still gave responses to other questions in the survey. All responses for “other” were for our work on ballot initiatives. 5 respondents reported that they had heard of our work on climate change, which was a placebo option. We do not believe that these respondents were disingenuous in their answers about this or other questions, but may have simply made a mistake. We should note that much of our work comprises a series of articles; the EA survey (1,2,3,4,5,6,7,8,9,10,11,12); invertebrate sentience (1,2,3,4,5,6,7,8,9,10,11,12,13); nuclear weapons(1,2,3,4,5); wild animal welfare(1,2,3) and so we do not know from this data whether respondents had read the entirety of each work or just a portion.

N.B. images can be viewed in full size if opened in a new tab.

Usefulness

44 out of 47 respondents stated how useful they found our work. The options were “not at all useful”, “not so useful” “somewhat useful”, “very useful” , and “N/A”. For 10 of the 13 options, 50% or more found our work somewhat or very useful. Those who selected "other" and wrote in an answer not listed (all writing ballot initiatives) only reported it as very useful, and our work on fish stocking received the next largest percent of somewhat or very useful ratings (82%). Our nuclear weapons series received the largest percent of “very useful” (50% of those who read it and responded to this question) and the EA survey received the highest percent of “somewhat useful” (58%) with ~80% of those who read at least one of these two and responded to this question rating them somewhat or very useful. In absolute numbers, our EA survey and invertebrate sentience work received the highest number of respondents rating them as somewhat or very useful (25 and 24 respectively). Our work on PriorityWiki and RC Grants received more not useful than useful responses. Of those respondents that opted to allow their comments to be shared publicly, we can see that our work in their most relevant field was judged to be useful. We will detail these responses in the cause area sections below.

Grouping our work into the loose cause area categories of animal welfare, longtermism, and all other work, 65% or more of responses for each category were that the work was somewhat or very useful. We break down these categories further in the next sections.

Changed beliefs

27 out of 47 respondents stated that our work had affected their beliefs. The EA survey was the most common response (15) followed by invertebrate sentience (13). PriorityWiki and the placebo option of climate change received the lowest response rate with 0 of those who read it and responded to this question saying it affected their beliefs.

Of those respondents that opted to allow their comments to be shared publicly, their changes can be grouped into the loose categories of information acquisition, cost-effectiveness updates, and prioritisation. Three respondents suggested that our work had provided them with new information about the number of animals affected, especially rodents and fish, while Max Daniel, Senior Research Scholar at the Future of Humanity Institute (FHI) , commented that the EA survey provided information on the composition of the EA community and our work on invertebrate sentience and nuclear weapons was “mostly a meta-level update that useful work can be done in these areas with relatively standard academic methods”. One unnamed respondent noted that rather than changing their mind, the research was “informative and a useful way to build my FAW [ed: farm animal welfare] knowledge without having to spend a lot of time finding this information myself”. An unnamed respondent noted that our work made them realize less is known about corporate campaign effectiveness than they realized, but still provided them with an estimate of cost-effectiveness, while another stated that they were less optimistic about how useful some cost-effectiveness estimates are. Finally, our EA survey work informed one repsondent’s beliefs about what geographic regions favor which cause areas, while another found that it decreased their confidence that EA is valuable. Our longtermism work caused one respondent to significantly update their expectations that nuclear weapons were threatening.

Changed actions

35 out of 47 responded to the question of has any of our research caused you to change your actions? [2] Our work was cited as an influence in 7 donations, 13 changes in research for the respondent (or their organization), 0 hiring decisions, 0 charities started or not charted, and 22 other actions, which were either not specified or not publicly shared. Our work on corporate campaigns influenced the most actions (10) followed by our work on invertebrate sentience and the EA survey (6 each). We provide further details on these changes, and responses that our work did not change their actions, in the sections below.

Animal welfare

In 2019 we continued our focus on farmed and wild animal welfare, as we still see animal welfare as an important, neglected, and uniquely tractable issue that our team is well-poised to address. Therefore, tracking our impact in this area is especially important. 41 of 47 respondents stated they had read our work on animal projects (corporate campaigns, fish stocking, invertebrate sentience, rodents fed to pet snakes, SI slaughterhouse ban survey replication, ballot initiatives, wild animal welfare). Our work on invertebrate sentience was the most read among these respondents, while the write-in option was the least read and the SI slaughterhouse ban replication, which is forthcoming, was the least read of the given topic choices.

Among these respondents, 38 of 41 responded to if our animal work was useful. Our invertebrate sentience work attracted the largest absolute number and largest percent of “very useful” ratings among those who read it and answered this question. None found our work on corporate campaigns, fish stocking, or wild animal welfare not at all useful. Across all our animal work, 61% or more found our animal work somewhat or very useful. Our work on rodents fed to pet snakes attracted the highest number and percent of not at all useful ratings, though 61% of those who read it found it somewhat or very useful. Of those respondents that opted to allow their comments to be shared publicly, David Coman-Hidy, President of The Humane League, found our corporate campaigns and invertebrate sentience work somewhat useful and Wild Animal Initiative researcher Hollis Howe found our invertebrate sentience work very useful and our wild animal welfare work somewhat useful.

Among respondents our animal work was cited as an influence in 5 donations, 9 changes in research for the respondent (or their organization) , 0 hiring decisions, 0 charities started or not charted, and 15 other actions. Note, respondents could select multiple changes to their actions resulting from a single piece of work and so totals may exceed the number of people who read each work. Our work on corporate campaigns affected the largest number of actions (10) ; 2 donations, 4 research inspirations, and 4 other actions. Our SI Slaughterhouse Ban Survey Replication, which is forthcoming, changed the fewest changed actions (1) , while invertebrate sentience attracted the most number of responses stating it did not change their actions (22).

You can see more comprehensive look (without first wave data) at the results of our survey for animal respondents here: https://tinyurl.com/rp-impact-survey-animal

Longtermism

In 2019, we also allocated time to working on reducing catastrophic risks to the survival of humanity, resulting in our series on nuclear weapons. 17 of 47 respondents stated they had read this work. Among these respondents, 14 of 17 responded to if our work was useful. 50% (7) of those who read it and answered this question found it very useful. 0 found it not at all useful. Of those respondents that opted to allow their comments to be shared publicly, Gregory Lewis of FHI found our work on nuclear weapons very useful and Max Daniel, also of FHI, found our work on nuclear weapons somewhat useful.

Among respondents, our nuclear weapons work was cited as an influence in 1 donation, 2 changes in research for the respondent (or their organization) , 0 hiring decisions, and 0 charities started or not charted, 1 other action. 10 reported that it did not change their actions. Max Daniel (FHI) reported that our work on nuclear weapons affected a change in research for the respondent (or their organization) , while Gregory Lewis (FHI) indicated it did not change any actions.

You can see more comprehensive look (without first wave data) at the results of our survey for animal respondents here: https://tinyurl.com/rp-impact-survey-longtermism

All other work

Of our remaining other work (the EA survey series, Priority Wiki, RC Grants: Review of Donational, RC Forward cost-effectiveness) , 34 of 47 had read at least one of these. Relative comparisons between these works should not be taken too seriously as they span a range of cause areas and are arbitrarily lumped together here.

Among these respondents 30 responded to if our work was useful. The EA survey received the largest absolute number of “very useful” ratings (7) and “somewhat useful ratings (18). One of those who read our work on RC Forward’s cost-effectiveness found it not at all useful, and 2 found it somewhat useful. Only 1 of those who read our work on PriorityWiki and answered this question found it useful, while the other 4 who read it and answered this question did not find it useful.

Of these areas of our work, only the EA survey changed respondents’ actions among the 24 who answered this question. Our EA survey work was cited as an influence in 1 donation, 1 change in research for the respondent (or their organization) , 0 hiring decisions, and 0 charities started or not charted, and 4 other unspecified actions. Our work on Priority Wiki, RC Grants: Review of Donational, RC Forward cost-effectiveness did not change the actions of respondents.

You can see more comprehensive look (without first wave data) at the results of our survey for other work respondents here: https://tinyurl.com/rp-impact-survey-allotherwork

Conclusion

This survey is one of the many ways we hope to capture our impact and we intend to continue running it each year. We expect to improve future surveys by offering greater ability and prompts to uncover what the many unspecified changed actions were as a result of our work. We also plan to more explicitly prompt respondents on the amount of money in donations which was influenced and the result of any research conducted that was inspired by our work.

Credits

This essay is a project of Rethink Priorities. It was written by Neil Dullaghan with contributions by Marcus Davis. Thanks to Peter Hurford for comments. If you like our work, please consider subscribing to our newsletter. You can see all our work to date here.

Appendix

The survey questions discussed above can be found here: https://tinyurl.com/rp-impact-survey-questions

  1. There were an additional 5 respondents who began the survey but did not provide answers for any question and thus were excluded from this analysis. ↩︎

  2. This question initially had an error that prevented respondents from selecting multiple options. 4 respondents were in a first wave before that error was corrected. As a result, the results summarized data in the PDFs shared are not complete. We still have this data and include their results in the numbers discussed, but the results are excluded from the summary analysis displayed in the shared documents PDFs and spreadsheets. ↩︎

Max_Daniel @ 2019-12-02T19:19 (+21)

Thanks for posting this! I'd really like to see more organizations evaluate their impact, and publish about their analysis.

Just a quick note: You mention that I indicated I "found [y]our work on nuclear weapons somewhat useful". This is correct. I'd like to note that the main reason why I don't find it very useful simply is that I currently don't anticipate to work on nuclear security personally, or to make any decisions that depend on my understanding of nuclear security. In general, how "useful" people find your work is a mix of their focus and the quality of your work (which in this case AFAICT is very high, though I haven't reviewed it in detail), which might make it hard to interpret the results.

MichaelA @ 2020-03-07T11:24 (+6)

Thanks for this - I found it very interesting both with regards to Rethink Priorities' impact in particular, and with regards to one way in which EA research organisations in general could evaluate their impacts.

We also plan to more explicitly prompt respondents on the amount of money in donations which was influenced and the result of any research conducted that was inspired by our work.

That definitely sounds interesting. As a reader, I'd have been interested in at least a vague sense of what ballpark the donation sizes were likely to be in - e.g., were most of the reported donations influenced from individual EAs who aren't on extremely high salaries, or from (decision-makers for) fairly substantial funders like the EA Funds or OPP. But I can also understand why any vague sense of that might be too difficult when you also want to maintain anonymity and not lead people to focus too strongly on a maybe fairly made-up guess.

Also, when I saw one of the "actions" was "Inspired you or someone in your organisation to research this topic", it occurred to me that perhaps an additional valuable action could be "Allowed you or someone in your organisation to use time and other resources for things other than researching this topic". This could occur if:

And this could be valuable if it "frees up" those people to do things that are more valuable than researching that topic (given that RP has done the research it has done).

It's possible that that would be one of the "Other" actions people indicated having taken.

I imagine "influencing donations" would've captured the equivalent for donations - e.g., people deciding to donate to something other than an intervention RP investigated, because RP's work negatively updated their estimates of that intervention's value. But it might be interesting to somehow check if respondents interpreted that "action" that way, rather than just as meaning "Influenced me to donate to this intervention that RP researched".

Two last quite minor points about things I found slightly unclear: