Effective animal advocacy bottlenecks surveys

By Jamie_Harris @ 2021-01-13T13:38 (+31)

This is a linkpost to https://www.animaladvocacycareers.org/post/effective-animal-advocacy-bottlenecks-surveys

Abstract

A better understanding of the problems that limit the efficiency and total effects of the animal advocacy movement is useful for advocates, movement-builders, and individuals making decisions about careers that help animals. We therefore ran a survey of leadership and hiring managers at effective animal advocacy nonprofits and a survey of researchers, grant-makers, plus others working on “meta” and movement-building services for the movement. Lack of funding was identified as a key bottleneck in both surveys, with lack of (qualified and capable) applicants for paid roles not far behind. Other results suggest that a high threshold is required for individuals to help animals more by choosing careers focused on donating money to nonprofits rather than donating time and labour. Organisations face the most substantial hiring difficulties for roles focusing on leadership or senior management, fundraising or development, and government, policy, lobbying, or legal tasks. There are few differences by role type in terms of retaining staff, though fundraising and development stands out as slightly more difficult than average. Headhunting and training for current staff of animal advocacy organisations were commonly suggested as possible solutions to talent bottlenecks. Some differences emerge when comparing responses from organisations based in the global South to those based in the global North, organisations operating primarily in Asia to those operating primarily elsewhere, or with large numbers of employees to those with small numbers of employees.

Introduction

Shortly after founding AAC, we conducted a brief survey about talent bottlenecks. That survey substantially informed the work that we carried out in 2020, so we were keen to improve the question wording, seek input from a wider variety of stakeholders, and update the results.

We therefore ran two surveys with partly overlapping questions:

The questions wordings were optimised for usefulness in AAC’s own decision-making.[1] However, we hope that the results can also be useful to:

Methodology

For the direct work survey, we contacted organisations that met the following inclusion criteria:

We also applied the following exclusion criteria:

For the direct work survey, we contacted 56 organisations and received 39 responses from 39 different organisations (70% response rate).[4] We asked for one representative response from each surveyed organisation, noting that this “should be filled out by someone in a top-level leadership role (e.g. CEO, ED, COO, VP) or a hiring manager/human resources staff member.” All except three respondents identified themselves as being in a “leadership (e.g. CEO, ED, COO, VP)” role.[5] You can see the questions used in this survey here.


We conducted subgroup analyses for nonprofits with each of the following focuses:

Most groups were included in more than one category. A group was only included if it seemed to spend a substantial amount of resources on this type of action.[8] This involved some subjective judgement calls; certain organisations did not clearly report their spending by intervention type.


We conducted several other subgroup analyses:

Given the small sample sizes involved and very large number of comparisons being made, we decided not to conduct formal significance tests to check whether the differences between subgroups were statistically significant. The results of the various subgroup analyses can be seen on this spreadsheet. That spreadsheet also includes a column where we treat nonresponses to a question as the lowest possible score (0, 1, or no selection) rather than just excluding non-responses from the analysis.

The main results reported below are either the mean score from respondents (for quantitative questions) or the percentage of respondents that selected a particular option. We were concerned that treating all responses to the survey equally (taking the mean of responses without any adjustment) might present an unrealistic account of the movement’s needs and bottlenecks. To address this concern, we also tried several weighting systems for quantitative questions:

These weighting systems made surprisingly little difference to most questions.[14] Hence, we decided to only include these weighting systems on the subgroup analyses spreadsheet, rather than including them in the main results tables below, as we had initially intended.

For the meta survey, we used less formal inclusion or exclusion criteria, contacting organisations working on research, grant-making, and other movement-building services whose input we expected to be useful. We did not suggest that we were hoping for the response to be representative of their organisation, though in most cases we still only asked for one response from each organisation, to reduce the time spent filling out the survey. We contacted 21 organisations and received 16 responses from 13 different organisations (62% response rate). You can see the questions used in this survey here. Given the small sample size, we did not conduct any subgroup analyses or apply any weightings.

Since a key goal of the surveys was to see how respondents’ answers differed from our expectations, we did not include our own responses in either survey; instead we jotted down predictions and rough notes and compared the results to those. We did not formally pre-register our analysis plans, but did informally plan most of our analysis in advance.[16] We decided not to provide the full dataset, in order to protect the anonymity of participants, though we are open to requests to conduct additional analyses.

Results and discussion

In the writeup below, we include qualitative comments about differences between subgroups or weighting systems that we thought were noteworthy, though you may prefer to ignore these and look at the results of the subgroup analyses for yourself. As well as a list of general limitations at the end, we note limitations that apply to particular questions as we go along. A single asterisk (*) signifies that this was a concern we had written down before seeing the results. Two asterisks (**) signify limitations that we had not necessarily thought of but that were highlighted by respondents to the questions themselves.[17]

The importance of different bottlenecks

We asked direct work respondents the following question: “To what extent does each item below limit your organisation’s efficiency or impact?” We asked meta respondents the same question but replaced “your organisation’s” with “the EAA community’s,” which was defined as “the group of people and organisations who are focused on maximising their positive impact for animals.”[18] We offered them the following options: “1 Not at all”, “2”, “3 Somewhat”, “4”, “5 Very much”.

Here are the average scores:

Differences by subgroups and weightings:

The following limitations should be borne in mind:

The value of donations relative to high-quality direct work

As the above results show, there are clearly many different types of bottlenecks that can affect the animal advocacy movement. However, one particularly salient tradeoff in many decisions — both for organisations’ leaders and for individuals seeking to maximise their positive impact for animals over the course of their careers — is the value of donations relative to high-quality direct work.

We asked direct work respondents the following question: “Imagine that someone has been working for 10 years building up experience and expertise that would make them an excellent candidate for one of the roles that is *hardest to hire for* in your organisation. Would you be more excited about that person applying for one of those roles at your organisation, or donating money to your organisation that was the equivalent of 50% of the salary of that role?” We then asked them the same question again but replaced “one of the roles that is *hardest to hire for* in your organisation” with “*a campaigns, corporate engagement, or volunteer management* role in your organisation.”[25]

For both questions, we offered them the following options: “Much more excited about them applying” (coded as 1), “Somewhat more excited about them applying” (2), “Roughly similarly excited either way” (3), “Somewhat more excited about them donating” (4), and “Much more excited about them donating” (5).

Here are the average scores:

These results seem like a vote in favour of careers in direct work in general, even for role types that are not necessarily “hardest to hire for.” We note below that the average salary seems to be around $50,000 or so in animal advocacy nonprofits in the global North, though the average salary might be higher for roles that are “hardest to hire for,” such as leadership and senior management roles. So you could interpret these results as suggesting that organisations would rather receive one additional very-high quality applicant for the roles that are hardest to hire for than receive $25,000 or more (each year for the length of time that the applicant might otherwise have been employed for).

The following limitations should be borne in mind:

As another way of understanding this trade-off, we asked meta respondents the following question: “Imagine an individual who is skilled and motivated enough to be a good (but not outstanding) candidate for roles in effective animal advocacy nonprofits. I.e., after a few applications, they are likely to secure a role, but they are not likely to be substantially better than the next best candidate, at least in their first paid role. How much money would you estimate that that person would have to be able to donate per year, on average, to effective animal advocacy nonprofits, to be indifferent (from an impact perspective) between focusing on a career “earning to give” vs. a career in animal advocacy nonprofits?”[28] The average answer given was $28,200, with a range from $100 to $90,000.[29]

The following limitations should be borne in mind:

The importance of different talent bottlenecks

We asked direct work respondents: “How difficult is it to *hire* high-quality candidates for the following categories of roles or types of expertise?” We then asked them the same question again, where “*hire*” was replaced with “*retain*”. We offered them the following options: “1 Not at all”, “2”, “3 Somewhat”, “4”, “5 Very much”.

Here are the average scores:

Differences by subgroups and weightings:

The following limitations should be borne in mind:

We didn’t explicitly ask respondents whether they saw hiring or retention as more of an issue, but the survey provides some evidence that respondents see hiring as more of an issue.[37]

Allocation of talent by sector

The above results focus on effective animal advocacy nonprofits. However, working in these nonprofits is certainly not the only way that an individual can help animals over the course of their career. Hence, we also asked meta respondents how they thought that “the next 100 highly motivated, highly competent individuals looking for careers where they can have a positive impact for animals” should divide their efforts between different broad career paths.[38]

Here are the average percentages given for each sector:

The following limitations should be borne in mind:

Organisational questions

We asked respondents to the direct work survey several organisational questions. Here are the average answers:

There were substantial differences between organisations based in the global North and the global South. Average salaries were $52,039 in the former[47] and $10,122 in the latter. Estimated hiring costs were $2,548 in the former and $377 in the latter. The average spend on staff development and training was apparently higher in the global South ($2,133.3) than the global North ($1,996), though with one outlier excluded (which we expect is a mistake[48]), the average for the global South is only $560.

Though the samples and methodologies do not match up exactly, we can compare these figures for respondents from the global North to figures from research into organisations in other contexts. In each case, the numbers seem surprisingly close:

The following limitations should be borne in mind:

Possible solutions

We asked respondents to both surveys: “Which of the following specific interventions would you be most optimistic about Animal Advocacy Careers offering? (Select up to 3)”

For AAC, these results updated us most substantially towards being more favourable towards headhunting and less favourable towards creating new skills profiles or improving our existing ones.

Differences by subgroups and weightings:

The following limitations should be borne in mind:

We asked respondents to both surveys slight variations on the following question: “If AAC could provide all-expenses paid, high-quality training for staff members in effective animal advocacy nonprofits whose job descriptions refer primarily to tasks in one of the following categories, which type of training would you be most excited about?”[52]

There are some confusing results here regarding management and leadership. Direct work respondents selected training for leadership and senior managers most frequently but training middle or junior managers nearly least frequently. Meta respondents did almost the exact opposite.

Differences by subgroups and weightings:

The following limitations should be borne in mind:

General limitations

Many of the limitations listed here could be influenced by our decisions in designing and running the surveys. So feedback on these limitations and decisions is especially welcome in the sense that it might help us to improve surveys in future years.

Footnotes

[1] In order to decide where to focus its interventions, Animal Advocacy Careers (AAC) needs to understand what the largest bottlenecks are in the farmed animal movement, i.e. the problems that most substantially limit its efficiency and total effects. We are most interested in understanding how different sorts of talent bottlenecks compare, but information about the importance of talent bottlenecks relative to other issues sheds some light on the usefulness of AAC’s work in general, as well the sorts of advice that we should give to individual job seekers.

[2] In most cases where this was the only reason for exclusion, these organisations were invited to participate in the “meta” survey instead.

[3] For example, many universities were excluded, where it would have been difficult to get survey results that had much relevance to their work on farmed animals. Some groups were excluded if they seemed to be primarily animal agriculture industry bodies; we expected these organisations to have very different needs and difficulties, though there were many borderline cases.

[4] The direct work respondents that gave us permission to share their organisation’s name were: ACTAsia, The Albert Schweitzer Foundation, Anima International, Animal Equality, Animal Kingdom Foundation, Animal Nepal, Animal Rights Center Japan, The Aquatic Life Institute, Brighter Green, Compassion in World Farming USA, Dyrevernalliansen, Environment & Animal Society of Taiwan (EAST), Equalia, Essere Animali, Eurogroup for Animals, Fair-Fish International Association, Fish Welfare Initiative, The Good Food Institute, The Greenfield Project, The Humane League, The Humane Society of the United States Farm Animal Protection, Mercy For Animals, New Harvest, People for Animals Uttarakhand, ProVeg International, Sentience Politics, Sentient Media, Sinergia Animal, Sociedade Vegetariana Brasileira, SPCA Selangor, Vegan Outreach, Veganuary, VegeProject Japan, We Animals Media, and World Animal Protection.

[5] We would have categorised all three of these individuals as leadership anyway.

[6] Unfortunately, due to the small number of respondents in this category, we had to exclude results of this subgroup analysis from the subgroup analyses spreadsheet to protect the respondents’ anonymity.

[7] This included legal actions, investigations, capacity-building programmes, journalism, and any group with a substantial focus on animals other than farmed animals.

[8] Where we identified financial reports or reviews from Animal Charity Evaluators with clear breakdowns of spending by programme type, we included an organisation in a category if it spent 10% of its budget or more on a particular intervention type. If we did not find this sort of financial data, we relied on descriptions on the organisation’s website.

[9] Note, the global North vs. global South divide is a socio-economic and political categorisation, not simply a geographic one. We followed this map for our categorisations.

[10] Unfortunately, the change in recommendations after we sent out the surveys meant that some organisations that are now recommended were not invited to participate.

[11] We gave the guidance: “E.g. you might count staff who work 20 hours per week as 0.5 each. Please include all current employees and independent contractors, regardless of the length of their contracts.”

[12] We asked the question “How many full-time equivalent, paid staff are there in your organisation?” and gave the guidance: “E.g. you might count staff who work 20 hours per week as 0.5 each. Please include all current employees and independent contractors, regardless of the length of their contracts.”

Participants’ responses to a particular question were multiplied by their response to the question about the number of FTE paid staff. This number was then divided by the sum of responses to the question about FTE paid staff across all organisations which participated in the question of interest.

Most surveyed organisations focus exclusively on farmed animals. However, this was not the case for some of the largest organisations surveyed. To prevent their answers dominating in the weighting system and to reflect the fact that we are most interested in the farmed animal movement, we counted these organisations as having only 51 full-time paid employees in the weighting. This number was chosen fairly arbitrarily (representing the boundary of the medium and large size categories) as we do not know the number of staff working specifically on farmed animal issues. Making this change had quite a substantial effect on some results.

For the one organisation that did not provide an answer about the number of full-time equivalent, paid staff that they had, we inputted an answer from a publicly available document which specified the number of staff they had in 2017. This number is presumably not accurate, but may still be roughly correct; using this number enabled us to include them in the weighted analyses.

[13] In weighting 2, current ACE “Top charities” were given a multiplier of 1.5, “Standout” charities a multiplier of 1.25, former ACE top or standout charities a multiplier of 1 and organisations that had not yet been recommended by ACE a multiplier of 0.75. This multiplier was then applied to their score from weighting 1, i.e. their response to a particular question multiplied by their response to the question about the number of FTE paid staff. This number was then divided by the sum of (responses to the question about FTE paid staff multiplied by their ACE recommendation multiplier) across all organisations which participated in the question of interest.

Weighting 3 was the same except that the multipliers used were 10, 5, 2, and 1, rather than 1.5, 1.25, 1, and 0.75.

[14] Across the 33 numerical scores for which we calculated an average score, the average difference between the unweighted score and weighting 1 score was 0.11. (For the purpose of this calculation, regardless of whether the unweighted or weighting 1 score was higher, the difference that was included in the average was a positive number.) The average difference between the unweighted score and weighting 2 score was 0.14. The average difference between the unweighted score and weighting 3 score was 0.25. (The differences in this footnote were calculated before we added in the responses for one late respondent.)

[15] For two of the organisations, we invited two individuals with very different roles. All four of these “duplicate” invited individuals provided responses. Two individuals from another organisation also both provided responses. The meta respondents that gave us permission to share their organisation’s name were: Animal Charity Evaluators, Animal Ethics, Charity Entrepreneurship, Faunalytics, The Greenbaum Foundation, The Humane League (Open Wing Alliance), Open Philanthropy, Rethink Priorities, Sentience Institute, and World Animal Net.

[16] We added weighting 3 as a robustness check after seeing how little difference there was between weighting 2 and the unweighted results. We added a subgroup analysis for Asian organisations compared to non Asian organisations after realising that several Asian organisations could be classified as being part of the global North, despite perhaps being in a very different context to other organisations in the global North. We added one late respondent to the analysis after seeing the initial results (which made little difference to the results).

[17] You might be inclined to take asterisked limitations more seriously; these are less likely to be spurious criticisms that we invented after seeing surprising results in order to justify our prior beliefs.

[18] We added that “this includes nonprofits (including but not limited to those recommended by Animal Charity Evaluators), companies producing or selling alternative foods to animal products, plus individuals pursuing careers in other areas (e.g. politics and policy, academia, animal law, or earning to give) and aiming to maximise their positive impact for animals.”

[19] Changed wordings for meta respondents were: “Unfavourable attitudes among targets of advocacy, e.g. companies, government, consumers),” “Lack of public awareness of the work of specific EAA organisations,” “Difficulties coordinating between EAA organisations and stakeholders,” and “Difficulties coordinating internally within specific EAA organisations.”

[20] This is perhaps unsurprising given the substantial support for corporate welfare campaigns from Open Philanthropy Project, Animal Charity Evaluators, and The Humane League’s Open Wing Alliance.

[21] Researchers have tended to agree that institutional tactics should be prioritised over individual tactics. This has been reflected in the funding from groups like Open Philanthropy and Animal Charity Evaluators.

[22] The range of average scores was wider for the South (1.8 to 3.8 rather than 1.9 to 3.3), though this could just be due to the smaller sample size meaning that outliers have a larger effect on the average score.

[23] After being asked, “(Optional) Is there anything you’d like to add? E.g. other factors that limit the EAA community’s efficiency or impact, elaboration on any factors you marked as a 4 or 5,” one respondent suggested each of the following factors:

[24] Exceptions might be if you think that the targets of advocacy just need behavioural support to actually act on their values or if the targets of advocacy are very specific groups of individuals whose actions are constrained by other stakeholders.

[25] The logic of the two options offered in the question wording — “working for 10 years building up experience and expertise” vs. “50% of the salary of that role” — was that these both seem like goals that many individuals in the global North could realistically achieve if they strove for it over a long period of time. Our “Effective Animal Advocacy Nonprofit Roles Spot-Check” identified an average salary among advertised roles of $42,000, or $50,600 if roles from India, Mexico, Brazil, and France are removed. Individuals in well-paid roles in the corporate sector might well be able to donate ~$25,000 a year if they focused their career on earning as much money as possible.

Additionally, we were keen to keep the question as easily comprehensible as possible to reduce difficulties of interpretation of the results, so the round numbers of 10 years and 50% were convenient.

[26] As shown in the section below, “Campaigns, corporate engagement, or volunteer management” had a moderate rating relative to other role types, in terms of difficulty in hiring and retaining high-quality staff.

[27] See also 80,000 Hours’ articles on the highest paying jobs and on earning to give more generally.

[28] We provided the following guidance: “Of course, there are many factors that could substantially influence this answer, but please try to answer just assuming your best guess or the “average” for those other relevant factors, rather than the best or worst case scenario. Please give your answer in US dollars.”

[29] One direct work respondent also commented (without having seen the wording of the question given to meta respondents) that “top talent is a lot more valuable than money for most hard-to-fill roles… until the person is giving at least $250,000/year, IMO.”

[30] Though using weighting on this version brings “Natural sciences” scores back up to a similar amount.

[31] This was more so for the role type that they were less specialised in, i.e. “Corporate/producer welfare campaigns” organisations struggled more with “Government, policy, lobbying, or legal” and “Political campaigns or engagement” organisations struggled more with “Campaigns, corporate engagement, or volunteer management.” Both also struggled a little more with “Research” roles.

[32] To a lesser extent, the same was true for “Campaigns, corporate engagement, or volunteer management” roles.

[33] This matches up with the higher ratings for limits in effectiveness and efficiency caused by “Lack of (qualified and capable) activists and volunteers” and “Lack of (qualified and capable) applicants for paid roles” for organisations working primarily in Asia (4.2 and 3.9) than elsewhere (2.1 and 2.6).

[34] For example, one respondent noted that they were “unsure” about the second part due to the newness of their organisation, but filled in every answer with “3” (in this case, we manually removed the responses to count them as non-responses, though others may have done this and just not told us). One respondent noted that if something didn’t apply to them, they filled in “1”. One of the respondents that we trialled the survey with verbally (before having added the guidance about skipping items) seemed to answer such questions based on their impression of the experiences of other organisations.

[35] The relevant comments were:

[36] The relevant comments were:

[37] The evidence is that:

[38] We noted at the top of the section: “IMPORTANT: For the questions in this section, imagine the next 100 highly motivated, highly competent individuals looking for careers where they can have a positive impact for animals. Imagine also that they are all completely flexible about which sorts of roles that they apply for, but that once they pick a career path, they are unable to switch out of it for at least 5 years. Imagine also that you have complete control over where to allocate their efforts. In your answers to the questions below, please ensure that your answers add up to 100%. I.e. each category should have a number between 0 and 100, with an average of 12.5.” We then asked “What percentage would you get to focus their careers on: X?” eight times, replacing X with each of the sectors in the table below.

One respondent provided answers that added up to 97%, rather than 100%; since the numbers are close, we decided to include this respondent anyway. One respondent answered one option with 90% and another with 10%; they left the others blank, but we manually filled these in as 0, since this appeared to be their intention.

[39] Specifically, four individuals gave the examples of overlap of “Related legal work” with other categories: “Nonprofits” (twice), “Politics and policy,” and “Academia.” One respondent also noted possible overlap between “Nonprofits” and “Politics and policy”; in this case, our intention had been for the latter category to represent roles in public institutions rather than nonprofits, though this was not clear.

[40] One respondent noted this and commented: “I would want many more people in China to work on starting a for-profit alternatives company, many more people in SE Asia working in policy, many more people in the US working in existing organizations.”

[41] The respondent noted that they would have redistributed half of their “Working at existing nonprofits” responses into “Setting up new nonprofits” if they were responding for individuals late in their career.

[42] For example, you might think that those with high competence and motivation should most often focus on working directly in nonprofits (or animal product alternatives companies), but that other individuals should most often focus on earning to give. Large numbers of such individuals could increase the funding pool of the movement, but they would be unlikely to be better than other candidates for direct work. I.e. earning to give might be their comparative advantage even if they do not have outstanding personal fit with it.

[43] See footnote 12.

[44] We provided the guidance: “If needed, multiply hourly rates by 2,000 to convert to the equivalent full-time, annual salary.”

[45] We provided the guidance: “Include the time of other staff involved in the process, but not the salary of the newly hired worker. Exclude onboarding costs.”

[46] We provided the guidance: “Include any salary costs from staff time (delivery and participation), operational costs, and payments to external providers.” In this figure, we have excluded one suspected error.

[47] Our “Effective Animal Advocacy Nonprofit Roles Spot-Check” identified an average salary among advertised roles of $50,600, if roles from India, Mexico, Brazil, and France are removed. (France was removed because the relevant data came from one organisation which seemed to be an outlier.) This matches the figure in this survey for the global North quite closely.

[48] There were actually two outliers, both from organizations based in the global South. We contacted both organizations for clarification but only one responded; this organization corrected the figure given from $20,000 to $1,000.

[49] Here are the main identified examples from academic research:

[50] If we focused only on respondents from “large” organisations in our survey (51 staff or more) and “small” organisations in the magazine’s survey (100 to 999 employees), then the figures are $3,288 and $1,511, respectively, suggesting that animal advocacy nonprofits might substantially outspend other organisations on staff training.

[51] Indeed, if we adjusted for this bias, then the responses of the meta and direct work organisations might look very similar, with the exception of much lower support for “Mentorships between staff in industry/government and effective animal advocacy organisations” among meta respondents and somewhat higher support for “Mentorships between people with more experience and less experience in the effective animal advocacy movement.”

[52] The reported wording is from the meta survey. The respondents in the direct work survey were asked about their organisation specifically: “If your organisation could have all-expenses paid, high-quality training for all the staff members whose job descriptions refer primarily to tasks in one of the following categories, which type of training would you be most excited about?”

[53] However, we did not previously expect that respondents would be biased in this direction, because you might expect that they would also have an inflated sense of their own talents and therefore not believe that they would need training on management and leadership. In fact, one of us expected that these respondents would be biased towards selecting “Middle or junior managers” over “Leadership or senior managers” for this reason.

[54] Though we have low confidence in these views, concerns of our’s include: