Does the US public support radical action against factory farming in the name of animal welfare?

By Neil_Dullaghan🔹 @ 2022-11-09T22:13 (+128)

Summary 

• Surveys from Sentience Institute (2021, 2020, 2017) and Norwood & Murray (2018) showed substantial levels of support in the US for banning slaughterhouses (~39-43% support). Evidence of this considerable public support for radical action has been suggested as a reason for animal advocates to push stronger messages and bolder proposals against animal agriculture. 
• A preregistered study that we present here casts doubt on how substantial support for such radical action against factory farming actually is. In an experiment of 700 US survey respondents, we found 7.9% support (95% CI [4.3% - 14.0%], weighted results), when arguments framed around animal welfare for and against are presented, and respondents are asked to explain their reasoning. We also found 20.4% support (95% CI [11.0%-34.7%], weighted results) in the control condition when respondents were not asked to explain their reasoning. 
• In the second survey of 2,698 US respondents, we found 15.7% (95% CI [13.0%-18.8%], weighted results) support for a policy banning slaughterhouses, when arguments framed around animal welfare for and against are presented, and respondents are asked to explain their reasoning. 
• The attitudes expressed by poll respondents in response to broad questions may not be reliable indicators of actual support for specific policies or messages. It would be better to test people's responses to more detailed messages and policy proposals, paying special attention to how radical messages compare to counterfactual moderate messages.

Methods
 

We ran two online surveys: survey 1 in July and survey 2 in August-September of 2022. In both, respondents were presented with a proposal that included a definition of slaughterhouses and arguments framed around animal welfare for and against the proposal. In survey 1, the treatment condition asked respondents to explain their reasoning, with this prompt removed in the control condition. In survey 2, only the treatment condition was presented. The question wording was as follows:

Some members of Congress are proposing that slaughterhouses (where farmed animals are killed to then be sold as meat) should be made illegal.

Supporters of this policy say that slaughterhouses should be banned because it is wrong to kill animals. There is no way to kill animals for their meat which is “humane,” so this should be banned.

Opponents of this policy say that people have a right to eat meat if they choose. The practices in place are humane and produce quality meat for consumers at an affordable price. It is possible to prevent animals being killed inhumanely without banning slaughterhouses.

Do you support or oppose this proposed policy? 
[Support/Oppose/Don’t Know]

[Treatment condition] 
Please explain your reasons for the answer you gave above.

Respondents were recruited using Prolific and surveyed using Qualtrics. In survey 1, respondents were filtered to be those currently living in the USA, fluent in English, de facto over 18 years old due to filtering Prolific does automatically, and to set gender quotas so that the sample is 50-50 female-male. The total sample size was 700 with 350 in the control condition and 350 in the treatment condition. In survey 2, respondents were recruited as in survey 1 and were weighted for representativeness using 5-year 2019 American community survey data and general social survey data. At the time of analysis, it had a sample of 2,698 US respondents after filtering based on an honesty check and a basic attention/comprehension check. Respondents were paid $1.38 USD for completing the survey. This was part of a larger forthcoming study on rodenticides (McAuliffe et al, 2022).

Results: Attitudes towards a proposal to ban slaughterhouses 

In the survey 1, with a sample of 700 US respondents:

• We found much lower levels of support in both control (20.4%, 95% CI [11.0%-34.7%]) and treatment (7.9%, 95% CI [4.3% - 14.0%]) conditions compared to the Sentience Institute (2021, 2020, 2017) and Norwood & Murray (2018) studies (~39%-43% when including “No Opinion” responses). (More results in the appendix.)
• In both the unweighted and weighted analyses, support was lower in the treatment condition than in the control condition. The share of “Don’t Know” respondents increased in the treatment in the unweighted analysis, while in the weighted analysis, the lost support seems to come directly from people choosing to oppose instead.

Weighted results for a proposal banning slaughterhouses (control and treatment)

Survey 2 had a sample of 2,698 US respondents (after filtering based on an honesty check and a basic attention/comprehension check). We presented only the treatment condition (asking for attitudes to the policy followed by open comment). 

• We found 15.7% (95% CI [13.0%-18.8%]) support in the weighted results.
• We found 67.7% (95% CI  [64.0%- 71.2%]) oppose, 16.6% (95% CI [13.9%-19.6%]) don’t know. (More results in the appendix.)
• Support could be higher in this study than in the experimental study because it was included in a survey that had a lot of other questions pertaining to animals (about rodenticide), so it might have nudged people in a socially desirable/pro-animal direction.

Weighted results for a proposal banning slaughterhouses (treatment condition only)

We think the measure deployed here is better, or at least not worse, than existing similar surveys discussed below. Just asking for support in the abstract may be vulnerable to some combination of acquiescence bias, social desirability and symbolic/expressive responding. Presenting arguments for and against the proposal may ameliorate some of these concerns. It may also be more comparable to what people would encounter in a real situation where this matter was actively debated or up for a vote (i.e., people would present opposing arguments as well). A limitation, of course, is that the results may depend on the specific positive/negative arguments which are employed, and further work could explore this. In Sioux Falls, there was a vote on 8th November 2022 on whether new slaughterhouses should be banned from being built and permitted to operate inside the city limits of Sioux Falls. The result was 52% against and 48% in support (Argusleader 2022, electionresults.sd.gov 2022).  However, “the group behind this is doing it entirely for NIMBY reasons and says they support slaughterhouses outside of city limits” (RyanBeck 2022). This non-animal welfare framing and the limited nature of this ban may have led to quite different results than a full US slaughterhouse ban framed around animal welfare arguments. 

Comparison with previous survey results

In 2017, Sentience Institute reported the results of a large US-representative survey (n=1094), examining attitudes towards animal farming and plant-free foods (2017).

Three results were particularly striking:

• “49% of US adults support a ban on factory farming” 
• “47% support a ban on slaughterhouses”
• “33% support a ban on animal farming” (ibid.)^[1]

Sentience Institute has since repeated its survey (2021, 2020) and again found substantial support ("49.1%" and "44.8%" support for banning slaughterhouses respectively- when excluding "no opinion"). These figures seem extremely surprising given that only around 1% of Americans appear to be vegetarians, with the percentage of vegans being yet smaller (Šimčikas 2018).^[2] It seems odd that large numbers of Americans, who routinely choose to eat animal products, would support measures which would likely severely impinge on the consumption of these products, either by massively restricting supply, significantly increasing price or making the consumption of animal products outright impossible.^[3] Of course, it is possible for individuals to have conflicting attitudes about such issues. For example, studies have shown a “vote-buy gap” regarding cage-free eggs; many people vote to ban a product they purchase (Paul et al 2019). Indeed, the Sentience Institute survey itself seems to show a clear and strong conflict with the attitudes their respondents appeared to evince. While 49% of individuals agreed with the statement “I support a ban on factory farming,” 97% agreed that “Whether to eat animals or be vegetarian is a personal choice, and nobody has the right to tell me which one they think I should do” (2017) - the strongest support for any statement in the survey by a wide margin. These results, therefore, seem in quite direct conflict given that bans on slaughterhouses or animal farming would remove personal choice on this matter.^[4] 

Norwood & Murray’s (2018) study seemed to suspect non-comprehension might be playing a role, as they write: “Perhaps many Americans simply did not understand what a ‘slaughterhouse’ is?” To test this, they included a question designed to probe whether individuals really understood what a slaughterhouse is. After being asked whether they agreed with the statement “I support a ban on slaughterhouses,” participants who agreed with this statement were asked “Were you aware that slaughterhouses are where livestock are killed and processed into meat, such that, without them, you would not be able to consume meat?”

A significant limitation of this approach is that participants are able, and have a strong incentive, to simply report that they understood the question whether or not they did. This might be to avoid embarrassment or self-delusion. If you have even a vague sense of what a term means, then when a question is explained to you, it might seem that you basically understood it all along; alternatively respondents may have been aware of some of these things about slaughterhouses, but perhaps not that “without them, you would not be able to consume meat” and so erred on the side of saying “yes.” Reported understanding may also be due to demand effects (thinking that saying they understood the question will please the researcher). Stronger forms of comprehension checks require that individuals actually understand the question in order to successfully answer (or at least make it much less likely that they can give the correct answer to the comprehension check without having understood the question).

Even with these significant limitations, Norwood & Murray (2018) found that more than a quarter (27.1%) reported that they did not understand the statement which they had just said they agreed with. We suspect that the true rate of non-comprehension would have been at least somewhat higher. Either way, having removed the 27.1% of individuals who self-reported not understanding what a slaughterhouse was, Norwood & Murray (2018) found that 34% of respondents (as opposed to 47% originally) supported a ban on slaughterhouses.^[5]

Given this apparent conflict between these results and individuals’ other attitudes and actual behaviors, there is a need to investigate how these results should be interpreted. Even if one supposes that these responses do reflect real support for these bans, it is necessary to work out how to make sense of what these conflicting attitudes (e.g., ‘I support a ban’ and ‘I believe personal choice should not be infringed’) mean and how individuals might be expected to act. 

Why examining this result matters

One reason why it is important to examine this result further and try to better understand the attitudes people hold about these questions is that we want to work out how individuals will actually behave. If 47% of Americans say they would support a ban on slaughterhouses, but they also say that whether or not to eat meat is a personal choice, should we expect them to vote in favor of a ban which would take away that choice? 

More generally, these results have been taken to carry practical implications for what messages and demands animal advocates can and should use. In the Sentience Institute report, Jacy Reese argued that the results suggest that “animal-free food advocates might be able to succeed with stronger proposals than they currently use” (2017).  Likewise, Lewis Bollard suggested that the results “may suggest broad popular support for reforms” (2018). Animal Ask has written “If this is accurate it could be taken as strong support for pursuing bold ballot initiatives” (2022). Tobias Baumann cited the results as evidence in favor of an institutional messaging approach: “Nevertheless, the results above suggest that animal advocates often make a serious mistake by using the framing (i.e. asking for personal dietary change) that elicits the most resistance and opposition” (2022). As such, the results are a potential crux for central ongoing debates within animal activism about milder reformist approaches versus more radical demands, and more individual vs more institutional approaches. However, if these results are misleading and large percentages of Americans do not really (or unambiguously) support such proposals, then they risk misleading us into supporting stronger messages and proposals than would actually be supported, with potential blowback and costly, failed campaigns.

The survey has also been taken as a potential indicator of changes in public attitudes towards animals and animal farming more broadly. Animal Charity Evaluators argued that the survey, if repeated yearly, may be an “important asset” in assessing the extent to which animal advocates are succeeding in changing attitudes (2018). As such, if the survey  - or one like it - is to be taken as an important bellwether for the influence of animal activists on attitudes, then it is particularly important that we understand what the responses to its questions actually mean.

Why does this survey need further replication, haven’t the results already been well-established?

One might think that even if the results matter for the reasons described above, that there is little need to examine them further. After all, the initial survey was large, pre-registered, used a representative US sample, and appears to have been well analyzed. Furthermore, the results have already been successfully replicated both by repeat surveys from Sentience Institute (2021, 2020) and by an independent group of researchers, who also thought that the original results “frankly, seemed outrageous” (Norwood & Murray 2018). So shouldn’t we be quite confident in these results? 

Even though we think that the initial study and replications were well conducted in the ways described above, this does not assuage our concerns about the validity and interpretations of these results. In brief, the described virtues of the original survey and its successful replications increase our confidence that if we were to replicate the procedure of the original survey again, we would find similar results. However, those results do not suffice to remove our concerns about whether the results of this procedure show that individuals really support these radical bans on typical farming practices. Nor do they allow us to make sense of the conflict between individuals’ stated support for the bans, with their belief that these are matters of personal choice.

To explain why the replications of Sentience Institute’s results don’t assuage our concerns, it is helpful to draw the distinction between direct replication and conceptual replication. Whereas a direct replication aims to exactly replicate the procedure of an experiment, a conceptual replication aims to test the same effect using a distinct procedure (Milkowski et al. 2018). As such, while a direct replication serves to increase our confidence that when we repeat the procedure we find the same result, it is ill-suited to highlight limitations internal to the procedure itself. Conceptual replication, conversely, is necessary to confirm the external validity of the effect and generalizability beyond a procedure (Lynch et al. 2015, Crandall et al 2015).^[6]

To take a concrete and germane example, suppose that when we ask people “Are you a vegetarian?” at least 60% systematically understand “vegetarian” in some way compatible with eating some meat, or else are biased towards answering “Yes” despite eating meat, due to social desirability or demand effects. If we are concerned about this kind of systematic distortion arising when we use the procedure of asking people whether they are a vegetarian, then no number of direct replications of this procedure should serve to reduce these concerns. Rather, we would need to attempt to examine the putative finding through other procedures, either asking individuals the question differently, for example, “Do you eat any meat products such as...?”, or supplementing our original question with distinct approaches, such as a food frequency questionnaire. As another example, research on intelligence that uses hair length as a measure of intelligence would be highly misleading; highly replicable gender differences in hair length would be interpreted as evidence that women are more intelligent than men. This inference would be false because hair length is not a valid measure of intelligence, even though the relationship between gender and hair length is highly replicable. Thus, even successful and replicable tests of a theory may be false if measures lack construct validity, that is, they do not measure what researchers assume they are measuring. (See Schimmack's recent (2021) validation crisis paper.) 

Concluding thoughts

A claim made about the original Sentience Institute study was that the evidence of strong public support for radical action suggests advocates can push for stronger proposals and messages.^[7] Insofar as we take the responses to slaughterhouse ban proposals as a proxy for 'support for more radical action,' if only 16% would in fact be supportive, this does not offer a strong basis for more radical action. (It may well be a net negative due to a majority of people opposing such moves. Recall that 68% opposed the slaughterhouse ban in our large study.)

Of course, treating responses to these items as proxies may itself be dubious. These responses may reflect a symbolic desire to signal that people think the current state of factory farms/slaughterhouses is bad, but not imply that people actually think that getting rid of them would be desirable. The attitudes expressed by poll respondents in response to broad questions may not be reliable indicators of actual support for specific policies or messages. It would be better to test people's responses to more detailed messages and policy proposals, paying special attention to how radical messages compare to counterfactual moderate messages. One could also test a radical ask (ban factory farming) and a moderate ask (labelling for cage-free eggs, say) each with a radical message ("meat is murder") versus a moderate one ("human/consumer welfare"). 

Acknowledgements

This post is a project of Rethink Priorities–a think tank dedicated to informing decisions made by high-impact organizations and funders across various cause areas. It was written by Neil Dullaghan. Thanks to contributions from David Moss and William McAuliffe, Jamie Elsey and Willem Sleegers for running the survey, analyzing the results, and providing data visualization, and Adam Papineau for copyediting. If you are interested in RP’s work, please visit our research database and subscribe to our newsletter. 

References

Amalia Zimmerman. (2018). Sentience Institute Survey Analysis. Animal Charity Evaluators. https://perma.cc/4EQB-46BT

Andrew S. Paul, Jayson L. Lusk, F. Bailey Norman, & Glynn T. Tonsor. (2019). An experiment on the vote-buy gap with application to cage-free eggs. Journal of Behavioral and Experimental Economics. https://doi.org/10.1016/j.socec.2019.02.005

Animal Ask. (2022). Pathways to victory: How can we end animal agriculture? Animal Ask. https://perma.cc/3WXU-E2MY

Argus Leader. (2022). 2022 Minnehaha County Results. https://perma.cc/FM3B-5KVW

Bailey Norman & Susan Murray. (2018). FooDS Demand Survey. Oklahoma State University. https://web.archive.org/web/20190806000018/http://agecon.okstate.edu/files/january%202018.pdf

Christian S. Crandall & Jeffrey W. Sherman. (2015). On the scientific superiority of conceptual replications for scientific progress: Journal of Experimental Social Psychology, 66, 93–99. https://doi.org/doi.org/10.1016/j.jesp.2015.10.002

Jacy Reese Anthis. (2017). Animals, Food, and Technology (AFT) Survey: 2017 Update. The Sentience Institute. https://perma.cc/NV2J-UK7M

Jacy Reese Anthis & Ali Ladak. (2021). Animals, Food, and Technology (AFT) Survey: 2020 Update. The Sentience Institute. https://perma.cc/JDX7-QV6A

Jacy Reese Anthis & Ali Ladak. (2022). Animals, Food, and Technology (AFT) Survey: 2021 Update. The Sentience Institute. https://doi.org/10.31234/osf.io/3v6ku

John G. Lynch Jr, Eric T. Bradlow, John C. Huber, & Donald R. Lehmann. (2015). Reflections on the replication corner: In praise of conceptual replications. International Journal of Research in Marketing, 32(4), 333–342. https://doi.org/10.1016/j.ijresmar.2015.09.006

Lewis Bollard. (2018). The 2018 Elections and Farm Animals. Open Philanthropy Farm Animal Welfare Newsletter. https://perma.cc/LAG6-RXU6

Marcin Miłkowski, Witold M. Hensel, & Mateusz Hohol. (2018). Replicability or reproducibility? On the replication crisis in computational neuroscience and sharing only relevant detail. Journal of Computational Neuroscience, 45, 163–172. https://doi.org/10.1007/s10827-018-0702-z

Ryan Beck. (2022). Comment on ‘Support for Sioux Falls Slaughterhouse Ban’. Metaculus. https://perma.cc/PLF5-2XQR

Šimčikas, S. (2018, August 16). Is the percentage of vegetarians and vegans in the U.S. increasing? Animal Charity Evaluators. https://animalcharityevaluators.org/blog/is-the-percentage-of-vegetarians-and-vegans-in-the-u-s-increasing/

South Dakota Secretary of State. (2022). Sioux Falls Results. Electionresults.Sd.Gob. https://perma.cc/7N2J-R7PA

Tobias Baumann. (2022, February). How the animal movement could do even more good [Forum]. Effective Altruism Forum. https://perma.cc/CW55-NLPM

Ulrich Schimmack. (2021). The Validation Crisis in Psychology. Meta-Psychology, 5. https://doi.org/10.15626/MP.2019.1645

William McAuffile, David Moss, Holly Elmore, Marcus A. Davis, & James Elsey. (2022). Rodenticides Polling. https://doi.org/10.17605/OSF.IO/3UNZB

Appendix

Survey 1, N=700

Survey 2, N=2698

1. ^{^}

   These figures are higher than the raw percentages agreeing because Sentience Institute reports the percentages agreeing out of those who either agreed or disagreed. But since many respondents selected “Don’t know” regarding the bans on slaughterhouses (11%), factory farming (12%) or animal farming (10%), the overall percentages supporting these bans are slightly lower: slaughterhouses (43%), factory farming (42%), animal farming (30%).

2. ^{^}

   This is the number of people that self-report being vegetarian and don’t report eating meat when asked. Between 2% and 6% of Americans report that they are vegetarians and yet report eating some meat when administered a food frequency questionnaire (Šimčikas 2018). Fewer than 0.4% of adults reported consuming no animal products in two non-consecutive 24-hour periods (Šimčikas 2018).

3. ^{^}

   In principle, a ban (within the United States) of these forms of animal product production does not necessarily entail that the consumption of animal products would be illegal. People might imagine that it would still be possible to import animal products and that they will still be able to (sometimes) consume animal products outside of the United States. Nevertheless, it seems clear that any of these bans would likely be a significant barrier to animal product consumption given that presently the vast majority of animals raised on farms within the US are raised in a factory farming system.

4. ^{^}

   One could argue that a ban on factory farming specifically would not entail telling people whether they could “eat animals or be vegetarian” tout court, since individuals could still eat animals raised on non-factory farms. Whether this is a plausible interpretation of respondents’ attitudes is, of course, a further question.

5. ^{^}

   After removing respondents who straightlined, almost a quarter (23.2%) reported that they did not understand the statement, and removing these uncomprehending respondents, the final support for a ban was 37% as opposed to 43% originally.

6. ^{^}

   We make no claims about the general superiority of either form of replication.

7. ^{^}

   Sentience Institute writes “if, for example, the public is much more opposed to factory farming and animal farming than we expect, that might suggest we should use stronger messages in our advocacy, e.g. ‘end animal farming’ instead of ‘end battery cages,’” and the results suggest that “animal-free food advocates might be able to succeed with stronger proposals than they currently use” (2017).