Effectiveness of a theory-informed documentary to reduce consumption of meat and animal products: three randomized controlled experiments

By Jacob_Peacock @ 2022-03-21T15:52 (+95)

Recommendations

Evidence does not support the use of informational documentaries.
Interventions that change intended eating habits may not result in actual change in eating habits. In our study, even very large changes in intention did not change actual behavior.
Future studies should ensure that participants are unaware of the study's purpose. They can do this with blinding.

Key Findings

Tested a 20-minute documentary "Good For Us" that highlights the environmental, human health and animal welfare harms of eating meat and other animal products.
- In a randomized controlled experiment, compared the documentary to a control video (a generic motivational speech). Participants were from the general population of the United States.
- Followed up 12 days later with a survey that was described as a different study. This helped to "blind" participants to the purpose of the study when collecting data, reducing potential bias.
Our first study found the documentary had no effect on a variety of different outcomes.
- Found no reduction in animal product consumption. The average change we measured in one study was less than a 1-ounce reduction in animal product consumption per week, with a 95% confidence interval ranging from a 6 ounce reduction to a 5 ounce increase.
- Found no change in moral valuation of animals ("speciesism").
- Found no meaningful increases in interest in animal activism or in perceived importance of environmental sustainability, animal welfare, or eating a healthful diet.
Our second study was deliberately designed less rigorously, to resemble previous studies that measured intended behavior. Immediately after the documentary, asked viewers if they planned to eat more or less animal products next week. Many viewers planned to eat less animal products in the week after seeing the documentary.
- Documentary made viewers 242% more likely to intend to reduce meat consumption than participants who viewed the control video. Critically, our first study suggested that these intentions do not actually translate to reductions in consumption.
- A previous meta-analysis suggested that comparable interventions make people about 22% more likely to intend to reduce their meat consumption. So our documentary was likely very convincing relative to other interventions, but still not effective at reducing consumption in our more rigorously designed study.
Our third study tried to make the documentary more effective. Added a pledge, goal-setting exercises, and reminder email; and participants were people interested in nutrition research. The documentary still was not effective by any measurement. Also looked at just people who attended at least a 2-year college and identified as Democrats, but still no effect.

david_reinstein @ 2022-03-21T18:09 (+11)

with a 95% confidence interval ranging from a 6 ounce reduction to a 5 ounce increase.

Thanks for including the CI bounds, that makes it much more interpretable as an 'actual tight null' rather than an underpowered study.

CF I think that is relative to an average US consumption of about 80 oz per week ... so under +/- 10% with 95% CIs ... and the 80% CI would be obviously even tighter

(Or am I wrong, maybe this isn't that tight)

Disclaimer: Jacob and I both work at Rethink Priorities

Peter Wildeford @ 2022-03-22T14:24 (+10)

Do you think your study is sufficiently well powered to detect very small effect sizes on meat consumption? It seems plausible that effects on meat consumption would be very small in expectation plus many people would not reduce meat no matter what, so you may be needing to detect a small shift within a small subpoulation.

It looks like you have 80% power to detect an effect size of d=0.24 - which is actually substantially larger than the effects we usually find for animal interventions even on more moveable things like attitudes/signing a petition/agreeing that "factory farms aren't great". Their null result on effect on meat consumption was not at all tightly bounded: -0.3oz [-6.12oz to + 5.46oz]

So I think the different results here seem possibly explained just by the fact that you could find effects on the moveable attitudes but were underpowered to detect differences in meat consumption. I'd be curious to estimate what effect size would we be looking at if say 3-5% of people stopped eating meat (an optimistic estimate IMO).

This is perhaps further confounded by a large amount of probable noise - how good are people at estimates oz of meat eaten in different time periods, is oz of meat something that is distributed in a way to corresponds to what a t-test is assessing?

Jacob_Peacock @ 2022-03-24T21:21 (+3)

Thanks for these Peter! (Note that Peter and I both work at Rethink Priorities.)

Do you think your study is sufficiently well powered to detect very small effect sizes on meat consumption?

No, and this is by design as you point out. We did try to recruit a population that may be more predisposed to change in Study 3 and looked at even more predisposed subgroups.

substantially larger than the effects we usually find for animal interventions even on more moveable things

I think we were informed by the results of our meta-analysis, which generally found effects around this size for meat reduction interventions.

Their null result on effect on meat consumption was not at all tightly bounded: -0.3oz [-6.12oz to + 5.46oz]

Obviously, this is ultimately subjective, but this corresponds to plus or minus a burger per week, which seems reasonably precise to me. The standardized CI is [−0.17, 0.15], so bounded below a 'small effect'. And, as David points out, less stringent CIs would look even better. But to be clear, I don't have a substantive disagreement here—just a matter of interpretation.

For even more power, we could combine studies 1 & 3 in a meta-analysis (doubling the sample size). Study 3 found a treatment effect of−1.72 oz/week; 95% CI: [−8.84,5.41], so the meta-analytic estimate would probably be very small but still in the correct direction, with tighter bounds of course.

explained just by the fact that you could find effects on the moveable attitudes

Just to clarify, we measured attitudes in all 3 studies. We found an effect on intentions in Study 2 where there wasn't blinding and follow-up was immediate. Studies 3 & 4 (likely) didn't find effects on attitudes.

I'd be curious to estimate what effect size would we be looking at if say 3-5% of people stopped eating meat (an optimistic estimate IMO).

Just roughly taking David Reinstein's number of 80 oz per week (could use our control group's mean for a better estimate) and assuming no other changes, 1% abstention would give a 0.8 oz effect size and 5% 4 oz. So definitely under-powered for the low end, but potentially closer to detectable at the high end. (And keeping in mind this is at 12-day follow-up; we should expect that 1% to dwindle further at longer follow-up. With figures this low I would be pessimistic for the overall impact. But keep in mind other successful meat reduction interventions don't seem to have worked mostly through a few individuals totally abstaining!)

corresponds to what a t-test is assessing

I wouldn't expect issues in testing the difference in means given our samples sizes. But otherwise not sure what you're suggesting here.

James Ozden @ 2022-03-21T18:19 (+7)

This is really interesting, thanks for this! In particular, it was really helpful comparing it to previous less-rigorously designed surveys, as I'm sure you expected pushback using those results. I had a few quite preliminary questions:

Do you think the effects of this could be different for different documentaries, and is this something you would consider testing in the future? Whilst in the paper you state that "Good For Us" uses psychological theory to make the documentary as compelling as possible to shift attitudes and behaviour, it feels quite hard to predict the emotional/attitudinal impact of a documentary. Some random thoughts I had was that maybe more sensationalist documentaries (What the Health, Cowspiracy, Dominion, etc.) could be more effective even though they ignore best practice, and it would be interesting to see how this stacks up against Good For Us. As these are touted as being the most effective/popular pro-animal documentaries, it would be interesting to see how these perform under the same controlled conditions.
Obviously whilst difficult to measure, do you think these documentaries might be important in shaping beliefs that later affect eating behaviour? A common analogy we hear is about "planting a seed" whereby one exposure to pro-animal content might not cause any behaviour change, but it primes them for later exposures which might then have more significant impacts on behaviour change. You talk about repeated exposures briefly in the paper but it would be interesting to hear your thoughts on how plausible you think this mechanism is (see point below)
- If repeated exposures to pro-animal content might be effective, we still might expect there to be some significant changes in this study as it should be repeated exposure for some people (unless you screened them out) so maybe this point isn't so strong
Do you think there are other long-term mechanisms that might be at play here e.g. the documentary causes more animal-focused conversations with friends and family, which might cause behaviour change past the 12-day mark? Do you think a follow-up after 2-3 months (for example) would introduce too much noise to have strong causal evidence?
More broadly, what implications do you think this has for the farmed animal movement in terms of funding documentaries vs other interventions, and where do you think more work is needed?

Jacob_Peacock @ 2022-03-21T19:22 (+13)

Thank you taking the time to engage, much appreciated! Forgive my responding quickly and feel free to ask for clarification if I miss anything:

Definitely, could be different results with different docs. But ours showed a much stronger effect than the average of similar interventions we found in a previous meta-analysis, suggesting Good for Us is pretty good. It is probably better than Cowspiracy on changing intentions, with longer studies of excerpts of Cowspiracy also finding no effect.
Agree especially with your sub-point. We also tried to recruit populations more likely to be effected in Study 3. Also, see sources in my previous point.
Maybe but doesn't seem likely since there wasn't change in importance of animal welfare or other measures of attitudes. I would generally expect effects to decay over time rather than get stronger; our meta-analysis (weakly) supports this hypothesis in that longer time points showed smaller effects. Usefulness of a 2-3 month time point would mostly depend on attrition in my opinion.
I would vote other interventions. Classroom education in colleges and universities seems good as does increasing the availability of plant-based options in food service and restaurants.

Ula @ 2022-03-22T11:45 (+4)

This might be a stupid question, but do you know if anyone looked at how many new activists can a documentary make? :) I met some people who were animal activists because they watched some inspiring movies about animal suffering. I wonder if despite the fact that the movies don't work as a thing that makes people change their diet, they might make some very specific people into animal activists? :) Probably the cost of producing them would not justify the results of getting like 100 activists out of one movie. But on the other hand, if the activist will later work in an organization that leads to a major policy change in a country, maybe it is worth it? :) Just some fun speculations about flow-through effects :) I wonder also about things like effects on the press coverage of animal-related topics, effects on influencers and celebrities, effects on the perception of animal suffering (e.g. would there more likely vote on a law that e.g. forbade using crates in farming systems, or more likely sign a petition to the government). We all know by now that habits are very hard to change, but opening minds to a different viewpoint - maybe not as hard.

Jacob_Peacock @ 2022-03-22T12:54 (+2)

Yes, we did and found no meaningful increases in interest in animal activism, including voting intentions. Full questions available in in the supplementary materials.