EA Market Testing

By david_reinstein @ 2021-09-30T15:17 (+81)

Authors: Josh Lewis, David Reinstein, and Luke Freeman

Acknowledgements: Thanks to peter_wildeford, jswinchell and others for pre-publication comments

Introduction

How can we get more people to give to effective charities? How can we identify those with the greatest potential to do good? How can we encourage people to take on more impactful careers? How do we spread ideas such as longtermism and a concern for animal welfare?  More broadly, can we identify the most effective, scalable strategies for marketing EA and EA-adjacent ideas and actions? We plan to conduct systematic testing and data collection aimed towards answering these questions and we want your feedback and ideas to make this effort as useful to the EA movement as possible.

Who are we?

We are the newly formed “EA Market Testing Team” – an informal group of professional marketers, business academics, economists, and representatives of EA organizations.

A non-exhaustive list of some of the people involved:

• Joshua Lewis (Assistant Professor of Marketing at New York University)
• David Reinstein (Senior Economist at Rethink Priorities, formerly a Senior Lecturer in Economics at the University of Exeter)
• Luke Freeman (Executive Director at Giving What We Can)
• Johnstuart Winchell (Senior Account Manager at Google) ^[1]
• Noah Castelo (Assistant Professor of Marketing at the University of Alberta)
• Dillon Bowen (PhD Student at the University of Pennsylvania)
• Jack Lewars (Executive Director at One for the World)
• Chloë Cudaback (Director of Communications at One for the World)
• Neela Saldanha (Board Member at The Life You Can Save)
• Bilal Siddiqi (Strategic Advisor at The Life You Can Save)

Our objective

We aim to identify the most effective, scalable strategies for marketing EA and, crucially, to share our results, data, tools, and methodologies with the larger EA community. We will publish our methods, findings, and data in both informal and academic settings.

Outreach to the EA forum: Seeking your input

We want to know any questions you have about marketing EA that our research might be able to answer, any ideas you have about what to test or how to test it, any risks you think we should bear in mind, and any other feedback that you think would be useful for us.  You can either fill in our feedback form HERE or leave a Forum comment below. More specific prompts are in the feedback form and pasted below as a Forum comment.

Background

Broad Testing Approach

We will primarily run experiments with effective-altruism-aligned organizations. We plan to test and optimize over a key set of marketing and messaging themes. We will evaluate these primarily through behavioral metrics; in particular, how many people donate and/or pledge to effective causes, how many potential effective altruists join aligned organizations’ mailing lists, and how much time do people spend consuming EA-related content.

We expect that behavioral measures (e.g., actual donation choices)  will be more informative than hypothetical ones (e.g., responses to questions about ‘what you would donate in a particular imagined scenario’).^[2] We expect that using the naturally-occurring populations will lead to more relevant findings than convenience samples of undergraduates or professional survey participants who are aware that they are doing a research study.

We will test strategies using:

• Advertising campaigns on Facebook, YouTube, LinkedIn, Google Ads, etc.
• The content and presentation of EA organizations’ websites^[3]
• Emails to mailing lists
• Surveys and focus groups

We focus on natural settings–the actual promotions, advertisements, and web-pages used by our partners, targeting large audiences – as our main goal is to identify which specific strategies work in the most relevant contexts. Ideally, EA orgs could directly apply our results to run cost-effective campaigns to engage new audiences and achieve sustainable growth.

Our second goal is to draw relevant generalizable conclusions about approaches to marketing EA. We will consider the robustness of particular approaches across contexts and particular implementations.

Approach to Working with EA Organizations

When we run tests with EA organizations, we will not charge for the time we spend on this project, nor on giving advice or support. ^[4]. Since running such tests (i) helps us understand how to market EA and (ii) may generate data that will contribute to academic publications, we have strong incentives to support this. Our combined experience in experimental design, empirical and statistical methods, advertising, and academic behavioral science, will enable us to improve current practice and enhance insight through:

• Suggesting targeting and messaging strategies based on our prior experience in marketing and familiarity with relevant literature. (E.g., we might suggest leveraging ‘social proof’ by placing greater emphasis on community in a website heading, saying “join a community of effective givers” instead of  “give effectively.”)
• Increasing the rigor and power of experiments and trials through, e.g.:
  • Performing more detailed and sophisticated statistical analyses than those provided by software such as Google Optimize and Facebook Ads Manager
  • Increasing the power of standard (A/B and lift) testing with more precise block-randomized or ‘stratified’ designs, as well as sequential/adaptive designs with optimal  ‘reinforcement learning’ while allowing statistical inference^[5]
  • Mapping the robustness of particular strategies to distinct frames and language variations using techniques such as ‘stimulus sampling’
  • Distinguishing meaningful results from statistical flukes and confounds; identifying and mitigating interpretation issues that arise from platform idiosyncrasies such as algorithmic targeting, etc.
  • Designing experiments to maximize their diagnosticity^[6]
• Considering heterogeneity and personalized messaging and evaluating the extent to which results generalize across audiences and platforms (e.g., someone Googling “what charities are most effective?” might be very different from someone getting a targeted ad on Facebook based on their similarity with existing website traffic).
• ‘Market profiling’ to find both under-targeted ‘lookalike audiences’ and (casting the net more widely) to find individuals and groups with characteristics that would make them amenable to effective giving and EA ideas

Prioritizing Research Questions

What concrete research questions should we prioritize? We are open-minded; we want to learn how to promote EA and each of its cause areas as “effectively” as possible, and we will test whatever is most useful for achieving this end.

We will sometimes address very specific marketing problems that an EA org faces (e.g., “which of these taglines works best on our webpage?” or “what is the most efficient platform on which to raise awareness of our top charities?”). However, we also aim to design tests that inform which broad messaging strategies are most effective in a given setting, e.g., “should we use emotional imagery?” or “should we express EA as an opportunity to do good or an obligation to do good?” We list some more specific research questions below.

If you have feedback on any one of them, (e.g., you think it’s particularly important to test or should be deprioritized, or you have an insight about it we might find helpful, or evidence we may have overlooked, etc..) please mention this as a forum comment or mention it in the feedback form (LINKED) .  In fact, we would encourage you to fill out this form before reading further; to share your original thoughts and ideas before being influenced by our examples and suggestions below.

Messaging Strategy: Example Research Questions

When and how should we emphasize, cite, and use...

• Support from sources of authority such as news organizations, respected public figures, Nobel prize winners, etc., if at all? Are these more effective in some contexts than in others?
  • For example, we might find that citing such support (e.g., testimonials) is useful for persuading people to target their donations effectively, but not for persuading them to take a pledge to donate more (or vice-versa). (This would argue for GiveWell using these approaches but not Giving What We Can, or vice-versa).
• Social esteem or status benefits of, e.g., being on a list of pledgers (relevant for GWWC) or working in a high-impact job (relevant for 80k hours)?
• Social proof or community?
  • For example, “join a growing community of people donating to our recommended charities” vs. “donate to our recommended charities”
• Effectiveness, e.g., estimates of ‘how much more impact’ effective altruist careers or donations have over typical alternatives?
• Personal benefits of engaging with EA material, e.g., such as easily tracking how much you are giving, finding a fulfilling job?
• A particular cause area?
• We might find that emphasizing global health is useful for persuading people to start donating money (as inequality and personal privilege is a compelling reason to donate to this cause), whereas having several distinct messages each emphasizing a different cause area may be more successful on platforms that can dynamically target different audiences
• Emotional messages or imagery?
  • These might be good for generating one-off donations but not for long-term commitments like pledges or career trajectories.

Images below: from www.givingwhatwecan.org, highimpactathletes.org, and thelifeyoucansave.org, respectively

Targeting and market-profiling: Example (meta-) Research Questions

• Which groups (perhaps under-targeted or under-represented groups in EA) do you suspect might be particularly amenable to EA-related ideas and to taking EA-favored actions; or at least particularly worth exploring? What particular contexts and environments might be worth testing and exploring?
• When should we use public lists, e.g., of political donors, perhaps donors to politicians interested in relevant cause areas?.
• When and how should we target based on particular types of online activity (watching YouTube videos,  participation on relevant Reddit threads, reading relevant news articles, visiting ethical career sites, visiting Charity Navigator, searching relevant terms, etc.)?
• How can we increase diversity in the effective altruism community (along various dimensions)? ^[7]

Above: "Psychographic profile of Canadians" from From Frank Graves on Twitter. Profiling, clustering, segmentation, and lookalike modeling approaches might be further applied towards attitudes and beliefs relevant to EA.

Risks, limitations to our work

1. Reproducibility and social science. The replication crisis in social sciences limits the insight we can glean from past work. Mitigating this: all the academics involved understand the severity of this crisis and have some expertise helping us to distinguish replicable from non-replicable research, and a commitment to doing so.^[8] We are committed to practices such as pre-registration to limit researcher degrees of freedom, to using appropriate corrections and adjustments in light of testing multiple hypotheses in a family (considering ‘familywise error’), and to fostering transparent analysis (through dynamic documents) and enabling replications. Also mitigating: we will undertake new work, our own large-scale experiments and trials on public platforms, working with practitioners to directly test the applicability and relevance of previous academic findings.
2. Limits to generalizability: We may end up over-generalizing and giving bad advice. Something that works on one platform may not work on another platform due to idiosyncrasies in the audience, the targeting algorithms, the space constraints, etc..  Something that works for one organization may not be transferable to other organizations with different target audiences or different aims. Mitigating: We aim to test each treatment with multiple implementations (see ‘stimulus sampling’) across multiple contexts. We will look for experimental results that are robust across distinct settings before basing broader recommendations on these.
3. Marketing is an art and not a science? Following on from the previous point, it may be the case that this ‘space’ is too vast and complicated to explore in scientific ways, and general principles are few and far between. Perhaps ‘nuanced appreciation of individual contexts’ is much more useful than systematic experiments and trials. Nonetheless, we believe it is worth putting some resources into fair-minded exploration to consider the extent to which this is the case.  We recognize that it is important to consider interaction effects and to consider context and external generalizability.  In the extreme case that ‘there are few or no generalizable principles’, at least we will have helped to improve testing and marketing in important specific contexts.
4. Downsides to broadening EA: There may be downsides (as well as upsides) to making EA a broader movement, as argued on this forum and elsewhere.  To the extent that our work focuses on
   • encouraging EA-aligned behaviors, particularly effective charitable giving, rather than on bringing people into the intellectual debates at the core of EA, and...
   • to the extent we are profiling people to e.g., get more participation on EA Forum and EA groups,

• ….  this need not imply watering down the movement. In fact our profiling work will also help better understand what sorts of people are likely to be deeply committed to hardcore EA ideas.

5. The possibility of mis-representing or negatively affecting public perceptions of EA through our marketing and outreach activities, or public responses to these  As researchers and members of the EA community, we are committed to a scientific mindset, openness (and ‘Open Science’), and upholding integrity and accurate communications (ss well as the other key principles proposed by Will MacAskill). We are particularly aware of the mis-steps and lapses of Intentional Insights, which provides an important counterexample. We will take careful steps to uphold generally-agreed EA community norms and values. This includes:
   • Obtaining the explicit consent of any organizations before linking their name to our work or promotions,
   • Representing EA ideas as faithfully as possible, and where we are unsure, consulting experts and the community (or otherwise making it clear that the ideas we present are not themselves connected to EA), and
   • Avoiding practices which mislead the public (through techniques such as ‘astroturfing’).

Conclusion

To reiterate: our mission is to identify the most effective, scalable strategies for marketing EA and EA-aligned activities using rigorous testing approaches (as well as surveys, profiling and meta-analysis).  Your ideas about ‘what to test’ and ‘how to test it’, as well as feedback on our current plans, will be immensely valuable.

Feedback Form (LINK)

We want your opinions, impressions, and experience. We have a few questions with prompts for open-ended answers. We do not expect you to answer all of the questions.  But please do enter any relevant thoughts, opinions, and ideas.  You can do so anonymously or leave your information if you would like us to contact you and/or acknowledge your ideas.

In answering these questions, please let us know your ‘epistemic status’ – do you have specific experience and knowledge informing your answers?^[9]