A BOTEC-Model for Comparing Impact Estimations in Community Building

By Patrick Gruban 🔸, Sarah Tegeler 🔹 @ 2023-03-14T06:26 (+49)

We are grateful to Anneke Pogarell, Birte Spekker, Calum Calvert, Catherine Low, Joan Gass, Jona Glade, Jonathan Michel, Kyle Lucchese, and Sarah Pomeranz for conversations and feedback that significantly improved this post. Any errors, of fact or judgment, remain our entirely own.

Summary

When prioritising future programs in EA community building, we currently lack a quantitative way to express underlying assumptions. In this post, we look at different existing approaches and present our first version of a model. We intended it to make Back-of-the-envelope (BOTEC) estimations by looking at an intervention (community building or marketing activity) and thinking about how it might affect participants on their way to having a more impactful life. The model uses an estimation of the average potential of people in a group to have an impact on their lives as well as the likelihood of them achieving it. If you’d like only to have a look at the model, you can skip the first paragraphs and directly go to Our current model.

Epistemic Status

We spent about 40-60 hours thinking about this, came up with it from scratch as EA community builders and are uncertain of the claims.

Motivation

As new co-directors of EA Germany, we started working on our strategy last November, collecting the requests for programs from the community and looking at existing programs of other national EA groups. While we were able to include some early on as they seemed broadly useful, we were unsure about others. Comparing programs that differ in target group size and composition as well as the type of intervention meant that we would have to rely on and weigh a set of assumptions. To discuss these assumptions and ideally test some of them out, we were looking for a unified approach in the form of a model with a standardised set of parameters.

Impact in Community Building

The term community building in effective altruism can cover various activities like mass media communication, education courses, speaker events, multi-day retreats and 1-1 career guiding sessions. The way we understand it is more about the outcome than the process, covering not only activities that focus on a community of people. It could be any action that guides participants in their search for taking a significant action with a high expected impact and to continue their engagement in this search.

The impact of the community builder depends on their part in the eventual impact of the community members. A community builder who wants to achieve high impact would thus prioritise interventions by the expected impact contribution per invested time or money. 

Charity Evaluators like GiveWell can indicate impact per dollars donated in the form of lives saved, disability-adjusted life years (DALYs) reduced or similar numbers. If we guide someone to donate at all, donate more effectively and donate more, we can assume that part of the impact can be attributed to us.

For people changing their careers to work on the world's most pressing problems, starting charities, doing research or spreading awareness, it’s harder to assess the impact. We assume an uneven impact distribution per person, probably heavy-tailed. Some people have been responsible for saving millions, such as Norman Borlaug or might have averted a global catastrophe like Stanislav Petrov.

Existing approaches

Marketing Approach: Multi-Touch Attribution

In our strategy, we write:

Finding the people that could be interested in making a change to effective altruistic actions, guiding them through the process of learning and connecting while keeping them engaged up to the point where they take action and beyond is a multi-step process. We expect most people to have multiple touchpoints with content and people along the way, like media, the EAD website, newsletter, books, local community events, fellowships, EAG(x) or 1-1s. Each is expected to affect the person’s engagement and the likelihood of taking the next step in larger commitments. Looking back at a person's steps until they made a large change, we can ask them what share of the decision they attribute to the touchpoints along the way. In marketing, this is called multi-touch attribution.

We describe a formula for calculating the impact that could be attributed to us as community builders. Still, ultimately, we conclude that it needs more work before it is ready to be used. After finalising our strategy, we learned that Open Phil had already worked on a more detailed model on similar principles two years ago.

Open Philanthropy: Influences on Longtermist Careers

In 2020, Open Phil conducted a survey of about 200 people working in longtermist areas, querying them in detail about community-building interventions that were important for them getting more involved and helping them have more impact. The respondents mentioned influences in free text, selected them from an existing list and gave them impact points. Additionally, the survey designers gave weights to the respondents according to the expected altruistic value of their careers on a scale of 1-1,000. This resulted in a list of interventions that could be ranked by their importance to the set of people.

While the survey has many influence factors, the level of detail seems insufficient to make informed predictions in comparing interventions within local or national groups: “Local EA groups, including student groups” is listed as one factor without splitting it up into the activities. It is also a descriptive model that looks at the influences on people before 2020. If we want to make assumptions about new interventions, we will need another approach.

80,000 Hours: Leader’s Survey on Value of Employees

A 2018 survey of leaders of EA organisations included a section that asked what donation amount organisations would be willing to sacrifice instead of losing a recent hire. The results were in the millions of dollars for senior hires, and the subsequent discussion led to a clarifying post. 

This approach addresses the altruistic impact that people might have in the future. 

CEA: Three-Factor Model of Community Building

Among other models, CEA uses “a three-factor model of community building where the amount of good someone can be expected to do is assessed as the product of three factors: resources, dedication and realisation.” In the case of donations, the resources could be a person's income, the dedication, the percentage donated, and the realisation of the effectiveness of the charity the money is donated to. Each of these factors can differ by at least an order of magnitude, and the model can be used to think about which of them to try to influence based on different target groups.

The model seemed very useful to us when looking at individuals taking an impactful action now, like taking the GWWC pledge or donating. For our purpose of comparing different interventions quantitatively based on their future results, we would have had to estimate the three factors before and after an intervention, leading to six parameters. While the CEA model looks at the present values of the factors, we would have had to predict the lifetime impact of someone making a career transition and discounting for them dropping out beforehand.

Ultimately we found the three-factor model to be too complex for our purposes.

CEA: Career Value

In April 2022, the CEA groups team gave a presentation where they estimated the counterfactual value of different actions.

Based on average salaries, times to retirement and drop-off rates, they roughly calculated $70,000 in expected donations for a GWWC pledge.

For a longtermist career, they estimated the number of people working in the field, the amount pledged to longtermist causes at that time and the willingness of organisations to hire a good candidate earlier rather than later. The result was a value of $18 million over the lifetime of a new employee. They noted that this value changed based on the potential contribution of the person and what role they would fill. 

For non-longtermist causes, they estimated 10x less counterfactual value as there was less money pledged and cause areas were not as talent constrained. In their presentation, they then tried to tease out intuitions of where community builders should focus if they assumed they were  10% responsible for each action outlined above. 

As the presentation is from April 2022, the numbers for career changes would be much lower after the reduction of the money pledged to longtermist cause areas since then.

Our current model

After looking into the existing approaches, we concluded that the level of uncertainty in predictions didn’t warrant a complex model. Reducing the parameters would make it easier to show our assumptions and focus on points where there might be disagreements. On a high level, we’re focusing on:

• Number of people affected by an intervention
• Change of average expected impact per participant

The last part is where it gets tricky, and we will discuss it below. Assuming we have the two parameters, we can calculate the impact change, factor in the costs and prioritise them accordingly.

Expected Impact

We assume people can have very different altruistic impacts. CEA’s article about the three-factor model, describes how each of the three factors could differ by at least one order of magnitude for people in Western countries. The Open Phil Survey uses an impact scale of four orders of magnitude between people working on longtermist causes.

Making sure that we’re guiding people in the right direction of having a high impact and focusing on those with this opportunity seems important. At the same time, we acknowledge that the future impact of people is tough to predict.

Looking at an intervention in community building, we can assume it had a positive impact if the average expected impact of each participant is higher after the intervention than before. In an ideal world, we could take two similar groups of people, apply an intervention on one and then record their impact over their lifetime. The difference between participants in both groups would be the change in expected impact.

For this model, we will assume that expected impact is the product of

• The maximum impact potential of a person over their lifetime
• The likelihood (or expected share) of reaching their impact potential

To put it into a formula:

n: Number of participants in the intervention

i: Average impact potential of participants

p₁: Likelihood of participants reaching their impact potential before the intervention

p₂: Likelihood of participants reaching their impact potential after the intervention

Impact Potential

We define the average impact potential of a group as a fixed numerical value. In an ideal world, every person would have the same potential to have a high altruistic impact. As we look at individual interventions in today’s world, we acknowledge that the impact of people is unevenly distributed. We can affect this number by target group selection. A mass media campaign reaching broad parts of the population is expected to have a lower average impact potential than, for example, a small event only for people already working on highly impactful causes.

Likelihood of Reaching the Impact Potential

The second value we consider is the likelihood (or expected share) of reaching their impact potential. We assume we can affect this through the intervention. Having a career 1-1 with someone might point them in a direction that increases the share of the impact potential they can reach. This can be a value between >0% and 100%.

Cost-Effectiveness

If we estimate the cost of an intervention (money and hours of work), we can calculate the cost-effectiveness:

We can now use this number to compare interventions.

Use Cases

The model can be used to compare different interventions without having to define all of the parameters universally.

Example 1

For example, we could compare the cases of

• Giving an introductory EA talk to a general audience of 100 people
• Giving an introductory EA talk to an audience of 10 entrepreneurs that have just sold their companies

We could assume the likelihood of audience members reaching their altruistic potential increasing through the talk will be similar, and the effort might be the same. In this case, we can ask ourselves if we see the impact potential of the entrepreneurs as more or less than 10x to prioritise the talk.

Example 2

Another example would be comparing two interventions at a local EA group. The group size and the average impact potential will stay the same, whereas

• Intervention 1 is highly engaging and needs 10 hours of preparation time
• Intervention 2 is mildly engaging and needs 1 hour of preparation time

Now we can ask ourselves if we think the percentage point increase of intervention 1 will be 10x higher than for intervention 2 (e.g. 10% to 15% vs 10% to 10.5%) to prioritise.

If we have more interventions, we could develop categories of target groups and estimate their relative average impact. For example, we could look at the general audience in a region, university students, mid-career leaders etc. We could also ask ourselves the current likelihood of them reaching their potential and how much we might affect it.

For younger people with fewer years of education, we might see a lower maximum likelihood even after optimal guidance, as they might still drop out before being able to take an impactful job or being able to donate substantially. At the same time, younger people might have a higher impact potential as they can change their career paths more easily.

Caveats and Uncertainties

This model currently does not address externalities, especially downside risks. We could see vetting against potential harms as a separate phase in the prioritisation process. Additionally, it also does not consider counterfactuals, with some interventions being more likely to happen anyways than others.

Given the high uncertainties, big error bars should be given around all parameters. While our model is designed to help prioritise interventions that help guide individuals to take significant actions, this approach might undervalue building a diverse and resilient EA community. The post “Doing EA Better” argues that quantifying might lead to worse results and weaken collective epistemics. We agree that this is a possibility and that it is easy to place too much emphasis on quantitative predictions that are merely based on intuition. Perhaps intuition without quantification might be better suited, or we might need models that let us formulate hypotheses so we can validate them.

We’re not researchers but community builders that need a framework to think about our decisions and will have to prioritise our next interventions soon. Given these constraints, we think it’s more useful to publish our current thoughts after a couple of weeks than to spend more time working on them alone. We’re curious to get feedback and are open to the possibility that this model won’t be useful to others or even to us after thinking about it more.
 

p_student_donates = beta(2.355081, 49.726) // 0.01 to 0.1, i.e., 1% to 10%, from <https://nunosempere.com/blog/2023/03/15/fit-beta/>
amount_student_donates = 5 to 20 // dollars

p_parent_donates = beta(3.287, 17.7577) // 0.05 to 0.3, i.e., 5% to 30%
amount_parent_donates = 50 to 500 // dollars

num_parents = 5 to 20
num_students = 15 to 90

expected_impact = num_students * p_student_donates * amount_student_donates + num_parents * p_parent_donates * amount_parent_donates