Which CEA-funded community-building events are cost-effective?

By OllieBase @ 2023-07-05T10:09 (+42)

Epistemic status: I’m ~75% confident in my bottom line – that EAGx events are more cost-effective than EA community-building retreats – based on this rough analysis (which took ~50 hours) and my 1.5 - 2 years of experience supporting EA community-building events. This post is part of the Community Events Retrospective sequence.

I lead the Community Events Programme (CEP) at CEA. This has consisted of two sub-programmes:

• EAGx events - locally-organised EA conferences supported by CEA.^[1]
• CEP grantmaking - making grants to support events, usually smaller retreats, usually also focused on EA movement-building.

I wanted to evaluate outcomes to see which of these programmes was producing more value per $ spent. In doing so, I hoped to also form a better sense of how to evaluate the impact of EA community-building events in general.

Bottom line upfront

The EAGx events I looked at^[2] were ~half the cost per person but the retreats I looked at^[3] produced a similar amount of value per person. This suggests that EAGx events are more cost-effective than smaller, community-building retreats. Specifically, taking the central estimates from analyses of attendee survey results, EAGx events seem to be ~1.4 - 2.1x more cost-effective than community-building retreats we funded via the Community Events programme across three metrics. ^[4]

This highlights the increasing returns to scale for these types of events and updates me towards focusing on larger events, but only when the goal of the event is to build the EA community by helping people build their networks. This finding is probably less relevant for events which aren’t focused on community-building, or which aim to have impact at the group level (e.g. by building consensus). Unless otherwise stated, the results don't apply to CEA's other events programmes, which often have different aims to the Community Events programme.

The impact of events is difficult to analyse. There are a wide range of potential outcomes; many outcomes might take a long time to materialise, and many will be fuzzy and hard to evaluate precisely (e.g. the benefits of meeting a new friend or a motivation boost). 

Gathering data is also difficult; we rely on self-reported outcomes, often from a minority of participants. Answers we receive are brief and we expect some respondents don’t interpret our questions as we intend. 

All of these difficulties mean you should take all of my claims in this report with a big pinch of salt. Some ways I could be wrong about all of this include:

• My analysis here has missed some important class of outcomes, perhaps because they’re difficult to analyse or don’t show up in our post-event surveys. For example, perhaps the value of the retreats I looked at will show up later, through more well-informed community-builders.
• Attendees who had an unusually valuable or unpleasant experience didn’t complete the survey and, if they did, it would materially change the topline result.
• The events I incorporated into this analysis were unusual in some way. I didn’t get results from a large enough range of events.

--

Process

My process, at a high level, was:

1. Survey event organisers to establish whether the event would’ve happened without CEA’s support;
2. Survey attendees on the value they got from events (n = ~500);
3. Subjectively score each case on how much value the event provided to the attendee (in impact terms), adjusting for the self-reported counterfactual likelihood of that outcome occurring without the event;
4. Use these scores in a BOTEC to calculate a 'impact per cost' metric;
5. Compare this metric between EAGx and CEP events, adjusting for the self-reported counterfactual likelihood of that event occurring without CEA’s support;
6. Supplement this with BOTECs based on raw connections data and “impactful” connections.

This survey went out over February and March 2023 and asked attendees about events that occurred in the second half of 2022, so attendees were being asked about events that took place 3–8 months ago.

Counterfactuals

To establish the counterfactual likelihood of whether the event would’ve happened at all, I also surveyed event organisers and asked:

• If CEA didn’t support their event, whether they would’ve sought funding elsewhere and what for;
• If they would’ve run the same event, where they might have applied, and how likely it is they would’ve applied;
• If they would’ve run a different event, which event, where they would’ve sought funding and how likely it is they would’ve applied;
• If they wouldn’t have run an event, why and what they would’ve done instead.

With this information, I eyeballed how much of the event’s value might have been captured in the worlds where I didn’t support the event. I estimate that ~16% of the value of EAGx events would’ve happened otherwise (if we didn’t support it), and ~35% of the value of CEP retreats would’ve happened otherwise. I expect these are both underestimates since some organisers claimed that there weren’t other funding opportunities available for them when there probably were (e.g. Open Phil, EA Infrastructure Fund). Since I expect both EAGx organisers and retreat organisers to underestimate in a similar way, I don’t expect this to materially affect the results.

This estimate of how much of the value of an event would’ve happened if we hadn’t supported it is baked into several parts of the following analysis and meaningfully affects the results. Specifically, my best guess estimate that EAGx events are ~1.8x more cost-effective on the specified outcome measures falls to ~1.5x if I assume that all of the events were equally likely to happen. Note that other EA-aligned funders reviewing this analysis should be wary of these counterfactuals: the existence of other funders is part of the reason why the CEP retreats were judged to be more likely to happen in the absence of the Community Events Programme.

Outcome measures

I compared the impact of EAGx events and CEP retreats across three outcome measures:

• Counterfactual “raw” connections created - We define a connection as someone an attendee met at the event who they now feel comfortable asking for a favour. This could be someone they met at the event for the first time or someone they’ve known for a while but didn’t feel comfortable reaching out to until the event. We ask attendees for the number of new connections they made at the end of the event, and I asked them for this number again in the follow-up survey. I took the mean average of responses from both questions for each event and multiplied this by the number of attendees to get the approximate total number of connections created at the event. I then multiplied this by the % of counterfactual value CEA can claim by supporting the event to get counterfactual raw connections for each event.
   
• Counterfactual “impactful” connections created - I introduced a new metric in the follow-up survey to try to capture certain event attendee connections that might be more impactful than other connections. I asked them for the number of connections which “might accelerate (or have accelerated) you on your path to impact (e.g. someone who might connect you to a job opportunity or a new collaborator on your work)”. I then followed the same process as above - taking the mean for each event and multiplying it by the number of attendees and the % of counterfactual value CEA can claim.
   
• Counterfactual valuable outcomes -  The number of connections isn’t the only source of value for event attendees. To approximate other sources of value, I assessed attendees' written answers to two open-ended questions about sources of value asked in the follow-up survey. I went through every written answer, scored each answer based on the system outlined in the appendix and adjusted this score based on the counterfactual value the attendee attributed to the event. Tallying these up, and extrapolating this out to the total number of attendees gave me a total score for each event which I called “impact points”, to introduce a spark of insight and a twist of originality. Note that this system is not intended to be a holistic scoring of the attendee’s potential impact, but rather a system for analysing the impact of the event according to the attendees’ survey responses. More on that in Appendix: Scoring system.

Costs

With three approximate outcome scores aggregated across the two event types, I calculated the costs for producing each event:

• EAGx events cost significantly more to run than retreats - 5 EAGx events in 2H22 cost ~$2.6m while the 7 CEP retreats I looked at cost ~$190k.
• I also spent significantly more time running EAGx events in 2H22; ~256 hours on EAGx events, and only ~24 hours on CEP retreats.
• I added in additional costs such as other CEA staff time and event organiser time
• Overall, the costs of running 5 EAGx events in 2H22 came to ~$3.14m while the cost of running 7 CEP retreats came to ~$203k.

Crucially, many more people attended the EAGx events I evaluated than the CEP retreats I evaluated.

• There were 3,568 attendees across the 5 EAGx events I looked at, but only 129 attendees across the 7 CEP retreats I looked at, a ~28x difference.
• This means the cost per person at EAGx events is ~half that of the cost per person at CEP retreats:^[5]
  • The cost per person for the EAGx events was ~$880 ($3.14m / 3,568)
  • The cost per person for the CEP retreats was ~$1,573 ($203k / 129)
  • This is a ~1.78x difference.
     

Comparison

So, before we even assess how well the retreats score on the outcome measures, it’s clear that the bar for retreats is high; to be cost-competitive on any measure, they need to generate almost twice the value per person than EAGx events. On the three outcome measures I used, the outcomes for retreats do not meet this bar:

Counterfactual “raw” connections 

Before introducing counterfactuals, it’s worth noting attendees report approximately the same average number of connections at EAGx events (7.1) as at retreats (6.8). Introducing the counterfactual value captured by the event suggests that EAGx events produced ~20,667 new connections, while CEP retreats produced ~655 connections. This means the cost per counterfactual raw connection at EAGx events is ~$152, while the cost per connection at CEP retreats is ~$310, a ~2x difference.

Counterfactual “impactful” connections 

Attendees report approximately the same average number of impactful connections at EAGx events (3.0) as at retreats (2.9). Introducing counterfactual value suggests EAGx events produced ~8,300 new impactful connections, while CEP retreats produced ~255 new impactful connections. This means the cost per counterfactual impactful connection at EAGx events is ~$378, while the cost per counterfactual impactful connection at CEP retreats is ~$797, a ~2.1x difference.

Counterfactual valuable outcomes

The impact points from each event varied quite widely, from an average of 0.4 per attendee (EAGxVirtual) to 2 (Longtermist organiser summit). The most impactful reported outcomes were 10 - 20x more valuable than the average (though the scoring system somewhat expected this heavy-tailedness), and many attendees reported outcomes that didn’t register as valuable on the scoring system.^[6] Overall, EAGx events produced ~2,573 counterfactual impact points, while CEP retreats produced ~119 counterfactual impact points. This means the cost per counterfactual impact point at EAGx events is ~$1,219, while the cost per counterfactual impact point at CEP retreats is ~$1,711, a ~1.4x difference.

That bottom line again: taking the central estimates from these three measures, EAGx events are ~1.4 - 2.1x more cost-effective than CEP retreats. This is driven by the following: 

• The cost per person at EAGx events is ~half the cost per person for retreat attendees;
• Outcomes per person are approximately similar in value;
• EAGx organisers reported that their events were less likely to happen without CEA support than retreat organisers. 

Uncertainty

I’m uncertain how to weight the different outcome measures - I trust the raw connections data most strongly because attendees gave approximately the same number when we asked them a second time. But the number of new connections does not capture the full picture of why people find EA events valuable. 

My impact scoring was more comprehensive in terms of capturing value because it is agnostic with regard to how attendees got value from the events. But the scoring system I created wasn’t rigorously designed, and it seems likely that there are other reasonable scoring systems or even that I would give different scores if I did the exercise again. 

Impactful connections is somewhere in between: it’s a measure which seeks to capture which connections were actually valuable, but we haven’t replicated this metric by asking the same people for this number at different times and it seems likely that different attendees have different standards for what they consider “impactful”.

My best guess is that I should put 40% weight on the counterfactual “raw” connections number, 20% on the counterfactual impactful connections measure, and 40% weight on the counterfactual valuable outcomes measure, leading to a best guess ratio of ~1.8x. Note that this is very close to the raw cost-per-person ratio of 1.78x, suggesting the difference in the cost-per-person figure more or less explains the cost-effectiveness difference because everything else cancels out.

My actual uncertainty is far wider than the ~1.4 - 2.1x ratio. Approximating a plausible lower and upper bound from these estimates suggests I actually think EAGx events are 0.7x - 3x more cost-effective than CEP retreats. 

• I put at least 10% credence on the claim that CEP retreats are actually more cost-effective than EAGx events. This might be because my analysis here has missed something important or because the events I incorporated into this analysis were unusual in some way.
• I put ~30% credence on the claim that EAGx events are more than twice as cost-effective as CEP retreats.
• I put ~60% credence on the claim that EAGx events are 1 - 2x more cost-effective than CEP retreats, with ~15% of this credence around the “it’s very hard to tell” range (i.e. ~1x or slightly above). 
• Overall, this means I’m ~75% confident in the fuzzier claim that EAGx events are more cost-effective than EA community-building retreats.

Commentary

The above numbers suggest a surprisingly large difference, but if we break it down into two key claims, it comes across as far more reasonable:

1. EAGx events are approximately half the cost per head;
2. Attendees get a similar amount of value at CEP retreats and EAGx events; 
3. Therefore, EAGx events are ~twice as cost-effective.

That second claim is perhaps the most surprising. I have heard (and made!) various claims like “retreats foster more impactful or deeper connections” or “I expect attendees get more value from a nicer, more intimate setting”. This might still be true, but these survey results should make us downweight these claims - I don’t see evidence that EA community-building retreats are more valuable for attendees than EAGx events, at least among the ones I’ve been involved with. 

I think the most salient thing this conclusion should point to is the importance of increasing returns to scale; perhaps retreats help those 20 - 40 people feel something special, but conferences provide a seemingly similar experience at ~half the cost per person. That retreat feeling needs to be really special to counter increasing returns to scale.

A reminder that this is just comparing events which aim to build the EA community, or connect community-builders. I still expect retreats to be more appropriate for goals such as:

• Achieving a specific outcome or agreeing on a set of goals;
• Creating a stronger / deeper social network among a small group of people;
• Creating stronger / deeper relationships between high-profile individuals.

But, for these kinds of retreats, I weakly advise against relying on metrics like connections in impact evaluations because, as shown above, retreats do not seem efficient for producing these outcomes.

Appendix: EAGxVirtual is unusually cost-effective

EAGxVirtual is something of an anomaly in the data. The cost per head for this event is substantially lower than any other, at ~$50 - $80 per person (depending on whether you count everyone registered or only those who were online at the event for at least a few hours). There is no real limit to how many people can attend this event either, so future iterations can probably improve on this number.

However, correspondingly, virtual conference attendees report less valuable experiences than in-person events:

• EAGxVirtual attendees reported an average of ~3.7 new “raw” connections. The in-person EAGx average is ~7.8.
• EAGxVirtual attendees reported an average of ~1.5 impactful connections. The in-person EAGx average is ~3.3.
• EAGxVirtual attendees scored an average of ~0.4 impact points. The in-person EAGx average is ~1.1.

But of course, in the cost-effectiveness game, costs matter. Although EAGxVirtual attendees reported getting ~half as much value at the event as in-person attendees^[7], EAGxVirtual costs ~10x less per person than in-person events, making EAGxVirtual far and away the most cost-effective event we ran in the second half of 2022. I don’t plan on switching entirely to virtual events, but this was an update in favour of virtual events and made me excited about EAGxVirtual 2023 (17 - 19 November 2023).

Appendix: Scoring system

I (Ollie) scored all the attendee responses on value from connections and responses on other sources of value, using the rough system below. ^[8]

• 50: Someone reports starting a project in an EA-aligned cause area
• 20: Someone reports getting an internship or job or full-time grant at an EA-aligned organisation
• 10: Someone reports shifting their focus to an EA-aligned cause area
• 5: Someone meets a collaborator who they intend to work with later on an EA-aligned project
• 1: Someone meets someone who inspires them to take an opportunity or cause area more seriously

The key thing about this system is that it’s essentially a logarithmic scale - the top of the chart is 50  times the bottom and intervals are ~5x. This is because I expect the value of event outcomes to fall into a heavy-tailed distribution or a power law distribution, in line with many other things (productivity, effectiveness of charities, athletes and researchers). After I started analysing the survey results, I feel more confident in this claim. For EAGxAustralia, three outcomes really stood out and the rest were good but unremarkable, according to the attendees. It seems likely to me that a good chunk of the value of the event was accrued by just a few people who had more life-changing things happen to them (e.g. getting a grant or job). 

Note that this system is not intended to be a holistic scoring of the attendee’s potential impact, but rather a system for analysing the impact of the event according to the attendees’ survey responses. I initially expected that cause prioritisation might have been an important factor for how outcomes were scored, but I ended up mostly focusing on whether the event was valuable for that person, based on their own prioritisation. If someone reports getting a job, that’s more impactful for that person than if that person just had a few interesting conversations.

As a result, I don’t expect disagreements in cause prioritisation to materially change many conclusions in this analysis. I find that EAGx events are cost-effective because they allow more interactions and learning to happen per $, so, naïvely, I expect this result to hold regardless of the cause areas represented at the event.

Appendix: Top impact stories

The impact reported in the follow-up survey I conducted suggests a heavy-tailed distribution (though note again that my scoring system somewhat expected this distribution, so this claim could be begging the question). The top 3 impact stories per event accounted for 20 - 55% of the impact of reported stories, according to my assessment, and this was true across various events.

Here are some quotes from the top stories reported, all of which received a score of >5 points. I’ve bold highlighted some text:

• Without EAGxSG, I would never have received funding to do independent alignment research, nor have the courage to apply to SERI MATS which I eventually got accepted to.
• [One connection] recommended that I apply for a planning grant for [organisation] and went on to become our grant evaluator - the grant was successful and enabled me to work full-time on the organisation.
• I have ended up collaborating with all 3 of them for a successful grant I received
• One person I met there introduced me to her organization and offered me to apply as an intern. Since I was on my way to shift my career to one with more impact anyway, I applied and actually got the spot.
• One [connection] has already led to a contract-based job offer, while the other has made further connections within my field which have proven really useful.
• I am working now in [organisation] because of these connections that I made during the event.

For comparison, here are some quotes from stories that received a score of 1 - 5:

• I made a connection with a speaker after attending his talk. I then approached him to deliver a talk to students from the program I work for and we had 68 attendees come and listen to his views on existential risk.
• We (those 10 [connections]) started a project together and meet weekly since the conference 
• Helped shed light on the opportunities and jobs within the EA community. Reinforced the idea that it can be valuable to work outside of typical EA orgs, and that working in non-EA roles can help bring new ideas to later work in EA orgs.
• Fellow AI safety researchers who helped me enter the field, explore research agendas, discuss strategy considerations, and help plan my US travels; I am still in contact with them and profit from these connections.

And here are some quotes from stories that received a score of 0 (this isn’t to say that these outcomes won’t be valuable, just that they haven’t led to anything tangible yet):

• I met a like-minded EA.
• They were very useful in directing me to resources and programmes in the areas of interest that I have.
• I had very interesting recommendations regarding my career plans, and discussions that taught me much about other EA topics.
• I found other people working on similar projects in their home countries. This led to fruitful discussion and we may collaborate in the future.

My thanks to Callum Calvert, Michel Justen, Sophie Thomson, Oscar Howie, Ben West, Eli Nathan, Ivan Burduk and Amy Labenz for comments and feedback.

 

1. ^{^}

   What’s the difference between EA Global and EAGx? EA Global conferences are organized by the Centre for Effective Altruism, and people from all over the world attend these. The team at CEA is responsible for choosing the content, processing admissions, and production of the conference. EAGx events, on the other hand, are community-organized with some support from CEA. The target audience for EAGx events is broader than EAG, but tends to have a more regional focus.

2. ^{^}

   EAGxAustralia, EAGxSingapore, EAGxBerlin, EAGxRotterdam and EAGxBerkeley.

3. ^{^}

   The EA Workplace/Professional Group Organizer Retreat, the Longtermist Organizers Summit, the Asia Community-builders retreat, the FERSTs (French x-risk) retreat, the Wild Animal Welfare Policy Summit, the African EA organisers summit and the Australia and New Zealand Group Leader's Retreat. The Wild Animal Welfare Policy stands out as one event not focused on community-building.

4. ^{^}

   This is the range implied by the central estimates from analyses of the three metrics I tracked. My actual uncertainty is far wider than this. Approximating a plausible lower and upper bound from these estimates suggests I actually think EAGx events are 0.7x - 3x more cost-effective than CEP retreats. 

5. ^{^}

   Note that this cost is not the same figure as how much CEA is willing to pay for a marginal attendee at an event. 

6. ^{^}

   e.g. “It was nice meeting people within my cause area in a more global setting”, and “knowing people to reach out to”.

7. ^{^}

   Note that, for these outcome measures, I’m only including survey respondents who presumably attended the event for enough time to consider themselves attendees.

8. ^{^}

   There were higher scores for more senior hires, but I ended up not using them.