Lessons from a full-time community builder. Part 1 of 4. Impact assessment

By weeatquince @ 2017-10-04T18:14 (+15)

NOTE: For the past year Sam Hilton has been funded by the EA community in London to grow, run and support the community. This is part 1 of a 4 part write up, broken down as follows:

Part 1. Impact assessment

Part 2.  General lessons on how to build EA communities.

Part 3. Specific lessons on running a large local community.

Part 4. Future plans and a request for funding

[Links will be provided to future articles when they are written]

Summary

Background: Sam was funded to work full time on the EA London community for a year.

What was done:

• Our theory of change was:.

• Raising awareness of EA -> engagement in our community -> positive behaviour changes

• 80% of project time went into events, publicity and marketing. We ran 70 events with 900 unique attendees.

• 20% of project time went into supporting other EA communities and projects in London, such as London's student groups, EA policy groups and one-on-one support.
  
  

What was the estimated impact:

• Our events caused 12 GWWC pledges and 6 other large behaviour changes and about 175 smaller but significant changes to behavior or beliefs.

• We believe we would have had 1/3 of this benefit if we were volunteer run.

• Our cost per pledge or equivalent is $2000. This is close to but not quite as effective as other EA meta-organisations (approximate average benefit of $1000 per pledge/etc).

• We have also had a variety of other benefits that are harder to put numbers on, such as the aforementioned sub-communities, projects created as a result of EAs networking, people retaining a engagement with EA ideas, and so on.
  
  

Conclusions:

• We did not reach a strong conclusion on if it is sensible to fund full-time EA community organisers to run non-student EA groups in large cities.

• We do think there is a strong case for targeted community outreach (such as to civil servants or poker players or start-up founders).

• We think there is a reasonable case for funding EA London for another year. 

Contents

- Background and initial plans

- Breakdown of activities and costs

- Impact – key data

- Impact – Further analysis

- Impact – Other benefits and stories of success

- Update on June 2017 – Sept 2017

- Conclusions

- Annex A: Impact – elaboration & data collection

- Annex B: London’s other EA communities

This document can be read as a Google Doc here.
This is part 1 of 4. Other parts cover community building, lessons learned and future plans. [Links pending]   

Background and initial plans

This impact assessment covers the activities done by Effective Altruism London from 3 June 2016 until 7 June 2017.

HISTORY

In April 2013, the first Effective Altruism London social event was held. The community grew slowly but steadily, drawing from outreach efforts in London and growth in the wider EA community. In June 2016 the community began funding Sam Hilton to act as a full time movement builder for a year.

THE VISION

Our aspirational vision was written collectively by the community and has helped guide actions. It is:

Everyone in London working effectively towards a better world

MODEL / THEORY OF CHANGE

We had a 3 stage model of how we create impact. Awareness of the community leads to people engaging which leads to people changing their behaviour to be more altruistic and effective:

This model has remained largely unchanged for the year, although we also tried to recognise (but do not have good measures for) alternative paths to impact. Including:

• Community benefits, such as networks and EA friendships leading to projects and actions taken.

• Benefits of keeping existing EAs within the EA community

AIMS AND PLANS

The aims set for this year were:

1. Work out the best ways of building an engaged EA community in London

2. Secondary aim: Support and inspire those in the EA London community to have a greater positive impact on the world.

3. Tertiary aim: Grow the EA movement in London
   
   

Plans were made about how these aims would be achieved.  For example, we planned to:

• Experiment with online marketing

• Experiment with a variety of events (Workshops, Giving Circles, Discussions, etc)

• Support individuals with their EA projects.

• Measure and track impact
  
  

Details of the initial plans can be found at: February 2016 _ Request for funding

Plans, adjusted following our mid-year progress review, are at: Jan 2017 _ Six month review

We had a budget of just over £30,000 to carry out this work

PREDICTIONS

Prior to the year beginning we predicted the following counterfactual impacts

• Awareness: New contacts (email, meetup, etc) of 1,500-2,500 people

• Engagement: Unique new event attendees of: 400-800

• Behaviour change: New regular attendees: 50-250. New GWWC pledges: 20-55. New significant career changes: 10-30

Breakdown of activities and costs
ACTIVITIES

The following chart shows the breakdown of key activities carried out in the past year.

This is roughly as expected from the initial plans, although perhaps with:

• less time going into structured experiments

• more time than expected spent on impact measurement

• more time than expected spent on growing EA London’s subgroups

COSTS

The following chart shows the breakdown of financial costs for the past year.

We spent £19,510 from 1 June 2016 until 1 June 2017. Adding in estimates for gifts in kind (books from CEA, printing) this brings the cost to £19,760. Sam took a low wage for the year. If accounting for the loss in salary of £15,000 the cost becomes approximately £35,000 (assumes 100% of additional salary earned would have been donated).

Note: We have not accounted for time spent by volunteers. This is difficult to do as it is unclear:

• how time would have been spent otherwise,

• how much they benefited in terms of career capital (overall a benefit is likely)

• what should be counted as volunteering as opposed to community engagement or an individual’s personal EA outreach actions.

Impact – key data

HEADLINE FIGURES

The total impact of Effective Altruism London this year is estimated to be:

• 18 people had large behaviour changes including 12 GWWC pledges.
  
  

• 174 people had some significant, but not large, changes to belief or behaviour

• Including 10 taking Giving What We Can’s ‘Try Giving’ pledge

• Including 14 impact-adjusted career plan changes at 80,000 Hours workshops
  
  

• 900 unique event attendees over the year of which 240 attended multiple events

• 3000 new unique individuals contactable on email list, Meetup and Facebook
  
  

• 7 new EA subgroups supported (LSE, UCL, animal welfare, finance, 2x policy, quakers)
  
  

• Additional reported but unquantified benefits for community members, in particular:

• Networking effects. Eg. meeting a contact who can help with work / a decision.

• EA Retention. Helping people retain an EA mind-set after university.
  
  

The above figures are calculated estimates rather than numbers known with certainty. We explain how they were calculated below in Annex A: Impact – elaboration & data collection.

IMPACT OF PROVIDING FUNDING

We estimate that if not funded we would have had 33% of the impact on behaviour and attendance, 25% of the impact on contactable individuals, and the finance subgroup would have been started. This is based on the amount of and size of events run in by volunteers in previous years.

This means that the impact on behaviour of funding EA London, rather than letting it be run by volunteers, is about 2/3 of the impact set out above, ie. 12 large behaviour changes of which 8 are GWWC pledges.

Additionally not being volunteer run means a reduced risk of community collapse, and the additional benefits created, as set out in the section: Impact – Other benefits and stories of success.

HOW THIS COMPARES TO OTHER EA ORGS

Our measured impact on behaviour changes suggests we are in the same order of magnitude as other meta EA organisations but somewhat less effective.

The returns to funding other EA orgs is in the ballpark of $1000 per pledge or equivalent (am uncertain estimate based on talking to people who work in EA organisations). We are getting a pledge level change in behaviour for about $2000 of funding (depends on how measured, see Cost benefit analysis section in Appendix A below).

Impact – further analysis

GROWTH

Growth rates are unclear but generally positive. Noise is added as the kinds of events being run has changed throughout the year as we have experimented with different things.

• Monthly attendance numbers have grown by 60%. We have moved from about 90 monthly attendees (average Jun-Sep 2016) to about 145 monthly attendees (average Mar-Jun 2017).

• If we look at data beyond the end of the year (in orange) monthly attendance has risen to 190 attendees (Average June-Sept 2017, staff time still being invested).

• Attendance at the regular monthly social has risen about 100%. These have gone from about 25 people to about 50 people.

On the other hand, we did not find additional evidence that we are changing behaviour or getting more new people engaged at an increasing rate. This maybe a sign that we are not successfully engaging more new attendees despite larger events, but might also be because the data set is noisy, too small and has missing data.

HOW OUR ACTIONS CREATED IMPACT

Roughly 80% of project time went into marketing and running events, which led to the behaviour changes we saw (18 large and 174 significant behaviour changes). It is difficult to know exactly which actions lead to which changes as people often do not change their behaviour straight away.

We can see from the event reports that social events led to the most new attendees (people for whom this was their very first event) returning (attending another event with 50 days):

NOTES: • Not adjusted to account for imperfect attendance data, total numbers returning is likely to be higher.
• Dates covered in the above table: 01 Aug 16 to 31 Jul 17.   • Returning means attended another event within 50 days.

More detail on exactly which events, campaigns and actions were more impactful than others is covered in part 3 of the write-up on Specific lessons on running a large local community [link pending].

Roughly 20% of the time went into supporting the sub groups, supporting individuals with their projects, and so forth. We believe this had other benefits, such as the existence of student EA groups at London universities. (Events at student groups and any resultant behaviour changes are not captured by our metrics, eg. we know of and have not counted 3 pledges at student groups).

WHY COUNTERFACTUAL IMPACT OF FUNDING WAS LOWER THAN PREDICTIONS

How our predictions match our results:

* Predictions made comparing to if volunteer run. Results are are the marginal impact, so 33% lower than headline figures..

We overpredicted the expected behaviour change. We have a good understanding of why this happened. The main problem was that we based predictions on self-reported behaviour change, but this did not match up to actual behaviour changes. For example, people said on surveys that attending EA London events has already had a drastic (8 out of 8) effect on their behaviour but upon questioning failed to give clear evidence of such changes.

Also, our behaviour change metric does not capture future behaviour changes. For example, changing a career plan (or donation plan, etc) would not be captured by our metric, unless we believe the person could give evidence that they had already changed their career (or donations etc). Anecdotal evidence does however suggest that people who are attending regular events and report an intention to change their behaviour may well actually do so.

Predictions of attendance and repeat attendance very closely matched results.

Impact – Other benefits and stories of success

As well as introducing new people to EA we have had significant other successes, as illustrated by the anecdotes below:

Founding and supporting sub-communities. Some examples:

• Starting EA student groups at LSE and UCL. The leaders of both these student groups decided to found them as a direct result of being involved in EA London.

• Supporting student groups. Also supported groups at SOAS, Imperial and Queen Mary. Grew students’ email lists by about 2000 emails. Arranged career workshops for 250+ students (with over 20 IA-SPC).  NOTE: Student group attendees are excluded from our metrics.

• Founding HIPE (High Impact Policy Engine) a highly promising initiative to support civil servants who want to have a positive impact with their careers.
  
  

Networking effects. Attendees have commented that they have benefited from the networking effects of the EA London community. Some examples:

• One of our past attendees used contacts she met at EA London events to help her set up an All Party Parliamentary Group (APPG) on Future Generations.

• We connected an EA climate scientist to a wealthy entrepreneur looking for his next project.
  
  

Retention effects. Attendees have commented that the EA London community helps make sure they continue to engage with EA ideas. However, this effect is hard to measure and we have no good evidence or stories of this. We have seen a few people attending events who had been involved in EA at university, not done much since and EA London gets them re-engaging with EA ideas.

Paved the way for future behaviour change. We have seen people who got involved in EA in part through EA London, prior to this year, play a bigger role in the EA community including:

• Michael Plant (likely to have come across EA anyway) is working on measuring happiness, drug policy reform, mental health issues and regularly engaging with the EA community.

• David Nash, now doing EA community building work for EA London

• Sanjay Joshi, founded SoGive - a social enterprise to give comprehensive charity evaluation data.

These are not benefits created this year but are evidence that impressive actions and significant life changes may take longer than a year to manifest. Already we have seen:

• A senior communications official connecting HNWs to CEA

• A significant number of EAs begin working at Founders Pledge in London, some following EA London career advice.

• Someone who got involved through the EA London community begin work at CEA
  
  

Paved the way for future growth of the London community. We should grow more quickly in the future as we have gained:

• Size. As evidenced above, more people are engaging with us each month.

• Infrastructure We are a registered charity and now have a regular email newsletter. The student groups and subgroups are providing excellent outreach opportunities.

• Lessons learnt. As we set out in parts 2 and 3 [links pending], we have learned a significant amount about community building

• Contacts and relationships with other philanthropy groups across London.
  
  

Downsides. It is likely some of the people who have put time and effort to advising and supporting EA London may have had significant benefit elsewhere.

Update on June – Sept

In the last 3 months we have continued to put time into EA London and to grow (captured by the orange bars in the charts in the section on growth). We have also been training new staff. Sam took a step back (and will shortly be moving into a Government role) and Holly Morgan and David Nash took over the EA London work. In these 4 months we have seen:

• The EA London Retreat 33 people attended of which 9 changed their mind on the top cause and 5 changed their career plans

• Our month with the highest total event attendance in a while (230) including our largest ever monthly social (66 people).

• 500 new people signed up to the email list at Just V show and about 3000 more have signed up to student groups at London freshers’ fairs (including in early October).

Conclusions

Overall the conclusions from this year are rather weak. As set out above we believe:

• EA London has been in the same magnitude as effectiveness as other EA movement building organisations, although our measured impact seems lower.

SHOULD EAS CONTINUE TO FUND THE LONDON COMMUNITY

 As a result of our limited success this year we intend to change our plans. We hope that with slightly different plans and more focus we can have an even greater impact in future.

Against funding:

• If Effective Altruism London is less effective than other organisations why fund it?

• It is hard to imagine that a local organisation has the same potential to grow as organisations doing mass outreach.

• It is possible that the people we would fund (especially Holly Morgan) could be doing more impactful work with their time.

For funding:

• A year is a short space of time in the philanthropy world. It is not obvious that it makes sense to pull the plug on a new organisation within a year if it is not as effective as more established organisations.

• Updated plans could mean more impact in future.
  
  

Overall, we think it depends on what the other opportunities for funding meta EA work such as movement building work are. We believe such opportunities are limited and that CEA are not considerably funding-constrained. We have, collectively as a community, taken the decision to develop future plans and seek funding for another year.

See part 4 for our future plans [link pending].

SHOULD EAS FUND FULL TIME LOCAL COMMUNITY BUILDERS IN MORE CITIES

We believe the larger EA organisations providing small amounts of funding and support to local communities is useful. That said, this year has not made a strong case either for or against funding full time local community organisers. Overall, we have no strong recommendation either way and it may depend on what other opportunities are available.

We do however think there is value in funding outreach to innovative community groups with specific targets, such as HIPE which is reaching UK civil servants, REG which reached top poker players or Founders Pledge which is reaching founders. This is defended by our ideas of how communities grow (see part 2 of write up on: General lessons on how to build EA communities) and analysis of our subgroups (see section below: Annex B: Subgroups). We will look to do more of this in London next year.

If people in the EA movement do fund local city organisers they should be prepared for the fact that compared to other movement building actions that involve mass marketing, growing a local community will produce less rigorous quantitative data but will provide a greater insight into the individuals impacted. 

RECOMMENDATIONS FOR CEA / LEAN 

A few things we think that CEA and/or LEAN should be considering in their work to build the EA movement:

• Funding. Continue to provide small levels of funding to local groups. Especially for things like copies of books or occasional events.

• Retention research. Build understanding on how people stay or leave the EA community over time and the impact of this would be useful.

• Joining up of groups in different locations. For example, connecting students to groups in cities following graduation.

• More high-quality support with the administration work for running a local group. A lot of the admin work done feels easily replicable. Building on the fantastic work already done, more support could be centrally provided, including support: with handling data, measuring impact, emails, newsletters, websites, support engaging with HNWs, etc.

Annex A: Impact – elaboration & data collection

You can read this here.

Annex B: Subgroups

You can read this here.