A Report on running 1-1s with EA Virtual Programs' Participants

By ES, Yi-Yang, Jay @ 2023-01-10T01:34 (+34)

TL;DR: 

We trialed two rounds of 1-1s and, in total, connected 44 EA VP introductory participants with members of the EA community, focusing on mid-career professionals and early-career professionals/students interested in making career decisions related to EA. We’re still a bit uncertain, but generally feel positive about the cost-effectiveness and value of organising 1-1s with VP introductory participants ^[1]. Our results suggest that 1-1s are providing a decent amount of value and are generally highly enjoyed by participants. We’re unsure on the longer term effects of 1-1s on engagement in EA. 

More specifically, here’s our initial research questions and the answers we found:

Q: How valuable^[2] are 1-1s with EA community members for Intro participants or is there any demand for 1-1s?

A: There is strong demand for 1-1s and it seems like 1-1s are valuable for VP participants. We asked "how would people feel if they weren't able to do 1-1s", 21% said they would be very disappointed, which is 19% short from the 40% benchmark. Although it's short of the 40% benchmark, on average 1-1s are rated 8.2/10. This makes us feel a little more confident that some 1-1s can be valuable. We anecdotally think they are valuable because it helps expand their network and exposes them to more of the community. 

Q: How cost-effective or impactful are 1-1s in the context of EA VP?

A: We feel somewhat uncertain about the cost-effectiveness of 1-1s, but we think that 1-1s with VP participants are ~3x as cost efficient as an EAG in leading to connections, but only 1/20 as scalable. Some notes on impact: 1-1s are very hits-based, impact can take a long time and it’s hard to measure counterfactual impact. 

Q: What types of 1-1s are most valuable (ex. career focused ones, general EA ones, social, etc)?

A: We think that both career-focused and general networking 1-1s are valuable. Career ones for helping people be open to changes, meet others and get advice. General 1-1s to expand people’s network and to serve as practice for reaching out to community members. 

Q: What type of mentors doing the 1-1s are best suited? 

A: We found a broad range of ‘mentors’ to do 1-1s – mentors who are very knowledgeable about EA and sociable (such as community builders) rate highly. Cause-area specific mentors (e.g. an experienced grant-maker for someone working in philanthropy, an AI safety researcher for someone working as a software engineer) to talk about specific career paths are extremely valuable as well. 

Q: What ‘type’ of participant are these most valuable for? E.g. mid-career professionals, those open to changing careers / jobs, students, people near to EA hubs, etc.

A: We’re very excited about mid-career professionals who are interested in changing careers and jobs. We also think social 1-1’s are valuable for anyone highly interested in EA to help them expand their network. A future research direction is to expand the number of VP participants who are offered a 1-1 – primarily test participants whose VP application we were less excited about.

Q: At what stage of the fellowship is it most valuable (beginning, middle, end, after, etc)?

A: We don’t think 1-1s at the beginning of the fellowship is best. We’re generally uncertain on whether there’s a difference between middle, end, and after. It’s either pretty equal or we need to do more tests. 

Q: Are repeated 1-1s, maybe some form of mentorship, more valuable than one-off 1-1s? 

A: We didn’t manage to test repeated 1-1s or mentorship. That’s a potential future research direction. 

We have a few main future goals and paths. 

1. Find ways to streamline the 1-1 process on CEA’s side to make it easier to scale to all of VP 
2. Run some more specific trials with people interested in making EA-aligned career pivots and interested enough to accelerate their involvement in EA
3. Consider running specific 1-1 programmes with In-Depth participants 
4. Scale up active, targeted outreach to particularly talented individuals already slightly associated with effective altruism to get them to apply to opportunities like EAG 

Background

We (ES and Jay) ran two pilot 1-1 programs with Virtual Program (VP) Introductory fellowship participants. We wanted to understand if 1-1’s (both general EA ones and career focused ones) with EA community members were valuable at accelerating EA connectedness and taking significant EA-aligned action.

Our reasons for why this was worth testing in the first place are that 1-1s historically have been used extensively in community building in groups. EA VP is an ongoing and relatively stable way people get introduced to EA. The primary pathway is to find out about EA → sign up for the Intro fellowship → finish → take the in-depth fellowship → ?[do independent EA work, join EA Anywhere, attend an EAGx or EAG, make EA informed career decisions]?

However, that’s quite a long timeline from someone interested in EA to making significant life and/or career decisions based on EA. In particular, we had anecdotal data^[3] that there were: 

A) promising VP participants who were new-to-EA but had existing career skills that made it highly valuable to accelerate their introduction into EA

B) promising VP participants who wanted to take significant EA-informed actions earlier in the pathway (e.g. After the Intro fellowship)
We were especially excited about A as a source of mid-career talent that could help diversify the community and immediately address talent bottlenecks at EA organisations.  Moreover, community building to mid-career professionals generally seemed neglected at this point (less so now with the advent of EA Pathfinder). Generally, there’s a large untapped pipeline (almost anyone can engage in EA virtually, only people with in-person groups near them can engage in an in-person group) for ‘producing new EAs’.

We also think that while the VP Intro Fellowship curriculum is strongly focused on EA philosophy – and less on the EA community and integration into the community. This is suboptimal (to some degree) because we believe that engagement and involvement in the EA community (feeling welcomed, connected, etc) is critical to helping people ‘take the leap’ and practise EA as a life philosophy (make informed career and life decisions). VP currently neglects this. VP also serves as perhaps the most core programme to help EA diversify (especially geographically). 

Hence, the 1-1 programme. You might be asking, why 1-1s? Targeted 1-1 connections are optimal between talented individuals on the periphery of the community (e.g. an EA VP intro fellowship participant) and people who are very involved in the community (working at a major effective altruism organisation) – in particular because we don’t think most VP intro participants reach out to get connected to other EA’s in a professional or social capacity. This programme helps facilitate that connection. 

Methods

We ran two time-based cohorts with VP Introductory fellowship participants from September - November. The general method was to identify VP participants on a variety of criteria (our excitement about them, interest in EA, career hingeiness), ask them if they’d like to participate and what type of 1-1 would be most useful, match them with a 1-1 guide (someone in the EA community that we thought they’d be a good fit to talk to), and let them connect. More details on the selection and matching methodology are here. 

We made a few modifications in our method between the first and second trial – mainly the guides we had, the demographics of the Intro participants, and the resources we shared with participants. 

Results

You can view all our data here.

Some qualitative feedback from our second round:

• A student considering careers in AIS said they felt really fortunate to have the opportunity to talk to someone who had worked at an AIS organisation they were considering joining.
• A mid-career participant applied to the Charity Entrepreneurship programme because of their guide’s nudge.  
• A mid-career participant who was hoping to pivot sharply into a more impactful career said that the 1-1 made them feel like they were already part of the community, despite only having learned about EA recently.  

We also did a rough BOTEC on cost effectiveness.  We found that this programme is about 3x as cost efficient as an EAG in forming connections. 

For the first round, 8/14 participants rated the 1-1 a 10/10.  The lowest score was a 7/10.  The most common positive feedback from the participant side was that they appreciated getting to talk to an EA community member who shared their interests and professional background.  

For the second round, 9/13 guides rated their 1-1 a 9/10.  The lowest score was a 6/10.  The most common positive feedback from the guide side was that they felt like their participant found the 1-1 valuable and that they enjoyed the call.  

The programme is a bit high cost to run (in terms of taking up time and resources) – so much so that it might not be sustainable or scalable. The most time intensive parts were communicating with mentors and participants, selecting participants and matching them to mentors, and then mentors’ own time costs. There are some potential solutions -– we could invest in setting up some strong automated infrastructure that reduces the organiser’s burden of selecting VP participants, doing 1-1 matching, and communicating with participants. We’d ideally like to see more conclusive data of 1-1s being useful (ideally at improving long-term engagement in EA) before investing in a scaled programme, but generally we’re excited by the initial results. 

Discussion

How valuable are 1-1s and for whom

After this project, we now think that one-off conversations are not effective as general on-ramps into effective altruism ideas or the community. We still think they might be useful if the person conducting the call is very clear-thinking and able to usefully direct the conversation to a place of true intellectual exploration, but these skills are rare and so would best be targeted at a particularly talented audience (e.g. if you meet one of the best physics students in the world, and they’re somewhat interested in the space of problems that the effective altruism community cares about, it’s probably worth introducing them to someone especially good at talking about those problems).  We sent interest forms only to people who had satisfied criteria such as having completed a certain number of weeks in the Intro fellowship, and there was additional self-selection in the people who ended up filling out the interest form. Therefore, our participants didn’t need much additional encouragement to seriously think about the ideas or want to engage with the community.  

1-1s for community

From our feedback form, it seems like the participants felt a strong sense of community through the calls and the guides felt like a lot of the value they were providing came from prompting the participants towards next steps. This makes us feel hopeful that calls can possibly serve as an effective trigger for participants to get more engaged with the community. Moreover, ~20% of our participants had the skills to start contributing to key cause areas within a few months and explicitly wrote that they were considering career changes related to effective altruism ideas^[4]. Therefore, we feel more excited now about projects that involve active, targeted outreach nudging participants to apply to opportunities like EAG or 80k Career Advising.  

Is there a formula for matching? 

There was a huge difference in retention rate and feedback quality from our first and second rounds from participants. We had a median and mean score of 7 in the first round. For the second round, we had a median score of 10 and a mean score of 9.  It seems like there are certain crucial factors that are essential to get right in order to make programmes like ours that involve getting people to interact with each other in value-producing ways successful enough to gain information value from. These include a relatively high level of professionalism in communication, lots of transparency around motives and setting clear expectations around the calls for both guides and participants, and matching people quickly (within a week of the interest form closing). We also had significantly better matching in the second round since we had amassed more people who could serve as guides, and this made the 1-1’s generally go much better.  

Summary: So where does that leave us? 

We still think VP is relatively neglected, undervalued, and there’s lots of low-hanging fruit. 1-1s are potentially valuable – both from a career and impact perspective and a community connectedness perspective. We think that a VP 1-1 programme is valuable, comparable on the order of an EAG (!!). However, in its current state, the programme is too time-consuming and the data a bit too unclear for it to be scaled up entirely^[5]. 

We’re left with some questions and programme changes to try out with the ultimate hope of having more clarity on who 1-1s are most useful for and what the lowest-cost way to run such a programme is^[6]. We also want to try some other low-hanging fruit tests entirely different to 1-1s to help direct and squeeze out EA VP’s value^[7]. If you have any ideas, let us know.  

1. ^{^}

    The cost is mostly the time of organisers and people doing the 1-1s than money directly. 

2. ^{^}

   Valuable in a sense of increasing EA connectedness, interest in EA, knowledge.

3. ^{^}

    Most of the anecdotal data comes from ES' experience doing 1-1s within her VP cohorts. There generally was evidence that it increased engagement in the fellowship and post-fellowship in EA more generally.

4. ^{^}

    This is likely an overreport since some people filling out the interest form might have thought that writing this would improve their chances at having the 1-1 or being matched with someone especially useful to them, but even 10% is a very good ratio. 

5. ^{^}

   We think our data is somewhat untrustworthy. We generally believe participants struggle to properly fill out feedback forms and access things (as verified by asking the same question with different wordings and getting different responses). 

6. ^{^}

    Specific alternative versions were interested in: 
   - Trialling a version with 1-5 ‘intro to EA’ mentors for participants who want it / have facilitators who are less connected to the community doing all the 1-1s
   - Reaching out to the most promising VP participants (or perhaps via some other metric such as willingness to change careers / jobs) and only connecting with them (doing 1-1s, helping them apply for 80k advising, etc)

7. ^{^}

    Such as: trialling opt-in group information sessions (ex. Writing on the EA Forum, Intro to EAGx’s and EAGs), trialling an email / slack account where participants can ask questions and get sent resources , upskilling facilitators (on their level of EA community engagement and knowledge) to equip them to do 1-1s, running a next steps programme after the In-Depth fellowship