CEA update: Q1 2021
By MaxDalton @ 2021-04-22T05:36 (+93)
This is a linkpost to https://www.centreforeffectivealtruism.org/blog/cea-update-q1-2021
This update covers CEA's work in the first quarter of 2021.
Background
Our mission is to build a community of students and professionals acting on EA principles, by creating and sustaining high-quality discussion spaces.
In 2019, we focused on stabilizing the organization and improving execution. In 2020, we clarified and narrowed our scope (by setting strategy and spinning off Funds and GWWC).
In 2021, we are focused on working towards our annual goals, as well as growing our team.
Program progress
These are brief summaries; you can find more details for each program further down in this post.
Groups
- Support
- We had around 100 calls and 120 in-depth email / Slack exchanges with group leaders. We received positive feedback on the calls (average likelihood to recommend >9/10). We increased 1:1 support for highly-ranked university groups, and helped to seed a group at Georgetown University.
- Fellowships
- We worked with Emma Abele (a contractor and CBG recipient) and EA groups at Oxford and Stanford to set up virtual fellowships. We also hired a contractor to provide 1:1 support for groups running fellowships.
- The number of people attending a fellowship grew 3x quarter-on-quarter (according to our records - there may have been some underreporting last quarter. We expect there was substantial growth in any case.)
- We’re working with Max Daniel to develop a new curriculum for the In-Depth Fellowship.
- Enhanced support
- We’re trying out enhanced support for law students, Black and Hispanic/Latinx community members, and group organizers in areas where the movement is just beginning to grow.
- Community building grants:
- We made several renewal grants and several new grants. We are looking to expand our capacity in this area.
Forum
- The number of hours people spent engaging with the Forum grew by 19% quarter-on-quarter, and we’re on track to exceed our target of doubling engagement time from 2020 to 2021.
- We published all content from the Introductory Fellowship[1] as a series of sequences on the Forum, which will make the Forum a better place to learn about EA.
- We’re hiring for a full-stack engineer to help us to develop features more quickly.
- We cross-posted a lot of old content (so that more high-quality content is searchable via the Forum), and began to build a wiki, in collaboration with Pablo Stafforini.
Events (EA Global: Reconnect)
- EA Global: Reconnect was focused on building connections between existing highly-engaged community members.
- 850 people attended, and they booked about 7 meetings on average. We recorded more meetings than we did in all of 2020, though that’s partly due to a different system of measurement.
Community health
- The team made good progress on a variety of small proactive projects, and continued to do reactive work on areas like media, interpersonal situations in the community, and reducing risk in spaces and locations where EA is newly developing.
Expanding capacity
My (Max’s) main focus in 2021 is on hiring.
Reasons for this:
- Stable base: We now have funding, strategy, and management capacity to build on.
- Important areas: We think that EA survey data + program data suggests that we are operating in some extremely important areas (e.g. university groups, events).
- Room for growth: We’re still taking only a small fraction of the opportunities available in those areas.
In Q1 we opened two rounds, for a finance lead and a full-stack engineer. We also began to redraft our careers page and made some improvements to our hiring process. We’re on track to open more rounds in Q2.
Other progress
- We have secured enough funding to cover our next two years of operation and expansion.
- We have moved into our new Oxford office, improved our cybersecurity, and streamlined a number of HR systems.
Program updates
Groups
We help local group organizers by advising them, providing resources, and creating online spaces where they can share resources and support each other.
Support and advice
- We had nearly 100 calls with groups (especially highly-ranked university groups), and received positive feedback on the calls (average likelihood to recommend >9/10).
- We helped to set up a new group at Georgetown University.
- We’re working with established groups, like Stanford and LSE, to build up capacity in less-established groups (like Chicago, UCL, Imperial).
Fellowships
- ~3x growth: We saw 1,082 people go through a fellowship, compared to 310 in Q4. (Some of this may be because we don’t have a full record of group-run fellowships in Q4, but there was still substantial growth.)
- In-Depth Fellowship development: Max Daniel is working with us to develop the In-Depth Fellowship.
- Virtual programs: We launched EA Virtual Programs. Over 600 participants (including 78 from highly-ranked university groups) and 100 facilitators took part.
- The fellowships are using a well-vetted curriculum, so we think the overall experience is likely to be high-quality (we’re currently analyzing post fellowship survey results).
- We think these programs are a scalable way to give group organizers around the world a good understanding of EA, which we hope will support high-fidelity transmission of EA ideas.
- For the first time, we had a person this quarter who was dedicated to providing 1:1 support to groups running fellowships locally.
- Note that while we provided the curriculum and general support for this work, the volunteer facilitators and fellowship organizers deserve the credit for actually carrying out the programs.
Enhanced support
We’re testing out providing additional support for specific populations within EA to see if we can foster better growth and retention there.
- Law: Matt Reardon, a contractor and former Harvard Law EA leader, launched a Legal Topics in Effective Altruism reading group with 59 participants and 6 facilitators as a mechanism to discover alumni and seed potential new EA law school groups.
- EA in new geographies: We connected organizers in geographic areas where EA is just beginning to grow with community members who had expertise in those areas.
Other
- We addressed 26 group-related community health issues, including four significant cases.
- We made many renewal grants to groups, and expanded support for groups in Brown University, the University of Oxford, Sweden, and the Netherlands.
Much of our work with groups happens in collaboration with group leaders, and some of this impact should be attributed to them.
You can find more details about this work in the appendix.
Forum/online
The Effective Altruism Forum (EA Forum) aims to be the central place for discussion about how to do the most good.
Progress
Key progress
- We have now posted the Introductory Fellowship content as a set of sequences on the Forum; this marks the first time that the Forum has offered a reasonably comprehensive and high-quality introduction to effective altruism.
- Our total engagement time grew by 19% over last quarter. March was our best month ever in terms of views and total engagement, and the week of March 29th was our best week ever. We continue to see a trend of increased Forum activity from top thinkers and researchers in a variety of cause areas.
Other progress
- We hired a contractor (Brian Tan) to encourage discussion on social media (e.g. answering introductory questions in the EA Facebook group).
- Pablo Stafforini agreed to host his in-progress EA encyclopedia as a wiki on the EA Forum.
- We have cross-posted a lot of material from other CEA projects (e.g. EA Global, EA Concepts) onto the Forum, and we are currently testing the impact on search traffic of forwarding EA Concepts visitors to those Forum articles.
- We enabled the EA Hub to share our authentication system, letting us track our impact across multiple platforms.
Technical progress has been slower than expected. We are in the process of hiring another developer. (The application period is over; we are currently conducting interviews.)
Metrics
This shows the number of views on the EA Forum by logged-in users. Each day's data point represents the average of the previous 30 days. We believe the spike in mid-2020 was largely the result of an issue where bots created false accounts before we made a technical fix.
This shows engagement for a variety of content projects over the last year.
Events
Events enable attendees to make new connections, learn about core concepts, share and discuss new research, and coordinate on projects.
EA Global: Reconnect
This quarter was spent planning and executing EA Global: Reconnect. This event was focused on increasing the numbers of connections between highly-engaged community members. We have not yet deeply analyzed the data, so we are sharing our initial impressions.
New ideas we tested:
- We sent invitations to register directly (via a shorter form) to EA organization staff and to people who had previously attended EA Global more than once.
- Group sessions
- We ran 58 in total.
- We introduced discussion groups where participants met in groups of 3-4 for three 18-minute blocks to discuss topics from a main session.
- The Ambassador program has previously arranged 1:1 meetings between more senior people in a field and mentees. This time, we also introduced private mixers for Ambassadors, where they met 5-7 specifically invited attendees for a 60-minute session of speed 1:1s.
- Less, higher-quality content
- We sharply reduced the amount of centralized content in order to focus on meetings, but we spent more time on each of the four sessions we organized. We clustered them together to make it easier for attendees to plan their weekends.
- Speakers received professional cameras, lights, and mics to ensure consistent production quality.
- Our sense is that the talks that we did have were a bit more interesting/high-quality than the average last year. On YouTube, the “Maintaining Motivation” session and Ezra Klein’s conversation with Rob Wiblin have both reached significantly more people than an average talk.
Outcomes
We did not hit the majority of the goals we set for the event, and we’re a bit disappointed in the outcomes, although we still think it was worth doing. We’ll reflect on this over the next month. While we’ve learned a lot about virtual events this year, we hope this will be one of the last events in which “Zoom fatigue” will be a major factor.
- Attendance: We had 851 unique logins to the event app, and 718 people scheduled at least one meeting.
- Connections: We focused on helping attendees make connections.
- The app shows 4,300 hours of meetings, more than what we recorded across all events last year. Some of this increase happened because we asked attendees to record meetings they scheduled in other platforms (like Calendly), in addition to those scheduled via the conference app.
- The mean survey respondent reported five new connections, which is higher than the mean for any event last year. Survey respondents were likely more engaged, so survey results probably overestimate the mean for all attendees.
- Diversity: We had a goal of introducing more community members to the work of EA leaders from underrepresented ethnic groups in EA, many of whom volunteered as presenters and ambassadors. 38% of speakers were from these groups, but we missed our target of >45% of speakers being female or non-binary.
- Other metrics:
- Likelihood to recommend: 7.7/10 compared to an average of 8.3 last year.
- Social media connections: The mean respondent reported gaining 3.5 new social media connections due to the event.
- Forum engagement: 11/70 (16%) of survey respondents who had a Forum account, but hadn’t used it in the last six months, logged in again. An additional 25 attendees created accounts.
We also began to process applications for teams organizing EAGx conferences, and we aim to support more EAGx conferences this year than last.
Community health
The Community Health team aims to preserve paths to value and mitigate key risks to the future of the EA community. Our work includes fostering a healthy culture, improving diversity, mitigating harm done by risky actors, reducing the harm of negative PR, and identifying risks to early field-building.
Reactive work
We handled:
- 10 inquiries or cases regarding media stories about EA
- 5 concerns around interpersonal problems such as sexual harassment
- 28 cases where we advised on situations in early field-building (geographical areas or academic/professional fields where EA is just getting established)
- 16 other situations where we advised groups, organizations, and individuals on situations like conflicts between staff or group members, online conflicts, best practices in HR, personal or mental health problems, and improving diversity, equity, and inclusion.
Examples of different cases from this quarter:
- We provided support and advice to three people concerned about colleagues struggling with personal/mental health problems.
- Interpersonal problems: We reached out to a community member about concerns/rumors regarding his behavior toward women. At his request, we met with him to discuss steps he’s already taken to remedy this and continued steps to take.
- Media: A journalist at a respected outlet planned to cover an EA-related topic they were somewhat unfamiliar with. We connected them with 2 interviewees who had relevant expertise and experience working with journalists (publication pending).
- Early field-building: At the request of a community member, we coordinated advice for a very popular YouTube creator who is brand new to EA and interested in creating EA-related content. Luke Freeman of Giving What We Can is now working with the creator on ideas for high-fidelity content that would still be of interest to their broad audience.
Proactive work
Brand
- We completed a brand survey with Rethink Priorities to see how many US college students had heard of effective altruism and how positive or negative their impressions were. Results:
- 10-15% of respondents had heard of EA. Awareness seemed higher among students at highly-ranked universities, but the sample size of that sub-category was very small.
- ‘A course’ was the most common source of hearing about EA.
- Almost every respondent who had any impression of EA had a positive impression of it.
- We provided mentorship and coaching to 2 early-career community members who have public communications experience and who are exploring doing more public comms in sensitive areas (meta EA and bio).
Diversity, equity, and inclusion
- We advised an EA organization that asked for guidance around better supporting their staff and recruiting a more diverse staff. We conducted interviews with three women familiar with the organization to get their advice for the organization on gender inclusion.
- Community health staff followed up with 70+ Black and Hispanic/Latinx[2] attendees of EA fellowships and/or the 2020 Student Summit to offer calls, information about upcoming events, or other connections in the community relevant to their interests. 15 students joined one-on-one calls and several attended the EA Fellowship Weekend.
- We co-hosted two icebreakers on peer mentorship to build connections between students in underrepresented groups. One was for Black and African students. The other was for people from underrepresented racial or ethnic backgrounds generally, co-hosted with a Fellowship Weekend organizer.
- Attendees had some similar experiences and some different bottlenecks. It may be effective to do peer mentorship events around shared minority experiences, such as first-generation college students, international students, etc.
- We advised a Black EA group organizer who is exploring thoughtful community building support for this cluster.
Culture
- We published a guide aimed at EA organizations on when mediation is or is not appropriate for problems including bullying and sexual harassment.
- Julia presented to organizers of fellowships and reading groups about handling conflicts that come up around EA’s relationships with other movements or views including social justice. Feedback indicated the material wasn’t specific enough; Julia is working on a written version that she aims to make more specific and actionable.
- Julia hosted a series of interviews at EA Global: Reconnect about factors that can alienate people from EA and how four community members stay connected.
Epistemics
- We want to support the community’s epistemics, to ensure that we think carefully and reason well. One way we hope to do this is to make it easier for newer community members to ask questions of more experienced members. In March, we beta-tested an ‘EA Librarian’ service on the Slack channel of the In-Depth Fellowship. People asked 16 questions, and while feedback was limited, it was quite positive (e.g. someone wrote: “I REALLY APPRECIATE THIS. It fills a crucial gap in the EA community support I have access to. Thank you!”). We hope to launch an iterated version for a larger audience in Q2.
- We provided input on the epistemics section of the In-Depth Fellowship content.
Reflections
- Given the positive results from the brand survey, we aren’t planning to prioritize further brand research on US college students. However, we think an important gap exists to do more proactive communication work (such as working with orgs to develop effective communication plans about longtermist and EA ideas, and differentiating approaches for different audiences or communication channels). We may seek to hire for this role in Q2.
- We didn’t make as much progress on epistemics as we originally planned (due to planning fallacy and a large reactive caseload). In Q2, we plan to better anticipate these potential obstacles.
Operations
The Operations team provides support across finance, legal, HR, grantmaking, office management, cybersecurity, and fundraising. This support enables CEA, 80,000 Hours, Forethought, EA Funds, and Giving What We Can to run efficiently.
Progress this quarter
Finance lead hire
- We’ve nearly completed a hiring round for a finance lead, and we think we’ll be able to find a suitable candidate.
Office refurbishment
- The office space is now functional, and initial staff feedback is very positive.
- Our dashboard is displaying real-time metrics from across the organization.
- We’re close to finalizing the cost allocation (this is complex due to the number of organizations involved).
Customer Relationship Management (CRM) software
- We’re aiming to integrate data from different programs, to allow us to do more personalized outreach, and to help us to assess our impact.
- We have determined the scope of this project after extensive internal stakeholder engagement.
- We plan to test this with two EA university groups in Q2.
HR and cybersecurity
- We revised our staff handbook and supporting policies, simplifying them.
- We revised contracts and NDAs.
- We developed an internal wiki for CEA staff.
- We completed a data mapping exercise and began a data audit.
Other
- We completed our US audit efficiently without any apparent issues, awaiting report.
- We completed benchmarking research for executive compensation.
Morale and retention
Average morale in Q1 was 6.45/10, which represents a slight drop from the previous quarter (6.61). Some of this may be due to a change in how we’re capturing this morale data, as we’ve shifted from using 15Five to using a weekly Slack poll.
Louis Dixon (finance lead) decided to leave to pursue study in computer science. We’re grateful that he gave us ample notice, and optimistic about finding a solid replacement.
Appendix: Additional groups information
Support
Advice
- We had nearly 100 calls with groups, and about 120 significant email/Slack exchanges. We received positive feedback on the calls (average likelihood to recommend >9/10).
- This included 37 calls with leaders at highly-ranked universities.
- At EA Global: Reconnect, we ran a workshop for group organizers which approximately 75 organizers attended, and which was highly rated. We also assisted in 1:1 connections and icebreakers for the organizers.
- Examples of group plans that probably wouldn’t have happened without our support:
- EA London School of Economics is planning to run a summer fellowship with the aim of finding future group leaders at LSE, Imperial College London, University College London, and King’s College London.
- EA Columbia identified a new group leader, and is organizing an additional Introductory Fellowship over the summer, which it wasn’t planning to do before.
- We encouraged the registration of EA groups at Stanford Law and Yale Law.
Resources
We’ve seen a gap between students completing an Introductory Fellowship and actually taking significant actions based on EA principles. As a result, we’ve supported / run the following activities:
- Career resources: We tested a career workshop with 4 highly-ranked university groups. The workshop was rated 8/10 for its usefulness in helping participants improve their existing plans.
- In-Depth Fellowship: We partnered with Max Daniel (FHI) to develop updates to the In-Depth Fellowship curriculum (to be piloted next quarter).
- Student projects: We provided funding to EA Brown and EA Harvard to run an intercollegiate projects program, SHIPs (Student-led High-Impact Projects), which has 30 students taking part.[3]
- Examples of projects students are working on include: a career guide on ending factory farming, a biosecurity infographics Instagram, and a post titled “Why is society concerned so much by illness and not, commensurately, by health?”
- Group model write-ups: James Aung coordinated organizers at CBG-funded university groups (Oxford, Cambridge, Harvard, Yale, Stanford, and Brown) to write up documents explaining how their groups work.[4]
Seeding new groups
- We worked with Yale EA to run a virtual Introductory Fellowship at Georgetown - one of the most important universities without any EA group. We sourced potential leads from local community members, 80,000 Hours, and CEA’s internal records. A professor we reached out to advertised it to his class, which resulted in around 5 new signups right before the deadline. 15 students at Georgetown are participating, and 10 have expressed interest in helping to grow the group, including the former student body president. 3 Introductory Fellowship participants have expressed interest in leading the group.
- We are now exploring using Stanford’s remote Introductory Fellowship to add capacity to groups at LSE, UCL, Imperial, and Chicago.
Funding
We continued to provide funding for group events and initiatives, at a similar level to last year.
Fellowships
Virtual programs
This quarter, Emma Abele, a contractor and EA Brown leader, launched EA Virtual Programs. Over 600 participants and 100 facilitators took part.
Group-run fellowships
- In addition, many EA university groups run Introductory Fellowships for students at their own universities. The number of groups running and adopting these fellowships has doubled (from 17 groups in Q4 to 35 groups in Q1).[5]
- We had informal chats with leaders running In-Depth Fellowships, which right now are operating fairly autonomously. We think most of our value add so far is providing input on various updates proposed by group leaders and Max Daniel (who is in the process of producing a new curriculum).
- Marie Davidsen Buhl joined as a contractor to provide advice to groups running Introductory Fellowships. In Q1, she provided 1:1 support to 15 groups.[6]
- This has allowed us to understand which resources are missing; we aim to spend more time filling these gaps next quarter (e.g. creating advertising materials and written guides for facilitators).
- We think most of the growth above is ‘organic / leader-driven’ growth, and most of our value in this space has come from improving the quality of the fellowships, rather than recruiting participants.
Community Building Grants
Community Building Grants (CBGs) allow group organizers to professionally engage in local community building activities.
Our main priority for this quarter was evaluating renewal applications for funding. Harri has decided to transition out of his role as Community Building Grants specialist. Over the next quarter, he will be exploring other possible roles at CEA and externally, and will continue to maintain the program until we hire additional team members.
We previously expected to reopen new applications for funding in January. We decided against this due to low capacity. We made a mistake in not communicating our timeline for deciding whether to re-open applications before the decision had been made. We have now made a public commitment to provide an update about our plans by June 1st, in advance of the June 13th EA Infrastructure Fund round.
In Q1, we granted ~$420,000, which was broadly on track with our expectations.
Renewal city / national grants made:
- EA Sweden, 2 FTE for 1 year (an increase of 0.7 FTE)
- EA Norway, 2 FTE for 1 year
- EA London, 1 FTE for 1 year
- EA France, 1.2 FTE for 1 year
New grants made:
- EA Netherlands, 0.8 FTE for 1 year
- EA Oxford, 1.9 FTE for 1 year
- EA Brown, 0.5 FTE for 3.5 months (spring term)[7]
- EA Germany, 1 FTE for 1 year
Group health cases
26 independent group-related community health issues were raised, and we spent significant time on four.[8]
Categories of cases (some of which fit into more than one category):
- 11 organizers who needed to develop greater skills or knowledge before successfully running a group
- 9 potentially reckless early field-building activities
- 7 conflicts between members of a group
- A few situations involving organizers with personal difficulties
The Introductory Fellowship (which will soon be referred to as the Introductory Program) is an eight-week program meant to introduce new group members to some of the core ideas in effective altruism. Dozens of groups have run this program for hundreds of people, but the content hadn’t been available in a publicly accessible format until now. ↩︎
People from Black and Hispanic/Latinx backgrounds are especially underrepresented in the community. We are prioritizing engagement of people from these backgrounds, based on the hypothesis that they may feel particularly isolated. We typically mean people who are living in places where they are in the minority, such as Black and Hispanic/Latinx people in the US or UK. ↩︎
This was proposed and driven by student leaders at Brown and Harvard. It might have existed without CEA funding, but we think funding helped organizers have enough time to actually run the project. ↩︎
Topics include: group history, how boards/committees are structured, current activities they run, open strategic questions, etc. ↩︎
As some students leave fellowships before finishing, and fellowships are run independently through groups, our estimates of the number of fellowships in Q4 and participants across Q1 / Q4 are somewhat uncertain. ↩︎
7 support calls, 2 meetings through EA Global, and 6 longer email correspondences. ↩︎
Brown is not one of our focus universities, but we believe that many of our focus university groups can learn from the Brown group’s innovative practices. ↩︎
By “significant time”, we mean interfacing closely with the Community Health team and holding at least one in-depth call with the organizer(s) involved. ↩︎
Peter_Hurford @ 2021-04-22T07:01 (+29)
This is really cool to see! Thanks for sharing this level of detail and transparency from one of the most important EA organizations!
BrianTan @ 2021-04-22T07:04 (+21)
This update and CEA's plans for 2021 mention the term "highly-ranked university groups" and "focus universities" a lot. Could you clarify what you mean by one or both of those terms? (i.e. are you looking at the top 40 universities globally based on a specific website)? Thanks!
MaxDalton @ 2021-04-23T07:58 (+8)
These terms are generally referring to 19 university groups which we give some additional support (e.g. we offer extra 1:1 calls with them, and we pilot some programs with them). This is on top of the support we offer all groups (e.g. online resources, funding for events, 1:1 calls, advice over Slack/email).
The groups are chosen primarily based on the university’s track record of having highly influential graduates (e.g. Nobel prize winners, politicians, major philanthropists). We also place some weight on university rankings, universities in regions with rapidly-growing global influence, the group’s track record, and leader quality.
Current focus university groups in no particular order: Harvard, Swarthmore, Oxford, London School of Economics (LSE), Cambridge, Georgetown, Stanford, Hong Kong University, Yale, Princeton, MIT, Caltech, Berkeley, University of Chicago, Columbia, Penn.
BrianTan @ 2021-04-23T10:16 (+2)
Got it! You listed only 16 universities though, but you mentioned you're referring to 19 groups. Do some universities have more than one group (i.e. Harvard and Harvard Law?)
MaxDalton @ 2021-04-23T12:46 (+4)
Yes, that's right.
Khorton @ 2021-04-22T12:56 (+13)
"We have ... improved our cybersecurity, and streamlined a number of HR systems."
Hurray! Well done
MaxDalton @ 2021-04-23T16:26 (+2)
Thanks - I'll pass this on to the people involved!
Habryka @ 2021-04-22T07:09 (+13)
Thank you for posting this!
Harry_Taussig @ 2021-04-23T21:24 (+12)
Are you able to reveal who this YouTube creator is? I'm surprised by how little EA YouTube content there is aside from recorded talks. I feel like an EA-related Veritasium or Kurzgesagt could be super helpful and popular as per this post.
sky @ 2021-04-26T17:23 (+2)
Thanks for asking. I’m not able to say more at this point about that specific creator.
I think you’re asking a good, implied question I share though: which comms channels would be most promising, for creating or sharing additional EA content?
I’m interested in analysis of those sorts of questions, and see them as part of the strategic comms role we’re hoping to hire for this year.
(I work at CEA).
james_aung @ 2021-04-22T17:20 (+11)
Would you be able to provide a Net Promoter Score analysis of your Likelihood to Recommend metrics? I find NPS yields different, interesting information from an averaged LTR and should be very straightforward to compute.
MaxDalton @ 2021-04-23T16:22 (+15)
Sure! I've asked the relevant people to respond with the NPS figures if it's quick/easy for them to do so, but they might prioritize other things.
Btw, I disagree about how useful NPS is. I think it's quite a weird metric (with very strong threshold effects between 6/7 and 8/9, and no discrimination between a 6 and a 1). That's why we switched to the mean. I do think that looking at a histogram is often useful though- in most cases the mean doesn't give you a strong sense of the distribution.
james_aung @ 2021-04-24T18:02 (+9)
Thanks! I guess I think NPS is useful precisely because of those threshold effects, but agree not sure that it handles the discrimination between 6 and 1 well. Histograms seem great!
MaxDalton @ 2021-04-25T10:00 (+17)
Hmm, I still think the threshold effects are kinda weird, and so NPS shouldn't be the main measure. (I know you're just asking for it as supplementary info, and I think we'd maybe both prefer mean + histogram.)
There's a prima facie case, that's like: the threshold effects say that you care totally about the 6/7 and 8/9 boundaries, and not-at-all about the 5/6, 7/8, 9/10 boundaries. That's weird!
I could imagine a view that's like "it's really important to have enthusiastic promoters because they help spread the word about your product" or something, but then why would that view want you to care not-at-all about the 9/10 boundary? I imagine 10s are more enthusiastic promoters, and it seems plausible to me that the 9/10 differential is the same or greater than the 8/9 differential.
And why would it want you to care not-at-all about the 7/8 boundary? I imagine 8s could be enthusiastic promoters, more so than 7s.
And similar comments for a view that's like "it's really important to avoid having detractors, because they put people off".
I could also imagine a kinda startup-y view that's like "it's really important to get excellent product market fit, which means focusing on getting some people to really love your product, rather than a large group of people to like it". But on that view, why ignore the 9/10 boundary? And why care about detractors?
I also think that maybe all of the above views make more sense when your aim is to predict whether your product will grow virally (not our focus), vs. whether it's generally high quality/providing something that people want (more our focus). So they might just not carry over well to our case.
james_aung @ 2021-04-25T20:43 (+5)
Thanks for explaining your view! I don’t really have super strong views here, so don’t want to labour the point, but just thought I’d share my intuition for where I’m coming from. For me it makes sense to have a thresholds at the places because it does actually carve up the buckets of reactions better than the linear scale suggests.
For example, some people feel weird rating something really low and so they “express dislike” by rating it 6/10. So to me the lowest scorers and the 6/10ers are actually probably have more similar experiences than their linear score suggests. I claim this is driven by weird habits/something psychological of how people are used to rating things.
I think there’s a similar thing at the 7/8/9 distinction. I think when people think something is “okay” they just rate it 7/10. But when someone is actually impressed by something they rate it 9/10, which is only 2 points more but actually captures a quite different sentiment. From experience also I’ve noticed some people use 9/10 in place of 10/10 because they just never give anything 10/10 (e.g they understand what it means for something to be 10/10 differently to others)
The short of it is that I claim people don’t seem to use the linear scale as an actual linear scale , and so it makes sense to normalise things with the thresholds, and I claim that the thresholds are at the right place mostly just from my (very limited) experience
MaxDalton @ 2021-04-26T05:26 (+2)
Thanks for explaining! The guess about how people use the scale seems pretty plausible to me.
MaxDalton @ 2021-04-23T16:24 (+5)
EA Global: Reconnect NPS was 20%
MaxDalton @ 2021-04-26T05:53 (+2)
For groups support calls, one staff member's NPS was 83% and another's was 55%. (They were talking to different user groups, which probably explains some of the discrepancy.)
David_Kristoffersson @ 2021-04-22T12:15 (+7)
Thanks for posting this. I find it quite useful to get an overview of how the EA community is being managed and developed.
BrianTan @ 2021-04-22T07:03 (+7)
Thanks for writing and publishing this! Lots of exciting progress. I have some questions, which I'll separate into different comments:
Group model write-ups: James Aung coordinated organizers at CBG-funded university groups (Oxford, Cambridge, Harvard, Yale, Stanford, and Brown) to write up documents explaining how their groups work.
Is it possible to get links to these documents? I and other student chapter leaders in EA Philippines would be interested to read them, and I think other student group organizers would be interested too. In particular, we think a lot about what the best organizational structure is for uni groups, and what strategies to use for a) goal-setting, b) deciding what projects to run, and c) dividing roles and responsibilities.
james_aung @ 2021-04-22T09:32 (+5)
Hey Brian. I'd have to ask the individuals who wrote up their docs, but the plan is definitely to eventually share more of these type of group writeups widely. They weren't written with a broad audience in mind, but I feel like several leaders would be keen to share their writeups more publicly after cleaning them up a bit. I'll nudge people on this and ask if they're keen
BrianTan @ 2021-04-22T10:28 (+2)
Got it, thanks! If they could be compiled and put on the EA Hub Resources, such as on this page, that would probably be the best place to compile them?
BrianTan @ 2021-04-22T07:28 (+6)
On Fellowship Data:
- If a fellowship starts in Feb but ends in April, does that count toward Q1 data or Q2 data?
- Regarding the number of participants for fellowships, I'm not sure how that data is collected, but maybe Marie Buhl reached out directly privately to fellowship organizers to collect this data. Anyway, there's a good chance that data for EA Philippines's chapters aren't accurately included in Q1 data.
For example, 2 of our student chapters in EA Philippines, EA UP Diliman and EA Blue, are both running intro fellowships since Feb and March respectively. They have a combined 67 participants (36 and 31 respectively), though a few have already dropped out or are not set to graduate. I assume both universities are non-focus universities. So if our participant data is part of your Q1 data set, then that means 29% of fellows (67/230) from non-focus universities are from our universities. I think that's too high, so my assumption would be most or all of their fellows are not yet in the data set. - Related to #2, I think it would be useful to have a public Google Sheet with rows for which groups are running fellowships, and columns for what country and university they're from, when they've started, when they will end, how many participants, and how many graduates (if data exists on that already). I assume most fellowship organizers would be okay with this data being public. I think having this public Google Sheet can let us easily know which groups are and aren't on the list yet. I think it's also good for others to know which universities/cities have fellowships - maybe they can recommend friends from those universities/cities to join those fellowships.
- I think it would be good to include data for both participants and graduates. Would be interesting to know what the avg. drop-off rate is for these fellowships.
- Also, a separate Google Sheet that lists which groups are going to run fellowships or reading groups in the next 1-3 months (and whether they accept people virtually or not) could also help people interested to know what fellowships or reading groups are coming up. But I think this is less important to do than the other things I suggested or mentioned above.
JoanGass @ 2021-04-23T17:11 (+4)
Hi Brian,
Great to hear about your enthusiasm for fellowships!
- Q1 data - we count fellowships based on when they start
- We collect data on this at the end of each fellowship, so the non-EAVP participant numbers in the report are guesses based on Marie’s conversations with group leaders. For groups Huw was not in touch with on a regular basis, Marie assumed an average of around 12 participants/group, so it’s possible that the number is higher based on EA Philippines numbers (although a number of fellowships are smaller than 12 participants).
- Marie is planning to make a spreadsheet like this from next quarter and will post it on the EA Groups Slack.
- We’re collecting data on both starting and finishing numbers of participants at the end of each fellowship.
- Marie is planning to include future fellowships in the spreadsheet. Adding a tab for other reading groups seems like a good idea.
Feel free to reach out to Marie directly on the EA Groups slack if you’d like to discuss more
BrianTan @ 2021-04-24T12:51 (+4)
On #1 and 2: Got it! I guess CEA should be more cautious (i.e. by putting significant caveats or not reporting the data yet) then about reporting participant data for non-EA Virtual Programs participants, since you collect data at the end of fellowships, and the data are just guesses before then.
On #3-5: Great!
JoanGass @ 2021-04-26T11:34 (+3)
On #1 and #2: In our report in footnote 5 where we reported this data we said: "As some students leave fellowships before finishing, and fellowships are run independently through groups, our estimates of the number of fellowships in Q4 and participants across Q1 / Q4 are somewhat uncertain."
I do think the benefits of reporting estimates are more valuable than only reporting precise information, but we do try to add additional detail about where the uncertainty comes from. I'll keep this comment in mind when we do our Q2 report as well.
BrianTan @ 2021-04-26T15:03 (+2)
Yeah estimates are probably better than nothing. Maybe making the caveat/uncertainties about the data more easily seen, i.e. as asterisks beside some numbers on the fellowship data table, rather than as a footnote might help. But yeah it's a minor thing!
Adam Binks @ 2021-04-25T23:22 (+3)
Thanks for this, really interesting! I am surprised that the total attendance of fellowships isn't even higher - do you have a feel for whether they're typically constrained by mentors or signups? In my experience helping run fellowships, many people are surprisingly interested but haven't heard about EA, have you looked at ways to reach more of these people?
JoanGass @ 2021-04-26T15:36 (+2)
For the large EA Virtual Program round, at first we were worried about having enough facilitators. But then we actually had quite a number of volunteer facilitators (over 100!) so then we focused on getting more participants. In the end we ended up having participant demand that matched available facilitators. As we mentioned, we're working to build more operations capacity for the virtual programs version of our fellowship. Once we do this, we hope to be able to offer them on a more consistent basis so more people can sign up.