Yale EA Virtual Fellowship Retrospective - Summer 2020

By jessica_mccurdy @ 2020-09-20T20:26 (+61)

Summer 2020 Virtual Fellowship Retrospective

**Update: In this post we recommend a selective fellowship using our selection criteria. Since then, we discovered that our selection process was not predictive of eventual engagement and have changed our recommendation. We still stand by the recommendations in the other sections of this post.

TLDR

During the Summer of 2020 I helped organize a virtual version of the Yale Fellowship with co-organizers from McGill, Northeastern, and Yale. Since McGill and Northeastern are new EA groups we hoped this could serve both as a recruitment mechanism and as practice for running their own Fellowships in the fall. We had unexpectedly good results with 87% of fellows giving 9/10 or 10/10 on the question “Would you recommend the Fellowship to a friend” (NPS - 87), an average attendance of 95% and 100%* completion. 5/6 Northeastern, 1/1 Yale, and 4/5 McGill fellows have joined their respective boards and one fellow from Northwestern is starting a Fellowship there as well. These are a lot better results than I normally get when I run the Fellowship at Yale** so I am writing this retrospective with my guesses on what went well.

My guesses are:

• This was COVID Summer - people desired social interaction and had a lot of time on their hands
• We have a selective process that only accepted those who were very excited and committed
• Only 16/80 Fellows were accepted after submitting an application and being interviewed. We spend a large amount of time on the selection process because we think it is really critical (see why)
• Fellows got to be in groups with a variety of other fellows and moderators
• We meet synchronously and had only 4 moderators (see why)
• Emphasis on the sensitivity of EA subjects and general tolerance for other ideas
• We hope this makes the group more welcoming and inclusive (see more)
• Focus on social community and getting to know each other at the start
• 1-1s where fellows gained information and resources

For Retention Specifically

• Fewer people were graduating
• We mentioned organizing for EA groups throughout the Fellowship including during interviews

*We had one person who was accepted end up deciding to not do the Fellowship. In this retrospective we focus entirely on the 15 Fellows who participated.

**Normally we have around 1 or 2 students drop out or miss more than 2 meetings and we usually only get around 2-3 fellows to join the board. In Spring 2020 (when we started calculating NPS) we got an NPS of 54.

Acknowledgements

This fellowship could not have been possible without my exceptional co-organizers Anna Mouland, Kaleem Ahmid, and Thomas Woodside. They are all great organizers and I was lucky to work with them. We also greatly appreciate help from Catherine Low in helping create the initial vision for the fellowship and helping to recruit organizers and advertise. The content of our fellowship is mostly based off of Joshua Monrad’s version of the Yale EA Fellowship. I am extremely grateful for everything he taught me about running Fellowships, moderating discussions, and creating inclusive welcoming environments.

Goals

This model of the fellowship had similar goals as most others but had the added goal of seeding new EA groups.

• We hoped new organizers would gain valuable experience running the Fellowship so that they could run their own in the Fall
• We hoped those organizers could recruit fellows as board members
• We hoped experienced organizers would be able to pass on useful lessons learned

Selectiveness

Edit: We are currently not recommending this process since we discovered it was not predictive of engagement.

We received 80 applications for 16 spots. The large number of applicants likely had to do with the fact that people were actively looking for things to do over the summer after many programs and internships were cancelled due to COVID. There is also the possibility that branding the Fellowship as the “Yale Effective Altruism Fellowship” made it more desirable to non-Yale students who wanted to include it on their resumes.

We would love to take as many students as possible but have found in previous years that by limiting it to a smaller group:

• Only the most excited applicants participate (less engaged fellows who have poor attendance or involvement can set unwanted norms)
• Fellows are incentivized show up and be actively engaged (since they know they are occupying a spot another person did not receive)
• We only need a few moderators that we are confident will be friendly, welcoming, and knowledgeable about EA
• Each Fellow can receive an appropriate amount of attention since organizers get to know each one individually
• We create a stronger sense of community amongst Fellows
• We don’t not strain our organizing capacity and can run the Fellowship more smoothly

Risks with Selectiveness

There is always a risk that someone might be permanently turned-off from EA if they are rejected from a Fellowship program. Additionally, being selective could have a negative image at schools where group organizations being selective is not a norm (unlike Yale). We take several steps to try and reduce risks.

1. During interviews we always start with this disclaimer:

We just want to let you know that this interview is not in any way to try and judge your competence or altruistic-ness or anything like that. But rather we are trying to figure out your personal fit within the Fellowship and how much we think you will get out of it.

We say this little disclaimer at the start of every interview because we don’t want you to think that we are saying it because of your interview performance or anything like that but we unfortunately did have a lot more applications than we can take. We would really love to take everyone and everyone who made it to interviews is really impressive but we only have the capacity to take 15 a semester.

However, we do run this fellowship every semester and people who re-apply get priority. Additionally, if the fellowship isn’t right for you we have a lot of other ways to get involved with the group and a lot of our members get involved through ways besides the Fellowship such as our intro workshop, speaker events, and socials.

Do you have any questions about that? Okay - now with that disclaimer out of the way we can get into the interview :)

“Note: Young people take interviews very seriously and may be a lot more nervous than you think they are. It’s incredibly important to be kind, patient, and encouraging. Intimidating them will be bad for them, for your group, and for the reputation of EA as a whole. Also, be mindful that people have different styles of thinking and talking, and it’s important not to be dismissive of those styles which are different than yours.” (From Joshua Monrad’s guide to running Fellowships)

2. Give rejected but good applicants a guaranteed spot in the next Fellowship as long as they re-apply.

3. Write personalized rejection emails encouraging the applicants to stay involved

Selection criteria

Edit: We are currently not recommending this process since we discovered it was not predictive of engagement.

When evaluating applicants for the Summer 2020 cohort, we rated them on the basis of a set of criteria that had also been outlined on our application:

• Altruism: Passionate about helping others
• Effectiveness: Ambitious in their altruism, with a drive to do as much good as they can. Potential to be aligned with the central tenets of EA.
• We used a question asking if they are driven to do as much good as they can rather than being satisfied with some good. Applicants often were confused by this question so we are changing it to ask applicants to compare two charities stressing that their ultimate choice is not important but rather their thought process when deciding.
• Potential Receptiveness: Excited to dedicate their career to doing good or to donate a significant portion of their income to charity. We could see them applying these ideas to their life in a high-impact way.
• We used responses to a question asking applicants to list possible future career paths as a way of gauging where they might lie in relation to EA priority paths. We were not looking for people to already be on priority paths but rather paths that seemed like they were relatively compatible with these paths.
• Open-mindedness: Open-minded and flexible, eager to update their beliefs in response to persuasive evidence.
• We asked people about something they recently changed their mind on and alsoused a sort of turing test by having people choose something they believed and then argue both sides of it.
• Enthusiasm: Willing and able to commit ~3-4 hours per week, took application seriously, came to interview well prepared (maybe having done some additional research and reading about EA), and excited
• Experience: New to EA and would learn new things from the fellowship
• Fit (Interviews only): How good a fit are they with the fellowship format? Will they be good in discussions?

The questions we used to judge these criteria can be found in our application

We rated the applicants from 1-5 on each of these criteria except for Experience which we ranked on a scale of 1-3 with 1 being most experienced. This method is based on the interviewing recommendations laid out in Thinking Fast and Slow.

Selection Process

Due to there being so many applicants we decided to automatically reject applicants from schools that had their own Fellowship programs and defer those to their own school.

• We only had one exception where we admitted an incoming first-year from Yale since they were particularly eager to get involved with the group early and then join the board in the fall. If schools have a way of advertising to incoming first-years and are doing a Summer Fellowship I would encourage recruiting some since they will be particularly excited to participate.

First Deliberation Meeting

We created a sheet where we blinded the names of the applicants to mitigate biases and conflicts of interest. In this sheet we made tabs for each organizer to score the applications based on the selection criteria above. These were separate tabs so that we were not influenced by each other’s scores.

We first went through a few applicants all together, reading their application and giving our own scores. Then we compared our scores to calibrate with each other. After we were sufficiently calibrated we had two organizers go over one half of all the applications and the other two go over the other half. We each ended up reading 40 applications.

After everyone had graded their portion, we ranked the applicants from highest to lowest mean score. We checked each organizer's average scores and compared between organizers grading the same applicants to make sure that we remained calibrated. We then proceeded to remove the least competitive applicants.

This meeting took 3.8 hours to complete. This was particularly long since we had so many applicants and a fairly long application. (Afterwards we decided to add a word limit to the application questions, and add a new question, “Why are you applying for this fellowship?, to aid in the scoring of the “Fit/enthusiasm” criterion.)

Interviews

We made a spreadsheet of organizer availability (with numbers assigned to each organizer rather than their name to mitigate the chance that applicants would choose familiar interviewers) and sent the sheet to 44 applicants to sign up for 30 minute interviews (many interviews ran between 30-45 minutes). Luckily we had a lot of time on our hands at the start of the summer and were able to do this.

For the interview, we asked applicants to read the Introduction to EA from CEA and asked them to “come prepared to share your reactions, disagreements or uncertainties.”

During the interview, we discussed the article and asked questions which we thought might help us rate them according to our selection criteria. You can read a sample interview here.

We had an experienced organizer join the first interviews of new organizers for calibration and feedback. After each interview the interviewer scored the applicants on the same scale and took notes before starting the next interview. We did this in a central google doc that had places to put reviews for each applicant

We strongly encourage groups to do interviews if they have the capacity, even if they don’t have that many applicants. For one, it gives groups a much better picture of the applicant pool and how much they would gain from the fellowship. Second, it sends a strong signal to prospective fellows that the fellowship is a serious thing with high standards.

Second deliberation meeting

In this meeting, given that we had 4 organizers, and 16 spaces available, each organizer picked their top 4 or 5 applicants that they interviewed. Organizers used their scores to inform this but not to determine it. We should have been better about making sure to take averages of organizers’ interview scores for calibration but it turned out okay.

We have had in the past where one organizer just gets many of the best applicants in their interviews. This sometimes happens and is why each applicant should be discussed before admitting or rejecting.

This only took about 1 hour this time around but has gone much longer in the past when we were really stuck between a few applicants for the last spot.

We then sent out acceptance emails and personalized rejection emails.

Other considerations on diversity and experience can be found in Joshua Monrad’s Fellowship guide. We highly recommend reading these.

Fellowship Structure

Overview

Fellows had weekly 75 minute discussions. Fellows were also strongly encouraged to sign up for 1-1s with at least one of the organizers. Each organizer made a calendly and we put them at the bottom of every email to make scheduling easier. There was a getting-to-know-you social and a concepts workshop at the start of the fellowship. In the middle of the fellowship we had an optional discussion on decision theory. At the end of the fellowship we had a final social.

Discussions

We structured our fellowship as a synchronous Zoom meeting of 15 fellows and 4 organizers. We would all first meet together and do announcements and a brief intro on the topic. This served as a convenient time to give disclaimers on sensitive topics, remind people to sign up for 1-1s, and to publicize any extra events.

After about 5 minutes of announcements we split Fellows randomly into four breakout rooms with one organizer in each. Every week we would start our breakout sessions checking in with each other. We would each go around saying a “happy” and a “crappy” from the week. This is a nice way to see how everyone is doing and create more of a sense of community.

Then, we had each fellow type in the chat a question they would like to discuss. Then Fellows could vote on which to focus on. For some weeks there were questions we as organizers wanted to make sure were discussed and we would just go over those first. This method gives the Fellows more control over the discussion and makes them feel more involved. However, this can lead to important questions not getting asked and unrelated questions possibly being upvoted.

At the end of the discussion we would all come back together in the main Zoom room. I would randomly call on someone from each group to either share something from their discussion or defer to another person in their group by saying “x had a really good point I will let them explain it”. This keeps fellows on their toes and paying attention because they might be called on but gives an easy out if the fellow does not want to answer. Meetings would usually end with podcast and/or book recommendations pertaining to topics that seemed to be of interest during group discussions, which was something that was appreciated by a few of the fellows.

You can find our readings directory here.

Benefits of this discussion structure

A benefit of this structure is that Fellows are never ‘stuck’ with one moderator or one group. Some groups of people are much quieter than other groups of people. Some Fellows tend to like certain organizer’s moderation style better than others or just get to experience more than one. Randomly mixing up the group each week gives variety and gets rid of the risk of a Fellow being stuck in a quiet group or with a bad moderator. (Our moderators did a good job but this a concern especially amongst those who are just learning how to lead discussions)

Another benefit of this structure is that it allows all of the fellows to get to interact with each other and know each other. We wanted to create a sense of community within our cohort which we thought would most effectively be done if everyone met at the same time. This does limit the audience of the Fellowship due to time-zones, however so that decision has to be weighed out.

1-1s

We opened 1-1s as a place where fellows could further discuss topics from the fellowship, ask questions about EA, find out how to get more involved in the global community, discuss career plans, and get resources and connections. It is also a time where organizers can get to know the Fellows better. Afterwards the organizer can send an email with helpful resources from places such as the EA Hub, EA forum, or various panlists and Facebook groups. Since Yale EA has been running Fellowships for several years now I was able to connect Fellows with Fellowship Alums who were now working in impactful careers. 14/15 Fellows signed up for at least one 1-1. We did not set a limit to the number of 1-1s a fellow could have. Seven Fellows had more than one 1-1 but none had more than four.

Optional Discussion on Decision Theory

In the middle of the fellowship we had an optional discussion with Mahendra Prasad, a PhD candidate at UC Berkeley. This discussion focused on decision theory and its implications on things such as voting and AI. We had fellows first watch this talk he gave at EAG and read this article. The discussion ended up mostly consisting of a short intro and Q&A.

Virtual Socials

We did two zoom socials that turned out better than I expected. Both had a similar structure:

• Before the meet-up I told everyone to pick an object from their house that had a story behind it they would be willing to share and bring it with them on the call
• We all logged onto zoom and broke into breakout groups - with the first group we did a happy and a crappy from our week and then "show and tell" with our objects.
• Then we came back to the full group and people who had particularly fun and interesting objects shared them with the whole group
• Then we went into different breakout groups and did two truths and a lie at the first social and found things they all had in common at the second social
• After that we came back together and people chose whether they wanted to go to an icebreaker room or stay in the zoom and play scribbl.io

Creating a good environment

Effective altruism includes a lot of sensitive topics. Not handling these topics in a delicate way can make EA feel unwelcoming to different groups of people. Additionally, it is important to note the privilege we have to be able to have these choices when it comes to things like donations and careers.

Note that people may be from countries you talk about in the global health and development portion of the Fellowship or come from cultures and backgrounds that heavily influence their approach to topics such as animal welfare. Careers, donations, and diets are all very personal topics that can be hard to discuss.

Additionally, note that all of these topics are extremely complex and this is just the beginning of us thinking about them.

Not addressing these topics sensitively can lead to Fellows dropping out and/or leaving with a negative image of EA. One year we had a Fellow drop out after another Fellow was insensitive during a giving exercise. Now we make sure to always have a moderator in break-out groups to make sure something like this does not happen again.

I highly recommend these notes on moderating

Oxford has also created an extremely helpful Fellowship Facilitator Guide and Discussion Norms Template

I also recommend checking out the disclaimers we always say and the intentional community building activities for certain weeks in the appendix.

Results & Feedback

I expect much of the value of this fellowship to come from the strengthening of the McGill and Northeastern groups and the number of Fellows who went on to join the board. 5/6 Northeastern, 1/1 Yale, and 4/5 McGill fellows joined their respective boards and as of a few weeks into the semester are still engaged. This is particularly impactful since it improves the sustainability of the two new groups.

We ran a pre-fellowship survey and a post-fellowship survey to measure changes in Fellows plans and opinions. Unfortunately, we realized when analyzing the results* that by asking so many questions we made it very difficult to achieve statistical significance with an N of 15. The only metric where we achieved statistical significance was for the question “To what extent do you agree that biosecurity is a cause that more people should be working towards improving?” Where more fellows chose that they “Strongly Agree” after the Fellowship.

However, we thought the graphs might still be helpful for people to see so we collated them into this results document of comparative data.

Some fellows gave us permission to share their written responses to the post-fellowship survey which we have added in this document which contains feedback and descriptions of impact.

• A big thanks to Thomas Woodside who analyzed these results much more thoroughly than we had in the past.

Future improvements

• Add an optional additional discussion time for people who want to keep the conversation going
• Have more socials and optional events throughout the fellowship (and plan these well in advance)
• Have different organizers be in charge of different meetings to distribute work and increase learning
• Have applicants add name pronunciations to their application so that we can say their name correctly in interviews and beyond
• Have groups be more artificially random (ie: with purely random groups you can end up having two people in the same group for multiple weeks in a row. In the future we would want to make sure that doesn’t happen too often)
• Include a non-profit in our giving exercise from outside of Africa
• Foster a more active group chat
• Have fewer questions on our surveys

Appendix: Moderator notes for specific weeks

1st week (Intro) -

• We dedicate time where each fellow introduces themselves with
• Name
• Pronouns (optional)
• What they do
• What they want to get out of the fellowship
• What they hope to contribute
• One thing that makes them happy

Note this takes a while but is important. It is important to try and build a foundation of community and take time for people to get to know each other

• We spend time at the start to have people brainstorm and flag discussion norms. We asked them what has worked well and poorly in discussions they have been a part of in the past and tried to implement those practices. That way Fellows felt more involved in the norm setting process.
• We explain that the purpose of the Fellowship is not to convince fellows of these ideas but to present them with arguments they might not be familiar with that might influence their decisions.

3rd week (Giving Exercise)

• Do not forget that these are real lives we are talking about and very complex issues
• Let’s acknowledge our privilege to even be able to do this exercise and not let that overshadow the actual needs of prospective recipients
• Please be considerate when discussing this in your groups. Some people have personal ties to these areas.
• We don’t use the phrase “Third World Countries” and instead use “Low and Middle Income Countries”
• Clarification: There are two distinct questions
1. What will end poverty?
2. What is the best thing I can do right now to alleviate poverty?

We do not think that these organizations could end poverty on their own. We know that ending poverty will require a lot of policy decisions in addition to philanthropic giving. However, right now we have some money and want to help the best we can.

4th week (Careers)

• A note on financial security: We are very lucky to be Uni students and not everyone can afford to think about their careers in this way. I think this is really important to acknowledge.
• A note on 80,000 hours: 80k has been wrong in the past, is not universally applicable, and is aimed at a particular audience. However, it can be extremely helpful when making tough career decisions.

5th week (Animal Welfare)

• Note - Some people are very uncomfortable with Peter Singer’s analogies. To be honest I am similarly wary and a bit uncomfortable comparing racism to specism. I think this is a good time in general to note the reason for this fellowship. We are not here to convince you of our moral convictions but rather we present you with influential works you may not have seen before and ask questions you may not have heard so that you can make your own decision on these topics
• I know that some people here are pretty unconvinced by this and I know it can sometimes feel uncomfortable to speak up when other people around you are convinced. But we respect everyone’s opinions here and want your voice to be heard so please don’t let that prevent you from sharing.
• You are always welcome to say “someone who thinks this might say”

8th week (Conclusion)

• At the end have people say one thing notable/memorable or something they were thankful for in the fellowship (Shoutouts to other fellows are particularly good)

David_Moss @ 2020-09-25T11:18 (+21)

Thanks for the post! This definitely isn't addressed at you specifically (I think this applies to all EA groups and orgs), so I hope this doesn't seem like unfairly singling you out over a very small part of your post, but I think EAs should stop calculating and reporting the 'NPS score' when they ask NPS or NPS-style questions. 

I assume you calculated the NPS score in the 'standard' way i.e. asking people “Would you recommend the Fellowship to a friend?” on a 0-10 or 1-10 scale, and subtracting the percentage of people who answered with a 6 or lower ("Detractors") from the percentage of people who answered with a 9 or 10 "Promoters"). The claim behind the NPS system is that people who give responses within these ranges are qualitatively different 'clusters' (and also the people responding with a 7-8 are also a distinct cluster "Passives" who basically don't matter and so who don't figure in the NPS scores at all) and that just subtracting the percentages of one cluster from another is the "easiest-to-understand, most effective summary of how a company [is] performing in this context."

Unfortunately, it does not seem to me that there's a sound empirical basis for analysing an NPS style scale in this way (and the company behind it are quite untransparent about this basis (see discussion here).  This way of analysing responses to a scale is pretty unusual and obscures most of the information about the distribution of responses, which it seems like it would be pretty easy for an EA audience to understand.  For example, it seems like it would be pretty easy to depict the distribution of responses, as we did in the EA Survey Community information post.

And it seems like calculating the mean and median response would also give a more informative, but equally easy to understand summary of performance on this measure (more so than the NPS score, which for example, completely ignores whether people respond with a 0 or a 6). This would also allow easy significance testing of the differences between events/groups.