UGAP Starter Program Retrospective

By jessica_mccurdy, Jesse Rothman, Joris P, Jake McKinnon, Uni Groups Team @ 2022-10-24T15:55 (+52)

TLDR

• The CEA uni groups team piloted running a starter program before the University Group Accelerator Program (UGAP) to:
  • Provide resources to more groups.
  • Gain more context on groups to see whether UGAP was right for them.
  • Improve our mentor-group matching.
• We had 81 groups participate in the starter program from over 20 countries.
• We faced challenges due to rapid scaling, capacity constraints, and trying to accommodate different target audiences. We are pleased with the results and are glad we scaled quickly but have noted many improvements to be made going forward.
• We found that having the starter program separate from UGAP was confusing for organizers and that we would like both programs to have the same criteria for admissions. We also were able to better provide specific start-of-the-year resources to newer groups.
• Therefore, next semester we will integrate the starter program into UGAP as one single program. This program will likely be open only to new groups. We are looking into different programming for existing groups such as a continued mentorship network (which will be explained further in a future post)

This post is part of our efforts to communicate more transparently what we are doing. This is the first post in a series of posts and will serve as mostly an update about the starter program specifically. Future posts will dive further into our strategy, Metrics & Evaluation (M&E), and our future plans.

What was the UGAP Starter Program?

We had a virtual two-week^[1] starter program that was open to a larger group of schools than UGAP itself. It provided groups with virtual training and resources to prepare for the semester. This was our first time running this program and 81 universities participated, of which 56 were accepted to UGAP.

Groups were given a starter program guide that pointed to specific resources, attended a kickoff session focused on strategy and getting ready for the semester^[2], and had a meeting with one of our starter program mentors where they discussed a plan for the coming semester. Participants who were preparing to facilitate fellowships were able to participate in facilitator training through EA Virtual Programs. We also offered two optional workshops, one on outreach and another on reasoning.

During the program, individuals from groups were invited to apply for UGAP where they could receive continued mentorship and a stipend. Ultimately, 56 groups ended up having at least one organizer participate in UGAP (which is going on now).

Why the starter program?

Overall, the starter program itself aimed to provide the following:

• Help advise and steer group strategy for new groups
  • We think that group strategy is one of the highest leverage ways to make a big difference in group outcomes. We go over the core of our current advice in this post.
• Increase motivation and excitement about running a group
  • We focused on why we believe community building at universities can be so impactful, tried to create the sense of being part of a much larger international project, and expressed encouragement and gratitude to organizers.
• Make sure groups were ready for the start of the semester
  • We think the start of the semester is one of, if not the most important time of the year for university groups. We wanted to make sure that groups were going into the semester with a plan and the resources they needed.

Running the starter program separate from UGAP was an experiment to do the following:

• Provide resources to a larger number of groups, unconstrained by UGAP mentor capacity.
  • Last summer we realized that there was a lot of interest in UGAP. While we only had ~30 groups participate in the Spring, our interest form had ~50 additional groups interested. We knew we would get even more applications than this and were unsure of how much mentor capacity we would have. We wanted to at least provide some of the resources to more groups.
  • Some of the interested groups were not brand new and we wanted to experiment with whether we could also have newer organizers from existing groups participate in the starter program.
• Help us to identify which groups were ready for and would get the most out of UGAP and improve our mentor-group matching after gaining more context on groups during the starter program.
  • Last semester we had some sub-optimal matching because we were matching groups with mentors based on limited information.^[3] We thought the starter program would give us a chance to gain more context on groups so as to get a sense of their readiness and match them better.

Results

Note: Since the starter program occurred before the semester started and most downstream metrics that we care about wouldn’t be accessible until later or the end of the semester, our feedback form mostly revolved around the user experience rather than outcomes. We will be running another survey at the end of the semester to focus more on outcomes.

UGAP is still ongoing so these results only reflect the starter program.

Overall

We had 81 groups participate in the starter program with generally good feedback about the program:

LTR = 8.63

(Purple is cut off but is “Strongly Agree”)

Overall, we think these results are decent but not great. We think this indicates that this program is generally helpful on these dimensions but has the potential to be more so.

Specific Aspects

More details are listed in Appendix A but some takeaways for specific aspects were:

Top 3 most helpful aspects:

• Mentor meetings were very popular and mentioned the most times as being the most helpful aspect of the program (mentioned by 18 organizers in free response)
• The resource guide was comprehensive and helpful (mentioned by 16 organizers in free response)
• The outreach workshop and materials were also popular (mentioned by 7 organizers in free response)

Top 3 least helpful aspects (pulled from multiple questions):

• Kickoff was repetitive (especially for more experienced organizers)
• Unclear communication around timings, dates, and general processes
• Some physical materials that were sent arrived late and/or were low quality

What we learned

• We had hoped to provide resources to a larger number of groups, unconstrained by UGAP mentor capacity.
  • We ended up not being bottlenecked by mentor capacity as expected.
  • We also found it difficult to provide useful common programming for groups in different stages of development.
    • We created UGAP originally because we have a decent model of how to start up a university group. We have some generalizable advice and lessons learned from our experience here. So, we were able to scale that advice via a program.
    • We are much less sure about what the ideal model of a more advanced group is and would like to see more experimentation. Therefore we have less generalizable advice for these groups,^[4] and so the general resources were less useful for these groups.
  • We think we are best suited to provide the type of support in the starter program to new groups and do not expect the number of new groups to exceed our mentor capacity. Therefore, we don’t expect the need to have a separate starter program in the future. We are hoping to provide a separate product to more experienced groups in the future.
• We had hoped that the starter program would help us to identify which groups were ready for and would get the most out of UGAP.
  • We realized that it made more sense to have higher expectations regarding EA knowledge and community building strategy of starter program participants than we initially believed. We now think the bar for receiving resources via the starter program should have been more similar to our bar for being ready for the full UGAP.
  • We think the starter program may have been too encouraging to a few groups who were not ready to start (i.e. because their main organizers needed to spend some more time learning about and engaging with EA before starting a group).^[5]

What went wrong

• Trying to do screening while providing resources and mentorship to groups was suboptimal.
  • We were not able to give as much support to groups and made some organizers feel more like they were in interviews than mentorship calls. This was confusing and unclear to a number of organizers.
• We underestimated the toll on capacity we would have from events and onboarding new team members.
  • We knew these would take a toll on our capacity but they took even more than we expected. This led us to miss some deadlines, have suboptimal communication with participants, and underdeliver on some resources.
  • We tried to scale really quickly while onboarding two new team members. While the timing wasn’t great, I don’t currently think that we scaled too quickly. I think it is a lot easier to notice mistakes than missed impact and my guess is a smaller program would have been cleaner and more put together but would have left impact on the table.

We have a longer list of mistakes we made and things we are hoping to do differently in Appendix B

Looking forward

As mentioned at the start of this post, we are planning on releasing a series of posts over the next few months detailing our future plans. Our next post will include what else the Uni Groups team will be doing and what other types of support we will be providing. Specifically related to UGAP and the starter program, we intend to run a narrower version of UGAP that is open to only new groups, which will include some components of the starter program. We do not intend to run the starter program as an independent offering next semester.

• This will allow us to keep providing a more tailored program with product-market fit.  
• We think it is a useful and important service to keep making it easy for new university groups to start (or restart) which we will cover more in a future post.

Appendix

Appendix A: Additional data from feedback survey

(Free response) What was the most helpful element of the starter program?

Summary

• Mentor Meeting 18
• Resources (general) 16
• Resources (outreach) 7
• Meeting others 4
• Strategy 2
• Facilitator training 2
• Slack 1
• Facilitator guide 1
• Outreach workshop 1
• Reasoning workshop 1
• Kickoff 1
• Semester planning 1
• Website template 1
• Psychological stuff? (motivation, accountability, inspiration) 1
• Structure 1

(Free response) What was the least helpful element of the starter program?

Summary

• Kickoff meeting 5
• Asana 4
• Reasoning workshop 4^[6]
• Mentor Meeting 3
• Communication 3
• Community bonding 1
• One meeting with a mentor was not enough 1
• Could have been good to have more iterations of the workshops run 1
• Perhaps the comprehensive-seemingness of it made us less likely to experiment (we had already run one intro fellowship) 1
• Notion 1
• Website template 1
• Decentralized information 1

(Free response) How did your group strategy change, if at all from the starter program?

Summary:

• 20 individuals listed strategy changes we were actively enthusiastic about
• These were usually shifting more towards the advice in this post
• 7 listed strategy changes which we think we are unsure about (but usually positive)
• These might include things like increased outreach for which our enthusiasm depends on context
• 1 listed a strategy change we were slightly concerned about.
• This was somewhat opposite to our advice in this post

(Free response) Was there anything else not on these lists that you found helpful/unhelpful?

Summary:

Helpful:

• Outreach workshop and templates 3
• Slack channel 2
• Facilitator guide  1
• Getting materials: 1
• Meeting other group organizers 1
• Funding 1
• Meeting someone at CEA 1

Unhelpful:

• Late physical materials: 3
• Poor communication: 3
• Kickoff session: 2
• Getting to know each other (Kickoff session): 1
• Too many resources (slack, asana, notion): 1

(Free Response) Any other feedback about the above resources or events?

Summary

• Unclear communication(timing, dates, process in general) 4
• Mentor-meeting felt like a job application/interview  2
• Things about basic EA ideas (in kickoff) were less useful/ kickoff was repetitive of readings 2
• Too much bonding in the training sessions 2
• Facilitator training was helpful 1
• Asana was more helpful last year 1
• The resources were nice 1
• Wants more discussions on concrete ideas regarding community building 1
• The readings were insightful 1
• Program recommended for new groups, not really for experienced ones 1
• It would be good to have leadership training 1
• Save-function on the form was not reliable 1
• The calendar was a useful template but was modified to show week by week 1
• The materials were late but glad to have them 1
• Too many different platforms to organize 1

(Free Response) What were your main takeaways from kickoff?

Summary:

• 19 individuals listed takeaways we were actively excited about
• 3 individuals listed their main takeaways as being negative
• i.e. because the presentation was too slow/repetitive or at an inconvenient time

Appendix B: List of additional mistakes and future improvements

This is mostly to be transparent about mistakes we made and plans to fix them. These are based on user feedback. We have additional internal improvements we are planning on making. We are planning improvements now and will have additional capacity to address them. However, we still anticipate being capacity constrained and will consider the program in perpetual beta.

1. ^{^}

    It ended up being longer than two weeks to finish all of the mentor calls since we had about 75 of them.

2. ^{^}

    All of the non-program specific resources can be found on the EA Groups Resource Centre

3. ^{^}

    Some mentors are stronger in helping make groups more organized and operationally healthy while others are stronger in helping guide overarching strategic decisions. Based on the groups’ strengths and weaknesses, we wanted to give them complimentary matches.

4. ^{^}

    One of the reasons we are thinking of focusing solely on mentorship for experienced groups in the near-future is because it is more easily customizable to different situations.

5. ^{^}

    Notably, we now think more strongly that suboptimal EA groups might not just miss out on value, but might actually be net-negative by turning off particularly promising individuals

6. ^{^}

    Our guess about the reasoning workshop was that it was too basic for most of the people who opted into it but would have been good for people newer to rationality techniques