What We’ve Learned from running an Impactful Policy Careers Workshop Series

By MvK🔸, Thijs Jacobs, Jana W.🔸 @ 2023-01-18T23:56 (+25)

Executive Summary (TL;DR)

In late 2022, we ran a workshop series on impactful careers in the European policy sector. Participants completed pre-work and attended live sessions to develop their career plans and get a better overview of their options. We built on a previous version of this which was run by Training for Good. Our preliminary evaluation suggests that this program had a substantial counterfactual outcome in changing the participants’ career plans and aspirations, had significant importance for building a EU Policy community within EA and we are very excited for this program to continue in the EA European policy space.

The following is a summary of what we did and what we learned from it.

We are writing this post hoping to increase the transparency of what we did and learnt, ideally to inspire others and reduce the time investment it would take for someone to run something similar in the future. We are also hoping to receive feedback and constructive criticism from the EA Community.

Looking Back, or: What We Did

The IPC Coalition (Impactful Policy Career) consists of Thijs Jacobs, Jana Wilke and Moritz von Knebel. We were brought together and supported by Jan-Willem van Putten, Co-Founder and Co-Director of Training for Good. After having worked with us each individually in different capacities, Jan-Willem expressed interest in handing over a project that Training for Good lacked the capacity to run due to some strategic shifts. Both Thijs and Jana had joined an earlier cohort of the IPC workshop series.

The materials we used were largely modeled after the previous version. This gave us an enormous head start and allowed us to focus on running the actual program with relatively limited effort rather than curriculum development.

Each of us was supposed to spend the equivalent of 5 hours per week on this, although in practice this was too low an estimate which we had to correct upwards in the majority of the weeks that we spent working on this. During this time, we were supported by a small grant of the EA Infrastructure Fund.

We announced the program on the EA Forum, several Facebook groups, via national community builders, a few Slack channels and it was also shared on LinkedIn. However, we noticed that there was a significantly higher chance of rejection for candidates who had heard of the program through LinkedIn. This was mostly because out of the 85 applications we received, we mostly had to reject people who didn’t have an adequate background for the program (i.e., there was only a small or no chance they could work in the European policy space in the next few years).

Subtracting a few dropouts due to personal circumstances, this gave us a cohort of ~70 participants, which might seem large for a workshop format. However, our goal was to cast a wide net, since this program is located at the early stages of the career funnel into an EA-related policy career. In a single case, we recommended an applicant to read up on core EA concepts, but overall we were positively surprised by the level of EA engagement in our applicants. (This might partly be a result of where we advertised for the workshop series.)^[1]

Content of the program

The content of the program was as follows:

• Week 1 - Introduction to Career Paths in Policy
  • Pre-work: Introduction to policy, general 80k-style career considerations, different policy roles and introduction Weighted Factor Model for evaluating career options.
  • Workshop: Introduction, general career considerations, explanation of Weighted factor model, feedback on pre-work.
• Week 2 - Personal Fit & Cause Area
  • Pre-work: personal fit and cause area considerations, country specific considerations, risks of doing harm in policy.
  • Workshop: Guest speaker on making impact in policy in an European country, introduction Theory of Change.
• Week 3 - Theory of Change for your Career in Policy
  • Pre-work: calculating impact, coming up with a Theory of Change.
  • Workshop: Introduction to Improving Institutional Decision Making (IIDM), reflecting on Theory of Change and a guest speaker on IIDM. 
• Week 4 - Q&A with EAs in Policy
  • Pre-work: prepare breakout rooms with experts
  • Workshop: EA’s with (2-10 years) experience in policy on an expert panel. Breakout rooms with 3-7 participants. Making a research or application plan.

The workshop lasted for two hours, with a rather diverse program including guest speakers, breakout rooms and explanation of key concepts. Partly the workshops overlapped with the pre-work to make sure that key concepts were well-understood. 

Outcomes & Theory of Change

By the end of the workshop, we expected participants to have:

• A better understanding of the paths to impact as an EA in policy making
• Knowledge about the different roles that influence policy
• An understanding of their personal fit for different policy roles
• Learned and applied one or more career decision-making tools
• Spoken to one or more EAs that are currently working in policy
• 3-5 career options that fit their personal criteria

Feedback and Impact Evaluation

[Disclaimer: The following is based on our experience running the workshop and the results of an exit survey, as well as some informal feedback we received via email. We acknowledge the susceptibility of these findings to various biases, such as selection bias (only participants who remained involved filled in the exit survey) or motivated reasoning (we all spent quite some time on this program and had an interest in its success.)]

• 27 participants responded to our review survey, which represents ~40% of the initial ~70 participants who signed up. Most of these respondents participated in all 4 workshops, we can therefore say that these have (expectedly) been the relatively active participants. 
• These 27 respondents reported spending an average of 2.8 hours weekly on the pre-work, plus 2 hours in the live sessions. 
• 14 people indicated short term changes in their career plans. The number of job applications our participants considered increased from 1.3 (an average we are taking from the application form) to 2.1. 16 respondents indicated the workshop changed their long-term career plans or the way they think about it. 
• We are hoping to be able to keep track of these developments by running a background search on peoples’ LinkedIn profiles ~6 months after the program and conduct interviews with some participants to understand attribution of career changes to the program, to give us a better idea of our counterfactual impact. 
• Satisfaction with the workshop was overall very high, and participants reported a high likelihood (average of >8 out of 10, similar to a Net Promoter Score) of recommending this program to people who are interested in having an impactful career in the policy sector.

The following results also indicated the program was successful in reducing participants’ uncertainties, increasing their awareness and confidence in choosing their career path. 

In addition to these quantitative metrics, participants were asked to elaborate on which elements they found particularly useful, and which ones they didn’t. Participants overwhelmingly praised the guest speakers, and emphasized the crucial role they played in providing an insight perspective. Participants were divided on some components of the program (like developing a Theory of Change and comparing the impact of different options), although we are unsure how much of this is a result of a lack of time, since the workshop series lasted only 4 weeks. Relatedly, it might have been better to communicate expectations ahead of the workshop more clearly, although respondents reported very high satisfaction rates with the communication around the program.

Another learning was that there was significant overlap between the pre-work and the live session, which may have caused less engagement and motivation, and could have been replaced by more time in breakout rooms. This would also work to foster the community-building and networking part of the program, and it’s a hypothesis we hope to validate by running a slightly altered version in the future that accounts for these considerations. An additional mechanism by which this might be achieved would be to create a “buddy” or “partner system” for the pre-work, which would also result in more personal connection and collaboration, which appears to be crucial in the policy space, and which seemed to bottleneck some of our participants.

Finally, one participant remarked on the lack of diversity with regards to guest speakers and participants. While this is partly a result of the strong geographical focus we have deliberately chosen to set, we hope to broaden our database of experts and guest speakers to include a bigger variety of perspectives in the future. (Send us an email if someone comes to mind!)

Call to Action (Expressions of Interest)

As previously indicated, our tentative impact evaluation suggests that this program was a valuable use of our time and energy, and of our participants time, too. Part of this is due to the high neglectedness, since we currently see no alternative or competing program in this space that could step in to fill this gap. In addition to some conversations we’ve had with other actors in this space, we are currently gauging interest for future iterations of this program. After our deadline closed, we invited people to submit their interest in becoming part of a mailing list where we’d announce a new program, you can still do so here.

The core team is pretty excited about running the program again, but we are even more excited about other people running it, given our commitment to other EA projects. We also believe running this program is a great way to explore your fit for community-building and entrepreneurship.

That’s why we’ve set up this form for you to express your interest in running this program in the future. We expect a range of time commitments to be possible, and we’re particularly excited about people who want to take our idea and run with it in some other part of the world. Our best guess is that this program could be run 2-3 times per year. We offer our support, mentorship and guidance, as well as a bunch of materials you can use.

(Finally, thank you to the EAIF for supporting our work with a small grant, which gave us more confidence in the value of the program. We’d also like to thank Jan-Willem van Putten for his ongoing support and mentorship and Sam Bogerd for his advice and assistance. Most importantly, we want to give a massive shoutout to our guest speakers who sacrificed their Monday evenings to speak to upcoming EAs who are passionate about pursuing a career in policy and who really appreciated your involvement. We couldn’t have done this without you!)

1. ^{^}

   We remain uncertain about the tradeoffs of having such a large cohort. We are moderately certain (~60-70%) that having a bigger cohort didn’t meaningfully reduce the quality or increase our workload, but that our inability to predict who would most benefit from the program is a strong argument in favor of having a more liberal admission policy. We did notice an unusually high amount of dropouts over the course of the program.