Does participation in Effective Altruism predict success when applying to CEA?

By Akara, Ben_West🔸 @ 2022-09-27T18:34 (+29)

Summary

• We analysed recruitment data for 8 roles open in 2021 and 2022, covering 492 applicants. 
• Those who self-reported that they attended EA Global or read an EA book were somewhat more likely to advance in our recruitment processes.
• However this relationship is noisy, and seems to recommend against hiring managers filtering strongly on self-reported engagement. Similarly, it suggests that people with some but limited EA engagement are not significantly disadvantaged in our hiring process.

Introduction

In CEA’s application form, we include a multiple choice question to ask applications about the extent to which they participate in effective altruism (EA). We are interested in whether any of these factors are good predictors of applicants progressing in the recruitment process.

We analysed recruitment data for 8 roles open in 2021 and 2022, covering 492 applicants. This post outlines these findings.

In summary:

• When applicants participate in a larger quantity of EA activities, there is a slightly higher association with progression in the recruitment process, which is statistically significant (p<0.001)
• When applicants have not participated in any of the named EA activities, they are significantly less likely to progress through any of the stages (p<0.001).
• Applicants who passed an initial sift were more likely to;
  • have read Doing Good Better, The Most Good You Can Do, or The Precipice,
  • or have attended EA Global or another EA retreat.
  • The associated odds ratios were 2.80 (p<0.001) and 2.52 (p<0.01) respectively.
• Applicants who passed the screening interview and trial task were more likely to have attended EA Global or other EA retreat. The associated odds ratios were 2.56 (p<0.05) for the screening interview and 2.26 (p<0.05) for the trial task.
• Surprisingly, recieving coaching from 80,000 Hours was not associated with significantly increased odds of advancement.

Context

We looked at applications to the 8 roles listed in this footnote^[1]. Applicants to the roles uploaded a CV and answered some basic questions. One question asked about applicants’ participation in EA; they could select more than one option from the choices provided, which are detailed in this footnote^[2]. 

We have removed some records from this analysis:

• Some roles were open as an EOI for a short period, before being advertised fully as an open role^[3]. The EOI form is different to the open role form; applicants to the EOI were not asked this multiple choice question about their participation in EA, so they are not included in this analysis.
• We have also removed candidates whose applications were withdrawn soon after applying.

Base rates for success

All applications first undergo a sift, in which the hiring manager chooses which applicants to invite to try further stages in the recruitment process.

In four of the roles the first stage was a screening interview and the second stage a trial task. In the other four roles these were reversed; candidates first attempted a trial task and then may have been invited to a screening interview.

When calculating base rates, we use as a denominator the number of people who participated in that stage.

Findings

We fitted logistic regression models to the data; with the dependent variable being whether a candidate passed a given stage, and the independent variables being their participation in each of the options outlined above.

We then calculated modelled odds ratios and probabilities associated with each "predictor". The results of this are shown below, with a data table in the appendix.

Number of ways of participating

We plot the number of ways in which applicants participate in EA against their outcome (1 for pass and 0 for not passing).

A logistic regression model suggests for each additional way in which applicants participate in EA, the odds ratio for success associated with each marginal way of participating in EA is:

• OR 1.21 for passing screening interview (p<0.001)
• OR 1.17 for passing the trial task (p<0.001)

Predictors for passing an initial sift

This model predicts whether all submitted applicants (N=492) would pass an initial sift and be invited to a stage in the recruitment process, with sensitivity of 68% and specificity of 67%.

• Applicants were more likely to be invited to screening interview if they had;
  • read Doing Good Better, The Most Good You Can Do, or The Precipice (OR 2.80, p<0.001)
  • attended an EA Global conference or other EA retreat (OR 2.52 p<0.01)
• Applicants were less likely to be invited to screening interview if they did not take part in any of these factors,  as labelled by the “intercept” (OR 0.16, p<0.001)

Predictors for passing screening interview

This model predicts whether all applicants who did not withdraw (N=492) would pass the screening interview. We count applicants who did not reach the screening interview as having not passed the interview either. This model has sensitivity of 79% and specificity of 66%.

• Applicants were more likely to pass the screening interview if they had attended EA Global or an EA summit retreat (OR 2.56, p<0.05)
• Applicants were less likely to pass the screening interview if:
  • they did not take part in any of these factors,  as labelled by the “intercept” (OR 0.13, p<0.001). The majority of this selective effect takes place before candidates are invited to the screening interview.
  • they said they had chosen a job or degree program on the basis of EA principles (OR 0.43, p<0.05).

Other predictors were not statistically significant.

Predictors for passing trial task

This model predicts whether all applicants who did not withdraw (N=492) would pass the trial task. We count applicants rejected at screening interviews as not passing the trial task either. The model has sensitivity of 84% and specificity of 59%.

• Applicants were more likely to pass the trial task if they had attended EA Global or a similar EA retreat (OR 2.26, p<0.05).
• Applicants were less likely to pass the trial task if
  • they did not take part in any of these factors,  as labelled by the “intercept” (OR 0.08, p<0.001).
  • they said they had chosen a job or degree program on the basis of EA principles (OR 0.39, p<0.01).

Commentary

By Ben West

• The base rates for success at trial task are imbalanced (with a low rate of success), which affects the models’ sensitivity and specificity. We would advise some caution when interpreting these results.
• The OR < 1 and lack of significance from receiving 80,000 Hours coaching is surprising: 80,000 Hours is somewhat selective, and I (Ben) would have expected it to be a relatively strong positive predictor (if only because hiring managers falsely believe that it’s a sign of the applicant’s ability and therefore are biased towards advancing candidates who received coaching). 
• The OR < 1 for choosing a job or degree program on the basis of EA principles fits with my anecdotal experience: people who aren't very involved in EA are likely to tick that box, whereas more highly engaged people have a higher threshold for "on the basis of EA principles", so this ends up having the reverse correlation that we intended.
• I would be interested in a follow-up study which makes some of these metrics more precise. E.g. instead of "I read the effective altruism forum regularly" measure "I read the effective altruism forum for more than one hour per week". (Or better yet: actually measure the amount of time they spend on the forum without needing self-reported data.)

1. ^{^}

   We looked at applications to the following roles:
   - Community Liaison
   - EA Strategy Coordinator
   - EA Virtual Programs Operations Specialist
   - Executive Assistant and Groups Support
   - Full-Stack Engineer
   - Groups Associate (Scalable University Support)
   - Product Manager (EffectiveAltruism.org)
   - Project Coordinator

2. ^{^}

   The question is Participation in effective altruism: Do any of the following statements apply to you? These are examples of activities that some members of the effective community participate in. Feel free to add any others that may apply!
   Choose as many as you like:
   - I have used the 80,000 Hours career guide
   - I have received 80,000 Hours 1-on-1 coaching
   - I attended an EA Global conference or a similar EA summit retreat
   - I donate some percentage of my income to effective charities
   - I have donated or pledged to donate 10 percent of my annual income to EA causes
   - I have spent 4 hours per week on EA organizing or other EA projects
   - I have participated in an EA group
   - I have led an EA group
   - I have read Doing Good Better, The Most Good You Can Do, or The Precipice
   - I have chosen a job or degree program on the basis of EA principles
   - I read the Effective Altruism Forum regularly
    

3. ^{^}

   Expressions of Interest (EOIs) represent roles which CEA is interested in eventually hiring for, but are not actively focused on at present. The website signposts that these roles have a higher bar for progression, and that CEA carries out much less outreach for them, but aims to make applying as easy as possible.