Cost-effectiveness of operations management in high-impact organisations

By Vasco Grilo🔸 @ 2022-11-27T10:33 (+48)

Summary

Following up on the challenge to quantify the impact of 80,000 hours' top career paths introduced by Nuño Sempere, I have estimated the cost-effectiveness of operations management in high-impact organisations (OM), which arguably include 80,000 Hours’ top-recommended organisations.

The results for the mean cost-effectiveness of various metrics in bp/G$ in terms of existential risk reduction are summarised in the table below for my preferred method. I present all results with 3 digits, but I think their resilience is such that they only represent order of magnitude estimates (i.e. they may well be wrong by a factor of 10^0.5 = 3). 

The full results are in this Sheet, and the calculations in this Colab^[1].

Acknowledgements

Thanks to Abraham Rowe, Dan Hendrycks, Luke Freeman, Matt Lerner, Nuño Sempere, Sawyer Bernath, Stien van der Ploeg, and Tamay Besiroglu.

Methods

I estimated the cost-effectiveness^[2] from the product between:

• The cost-effectiveness of the high-impact organisations, which I assumed equal to that of the effective altruism community.
• The multiplier of OM, which I defined as the ratio between the cost-effectiveness of OM and the high-impact organisations.

This method assumes the cost-effectiveness distribution of the high-impact organisations is represented by the one theorised for the effective altruism community in the next section. Moreover, the cost-effectiveness estimates are only accurate to the extent that future opportunities are as valuable as recent ones.

The calculations are in this Colab^[1].

Cost-effectiveness of the effective altruism community

I calculated the cost-effectiveness of the effective altruism community from the mean cost-effectiveness weighted by cumulative spending between 1 January 2020 and 15 August 2022 of 4 cause areas:

• Global health and development.
• Longtermism and catastrophic risk prevention.
• Animal welfare.
• Effective altruism infrastructure.

These are the areas for which Tyler Maule collected data here^[3] (see EA Forum post here). I adjusted the 2020 and 2021 values for inflation using the calculator from in2013dollars.

I computed the cost-effectiveness of each area using 3 methods. All rely on distributions which are either truncated to the 99 % confidence interval^[4] (CI) or not truncated, in order to understand the effect of outliers. The parameters of the pre-truncation distributions, which are the final distributions for the non-truncation cases, are provided below.

Method 1

I defined the cost-effectiveness of longtermism and catastrophic risk prevention as a truncated lognormal distribution with pre-truncation 5th and 95th percentiles equal to 1 and 10 bp/G$ in terms of existential risk reduction. These are the lower and upper bounds proposed here by Linchuan Zhang.

I assumed the ratio between the cost-effectiveness of i) longtermism and catastrophic risk prevention and ii) global health and development to be a truncated lognormal distribution with pre-truncation 5th and 95th percentiles equal to 10 and 100. These are the lower and upper bounds guessed here by Benjamin Todd for the ratio between the cost-effectiveness of the Long-Term Future Fund (LTFF) and Global Health and Development Fund (search for “10-100x more cost-effective”).

I considered the ratio between the cost-effectiveness of i) animal welfare and ii) global health and development to be a truncated lognormal distribution with pre-truncation 5th and 95th percentiles equal to 270 μ and 211. I computed these multiplying:

• The 5th and 95th percentiles of 0.0436 and 34.1 k I estimated here for the ratio between the cost-effectiveness of corporate campaigns for chicken welfare and GiveWell’s Maximum Impact Fund^[5], which is now designated Top Charities Fund.
• A factor of 1.07 m to adjust the moral weight downwards, which I calculated from the reciprocal of the product between:
  • The mean moral weight of chickens relative to humans of 2.41 I obtained in the analysis mentioned just above.
  • The ratio of 389 between the number of neurons of humans and red junglefowls (similar to chickens), which I took from Wikipedia.

This adjustment is analogous to assuming the mean moral weight is directly proportional to the number of neurons.

I set the cost-effectiveness of effective altruism infrastructure to the mean cost-effectiveness weighted by cumulative spending between 1 January 2020 and 15 August 2022 of the other 3 areas.

Method 2

I obtained the cost-effectiveness of each area based on the 27 answers regarding the mean cost-effectiveness of the Effective Altruism Funds given in the EA talent needs survey - 2018. Such answers are in the table below, whose last column was calculated by me.

I defined the cost-effectiveness of longtermism and catastrophic risk prevention as in method 1.

For the other areas, I assumed a truncated lognormal distribution with pre-truncation 10th and 90th percentiles of the area relative to those of longtermism and catastrophic risk prevention based on the 10th and 90th percentiles in the table above:

• For global health and development, 6.25 % (= 1/16) and 22.3 % (= 63/283).
• For animal welfare, 18.8 % (= 3/16) and 37.8 % (= 107/283).
• For effective altruism infrastructure, 625 % (= 100/16) and 35.3 % (= 100/283).

Method 3

I defined the cost-effectiveness of each area from the mean between those of methods 1 and 2. My best guesses regard the truncation case of this method.

Multiplier of operations management in high-impact organisations

I defined the multiplier of OM as the median of the 11 distributions described in the table below, and also experimented with truncating to the 99 % CI the component distributions of each of them. I used the median with the intention of following Jaime Sevilla’s best guess on how to aggregate forecasts:

• The assumptions going into each of the estimates are apparently not mutually exclusive.
• There are outliers:
  • The interquartile range of the mean multipliers is 23.0 (= 30.0 - 6.99), which is the difference between the mean multiplier of the 9th and 5th estimates.
  • Consequently, the 4th and 8th estimates are outside of the Tukey’s fences, as their mean multipliers are higher than the 3rd quartile by more than 1.5 times the interquartile range (1.36 k > 136 > 64.5 = 30.0 + 1.5*23.0).
• However, these are not necessarily “poorly calibrated”^[6], so I opted not to exclude them.
• According to Jaime’s flowchart, the 3 choices above justify using the median^[7].

I obtained the distributions via asking i) 75 people working at 80,000 Hours’ top-recommended organisations (on October 30 and 31), and ii) 259 people in the Slack “EA Forecasting & Epistemics” (on November 2) for the multiplier of OM of their organisations and the effective altruism community. You can see here the list of emails I contacted, and the messages regarding i) and ii) (see “Emails” and “Slack message”, respectively).

I should emphasise the multiplier of OM may depend a lot on the organisation (e.g. its size, maturity, cause area, and what it understands as operations^[8]), specific position (e.g. seniority), and personal fit^[9]. Consequently, aggregating all estimates as I did has serious limitations.

The low number of estimates is evidence of:

• The difficulty and lack of quantification of the marginal cost-effectiveness of positions. I guess I would receive more estimates for the multiplier of OM if these could be readily obtained from internal data.
• Me having framed the questions poorly.

In addition, what OM refers to is somewhat unclear. Based on what 80,000 Hours describes here, I think it can refer to both operations more broadly, or to senior operations positions which are further down the career path.

I also thought about estimating the multiplier based on the number of vacancies and candidates for operations management and all positions, but decided not given their unclear relationship with value. As vacancies decrease and candidates increase for a given position, the difference between the factual and counterfactual decreases, but the value of the factual increases.

Results

The tables below have the results for the mean, 5th percentile, and 95th percentile of the multiplier of OM and cost-effectiveness metrics. This Sheet contains more results (see tab “TOC”).

Multiplier of operations management

Cost-effectiveness

Method 1

Without truncation

With truncation

Method 2

Without truncation

With truncation

Method 3

Without truncation

With truncation

Discussion

Multiplier of operations management

The mean of the multiplier of OM for the non-truncation and truncation cases are 4.55 and 4.53. I thought organisations would organise themselves such that the expected (marginal) cost-effectiveness would be similar for all positions, so I was somewhat surprised to get values 5 times as high as 1.

The p-values for the null hypothesis that the OM follows distributions with the same shape as the ones I obtained, but with a mean of 1, are 1.27 % and 1.16 % for the non-truncation and truncation cases^[15]. So one can be reasonably confident that the multiplier is higher than 1, but only if the 11 answers I got are representative of the effective altruism community, which is far from certain.

The mean multiplier of OM for the truncation case is 99.5 % the one for the non-truncation case. This means the outliers of each of the individual estimates practically do not affect the results.

Cost-effectiveness

For the truncation case, the mean cost-effectiveness metrics as a fraction of that of longtermism and catastrophic risk prevention for methods 1, 2 and 3 are:

• For global health and development, 3.96 %, 17.8 % and 10.9 %.
• For animal welfare, 49.3 %, 32.7 % and 41.0 %.
• For effective altruism infrastructure, 33.0 %, 1.29 and 81.0 %.
• For the effective altruism community, 33.0 %, 45.2 % and 39.1 %.
• For OM, 1.50, 2.05 and 1.77.

Consequently, for the truncation case:

• Longtermism and catastrophic risk prevention is the most effective area for methods 1 and 3, and the 2nd most effective behind effective altruism infrastructure for method 2.
• Global health and development is the least effective area for all methods.
• OM is more effective than all areas for all methods. Its mean cost-effectiveness is 1.77 times as high as that of longtermism and catastrophic risk prevention for method 3.

For the non-truncation case:

• Animal welfare is the most effective area for methods 1 and 3, and the 2nd least effective in front of global health and development for method 2.
• Global health and development remains the least effective area for all methods.
• OM remains more effective than all areas for all methods. Its mean cost-effectiveness is 3.83 times as high as that of longtermism and catastrophic risk prevention for method 3.

I believe the results for the truncation case are more accurate because it is hard to represent outliers well based on subjective 90 % CIs. For example, I think the cost-effectiveness of animal welfare for the non-truncation case is too heavy-tailed, with its mean being 9.98 k (= 41.1/0.00411) times its median. The heavy-tailedness of this same metric for the truncation case seems more reasonable, with its mean being 474 (= 1.95/0.00411) times its median.

The mean cost-effectiveness of OM for the truncation case as a fraction of that for the non-truncation case is:

• For method 1, 26.3 %.
• For method 2, 96.2 %.
• For method 3, 45.3 %.

This means the outliers have a material effect for methods 1 and 3, but not for 2.

The 5th percentile, median, and 95th percentile of the cost-effectiveness of OM for method 3 with truncation are 17.0 %, 58.1 % and 3.09 times the mean of 7.01 bp/G$. I expected the distribution to be more heavy-tailed, but I arguably had in mind the wider distribution of potential applicants instead of the narrower one of those selected for working in the positions.

Further work

Some potential avenues for further work are, in my descending order of importance:

• Determining and quantifying the effect of the major drivers of the multiplier of OM, such as the size, maturity, and cause area of the organisation, and scope, seniority and personal fit to the specific position.

• Collecting additional estimates for the multiplier of OM, as a sample size of 11 may well not be representative.

• Thinking about ways to estimate the multiplier of OM based on metrics which organisations have readily available (e.g. number of opportunities and applicants).
• Understanding how the multiplier of OM will vary in the future. In general, I wonder about:
  • How much the ranking of 80,000 Hours’ highest-impact career paths is going to change in the future.
  • The extent to which expected future variations of the effectiveness of these career paths have already been taken into account, and integrated into the current ranking.
• To follow up on the potentially outdated EA talent needs survey - 2018 used in method 2, asking the leaders of organisations aligned with effective altruism about the cost-effectiveness of the 4 EA Funds. For example, one could request estimates for the 90 % CI:
  • Either in terms of increasing the expected value of the future.
  • Or as a multiple of the mean cost-effectiveness of the LTFF, and this mean in terms of increasing the expected value of the future, similarly to what Linchuan Zhang did here.
• Reflecting about how much weight to give to methods 1 and 2 in method 3, instead of defaulting to giving the same weight to both.
• Defining the significance level of the truncation case in a systematic way. I selected 99 % because its complementary of 1 % is one order of magnitude below 10 %, which is the complementary of 90 %, and most distributions I modelled are based on 90 % CIs.
• Assessing the influence of the FTX crisis on the results/interpretation of my analysis, which was conducted before it started. I expect:
  • The cost-effectiveness of longtermism and catastrophic risk prevention, and effective altruism infrastructure to increase relative to global health and development, and animal welfare^[16].
  • The cost-effectiveness of the effective altruism community to increase^[17].
  • The multiplier of OM to decrease in the short-term, due to less funding being available to start and scale projects.

1. ^{^}

    For 10 M random samples, each truncation and non-truncation case takes me 5 min to run and save the results.

2. ^{^}

    In this text, cost-effectiveness refers to marginal cost-effectiveness.

3. ^{^}

    This is a link to my copy, which contains data last updated on August 15. You can find Tyler’s Sheet here.

4. ^{^}

    If X and X_pre_trunc are the truncated and pre-truncation distributions, and p is the probability of X_pre_trunc being between the minimum and maximum of X, the probability of X being between a and b is 1/p times as large as that of X_pre_trunc being between a and b, which are 2 values between the minimum and maximum of X.

5. ^{^}

    These 2 values consider a wide moral weight distribution whose 95th percentile is 60 k (= 17.2 / (270 μ)) times as large as the 5th percentile.

6. ^{^}

    According to Jaime:

   When the data includes poorly calibrated outliers, if it's possible exclude them and take the geometric mean. If not, we should use a pooling method resistant to outliers. The median is one such popular aggregation method.

7. ^{^}

    The median significantly attenuates the effect of the outliers. For the truncation (non-truncation) case, the mean multiplier with all estimates is 1.54 (1.97) times that without the 4th and 8th estimates using the median, but 11.0 (12.5) times using the mean.

8. ^{^}

    According to Stien van der Ploeg, Animal Charity Evaluators’ Executive Director:

   Some organisations consider any non-program related positions to fall under operations, including communications, strategy, HR, finance, and fundraising roles. Other groups only consider specific administrative jobs like finance, personnel, and organisational support as operations, and some interpret it even narrower.

9. ^{^}

    The mean person working in OM has a much better fit than the respective mean applicant, but there may still be material variation amongst workers.

10. ^{^}

     The 1st/2nd distribution represents the marginal impact/cost per unit time of OM relative to all positions.

11. ^{^}

     The respondent mentioned this type of distribution was an approximation.

12. ^{^}

     The 1st/2nd distribution represents the marginal impact/cost per unit time of OM, and the 3rd/4th that of all positions.

13. ^{^}

     The 1st distribution represents the marginal impact per unit time of OM as a multiple of the mean marginal impact of all positions, and the 2nd/3rd the marginal cost per unit time of OM / all positions.

14. ^{^}

     The 1st/2nd distribution represents the marginal impact/cost per unit time of OM, and the 3rd/4th that of all positions.

15. ^{^}

     Calculated in J2 of the last 2 tabs of the Sheet.

16. ^{^}

     According to the data collected by Tyler Maule, the spending as a fraction of the total of the FTX Foundation between January 1 and August 15 on longtermism and catastrophic risk prevention, and effective altruism infrastructure was 73.5 % and 26.5 %.

17. ^{^}

     According to the data collected in July 2021 by Benjamin Todd here, the “FTX team” represented 35.8 % (= 16.5/46.1) of the funds committed to effective altruism. Assuming the cost-effectiveness is inversely proportional to the committed funds, losing those from FTX leads to it being 1.56 (= 1/(1 - 35.8 %)) times as high.