Part 2: The advantages of agencies

By Arepo @ 2021-07-25T13:04 (+19)

Introduction to part 2

In part 1 I argued that there are three predictably recurring problems with EA tech work: it being bad for developing technical skills, inefficient, and difficult to hire effectively for. In this part I argue that each problem could be mitigated or even fixed by consolidating the workers into a single agency. I focus here on the benefits common to any form of agency - in the next part I’ll look at some of the unique advantages of an agency which itself was a donor-funded EA org.

Stepping through the issues in part 1:

Agencies would make the tech work better for coders

Multiple coders

At any given time in the last few years, there have been perhaps 5-10 software developers working full time in EA nonprofits. Most of them have been working in teams of 1.

If instead of splitting them all this way, even two were extracted to an agency, this would provide a huge improvement to their skill development:

• it allows code review, which is widely considered best practice, both for improving software quality and software developer quality;
• it allows pair programming, which can be very helpful both for making progress on difficult problems and making dull work more engaging;
• it diversifies the skillset, allowing each to learn from the other: working on one’s own can create a feedback loop in which a developer has to work primarily with the tools they know best and so only gets to know those better;
• it gives them someone on the same wavelength to talk to. This emotional outlet can be very important when they’d otherwise only work with people who don’t have an intuitive grasp of what they do.

Multiple technologies

Working at an agency would give a developer exposure to all the technologies of the EA orgs who they worked with. This has upsides and downsides:

• If organisations or projects are committed to ‘stale’ tech stacks (cf the Wordpress comment in part 1, and also the rating of 3/7 for tech stack by one of the EA org workers), you would still have to work on those technologies
• More diversity is good for developing your knowledge of the advantages of different technologies, even if some of them are stale

A downside of this diversity is that if existing EA tech departments did want to shift towards relying on an agency, there would be either be a substantial upfront cost from shifting their codebase to a framework the agency was familiar with or an ongoing cost of the agency needing to ensure its competencies extend to the different frameworks. Nonetheless, I would expect most developers to substantially prefer the variety overall to working on a stale tech stack for a single EA org.

Better balance between building and maintaining

Some minor effects in this respect: 

• In principle, the building/maintaining ratio would be the same regardless of how the work was split
• The tedium of maintenance would be somewhat offset by the greater flexibility to move people around between tasks, so that no-one was stuck doing it constantly
• In practice maintenance work is often relatively low value incremental improvements, which developers are pulled from as soon as a more important project arrives. So I would expect better prioritisation of work across the sector (see ‘Agencies would be make the tech work more efficient’ below) to lead to more time spent building
• Over time, the better practices of an agency (see above sections) should reduce the maintenance cost of projects compared to an in-house developer building them on her/his own

Better CV optics

All the positives above should translate to a CV - moreover, working for a tech agency (even an EA-focused one) likely sounds more impressive than working for a ‘normal’ charity. 

Perhaps it would not be as impressive as working for a traditional for-profit agency, at least until it had established a good reputation. But with some clever marketing, it might be possible to brand such an agency as an ‘elite’ organisation which hired the best engineers to work on the most important tech problems in the world. 

Potential for more workplace flexibility

Another relatively popular option in the polls cited in part 1 was ‘Not enough remote positions/the main hubs are too far away from me’.

In the current climate remote work is necessary - but even if covid is eliminated, tech work will still be one of the most remote-friendly professions. Part time work may also be more feasible at an agency.

(Developer wages would be about the same - ie close to market rate)

On average, I would guess some developers would be willing to work for an agency for less than they would directly for an EA org simply because of the points above - larger tech depts with more interesting work - perhaps more so earlier on from a drive to make the agency successful. I don’t think this would allow huge cost-savings and wouldn’t want to rely on developers’ willingness to self-sacrifice longer term - they can always choose to take a below-market rate salary or to automatically donate some amount back to the agency - but, if I was right in part 1 about typical EA salaries already being closer to market rate than many EAs think, salaries would be a nonissue.

Agencies would make the tech work more efficient

What follows is a summary. See the appendix to this part for the full calculation/set of assumptions. 

I stated in part 1 that we lost the value of approximately half of a developer’s cost - in either actual wasted salary or lost productivity - for each single-developer org. To be more precise, I estimate it at between 0.33–0.66 devs' costs. This comes from the mismatch between their relatively fixed capacity for work and the fluctuating amount of valuable work lined up.

These costs represent the difference between individual departments and a theoretically perfect allocation of resources, so an agency wouldn’t avoid them entirely. But for an agency combining the developers and tasks, the relative fluctuation would be smaller than for each of the tech departments working individually. Whereas the total cost to the movement given N organisations with individual tech departments is roughly proportional to N, the cost if those departments coalesced into an agency is roughly proportional to the square root of N - thus the potential savings per organisation increase with the number of organisations.

Looking at the example of single-developer orgs, and assuming that there are eight such EA orgs at the moment, I conservatively estimate that combining their developers into an agency would save approx 0.11–0.43 of the cost of a developer per organisation, ie saving the movement as a whole 0.88–3.44 developers in total, or equivalently increasing its productivity by that value. Translating this into actual money, if we assume the average EA software developer costs somewhere from $50,000–100,000 (assuming that most EA software developers come from the developed world or eastern Europe - since that’s where the vast majority of EAs come from. This is a reliably conservative cost range, since the cost of a hire is substantially greater than their salary), such an agency would save the EA movement or equivalently increase the value of its output from $44,000–344,000 per year (after taking into account the costs of the actual developers).

There are also many more EA orgs that don’t have sufficient tech work to justify a developer - I count about ~80 - and so have to choose between hiring a normal for-profit agency at a much higher pro rata cost or just not getting their tech work done in good time. I estimate that each of these organisations costs the EA movement 0.04–0.17 of the cost of a would-be developer in lost output.

If an agency primarily served these organisations, I estimate, albeit less confidently, that it would save the movement as a whole value equivalent to about 0.04–0.15 developers per such org so a total saving of/productivity increase worth between $160,000–$1.2million per year - again after accounting for the costs of developers.

Each of these estimations is made more conservative for having ignored the other. And none of this includes the additional benefits of being able to work on the kind of pico-orgs mentioned in part 1, which I don’t know of a way to sensibly estimate.

An agency might also need a project manager, but it’s unclear whether this would be a counterfactual cost or saving. On the one hand the developers need to be individually managed in their current orgs, and in an agency the manager would be more tech-specialised, on the other, in current orgs the spare management capacity may already be available.

Agencies reduce the problem of effectively evaluating tech hires

• The agency will have more software developers with more total experience, so they will better at evaluating new developers.
• For other orgs that still wanted to hire their own staff, evaluating a permanent staff member would be a high value project that the agency could easily help them with.
• If two EA orgs are strictly cause neutral (or strictly share terminal goals), one can donate resources to the other. In particular, if someone working for an EA agency decided they would rather work directly on an organisation that wanted its own staff, there’s no reason why they shouldn’t. And if they’d been working with that organisation via the agency beforehand, that would give a great indication of their cultural fit. Obviously this would need to be done with regard to how much it would impact the agency’s commitments to the rest of the EA space.

A bonus: incentivising high value projects

Giving high value orgs low-friction access to tech workers with >= contractor flexibility will probably incentivise more such orgs and more proposals from them. If agencies are sufficiently reliable at picking out good projects, this will further increase the net value of the EA movement.

Next: Part 3 - Comparing agency organisational models

Wherein I’ll look at the profound implications different funding models have for an agency's scope.

Appendix: Modelling the efficiency gains of an agency

The amount of work of any organisation fluctuates over time faster than the org can adjust the size of its workforce and will have an expected value μ and standard deviation σ.

To find the balance between not wasting people’s time (or rather, in $ terms, their salary), and not neglecting valuable work, we want to hire around μ employees. 

But we will still have periods where there is too little or too much to do. The bigger the fluctuation σ, the bigger this mismatch will be. The dollar value of the mismatch between the work to be done and the work that the employees can do, is the loss of value (LoV).

We know from probability theory that the sum of N identical independently distributed random variables will have an expected value of N times the expected value of one of them. Meanwhile the standard deviation will only be $\sqrt{N}$ times the standard deviation of one of them.

So combining all the N actual EA organisations into one agency will give a linear scaling of the expected work (ie Nμ), but the standard deviation will only scale with the square root of N: $\sqrt{N}$σ. So the LoV of the agency would be only around $\sqrt{N}$ times that of one individual organisation. And not combining them means we currently get N times the LoV of one individual organisation.

Estimating the loss of value for one organisation

Let’s now approximate the loss of value of an individual organisation. We will look at a hypothetical org, and assume that the valuable work they have available will vary between 0 and K full time equivalent workloads (FTEs). To estimate the LoV, we will imagine that each existing organisation has between 0 and K FTEs worth of tech work. We’ll assume that it’s uniformly distributed in this range. Let’s call the salary of the employees S and the potential value an employee could add to the organisation if he/she were doing full time useful work a. We'll assume that S < a, otherwise employees could never be worth their salary!

Let’s call n the number of full time tech staff we employ. We want to calculate their total expected value. This value has two components: the value they create by doing useful work minus the value of their salary. Let’s call the work done by the n employees $W{D}_n$. Then:

$E\left(V_n\right)=E(a\cdot W{D}_n-nS)$

$=aE\left(W{D}_n\right)-nS$

$=a[E(W{D}_n|W>n)\cdot P(W>n)+E(W{D}_n|W<n)\cdot P(W<n\left)\right]-nS$

When W > n, the n employees can only do  n amount of work, so $W{D}_n$=n. When W < n the amount of work they can do is W. So now we have:

$=a[n\cdot P(W>n)+E(W|W<n)\cdot P(W<n\left)\right]-nS$

Given the uniform distribution in W between 0 and K, we have $P(W>n)=\frac{K-n}{K}$, $P(W<n)=\frac{n}{K}$ and $E(W|W<n)=\frac{n}{2}$.

  $=a[n\cdot \frac{K-n}{K}+\frac{n}{2}\cdot \frac{n}{K}]-nS$

  $=\frac{anK}{K}-\frac{a{n}^2}{K}+\frac{a{n}^2}{2K}-nS$

  $=an-Sn-\frac{a{n}^2}{2K}$

  $=(a-S)n-\frac{a}{2K}{n}^2$

So now we know how much value we will create given n FTE employees. We want to maximize the value so we need to find the optimal value of n. The maximum in this parabola is halfway between the 2 zero points. The first zero point is when n = 0. To find the second:

$0=(a-S)n-\frac{a}{2K}{n}^2$

$=n\left(\right(a-S)-\frac{a}{2K}n)$

$=-\frac{a}{2K}n(\frac{a-S}{-\frac{a}{2K}}+n)$

$=-\frac{a}{2K}n(n-2K\frac{a-S}{a})$

$=-\frac{a}{2K}n(n-2K(1-\frac{S}{a}))$

So the second is when n=$2K(1-\frac{S}{a})$. Halfway between 0 and this is the optimal n:

    $n_{opt}=K(1-\frac{S}{a})$

Filling this in gives:

$E\left(V_{opt}\right)=(a-S)\cdot K(1-\frac{S}{a})-\frac{a}{2K}{\left(K(1-\frac{S}{a})\right)}^2$

$=K(a-2S+\frac{S^2}{a})-\frac{a}{2K}(K^2-\frac{2S{K}^2}{a}+\frac{S^2{K}^2}{a^2})$

$=aK-2SK+\frac{K{S}^2}{a}-\frac{aK}{2}+SK-\frac{K{S}^2}{2a}$

$=\frac{aK}{2}-SK+\frac{K{S}^2}{2a}$

We then consider value created in the perfect case where we would always have exactly the right number of employees:

$E\left(V_{Perfect}\right)={\int }_0^K\frac{1}{K}(a-S)ndn$

$=\frac{1}{K}(a-S){\left[\frac{1}{2}{n}^2\right]}_0^K$

$=\frac{aK}{2}-\frac{SK}{2}$

And finally we compare the perfect case with the optimal case to see how much value an optimally staffed organisation loses in expectation:

$LoV=E\left(V_{Perfect}\right)-E\left(V_{opt}\right)$

$=(\frac{aK}{2}-\frac{SK}{2})-(\frac{aK}{2}-SK+\frac{K{S}^2}{2a})$

$=\frac{SK}{2}(1-\frac{S}{a})$

Benchmarking the loss of value for an agency vs multiple organisations

This is a simplistic model, so values should be taken with a grain of salt. Nonetheless, if we fill in some sample values, it can give us an approximate upper and lower bound on how much an agency could improve efficiency all else being equal.

Let S be 1, since it’s relative to a, and so that our final answer will be in units of developers’ salaries - then the latter will tell us how many multiples of a developer’s salary a developer is likely to be worth to the organisation.

Picking a sensible upper bound for a is difficult, but we we have some data to guide us. Firstly, there’s the 80 000 Hours talent gaps surveys, which they seem to have last run in 2018. This suggests that the median EA org would require $450,000 to forego 'a typical recent junior hire’. The average software developer in the US, where the plurality of the organisations are probably based, earns about $100,000 per years based on a quick Google search, so at face value would set a to at around 4 (compensating somewhat for their salary being less than their cost, and treating all software developers as equivalent in value to junior hires).

A second approach is to look at 'very softwarey’ companies, divide their annual revenue by their number of employees, and divide that again by the Glassdoor 'average software engineer’ salary (it doesn’t list or differentiate ‘software developers’). Then we get:

• Google: a = 9.3
• Facebook: a = 8.9
• Microsoft: a = 7.1
• Palantir: a = 3.1
• LinkedIn: a = 2.6

These seem likely to be biased upwards, firstly because these companies (especially the first three) have their pick of the world’s best software developers, and secondly because this is ‘base salary’ and probably won't include other compensation and costs. Pushing the other way, most of the value a software developer produces is probably realised a few years after their relevant work, so assuming these companies are expanding, revenue probably lags costs. Also, since the for-profit space is far more saturated than the nonprofit space, the marginal value of even the best for-profit may be lower than of an effective charity. Nonetheless, my instinct is to assume the first two points outweigh the latter two.

A more conservative third approach, which seems soundest to me, is to observe that existing EA orgs can hire 'custom-time' software developers more or less on tap from reputable for-profit consultancies. So if EA orgs aren’t hiring an agency to perfectly match their tech needs, we'll assume the value of the work isn’t enough to justify their cost.

To get a sense of what that difference in pricing is, I posted the question ‘Roughly how much more would you expect software consultancies to charge for equivalent jobs than permanent staff of the same skill?’ in the EA software groups.[1] The answers ranged from 1.12x to 20x, with a mid-range median of 3.5 - I’ll take this and round it down to 3, to again account for the fact that the cost to an org of a staff member is somewhat higher than their nominal salary. So that gives a=3 as a plausible but conservative maximum estimate.

For a lower bound, given that successful negotiators seem to be able to consistently raise their salary by 10% above the rest of us, and that companies are probably risk averse about hiring, let’s say a= 1.2.

For the moment let K be 2, such that the likely number of software developers is 1 - this has been most EA organisations that have had >0 software developers for most of the last decade. Further down, I’ll look at the organisations with 0.

Finally, let’s say that N is 8 - my best guess as to the number of EA nonprofits that have actually had 1 full time developer (to my knowledge,  CEA, 80K, Founders Pledge, Humane League, AMF, GiveDirectly, Givewell and Giving What We Can who’re currently hiring their first) though this needn't be an exhaustive list for this rough estimate. There might be nonprofits I’ve missed and there will certainly be more if we consider also consider for-profits. But in practice there will also be diminishing returns from throwing together many more developers into a single agency much larger than this. You’d probably want to split them into multiple teams of not much more than 10. 

So on these estimates our ‘maximum’ loss of value given these eight orgs working individually would be

$8\cdot \frac{1\cdot 2}{2}(1-\frac{1}{3})$

which is ≈$5\frac{1}{3}$ devs’ salaries or 67% of one dev’s salary per org. If we imagine the same eight tech departments as a single agency, we get

$\sqrt{8}\cdot \frac{1\cdot 2}{2}(1-\frac{1}{3})$

roughly 1.9 software devs’ salaries or 24% of one per org - so a saving of about 43% of a developer’s salary per org.

The same calculation for the ‘minimum’ value, a= 1.2, yields a saving of about 11% of a developer’s salary per org. So we have an approximate estimated saving of 11-43% of a software developer’s salary per org, if all of the EA tech departments currently employing developers combined into one agency. (Weighted towards the upper end: taking the mid-point of our range for a, ie 2.1, would give us 35%)

There are many more EA orgs with no developers, and this model doesn’t let us easily aggregate the savings of those with those of single-developer orgs. Instead, let’s do a separate comparison. Where the first was imagining an agency which primarily serviced ‘major’ EA orgs that would otherwise have had a developer, the second imagines an agency which primarily works with orgs that haven’t yet reached the point where they hire a developer, but would benefit from some amount of tech work. Each is inherently somewhat conservative for having ignored the other.

Let’s let K = 0.5, to approximately represent these orgs (this is hopefully reasonably conservative; if we assume rational hiring and that K = 2 for 1-developer orgs, then K would just be <2 at <1-developer orgs - but there’s probably a skew at any given time towards smaller orgs with less overall work). At the time of writing there are 88 total listed EA orgs. Subtracting the eight assumed for the previous example, N= 80. Given the same upper and lower bounds we used above, we would expect an agency to save 4–15% of a salary per org.

I owe the mathematical model in this appendix to the awesome Johan Lugthart, without whose help these would have been far sillier estimates.

Link to Part 3 in case you missed it above

[1]: The answer seems to be ‘a lot’. At the time of writing, 6 people have answered, ranging from 1.12x to 20x!:

• ‘12-45% higher cost than permanent staff’
• ‘around 2x more when talking about seniors? gap for junior and mid level developers' pay and day rate what agencies are asking is even bigger (5-7x).’
• ‘I think in practice it varies widely, anything from 2-20x. At a consultancy I worked for previously it was 10x for a junior software dev.’
• About 3-4x: ‘the consultancy I worked at, that promoted itself as a premium consultancy I suppose, charged 950 per senior consultant per day - meaning roughly 20567 pounds per month. They paid their employees fairly standard wages, so senior consultant would be paid between 60 - 90k per year, meaning 5-7.5 k per month including tax’
• ‘In my experience, between x1.5 and x6.’
• ‘About 1/3 extra’

    My own estimate (included for the median) was about 1.4-3x, decreasing with seniority.

    I would expect such agencies to offer some charitable discounts, but for those to be substantially less than the multipliers above. I was able to ask the founder of one successful agency what their charity discount was, and he said 10%.