Part 3: Comparing agency organisational models

By Arepo @ 2021-07-25T13:03 (+14)

Back to part 2

Introduction to part 3

So far I’ve contrasted in-house hiring with outsourcing to a traditional client-funded agency, albeit one aimed specifically at EA orgs. An EA-specific agency would have to be low-bono, offering major discounts to EA orgs - otherwise it would be indistinguishable from the countless existing for-profit agencies.

This post explicitly compares the low-bono option with various others on two axes: on entity type (ie individual or agency) and on different funding models. It argues that a donor-funded model would have many unique upsides over a low-bono one (though I want to explore the potential value creation it allows rather than any specific implementation. There will be many open questions about eg what type of projects it should prioritise since the scope is so much wider and the territory so much less explored than client-funded agencies - hence the appeal): it would be more rewarding for developers to work on, and potentially be able to generate a lot more value from its wider domain. It also looks at the unique challenges donor-funding would present, and how it could mitigate them and in some cases turn them to its advantage.

I’ll discuss each of these below, but the italicised entities are those which already exist in reasonable supply. For this reason, though they might have an important part to play in getting EA tech work done, it seems very unlikely that they could solve the problems described in part 1. If they did, those problems wouldn’t exist: there would be no EA orgs struggling to recruit developers when they could just work with volunteers, contractors or traditional agencies for the work they need; and no EA developers opting to work in-house despite anxiety about career capital.

I’ll examine the limitations of each of the existing options to inform the subsequent discussion on what I see as the comparative strengths of donor funding vs low-bono.

Individuals

We can think of an individual as having a <base market rate> based on the supply/demand for their skills, this value being what you’d expect to pay someone with their skills as an in-house salary.

Traditional contractors

These are flexible, but their fee multiplies the <base market rate for inhouse staff> by something like an <admin factor>, by <1/a sector utilisation factor>, which together determine the cost of providing the service, and perhaps by a <premium factor> - some added value which increases their value over their competitors.

The admin factor encompasses the work required to set up projects - identify the correct skills, agree the scope, arrange contracts, and adverse tax implications such as value added taxes (VAT) etc.

The sector utilisation factor is the proportion of time contractors in your field are employed on average - the lower this is, the higher the cost of their working hours.

Both these factors determine the cost of the developer’s productive time to the developer, so set a lower limit on her/his expected fees. A premium factor is anything or any things beyond fungible skills that brings more value and so allows them to charge more than their competition - industry insight, a relevant network, perhaps likeability to some class of people, perhaps reputation, etc. The distinction between this and base rate is hazy - both encompass a wide range of factors - but you could think of the premium as applying to specific sectors and the base rate as what they would charge (before the other modifiers) if the work from those sectors dried up.

Low bono contractors

These will waive some or all of the premium factor, and perhaps also lower their base rate. These discounts have to come from somewhere, however - all the time they spend doing discount work either reduces their own capacity to donate, or just harms their future selves. This harm disincentivises the contractor from doing this and future EA work, so over time will reduce the number of developers with EA-sector-specific knowledge (ie with a premium factor to waive); while this might be a good trade, we should remember that it is one, even if we exclude the developers’ welfare from our cost-benefit calculation.

Volunteers

These are the de facto form of donor-funded individuals. In principle someone else could pay for their work, but the costs of evaluating each individual who wants money are sufficiently high that this almost never seems to be worth doing at such a granular level.

Volunteers could potentially be helpful at any scale, but several people have discussed the costs and difficulties of getting sustained high quality input from volunteer developers. It also just shifts the costs from EA orgs to EA individuals, which as with doing so to paid developers has long-term drawbacks. Because volunteer work tends to be spread out into tiny weekly amounts, I suspect that the context switching for both the volunteers and their managers actually makes this the most expensive form of workforce, when all costs are considered.

Inasmuch as volunteers are useful, it's probably for very small and relatively low-skill projects, extremely time-insensitive ones, or ones that can absorb an extremely large amount of work without the returns diminishing too much (eg major open source software libraries). But most low-skill/time-insensitive projects will be comparatively low value - otherwise someone else would probably have done them already. However, in cases where volunteers are available and useful, if we simply think of them as staff with some low FTE fraction, many of the arguments for consolidating paid staff into an agency also apply to volunteers.

Traditional Agencies

These seem to cost a client at least twice as much as in-house hires, usually more, and substantially more than independent contractors (see footnote in part 2). Some of this probably comes from higher admin, possibly some from lower utilisation if they tend to take shorter-term contracts or if the seniors are spending a lot of their time on business development. The rest is by definition from a higher premium factor. The premium factor may differ across agencies, but my best guess is it’s a combination of greater assurance of quality, persistence (they’ll more likely be available if you need product development or maintenance work or have questions in future), and a consultancy-esque higher-level view of many projects. Concretely, since they have more net experience, they might be able to point out a widget or practice you didn’t know you needed, or perhaps more valuably point out why you don’t need one you thought you did.

Low bono vs donor funded agencies

Both a low-bono and donor-funded agency will waive some or all of the premium factor. They might also lower their base rate, though this would have the same drawbacks as it would for a low-bono contractor.

In part 2 I laid out the generic benefits I see of moving tech work to an agency, whether client- or donor-funded. In the rest of this post, I’ll make the case for donor-funding.

I don’t mean to imply that a low-bono agency has no advantages over a donor-funded one. It would be trivial to set one up, and it would have intuitive market-driven incentives. But for the reasons below, I think a donor funded agency could both do more total good, and be a more fulfilling place for EA software developers to work.

Outside view comparison

The very ease of setting up an EA-oriented low-bono agency should give us pause, since it raises the question of why no-one has done so.

Although no agency has been created specifically for low-bono work on EA ideas, there seem to be several non-EA tech support agencies that are themselves or are geared towards nonprofits; and EA has always been laden with developers who could have set one up. So while the case is less clear than for the italicised entities in the table above, the fact that the problems of part 1 still exist should raise our credence that there's either a) some need that low-bono agencies don’t meet for potential clients & donors or b) some reason that they’re not appealing enough for developers to create.

Inside view comparison

First let’s look at a). It’s hard to say whether there’s a specific need low bono agencies don’t meet, but I see several net advantages to the EA movement of a donor-funded model. In very roughly descending order of importance:

1. Increasing the viability of pico-orgs: Pico-orgs of the sort described in part 1 may not have anything resembling an owner who’s going to want to take responsibility for applying for funding then hiring and coordinating staff. The grant-application/low-bono model really doesn’t fit such projects: they may not have anything resembling an owner who’d be able to apply for funding let alone coordinate staff; the originator may not even know there was funding to apply for, and even if someone spontaneously decided to fund the project there might be no entity to grant money to. So they usually rely on volunteer work (meaning many don’t get done). While any agency would incentivise more high value macro-org projects, a donor-funded one would also incentivise more high value pico-org projects.
2. Exploring and exploiting synergies between existing orgs: Donor-funding allows you to work in the gaps between organisations on communal services that would have high net value, but which standard prisoners’ dilemma logic would discourage. The extent of this effect depends both on how likely EA orgs would be to cooperate if they found such opportunities (typical EA impact metrics don’t actually incentivise such cooperation well if they’re a basis for funding^[1]) and how likely they would be to find them (ie how siloed their tech depts are). It might also be possible to use these services to share overheads such as access to expensive APIs, which can cost thousands of $s per month.
3. Avoiding VAT losses: If you buy a service from an agency, they have to add VAT, typically around 20%. I suspect this would be possible to avoid for a (charitable?) non-profit funded by someone other than the beneficiary, though the details would differ between countries. I’d be interested in hearing legal advice here from anyone who knows the subject (this would be the most concrete benefit but may not apply - I discuss some caveats with VAT concerns in more detail below).
4. Guaranteeing service across cause areas: A donor-funded agency could commit to splitting its time as evenly as possible across cause areas, or weight them by donor preference, or test other such strategies. Conversely, suppose most of the employees of a low-bono agency were committed longtermists. Then rather than offering low-bono services to all EA orgs, it would make sense for them only to do so to strongly longtermist orgs, and then seek full-fee paying clients the rest of the time, to subsidise higher/more frequent longtermism discounts. Given that people tend to cluster with others who think like them, you would eventually expect this outcome from a low-bono agency. However, if it turns out to be trivial to find enough committed developers to set up low bono agencies to cover each cause area, this concern would be immaterial.
5. Increasing the bandwidth for cost-benefit analysis: Many people have argued that EA is talent or vetting constrained. This means that donors such as OpenPhil et al have limited bandwidth to do cost-benefit analysis on small projects. Donor-funded agencies partially relieve this bottleneck by offloading some of the ‘cost-’ side of the analysis to people specialised in the type of work that constitutes the costs in question.^[2]
6. Deferring resource commitments: In exchange for committing to a particular class of human resource earlier (in this case, tech) and for putting some trust in the agency’s analyses, donor-funding allows allocations within that class to be made much later, ie with maximum versatility. Eg if an agency is unsure about the complexity of a project, it’s much easier for donor-funded staff to test the water, planning to reprioritise it if it turns out to be more difficult than it’s worth. With client-funding, the client has to apply for sufficient funding in advance, before having this feasibility test. In practice they will often have to do so several months in advance.^[3] And circumstances may change after funding decisions have been made, reducing the value of projects that received funding or increasing the value of ones that just missed out on it. With donor-funding, there’s no fundamental restriction on when you can reallocate resources if something more valuable comes up. You could even do so mid-project: this should be rare, but with sensible prioritisation, having the option will be strictly better than not. Similarly, if a project turns out to require more work than was originally budgeted for and you need to decide whether to cut your losses or run over, this decision can potentially be made earlier - and putting it on temporary hold is more realistic.
7. Reducing per-project admin: there would still be communication costs, but less bureaucracy. In practice you might still want a nominal contract for some or all projects, which acted as a clear public statement of intent, perhaps having a token value of eg $1 even though it would never be legally enforced.
8. Exploring a new organisational model: Donor-funding an agency seems uniquely possible in the EA space, because of our somewhat shared but flexible EA goals.^[4] No-one has set up an agency with either funding model, so there’s exploratory value in both but whereas anyone could have set up a low-bono agency just by starting to sell their services, setting up a donor-funded requires solving less familiar problems (see below), not to mention making a strong case to a very limited group of major EA donors. In ITN terms, agencies are a neglected organisational model, with donor-funded agencies the neglectedest.^[5]

And those advantages help explain why I think a donor-funded agency would be more appealing for developers throughout various points in their career than a low-bono one. In no particular order, since each is extremely subjective:

• More coworkers: Points 1, 2, and perhaps 7 of the previous list mean you would expect there to be more high-value tech work than for a low-bono agency, meriting more staff (point 3 might also help afford them) and therefore a better environment for developing technical skills. Most of the concerns with EA tech work described in part 1 boiled down to or were fungible with ‘working with more developers’ - one of the most popular poll options there was wanting a team of five+ developers (getting 10 votes, vs just 1 vote for a team with 2-4), so the sooner an agency could reach that size the better.
• More personal engagement with projects: Working on a project which you helped select as one of the highest value in the whole space seems extremely rewarding. Working on a project with an extra degree of removal - ie one that an overall high value org thinks is the highest value of their projects - seems much less so. (more on the possible tradeoffs of developer autonomy below)
• More variety of experience: Working on smaller orgs’ projects allows developers to choose their tech stack each time. This will give them more variety of experience than only working on the ageing tech stacks of a small number of large paying clients. However, this would potentially bias developers away from working on the highest value project, which would need to be weighed in.
• Greater compatibility with part-time work: Either type of agency could probably support some part-time or irregular work, but if one-off projects turned out to be a substantial part of the total work, this might be especially amenable to it. Two of the respondents to the poll in part 1 (relink) were keen on such flexibility, and I suspect many people would want it at some stage if it were feasible.
• More part-time work = more total knowledge: Having one or two (but not all) permanent part time coders might also help with the problem of having too few coders initially to get the full benefit of a large tech dept. In the poll above, I asked some of the respondents who’d picked that concern whether splitting N FTE coders into >N including some part timers could improve things in this respect. The answers were nuanced and mixed so I won’t try to summarise them here, but my impression was that overall there was modest support for the idea, given specific parameter restrictions (eg part timers working closer to full-time hours than just a couple of days a week).
• More compatibility with mentorship and education: If funders were willing, donor-funding would allow the agency to have an educational purpose. Many EAs move into coding through software bootcamps or self-study, and for people in this position it would be really helpful to have an organisation that could help them learn enough to get their early software jobs while focusing on projects that actually matter. This would require specific willingness from the donors and might not be worth diluting the mission goal for, but if it were, it would be easier to do it than with client-funding.

I’m interested to hear how much other EA developers find these arguments compelling, and hear other reasons why they would prefer to work for one over the other.

All this said, it’s possible that both low-bono and donor-funded agencies are viable and valuable. You could perhaps have a donor-funded agency specifically servicing the pico-orgs and a low-bono one servicing the larger ones. However, this seems antisynergistic: sometimes the donor-funded agency will likely think doing ‘free’ work for larger orgs is better value than any current pico-org project, which would mean they’d end up in unfair competition with the low-bono agency. So this dynamic would rely on pico-orgs reliably having the highest value projects (which seems much less likely than a mix), or being able to work in close cooperation, since they have the same goals - but then the value of having two different organisations is diminished.

Hybrid models

Rather than getting all the funding from one or two philanthropic organisations, it might be easier to get something like a retainer from multiple orgs seeking intermittent development work. This would surely lose VAT benefits, but otherwise be something between donor- and client-funding. How far between would depend on the retaining orgs: if they were expecting you to allocate your time in proportion to their funding, this would be hard to distinguish from low-bono - if they were highly flexible it would be more like donor-funding. If they wanted to go peak EA, the retainer could buy quadratic voting rights for projects.^[6]

A simpler hybrid model would be to get some of the funding from donors, and some from paid work for EA orgs. I don’t think this would provide any theoretical advantage, though. You couldn’t use the fee payments as a windfall to subsidise work on smaller projects you thought were higher value, for example, given that you’d presumably still need the fees from low-bono work to stay in business. It would also lose some of the key benefits of pure donor-funding described above, such as some of the VAT savings and much of the job satisfaction - perhaps all of them if its incentive was always to find paid work.

A third option would be an org that did a strict split of full-fee paid work for non-EA orgs and pro-bono work for EA ones. Again, this would lose some of the key benefits of pure donor funding - long-term value alignment, and probably job satisfaction. And for this model to be optimal, there would need to be diminishing marginal returns on EA tech work such that a) it was worth doing some such work (or there’d be no need for any EA developers), and b) the marginal value of that work quickly diminished to the point where doing paid work was higher value than the EA work itself. In practice this would be the for-profit arm of the organisation donor-funding the EA arm while putting itself at a substantial competitive disadvantage against other tech agencies that could plough all their profits back into admin, marketing, higher salaries etc; so again, this seems like a poor compromise.

Nonetheless some form of hybrid model might be the pragmatic option if direct philanthropic funding is available but insufficient.

The unique challenges of donor-funding

Incentives and feedback mechanisms

These are by far my biggest long-term concerns - if the people you’re working for are not the people paying you, it potentially weakens the feedback loop, reducing your incentive to do a good job. The obvious solution is to solicit reviews from the clients you work with. You could offer some proportion of the maximum possible compensation to developers as a bonus, subject to the review being positive enough. This could be substantially higher fidelity than market feedback (eg 1–5 stars on potentially many axes vs the 1–2 star ‘hire you again or don’t’ system), but it has some substantial challenges:

• A client who has received something for nothing is likely to be happy even if it was slowly or poorly done. You might need to give them an estimation of the real cost of the work and ask that they imagine they’d paid that much for it, reviewing you on that basis.
• It will probably be subject to greater social desirability bias when the client has very little at stake. There is some literature on counteracting this bias, but it has at best partial success and often relies on the respondents not knowing the real purpose of the questions.
• It’s less effective the less the client is equipped to appraise the work. If they only assess you based on speed and a functioning end result, for example, this will incentivise bad coding practices (though any type of agency has to deal with this problem - a for-profit arguably more so, since their clients won't fully understand the long term costs). Ironically, this means orgs with in-house developers will tend to provide more accurate signals on code quality - and pico-orgs might not be able to provide any feedback at all.
• It incentivises developers to work for generous graders - and hence incentivises people wanting tech work done to signal willingness to give overgenerous grades. This would only matter inasmuch as developers were involved in prioritisation (see next section)
• You could eventually combat this by normalising by the average rating an org gives, though only after they’d been able to give a few reviews to different coders.

One other upside to this approach vs client-funding is that the main feedback mechanism would come after all the coding was finished rather than when the last paycheque was signed, making it a more up-to-date signal. This effect would be stronger still if clients were able to edit their reviews later on, to account for bad (or particularly good) post-project support, retrospective discoveries of low quality code, etc. Nonetheless, I think this is the deepest problem a donor-funded agency would face. I don’t see any reason why it should be insurmountable any more than it is for other EA orgs, but it would require a high level of transparency and probably repeated iteration to overcome.

Prioritisation

Another question is who would prioritise projects. Major donors would probably want some say in it - at the very least, an agency would need some commitment to following the values and research of EA-aligned organisations, and it would probably need to develop some reporting metrics to show how it was doing so.

As discussed above and in part 1, I would expect EA software developers to be strongly motivated by having at least some say in the prioritisation - of the four who I interviewed in part 1, the three who reported high job satisfaction all had an active say in their work, and the one who seemed less satisfied did not. Developers also know their comparative strengths better than anyone - which could have a profound impact on the real cost of each project. One hitch is that they might be biased towards working on technically interesting projects, which won’t strongly correlate with high value projects. Value-aligned coders will hopefully be strongly motivated by that value (at least one anecdote supports this), but there may be a limit to this motivation. Allowing coders some flexibility to work on high-value-but-suboptimal projects they found interesting might have enough long term net benefit on their motivation and competence to be worthwhile in the long run, but the parameters would need to be discussed carefully.

The prioritisation policy needn’t be the same for all coders, even those of equal ability. I imagine the agency would mostly hire EA coders in practice, but if strong non-EA coders wanted to work for it, they could do so without any understanding of EA principles and research if they were willing to either develop such an understanding or to delegate the prioritisation. There could be a spectrum of involvement, running from product owners with both a deep understanding of EA cause prioritisation and strong technical knowledge through to dedicated coders who just wanted to work on something socially beneficial.

Commitment

Clients would want some reassurance that when someone worked on their projects, they’d see them through to a particular milestone. In many cases they might also want indefinite low-level maintenance.

• Reputational damage from quitting early might be a good incentive to finish the projects, but that would rely on having an effective reputation system.
• Prioritisation should account for a project’s long-term commitment needs. There’s nothing stopping an agency committing long term in principle if the client made that requirement clear from the start (cf token contracts above). And any work should involve a tacit agreement to at least bugfix any problems it had caused.
• Such long-term low-level maintenance would probably be easier for a donor-funded agency to offer than a client-funded one, since there’d be no need to negotiate around what point you need to start charging for it.

Setting up

• Finding the right founders might be difficult. It would need at least two developers to start with, otherwise it would still have most of the problems described in part 1. (another prompt to developers: please reply or PM me if you’d be potentially interested)
• At least one of the founders would ideally be a senior developer. There might be a Catch-22 where getting commitment from a senior developer would be very difficult without reliable funding, and reliable funding difficult to get without a committed senior developer (senior devs: reply or PM if you would be interested in conditionally committing to such a project subject to funding)
• Setting everything up legally might end up being very time consuming, eating up a lot of the time of one of the developers, or requiring someone dedicated to doing it. This would probably be less of an issue if you weren’t set up as a charity (see next section)
• Getting funding might be be even more time consuming, at least initially, especially if you wanted to pay close to market rates right from the start - though hopefully the founders would be willing to start low while proving the project’s worth
• You’d need to find enough projects to justify 2–3 initial developers. I think this would be trivial. Rethink Charity used to maintain a spreadsheet of EA projects looking for tech support. It looks defunct now, but a) I suspect many of the projects are still active, b) with so many organisations in the EA space I think it would be trivial to find this level of interest if you were actively cultivating it, and c) in the unlikely event that this wasn’t enough, depending on the founders’ backgrounds and their pro rata salary they might be able to work part time while building up momentum (or rather, take reduced wages while they focused their efforts on finding new projects and clients).

Restrictions of charitable status

• Becoming a charity at least in the UK is hard and seems to be getting harder. You need specific knowledge of what the Charity Commission expects. A service charity isn’t unheard of, but is still quite unusual, so it might have an uphill battle.
• Again in the UK, being a charity imposes substantial restrictions on who your beneficiaries can be, and what benefit you can provide them. For example, I strongly suspect it would preclude providing free work for any for-profit org, no matter how socially beneficial it was (or even a non-profit that wasn’t a registered charity). Perhaps you could charge them a very low market rate, though you’d need to be sure you were on firm legal footing; this would also increase friction, since now you’d need to act like a regular tech agency, contracting to do a set amount of work even if it turned out to be low value, etc.
• Given the above, it might be better not to be a charity. That might lead to loss of tax breaks from individual donors (25% in the UK, or more for higher rate taxpayers), though if most of the funds came from charitable foundations they presumably can't duplicate tax breaks by when they regrant to another charity, so the question would be whether there were legal limits on them donating to a noncharity-nonprofit. There’s also the question of whether charitable status would have any relevance to VAT, or whether the only issue would be whether the client is the one paying.

If anyone has knowledge of relevant charity law in the UK, EU and North America, clarification here would be great!

Organisational structure

My first instinct is that such an org should be something like a teal organisation, ie minimal if any hierarchy, maximally flexible working conditions. But I don’t have much evidence for this beyond the idea that it would just make it a more appealing prospect for developers.

The agency would likely need at least one person to do the admin, get funds etc, and they might also need to double as some sort of product owner/head prioritiser (although these might be better as separate roles), who might need some amount of authority to decide contested issues, ensure incentives were working, etc. That person or those people wouldn’t necessarily need to be full time or even on the payroll if the role(s) were comparatively small - it might be a job for a board of trustees, for example.

Last: Part 4 - Intra-organizational and non-tech agencies

Wherein we’ll conclude with a look at one more hybrid mode - an in-house agency of sorts; and the possibility of non-tech agencies.