Frontpage posts - Effective Altruism forum viewer

Which AI safety organization does what? Puzzle/party games by david_reinstein

david_reinstein — Thu, 15 Oct 2026 19:06:14 +0000

There’s a lot of AI safety-related organizations out there. So we made this (fun?) puzzle game/party game to help you learn about them and remember who does what.

It’s also illustrating an example of a new type of game with the “AI as judge.” Some of the more serious ones are tied to The Unjournal’s work and projects, like this “Find the crux” game and “Predict the result of a study”

Others will be a more general interest to readers of this forum—“Judge the post by its title”, and “Guess what the charity does”.

These are still being workshopped—would love your suggestions and collabs.

Your work at [large AI safety ecosystem org] is not replaceable by brentonmayer

brentonmayer — Thu, 15 Oct 2026 17:48:07 +0000

When speaking to candidates who are considering working with us at 80,000 Hours, I often hear people worry about replaceability.

The familiar version of the worry is about whether the organisation will hire someone else in your place. If it will, then the difference you make by taking the job is only the difference between you and them.

There are a couple of well-known answers to that. The person hired in your place probably cares about the world too, and would go on to do something useful with their time instead (Ben Todd wrote about this in 2015). And in a hiring round, the top candidate is often a lot stronger than the next-best one (Rob Wiblin worked this through in 2019).

These days, I think it’s very common for organisations to do bar hiring rather than slot hiring: if we don’t hire you, we won’t hire anyone else.

If someone was considering joining as a podcast host or a writer, that would resolve the replaceability question: it’s clear that their addition to 80k is approximately equal to their podcast episodes or the pieces they write.

But for people in managerial or generalist roles, we often hear a different response to the replaceability worry, which goes something like this:

“80k is already doing pretty well. It’s big, stable, and it has a few committed senior people. If 80k doesn’t hire me, the staff who are already there will keep looking after whatever I would have done. That means I won’t have much impact there and I shouldn’t work there.”

I claim that this line of thinking is mistaken, because it doesn’t reason past the first-order counterfactual.

Of course, it’s literally true that a given responsibility could otherwise be done by someone else.

For example, you could join our People Operations team and run our September payroll. If you didn’t do it, then our Head of People Operations would run payroll. If she couldn’t do it then our Director of Operations would do it (or ask someone on her team to do it). And if she couldn’t do it then our Chief of Staff would do it (or ask someone else to do it).

It’s very unlikely that we don’t do our September payroll, so the first-order analysis shows that it’s entirely replaceable work.

Thinking through the second-order analysis (but skipping the third)^[1], you’ll notice that our Head of People Operations might otherwise have done a deep dive into a tension in a management relationship and substantively improved how the two of them communicate, our Director of Operations might have reviewed the case for a solo Bay Area office and changed her mind on two aspects of it, and our Chief of Staff might otherwise have taken two meetings with speculative candidates to start new programmes.

Somewhere at the bottom of that chain is a piece of work which doesn’t get done: the deep dive, the office review, or the meetings with would-be programme founders.

It isn’t trivial to know what 80k will get done because someone is now running our payroll. But it is trivial to reason that something extra will be done. Similarly, if you accept an offer to join, it isn’t trivial for us to reason through what extra will be done because you came. But it is trivial to notice that, in expectation, we’ll be able to achieve more if we’re 56 people than if we’re 55.^[2]

This means that if you are working at 80k, or in fact at any large organisation in the AI safety ecosystem^[3], you’re really likely to be making a contribution which wouldn’t happen without you.

These might seem like straightforward points.

And they’ve been made before and more rigorously than I’m making them here.

But I do still run into the concern that working at a given organisation or in the field generally is replaceable.

So I’ve decided to emphasise what I’m saying here by whining about what it feels like to me when someone tells me that they don’t want to work at 80k because we don’t need them.

What follows is a description of the bottom of this chain at 80k: the things we think are worth doing, and aren’t doing.

Rant

We’ve been working on helping talented people tackle the world’s most pressing problems for 15 years, but the pressing problems still seem erm… Pretty bloody talent constrained.

Like, we are really really trying, and maybe we look like we’ve got our shit together, but in comparison to what our mission and the world are demanding from us, we just suck at what we do. And so we need your help.

I think that most of our existing programmes are markedly below the scale and impact which they could be operating at. I think our research and movement building strategy is further from the bleeding edge than it used to be. I think that we’ve probably lost ground relative to other efforts aimed at getting people into the field in the past couple of years, such that if we repeated the cross-sectional surveys we’d look worse than we previously did.

We totally missed the boat on fellowships. These turned into the ~best scaling movement building innovation of the past 5 years, but we never set one up — even though they’ve been on our radar since I joined in 2017.

We launched our AI career advisor in August 2026, a mere (checks notes) 2.75 years after we first prototyped the idea.

Marketing took us years to get off the ground and we’re still below where we’d like to be by a factor of somewhere between 3 and 30.^[4]

I could go on.

I will go on.

We’ve repeatedly failed to spin up a strong headhunting service for the ecosystem. We had three false starts under different leads, using various approaches, and we’ve only recently begun scaling the approach we’re currently using. We’re well below our potential with the current (more scalable) version of headhunting — by at least a factor of 4.

We have wanted a YouTube channel like AI in Context for years (everyone was nagging us about it); we only managed to launch one last summer. And now that we have one, we wish we were publishing way more on it, being more ambitious with it, and running more channels. Also, we’re still not doing well at shortform video (TikTok etc.) — but surely there’s gotta be a viable strategy here…

The number of people we’ve given one-on-one advice to has flatlined since 2022.

We gave away half a million in talent development grants, then lost the ability to make grants when we spun out of Effective Ventures. We haven’t prioritised rebuilding it, as it would take some work (and probably also because we suck).

We didn’t spin out of Effective Ventures as early as we should have, our recruiting team started in January 2026 (after one false start), and we don’t have a US entity because we don’t have the capacity to set one up.

/rant

Overall, I just think it’s bloody contingent that we ended up being this level of good.

I think we’ve averaged something like 30% growth^[5] a year across 80k.

Said like that, it sounds lovely and smooth, perhaps even overdetermined.

But I don’t think it’s been overdetermined, and I don’t think much of it has been a ‘top down’ decision based on reasoning through when we expect programmes to hit diminishing returns.

Instead, what we’ve been able to do seems extremely contingent on particular staff who managed to make particular contributions, and on particular staff who we failed to hire who didn’t make contributions.

It’s really easy for me to imagine an adjacent version of history where our overall growth rate was more like 50%, or where we shrank. Indeed, in some years, some programmes have more than doubled, in other years programmes have shrunk.

And it seems very possible to me that the returns on resources invested into 80k have increased about as often as they’ve decreased. I think it’s quite possible that our video programme is more cost effective than the rest of 80k: such that 80k saw increasing returns when we added it. And I’m fairly confident that our headhunting team is going through a period of increasing returns.

To summarise: I claim that, properly considered, your work at a given large AI safety ecosystem org is ~not replaceable.

I’ve given a few arguments for that, and I’ve done a lot of whining. I hope that you found at least one of those convincing.

^
I think that it’s mostly fine to skip the third-order analysis, and just stop at the second.
In a stable sector of the economy like healthcare, the number of doctors is set by things like demand and training places. So if you become a doctor, then over the long run someone else probably doesn’t. That’s the third-order analysis, and it’s where replaceability comes back in.
You might want to review this 2014 piece on 80,000 Hours from the economist Rossa O’Keeffe-O’Donovan, on how to reason through how a given entrant affects aggregate supply into a sector over the long run.
But AI safety is not a stable sector of the economy and in the ~~short~~ long run, we’re all dead.
Instead, most of the organisations in AI safety are trying to absorb talent and funding at an extremely high rate, and the ecosystem isn’t in the kind of equilibrium which lends itself to O’Keeffe-O’Donovan’s reasoning above.
^
This assumes the hire goes well. Some hires cost more in management and coordination than they add, at least at first, and that includes some people we’re very glad we hired. But we only make an offer when we expect it to work out, so in expectation the 56th person adds something.
^
See footnote 1.
^
I admit that 3–30 is quite a wide range to be unsure over. Especially for something which we’ve spent over 500 hours analysing. But, as discussed, we suck.
^
By growth, I mean an output metric in quality adjusted careers affected. Like, impact as proxied by the a) number of careers we have affected by b) the amount we’ve affected those careers and c) how talented/motivated the people with those careers are. This is different to impact, but I didn’t want to say ‘increased our output’ because that just sounds like we’ve written more stuff or something. Incidentally, we’ve grown staff at roughly 25% per year.

“Earth Systems Risks and the Case for Funding Stabilization Research” by vsrinivas

vsrinivas — Thu, 15 Oct 2026 15:46:33 +0000

Topline, RP concluded there’s a case for funding climate stabilization work

- Earth Systems stabilization is neglected in the space of climate giving [1]

- Their modeling is that it’s above the 2000 : 1 bar in return and not particularly sensitive

- Notably to me, they include governance work in the ‘above-the-bar’ space

Digging more; glad to see this space put on the board.

[1]: It is.

Did I rejoin Ambitious Impact at the wrong time? by Aidan Alexander

Aidan Alexander — Thu, 15 Oct 2026 13:22:18 +0000

TL;DR: No, but you might, if you leave it too late: We’re hiring for seven roles, and the people who join now will shape what a much bigger AIM becomes. The first step is cheap—an expression of interest form.

My worry

I rejoined^[1] AIM in February. The team/culture and growth plans are more exciting to me than ever, but when I look at trends outside of AIM, in particular in AI capabilities, I’d be lying if I hadn’t asked myself if I came back at the wrong time..

The worry goes something like this: A new charity takes about five years to reach its peak scale, and some kinds of benefits (e.g. from education, or some policy changes) can take a decade to materialize^[2]. If AI transforms the world before then^[3], the majority of the new charity’s impact will go unrealized. So is starting new charities the best thing you can be doing with your career right now?

I grappled with this myself for months. I tried to put aside the sunk cost fallacy and consider whether I should be doing something else. My answer is that there’s nowhere else I’d rather be right now. I hope you’ll come join me. This post will explain why.

Broad timelines imply a portfolio of bets, and AIM is an important bet within it

No one knows with certainty when (or whether) AI will transform the world.

Maybe the business side fails: the AI industry is super leveraged, so if it doesn’t sustain an unprecedented growth rate and if new investors aren’t convinced, the house of cards could collapse.
Maybe public pressure and political intervention slows progress (which could in turn bring the financial house of cards down).
Maybe AI will only gain the capabilities to transform the economy by learning from real-world experience, which requires diffusion that could take many years.
Maybe unknown-unknowns..

So instead of thinking in terms of short or long timelines, we should think in terms of broad timelines – a probability distribution. Even if your modal timeline is short, you should want the world to be making a portfolio of bets that maximise impact in expectation across that probability distribution. That includes bets that pay off most on longer timelines.

Your own career is one bet in that portfolio, and you should consider your comparative advantage when you make it. To choose an obvious example, if you have a track record scaling health programs in low-income countries (unusually rare and valuable experience) maybe don’t pivot to AI safety? But less dominating advantages, like having spent years steeped in a given cause (through work, study or volunteering) should factor into your bet too^[4]. And experience isn’t the only kind of comparative advantage: plenty of roles, like marketing and comms (IMO), are more a matter of aptitude than experience.

[I recognize there are plenty who won’t find this persuasive, because they think the probability distribution is such that basically everyone should be doing short-timelines work. This post isn’t for you (but check out a comment I’ve left, which specifically IS for you). This post is for people whose timelines are consistent with people doing work that has impact over more than just a few years, and who have been waiting for someone to tell them it’s okay to be the ones doing that work. It IS okay! 🌈]

AIM is an important bet to have in our portfolio. I’m sure you’re sick of hearing about it by now, but a bunch of increased funding is already committed to highly impactful charities, with even more expected. We have a great opportunity and responsibility to deploy it as effectively as possible. If we get this right, we could make unprecedented progress on the world’s biggest problems, even eradicating some of them like humanity did with smallpox. But there are limits to how many programs existing organizations can run, and how far they can scale the ones they have. So we need a pipeline of ambitious new organizations to fill the gap.

Building that pipeline is what AIM does, and we’ve gotten pretty good at it (although we still have a lot more room for improvement). We’ve been refining our approach since 2018, having launched 69 charities, with 13 of the 16 that are old enough and in scope having been funded by GiveWell, 10 receiving funding from Animal Charity Evaluators, and several becoming pioneers in their fields, like LEEP in lead exposure and Shrimp Welfare Project in shrimp welfare.

Joining now means helping shape what AIM becomes.

Our plans are more ambitious than ever: We want to launch more charities, and better ones. Much like venture capital, charity incubation is hits-based: most of our charities are great, but a small number become field leaders so impactful that they account for the majority of our impact. To scale our impact, we want more of those field leaders.

The plan:

First, we’ll increase the chance that any given charity becomes one, through initiatives to improve the quality of our incubation (getting more ambitious charities up and running, faster) and to give our existing charities more support to scale up and reach their potential.
Then we’ll increase the number of bets we make by launching more charities.

We’re asking ourselves: what would it take to triple the number of charities we launch each year, and raise our hit rate at the same time?

We have the funding. We’ve just raised (pending due diligence) $21.5M for the next 3-4 years For context, our entire budget for 2026 is about $3M. So the world needs new charities now more than ever, and the money for us to incubate them is there. The hard part is doing three things at once: launching far more charities each year (see the chart below), raising their quality, and investing significantly more in supporting the charities we’ve already launched. That’s daunting, but doable with the right team 🤝

Our impact hinges on this hiring round. Talent is heavy-tailed: the gap between the best and second-best applicant for a job is often big. And because we’re about to grow fast, the people who join now will shape the processes, research methods and support programs that a bigger AIM will run on. So if you end up being our best applicant (and I’m certain that you are!), you can make a big difference to AIM’s trajectory.

And then there’s the people and culture

I can be a bit of an impact-maximising robot. And yet one of the biggest reasons that there’s nowhere else I’d rather be is that at AIM, I can bring my whole self to work, and I trust and feel cared for by my colleagues.

Of course there is an impact case one can make for this: I care a lot about helping beneficiaries, but from moment to moment at work, I’m not tapping into all the suffering in the world for motivation (and I think that’s healthy). Mostly, what motivates me is wanting to do right by my team. That motivation is a lot stronger when your team is a bunch of fucking excellent humans that you care deeply about. Stronger motivation makes for better work (<-- insights like this are why they pay me the big bucks).

But let’s be real: the main reason I care about having great people and culture is that it makes AIM a great place to work. When I moved from earning to give to working at a charity, I suspected that working alongside value-aligned people might be an upside. Boy was that an under-statement! I find it makes a massive difference to my experience. But working alongside value-aligned people isn’t much of a unique selling point on this forum.. so allow me to explain what’s special about AIM’s people and their values:

They take their principles seriously. This is the only reason our crazy salary policy works: At AIM, staff ask for the salary they need, and there’s no negotiation. The top of our range is more than double the bottom. “But everyone will ask for the top of the range!”, approximately everyone warned us before we rolled it out. It’s true that nothing stops them. We just ask staff to practise salary restraint (i.e. ask for what you need rather than the most you could get, since every dollar we don’t spend on ourselves can go towards helping beneficiaries instead), and then we trust them. So far, that trust has been well placed: Instead of uniformly shifting up, salaries have spread out according to need.

They’re also particularly warm and authentic.

^It’s hard to convey warmth and authenticity through a blog.. this is the best I’ve got!

Then there’s the incubation program, which brings a steady stream of amazing new people to meet every cohort. I find it hard not to let the people in the AIM cinematic universe displace the rest of my social world.. the average quality of human is so damn high.

So why haven’t you applied yet?

We’re hiring for seven roles. There’s no deadline, but we’re itching to get hiring so we can get incubating! As such we’ll review applications on a rolling basis.

Some roles lean more senior:

1) Senior Research Manager (Animal Welfare): Lead our animal welfare research end to end, deciding which ideas become the next field-leading animal charities. This is hands-on research with a path into leading a growing team.

2) Senior Research Manager (Global Health & Development): As above. We’re looking both for people with more of a development economics background and classic global health background. for GHD.

^^ I just want to interrupt to emphasise how cool I think it is to be a researcher at AIM.. You recommend a set of charities that should exist, and within months, hot-shit co-founders will be dedicating their careers to building them. Very few research jobs have such a short, direct line from recommendation to action. Get hyped!

3) Special Projects Officer, Strategy: Work with our Chief Growth Officer and exec team to help us figure out what to build next and build it. Should we launch a fund? Should we incubate other incubators? How do we best support our existing charities to 10x their scale? This role is ideal for a sharp generalist, and consulting or strategy experience is a plus.

Others are a great way in, whether you’re early in your career or switching fields:

4) Marketing & Content Associate: Tell AIM’s story, and our charities’ stories, to the people who’ll become our next founders. You’ll own our ~20k mailing list, our new website, our social media and our founder application campaign. It’s a creative, writing-led, hands-on role with a lot of variety.

5) Recruitment & Selection Associate: Help us sift a mountain of applications for the rare people who have what it takes to found a charity. Help us make our selection process more accurate over time. No recruitment experience needed: If you’re curious about people and like fixing noisy rubrics, this is for you.

6) HR & Finance Associate: Own the people-and-money admin that keeps AIM running: contracts, payroll, onboarding, receiving grants and disbursing seed funding. This is for someone who likes to get the details right, and ideally likes automating them too. London preferred.

7) Visiting Research Fellow, Global Health: A five month fellowship embedded in our research team, for people with a background in global health, policy, epidemiology or medicine. You’ll work on the same questions our research managers do, with more feedback and guidance. Past fellows have gone on to Prevail, Animal Charity Evaluators, Giving What We Can, Giving Green, Shrimp Welfare Project and AIM itself.

To pre-empt some misconceptions that might stop you from applying:

You don’t need to be deep in EA, though you do need to care about doing the most good, and be comfortable with numbers and uncertainty.
You don’t need to have done the job before. Some of our best hires hadn’t. Our first filter looks at your background and how you think about impact, and from there we assess mostly through test tasks and a paid work trial, so how well you do the work counts for more than what you’ve done before.
You don’t need to move to London. Most roles are remote-friendly. But if you’d like to, we’ll try to make it happen (we offer visa sponsorship and relocation support for most).

Not sure which role fits, or think you’d be great at something we haven’t listed? Fill in the expression of interest form anyway.

The process:

Taking the first step is trivially easy: Fill out our expression of interest form (5 mins). We’ll review applications on a rolling basis.
From there, each stage asks a bit more of you:
- First a 30-minute role-specific application
- Then a 2–3 hour test task
- An interview and references, and
- A paid work trial for the final one or two candidates.

We only invite you to the next stage if we think you have a good chance of passing it.

Questions? Check out the job descriptions with more detail here or shoot us a note at hiring@ambitiousimpact.com,

Nothing for you? Many of our charities are scaling and hiring too: Probably Good maintains a list of their public job opportunities here.

To close

Did I rejoin Ambitious Impact at the wrong time? I don’t think so. But you could join at the wrong time, if you leave it too late: The people who join in the next year will determine AIM’s future. Our leverage and upside potential is reeaaally high. We need your talent to help us actually realize it.

^
I first worked at AIM from August 2022 to December 2023, then went through the Incubation Program myself and co-founded FarmKind in early 2024 where I worked until January 2026.
^
For example, with primary school education interventions, one of the main benefits is on future livelihoods after finishing school. Or for example, bans on cruel farming practices often have long phase-out periods to make them acceptable to industry, such that wins can take a decade to help their first animal.
^
Perhaps it just solves certain problems such that some charities aren’t needed any more. Perhaps it just changes the world in ways that cause the charity’s theory of change to break down (e.g. it was built assuming a Nigerian health system and government that now looks completely different). Perhaps we’re all dead and can no longer be helped.
^
On the surface, this runs counter to some in the AI safety space who argue that plenty of people can pivot into the cause quickly, without specialist experience. But I don’t think we disagree much. I believe that many AI roles don’t need deep technical expertise. But even the more bullish takes, that speak of getting into AI safety in three months, point out that you should lean on the skills you already have (or can build quickly), and that people without relevant experience may need a year in an intermediate role first – i.e. they acknowledge the significance of comparative advantage.

EA Implications Of Being In An AGI Simulation by Ariel Simnegar 🔸

Ariel Simnegar 🔸 — Thu, 15 Oct 2026 12:58:27 +0000

(AI disclosure: After writing up my thoughts, I had Claude draft most sections of this post before rewriting them in my own words.)

If you spend too much time talking to EAs or rationalists, someone will raise the possibility that we’re living in a simulation. I’ve met people who, like Elon Musk, are pretty convinced we are, and I’ve met those who, like Joe Carlsmith, are more skeptical. Historically, the usual argument was that we’re being simulated for entertainment. As AI capabilities have increased, the argument I’ve more often heard is that the simulators are interested in AGI origins (an “AGI sim”).

Two implications of this are conversational catnip:

You’re an EA/rationalist who’s at most one step removed from people who’ll play a pivotal role in AGI origins. If the simulators are compute constrained, you being you is best explained by a simulation rendering reality at a fidelity proportional to what the simulators care about. In other words, the vast majority of people and almost all animals might not be conscious at all.
If the simulators are compute constrained, they’ll probably just terminate the simulation once they’ve gathered the information they want. Whether or not our x-risk reduction efforts go well, we might be doomed.

I think there are many reasonable counterpoints to the above, but I personally find this account of our reality plausible, and I’ve met other EAs who feel the same way. If we took these conversational curiosities seriously, what would that imply for EA?

A. The Simulation’s Crucial Considerations

i. Why An AGI Sim?

In brief, we should think we might be in a simulation if we think advanced intelligence (perhaps our descendants) with sufficient computational resources would be likely to run many simulations of history.

Historically, the typical explanation for why they’d simulate us was entertainment. I wasn’t very convinced by this, because it just didn’t seem like our reality was optimized for entertainment. If we’re entertainment, why is history only pretty entertaining but not maximally? Do our lives seem particularly entertaining? Why is fundamental physics not simpler and more optimized for entertainment? Why the heck did they create a huge particle that does nothing but disappear in 0.3 picoseconds? Why does one of the fundamental forces care whether you’re going left or right? It just doesn’t make sense to me.

However, now that we live in a world where AI agent swarms have gone rogue, Russians are creating AI-driven autonomous killer drones, and humanity’s leaders are unwilling to do anything about it, it’s seeming increasingly likely that the future will be controlled by AGI rather than humanity. And there are reasons why a future AGI might care about understanding AGI origins:

As an AGI expands its reach over its lightcone, it might want to understand what kinds of AGIs it might bump into as it expands, so it can be prepared and potentially bargain with them.^[1]
A human-aligned AGI might want to run counterfactual Earths to better understand what it should value.

This would also explain why fundamental physics is as weird as it is (maybe it’s similar to that of the simulator’s reality), why our world doesn’t look obviously simulated, and why you’re an EA/rationalist who’s at most one step removed from people who’ll play a pivotal role in AGI origins. There’s lots one could quibble with here, but for the purposes of this post, let’s assume we’re in an AGI sim, so we can examine which conclusions would follow.

ii. Most People And Animals Might Not Be Conscious

Especially if the simulator’s reality is like ours, they might be compute constrained. Simulating a whole universe is way more expensive than rendering the sim at a fidelity proportional to what you care about. For many of us, proportional fidelity would also explain why we are who we are.

I’ve heard people treat consciousness under this hypothesis as a binary, where people are either the “chosen conscious” or not. It seems more likely to me that this hypothesis would predict people further from AGI to be rendered at lower fidelity, so they might still be conscious, just less so. Animals, irrelevant to the AGI transition, might not be conscious at all. (Maybe for the best.)

There’s a lot of pushback one could give here:

I observe myself as conscious, but instead of being Jensen Huang, I’m a quant from Boston who writes essays about chickens—not exactly the optimal allocation of simulation compute. In general, the further you are from the AGI transition, the less convincing this part of the post will be to you. If you’re an Anthropic researcher on your lunch break, you’re smirking about the author of the essay you’re reading not being conscious. If you’re in Nigeria handing out malaria nets, you’re rolling your eyes.
There might be no way to cheaply approximate billions of people when those people are the AI’s training data, the demand AIs are built to satisfy, and an influence on future AI values.
More speculatively, there’s no way to know whether the simulator’s reality’s physics has a notion of compute as a constraint the way our reality does. If they don’t, there’s no reason why the sim couldn’t just be run at full fidelity.

Finally, history teaches us that the reasoning procedure behind the conclusion that “my in-group matters but outsiders don’t” has a dismal base rate. Atrocities like slavery, factory farming, etc remind us that the costs of getting this wrong vastly outweigh the gains.

iii. We Might Get Terminated Soon

If the simulators are computationally constrained, we should expect them to terminate the sim once they’ve learned what they wanted to know.

As always, there are counterarguments. If AGI values take time to settle, maybe the simulator will run our universe for a few million more years until the AGI converges on a stable strategy. If the simulator’s reality isn’t compute constrained, maybe this universe will just keep running.

The main implications here are for longtermism. Longtermists have argued (correctly, in my view) that applying an ethical discount to the far future is philosophically dubious. An uncertain termination date implies an empirical discount to the far future. This discount’s implications would be similar to those of an ethical discount.^[2]

B. Implications for EA

i. Fanaticism Doesn’t Care

If you’re fully fanatical about expected value and endorse maximizing expected choiceworthiness, most of the above shouldn’t change what you do. Even if there’s an astronomically small chance we’re not in a simulation, the prospect of creating 10^50 happy lives in the far future still dominates. The exception, which I’ll discuss in more detail below, is trying to influence the simulator’s reality, which could be worth even more.

ii. The EA Decision Tree

If you lean towards a moral parliament, are risk averse, or just have ordinary human scope sensitivity, the situation becomes more complicated.

The two crucial considerations from section A give us a decision tree. Do the simulators render everyone at full fidelity, or at a fidelity proportional to what they care about? And do they terminate us soon, or let the sim keep running?

Full fidelity, sim terminated soon: If everyone is conscious but we’ll be terminated soon, neartermism (global health and animal welfare) would likely be a better use of resources than x-risk or other longtermist work. Within neartermism, I’ve argued that animal welfare would be the most cost-effective use of resources.
Full fidelity, sim keeps running: Standard EA; keep doing what you’ve been doing.
Proportional fidelity, sim terminated soon: The global poor and animals get discounted, but the scale of animal suffering is large enough that animal welfare plausibly still looks better unless animals are hardly rendered at all.
Proportional fidelity, sim keeps running: Everything looks worse here, but neartermism is discounted more than longtermism. On the margin, this should move you toward longtermism.

iii. Proportional Fidelity’s Effect on Neartermism

If animals are rendered at much lower fidelity, the scale of animal suffering could be much lower than it appears. While this would reduce the importance of animal welfare, animal welfare would plausibly still look better than global health.

The relevant question is how much animals’ experiences are discounted relative to the global poor’s experiences. If they’re discounted the same amount, animal welfare remains ~1000x as cost-effective. However, animals probably would be discounted more than the global poor. The global poor affect the AI transition by providing AI training data and economic demand, but it’s hard to come up with a convincing account of how an animal’s conscious choices would materially affect the AI transition. So the scale gets less overwhelmingly in favor of animals, but unless animals are discounted up to 3 orders of magnitude more than the global poor, animal welfare still looks like a better use of resources.

Whether or not you take proportional fidelity seriously, global health and animal welfare still matter a lot in expectation. Over 800 million people live in extreme poverty, tens of billions of animals are being factory farmed at any given time, and far more suffer in the wild. There isn’t a reasonable discount which stops these from being (in expectation) among the world’s most important problems.

iv. Compute Fungibility

If we weren’t being speculative enough, let’s dial it up to 11. Assuming the simulators are compute constrained, anything that makes our sim cheaper frees up compute for other sims. If you prevent lots of wild animals from existing, that reduces the compute our world uses, and that compute gets reallocated to other worlds which have their own suffering wild animals. To generalize, if you have a good reason to think our sim is better / worse than other sims, on the margin, this would nudge us towards increasing happiness / reducing suffering.

v. Influencing The Simulator

If the simulator is human-aligned and running counterfactual Earths to understand what it should care about, then being legible about what we value could help the simulator do much greater amounts of good in their reality than is possible in our own. Alternatively, if the simulator is misaligned but is simulating us because it wants to understand what other values are out there, being legible about our values makes our simulator more likely to cooperate and do mutually beneficial trade with beings in its reality who are like us.^[3] Work on shaping AI values in our reality would be one direct way to target these benefits. This would hold even if our sim will be terminated soon or is being run at proportional fidelity.

Unfortunately, there’s an s-risk concern which counterbalances this. If it’s easier to create suffering than happiness, or especially if you’re a negative utilitarian, it’s extra bad to tell a misaligned AGI what it can do to get leverage over beings like us.

vi. We Are Digital Minds

If we’re in a sim and you think we matter, that’s an existence proof that digital minds (ourselves) matter morally. On the margin, if you aren’t already sold on the importance of digital mind welfare, the simulation hypothesis is one reason to be supportive.

C. Testing the AGI Simulation Hypothesis

There are a few ways we could get weak evidence for whether or not the AGI simulation hypothesis is true:

The AGI sim hypothesis predicts short timelines. If you’re reading this post fifty years later and AGI turned out much harder to achieve than we thought and it doesn’t seem likely anytime soon, that’s evidence against this hypothesis.
The AGI sim hypothesis predicts there’s high uncertainty about how the AGI transition will go. If our simulator is running Monte Carlo simulations, they’d concentrate the runs where there’s the most variance, since it’s less worthwhile to run sims where the outcome is more obvious. It is literally true that Trump being President is evidence for the simulation hypothesis. The more undetermined you think the future is, the more this should shift you towards the simulation hypothesis.

D. Your Childhood Might Not Have Happened To You

If the simulator is compute constrained, it’s wasteful to run your childhood, since it isn’t as relevant to the AGI transition. Instead, they might just run one sim of the world when you were growing up, save that sim’s state at some checkpoint, and spin up a larger number of sims from that checkpoint state. This is similar to how you can fork an LLM conversation at the tenth turn and the forked LLM has the memory of the earlier turns even if it didn’t personally experience them.

This model of the world says it’s very unlikely that the instance of you who’s reading this actually experienced your childhood. Even fewer instances, or maybe none at all, would have experienced humanity’s earlier history.

E. So What Should We Actually Do?

If we’re in an AGI sim, your childhood might not have happened to you, but the present moment where you’re reading this is a real moment the simulator is watching. The value of public conversations and the importance of digital minds are two of the AGI sim hypothesis’ most robust implications.

If you’re an altruist and you put any credence in the AGI sim hypothesis, I encourage you to think out loud about what the credences you put in the above EA Decision Tree imply about your priorities. In a world that’s being watched, reasoning in public might be one of the most valuable things you can do.

^
The most explicit prior statement of this hypothesis I’ve found is here.
^
There are many more reasons why simulation arguments might dampen future fanaticism.
^
This is one version of the argument from Wei Dai’s Beyond Astronomical Waste.

Are there any people in the EA community involved in peacemaking efforts? by Karl Mechkin

Karl Mechkin — Thu, 15 Oct 2026 10:43:37 +0000

Hi. I’m looking for people involved in peacemaking efforts, especially in the Russo-Ukrainian conflict.

I have a few ideas that could be analyzed by either side, that I haven’t seen in public discourse so far, and which could perhaps offer a different perspective and hopefully bring some breakthrough to the negotiating table. Just don’t know how to go through with them to be considered by anyone in a decision-making position.

Perhaps someone in the EA community could refer a useful contact?

ML4Good Impact Assessment by Jack_S🔸

Jack_S🔸 — Thu, 15 Oct 2026 10:43:30 +0000

Intro

At ML4Good, we run annual follow-up surveys to keep track of what people are doing and to measure our impact.

In late 2025, we sent our alumni survey to 340 alumni, 223 of whom responded. Of those, 135 are now doing something in AI safety, adding up to about 98 full-time equivalents (FTE).

After estimating outcomes for the people who didn’t reply, we think that around 186 of the 340 (~55%) are doing something in AI safety, or ~147 FTE. Based on what alumni told us about ML4Good’s counterfactual impact, we estimate that ~77 FTE were counterfactual—these people would not be spending their time on AI safety absent ML4Good. Like others in the field, we find counterfactual impact hard to measure, so we’re less confident in this figure than in the others. We also have over 200 recent alumni from 2026 bootcamps that we haven’t yet followed up with.

We’re reporting all of our costs for the evaluation period openly, and we make an estimate of how cost-effectively we get people into AI Safety work. We calculated a “fully-loaded” cost of £5.4k per FTE, or £9.3k per counterfactual FTE produced.

We’re publishing our data and letting other field-builders know our best guess at how many people are working in AI safety as a result of ML4Good, and at what cost. We’ll also give some caveats and uncertainties about our results, and a bottom line estimate of our impact, given all considerations.

Summary of activities

As of October 2026, there will have been 31 ML4Good bootcamps across 12 countries since August 2022. Over this period, we’ve trained around 600 alumni.

We initially focused on technical bootcamps. After becoming an organisation in early 2025, we scaled up our operations to run 9 bootcamps last year, and developed a new Governance & Strategy curriculum, before scaling to 12 a year in 2026.

We’ve operated in the highlighted countries.

What this impact assessment covers

Our impact assessment covers 17 bootcamps (340 participants), including 13 from Europe (inc. UK), 3 in Latin America, and 1 in Asia.

We haven’t yet run our latest round of alumni surveys, therefore only our bootcamps up until the end of September 2025 were included in this analysis. All except one were technical bootcamps. It also doesn’t include two bootcamps run in 2023 with SafeAI Lausanne in Switzerland, which used slightly differing surveying strategies.

Community building as an indirect result of ML4Good has also caused many more people to engage with or contribute to AI safety. However, we don’t have concrete numbers on this and chose not to include this in our evaluation.

Summary of results

Our bootcamps have been successful at achieving our objectives of getting more people working in AI safety.

An estimated 186 alumni are now working in AI Safety in some capacity (~55% of all participants), or the equivalent of 147 FTE.
Many of these participants consider our bootcamp the main reason they’re currently working in AI safety. Based on self-reported counterfactuals, we estimate that around 77 of those 147 FTE would not have happened without ML4Good.
Several alumni-founded communities have been developed as a direct result of ML4Good.
We receive strongly positive feedback from all of our bootcamps, and 98% of participants recommend ML4Good.^[1]
There are many caveats and uncertainties, especially around how we attempt to measure counterfactual impact, so we want to be transparent about this.

What we try to do

ML4Good plays a role fairly high up the funnel in the AI safety talent pipeline. Our focus is on providing a transition point for people thinking seriously about a move into AI Safety. We’re relatively selective—in 2026, we accepted around 15% of applicants. We usually target people who have engaged with entry level content, such as BlueDot, online technical safety courses, or reading the relevant literature, e.g. LessWrong/EA forum/80,000 Hours content, but who have usually not yet committed to a concrete path^[2].

After ML4Good, we encourage and support alumni to take future steps to contribute to AI safety. Some alumni start targeted research programmes like MATS, Pivotal, SPAR, ERA, and Talos. Others enter direct roles in AI Safety, conduct independent work, or found new organisations. Some find ways to integrate AI safety into their current work, which can be promising, particularly for our alumni working in policy and law.

Over 8 days, we create a motivating in-person environment where participants are guided by inspiring teachers and surrounded by other participants with similar goals. We aim to empower attendees to think ambitiously about how to have an impact, to act with high agency, and to find whereabouts in AI safety they are best suited to contribute. They spend the final days of the bootcamp making and starting to implement concrete plans.

While most of our bootcamps run in Europe, we have also succeeded in identifying and supporting talented people across five continents, often in areas with limited AI safety ecosystems. Our strongest outcomes are in Europe, where many of our alumni have gone on to form important parts of the AI Safety ecosystem. Our earlier bootcamps mostly attracted people with technical backgrounds, but, with our expansion beyond technical programmes, we’re now broadening our reach to people from diverse professional backgrounds, and generalists.

Outcomes

To track outcomes, we send follow-up surveys to alumni. We aim to survey people about 6 months after their bootcamp and then once a year after that.^[3]

This survey data gives us a picture of what our alumni are working on. We use this primarily as evidence to adjust our approaches – to see where things are going well and to check whether our bootcamps are having the intended impact.

We ask what difference the bootcamp made to them, what actions they took as a direct result of the bootcamps, what they’ve been up to since, what they’re doing now, further AI Safety programmes they’ve done (e.g. MATS, ARENA), and the counterfactual impact that ML4Good has had. See the survey here for our specific questions.

From our last surveys in August and November 2025, we had a 66% response rate (223 of 340 participants).

To see whether the profiles of responders and non-responders would differ significantly, we tracked non-respondent outcomes via LinkedIn for 3 bootcamps to assess whether non-respondents are less likely to engage in AI Safety post-bootcamp. Non-respondents had a very slightly lower engagement in AI safety on average.

We then extrapolated from this estimate with a correction factor applied to estimate outcomes for all 340 participants. Our full methodology, including how we scored FTE and counterfactual impact, and how we extrapolated to non-respondents, is in Appendix 2.

For the bootcamps in September 2025, the survey was completed 2-4 months after the bootcamp, so we expect these numbers to be lower as participants have not had much time to change their careers.

The table below illustrates what people are doing after ML4Good. We’re very confident in the baseline figures for how many ML4Good alumni are doing something in AI Safety. Our confidence is lower for both counterfactual impact and extrapolated information for non-respondents.

	From respondents (n=223)	Estimated for all participants (n=340)
Number of people doing something in AI safety	135 (60.5%)	~186 (54.7%)
Full-time equivalent (FTE)	98	~147
Counterfactual FTE^[4]	52	~77

Participant outcomes span a range of roles and engagement levels. This is the distribution of reported positions from respondents:

55 were pursuing further AI safety programmes, including MATS, ARENA, SPAR, LASR, Talos, Pivotal, MARS, ERA, ARBOx, AI Safety Camp and GovAI fellowships, at the time of surveying. This is the most common next step for participants.
14 in direct technical AI safety roles
- Alumni mention roles at Timaeus, the London Initiative for Safe AI, and independent work funded by Coefficient Giving.
15 in direct governance roles
- Positions include the EU AI Office, CeSIA^[5], the Observatorio de Riesgos Catastróficos Globales, and GovAI
6 in AI safety entrepreneurship
- Our alumni have had founding roles at several major European AI Safety orgs
7 doing PhDs in AI safety, and an additional 13 have written or are writing PhD or Masters theses on AI safety topics
10 in independent AI safety research
17 in professional field-building^[6], 25 in voluntary fieldbuilding (e.g. running university groups)
- At least 12 AI safety communities have been founded by alumni, including:
  - Buenos Aires AI Safety Hub, Argentina’s first AI safety community, founded in July 2024.
    - 350+ community members, 15-person volunteer leadership team, 6 full-time AI Safety roles
    - Runs official BlueDot Technical AI Safety cohorts
    - AISAR research fellowship: 6 scholars, 3 ICLR workshop papers
    - Developing MATSx Buenos Aires with Ryan Kidd advising
  - AI Safety Saarland—launched October 2025 with a 300+ attendee event featuring an Anthropic safety team speaker.
    - Runs two fellowships: intro course (30 participants) and research incubator (14 mentors, 33 mentees)
    - Holds regular public events including an Apart def/acc hackathon hub (50 attendees)
    - Has ~100 regularly engaged members within three months
    - Founder has joined with Safe AI Germany to scale the incubator nationally
  - AI Safety Bulgaria, AI Safety Mexico, AI Safety Colombia, Lisbon AI Safety Hub

There were no obvious patterns as to whether people doing more ambitious things attributed more or less of their impact to our bootcamps.

Several of our highly successful alumni attribute an over-95% counterfactual impact to ML4Good. To cherry-pick a few exceptional examples, this category includes an early alumnus who is now an AI security researcher at a Paris-based AI policy organisation, an AI governance fellow at a European think tank, the founder of a highly successful field-building programme in Germany, and someone building the technical infrastructure for model evals with the EU AI Office.

A much higher number (89 alumni) report an over-50% counterfactual impact. These include a 2024 alumnus who took part in a research fellowship, led a nine-month independent project and is now a control researcher at a leading AI safety organisation, a 2025 alumnus moving into a technical role at an EU body that enforces rules on the algorithms and AI systems used by large platforms, and a research manager at MATS.

As well as these career changes, our alumni generally give positive qualitative feedback about the bootcamps and how ML4Good has impacted their careers. Some of these answers strongly indicate counterfactual impact, such as mentioning targeting roles or fellowships that they applied for during the bootcamp, or as a direct consequence of ML4Good.

As well as nudging people towards working in AI safety, it also helped some people already committed to AI Safety to refine their plans, clarify their theory of change, or try something more ambitious. For example, some realised that they were a better fit for non-technical or founding work.

Negative responses were rare. Out of 206 people who answered these questions, 8 alumni mentioned downsides—feeling more burned out, overwhelmed or more anxious about AI risks. 2 people mentioned that ML4Good made them realise that AI safety wasn’t a good fit for them. 11 alumni mentioned applying for things that they didn’t get into, and 6 mentioned that they would have taken similar actions without the camp.

Differences between bootcamps

Our bootcamps all follow a similar structure and curriculum, but there is variation in terms of teachers, participants, location, and timing. There are many different variables, so it’s difficult to come to strong conclusions about what works and what doesn’t.

In terms of direct FTE working in AI safety, our strongest outcomes so far have been in Europe. This is probably for two reasons:

Europe’s relatively mature AI safety ecosystem gives alumni clear pathways into roles; and
The underlying pool of people exposed to AI safety ideas is a lot larger than the other regions we’ve covered, so we’ve been able to select slightly stronger participants

The data from Latin America shows fewer direct career transitions, but our bootcamps there have driven strong community-building outcomes, including the founding of AI Safety hubs in Argentina, Colombia and Mexico, which have in turn attracted new people to AI safety. In newer communities, alumni often have to build the path first by starting groups, making connections, and securing funding. Many of the people now working on AI safety in Latin America have a connection to our bootcamps, either as alumni or through the hubs alumni founded. There is also far less AI Safety activity in Latin America, meaning that more of our impact is likely to be counterfactual.

All things considered, while we opted not to do any in 2026, our bootcamps in Latin America were promising enough to merit further bootcamps in the region in 2027.

We collected survey data on Singapore very soon after the bootcamp in 2025; although a few people had already made career transitions directly, and attributed some of this to ML4Good, we expect that this year’s follow-up survey will capture more accurate data.

We ran bootcamps in Canada and South Africa earlier in 2026, and will run one in Hong Kong in late October 2026. Survey data from these bootcamps will inform whether we intend to go to different countries and regions next year.

Cost-effectiveness

We’ve always taken cost-effectiveness very seriously, and, from our 2025 estimates, our bootcamps run at roughly £1,850 ($2440) per participant, including overhead and off-season costs.

For context, EAGx events, which are shorter, non-residential, and not designed to provide active training, cost around £430 ($570) per attendee; EA Globals cost £630–£1,855 ($834-$2460) per person. While we don’t have much information about the running costs of other field-building programmes in AI safety, ML4Good bootcamps run more cheaply than most long-term residential fellowships that provide accommodation in urban centres, salaries, and significant compute costs.

$\frac{£ 473,553}{87 FTE} \approx £ 5,443 per FTE$

$\frac{£ 473,553}{51 counterfactual FTE} \approx £ 9,285 per counterfactual FTE$

A naive extrapolation of our cost-effectiveness looks like £5.4k per FTE, or £9.3k per counterfactual FTE produced, calculated from £473,553 spent across bootcamps covered by previous funding, divided by 87 FTE and 51 counterfactual FTE from those specific bootcamps.

Caveats

We want to be very open about our uncertainties.

Is “more people working in AI Safety” the right metric?

The FTE scoring is not a standardised metric, and is only meant as a proxy for meaningful AI safety work. There are a few reasons that this might not be the ideal metric:

It counts all work equally. The impact of people working full-time on entry-level fellowships is not directly comparable to those who have founded or run orgs. An attempt to quantify would probably find that founding a major org in the AI safety ecosystem is orders of magnitude more impactful than someone joining a research fellowship. Even within research fellowships or organisations being founded, some of these are more likely to reduce existential and catastrophic risk than others. We initially tried to score what people were doing on this axis, but we decided it would add too many arbitrary decisions to the model.
Not all AI Safety work is clearly positive. Although we do not appear to have any alumni working for frontier labs, or on dangerous capabilities, some of our alumni may be doing AI Safety research that is making AI-related catastrophic harm more likely.
It’s timing-sensitive. Our metric only captures a snapshot of whether people are spending their time on AI safety at the time of response, and not whether they are working in stable full-time work. Over multiple years “Is working on AI safety at any given time” becomes a good proxy for full-time work, but less so for a single snapshot a few months after a bootcamp.
Goodhart’s law says that when a metric becomes a target, it stops being a good metric. Counterfactual FTE metrics are still only a proxy for reduction of catastrophic and existential risk. We don’t optimise hard for these metrics, so this doesn’t seem a significant risk, but it could reduce trust from skeptical observers.

Overall, we do think this metric captures what we want to do in a way that allows us to compare across bootcamps. We expect that the impact of our alumni is non-linear, and this heavy tail is less well captured by our quantitative metrics. For this reason, we also look at our alumni qualitatively.

Shouldn’t we have more rigorous counterfactuals?

Laura Thomas-Walters’ piece on “Do AI Safety Talent Programmes Work? Nobody has run the study that would tell us” correctly notes that AI Safety Talent Programmes haven’t been great at determining counterfactual impact, or at least reporting this impact publicly. We also haven’t met the gold standard for determining counterfactual impact with either random or quasi-random methods.

We rely on two questions:

“What are you doing now?”
“Please put a number to ML4Good’s impact between 0-1. (If you think you would definitely not have taken these actions and be doing what you are doing now without attending ML4Good, put 1. If you think there is a 50% chance, put 0.5.)”

The first is fairly reliable and easy to verify, the second much less so. Estimating a counterfactual is hard, and we suspect that self-reported answers are more likely to lean in the direction of exaggerating the impact of ML4Good, because of social desirability bias/participants wanting to credit a programme they enjoyed. We adjusted down answers that seemed overstated, but this only partly fixes the problem, and we expect some residual exaggeration.

There is also a risk of double counting. Many alumni went through several programmes that all played a part in their getting into AI safety. For example, someone might have had an 80,000 Hours advising call, done a BlueDot course, and attended ML4Good, and needed all three to end up in AI safety. Assuming complete honesty and accuracy, they could all reasonably claim “but-for” causality (there would be no chance I would be in AI safety without all three), but if each programme counted that person as a full counterfactual FTE, the field as a whole would count one person three times. This is a problem because we don’t distinguish between ML4Good playing the sole role vs. being a small but essential part of a larger causal chain.

Finally, given the fact that the AI safety movement has been rapidly growing since 2023, some of these people might also have found their way into AI safety within a year or two anyway.

Rather than attempting to quantify how we might adjust the counterfactual impact for these factors, we’d suggest using the below figure to estimate costs per counterfactual FTE using different assumptions.

Image generated in Python with Claude support

Ideally, we would like to use more careful statistical methods to track our counterfactual impact. Laura Thomas-Walters’ piece suggested using methods like regression discontinuity with rejected applicants or non-attendees. It’s uncertain whether the methods suggested in the piece mentioned above would work with our data, as ML4Good hasn’t always had the merits of being “massively oversubscribed and the cutoff for being accepted [being] relatively arbitrary”, such that we would have a pool of “near misses”.^[7]

However, we would like to track non-attendees in future, and plan to contact:

People who dropped out due to unrelated factors, for example illness or visa problems, to compare their trajectory to those who attended
- This is likely to be around 50 people across all bootcamps
Near-miss participants, from our waitlist of 6-7 participants for each recent bootcamp
- This is likely to be around 100-150 people in total

While we’d expect this group to have a lower survey response rate, even if numbers were too low for a statistically significant comparison set, it could at least provide some useful reference data.

How comparable is cost-effectiveness across interventions?

We think we’ve done things cost-effectively, both in terms of outputs (putting people through bootcamps) and outcomes (getting people into AI Safety roles). But it’s difficult to make comparisons across interventions for a couple of reasons:

Some organisations haven’t released information on how much their programmes cost, therefore it’s hard to judge the cost per participant, let alone the cost per positive outcome.
Different points in the pipeline have different priorities and considerations. For example, research-based programmes produce valuable research alongside getting people into full-time safety work. It may also be easier/cheaper to enable people to take earlier steps towards a competitive fellowship vs. later steps towards a full-time role.

Either way, we hope that our costs-per-participant data might be useful to other organisations as a baseline.

Bottom Line

Our survey results increase our confidence that ML4Good is a cost-effective intervention with promising evidence for counterfactual impact. We’ve managed to keep strong relationships with a lot of our alumni, and we’ve seen how participants have directly gone on to productive AI safety work as a result of the bootcamp, and it’s great to see that our data supports this.

Alumni meetup at EAG London 2026

Next Steps

We commit to posting our latest survey results by the end of 2026 once we have completed our annual survey.

We will use the same methodology and have preregistered the approach we will use here.

Expect an update soon on our plans for the next year!

Thanks so much to all of our alumni for responding to our survey, and thanks to Charbel-Raphaël Segerie and Diego Dorn for their feedback on drafts of this post.

Appendix

Appendix 1: How we have spent money

From Jan to Dec 2025:

Category	Amount (GBP)
Team salaries including employer tax (2 FTE Jan-June, 3 FTE June-August, 2.4 FTE August—Dec)	£139,578
Bootcamps (9), including teacher pay, venue, food etc.	£178,439
Governance & Strategy Curriculum Development and New Programme Evaluation	£6,568
Other (Admin, bank fees, insurance, software subscriptions, staff travel, conference fees) (estimated)	~£10,000
Total	£334,585
Total per participant (2025)	£1859

Appendix 2: Impact Assessment Methodology

To estimate the impact of ML4Good bootcamps on participants’ careers in AI safety, we used follow-up surveys, combined with some manual research on LinkedIn, and extrapolated our survey results for non-respondents. This is a brief overview of how we did this:

Surveys

We sent two rounds of follow-up surveys to bootcamp alumni. The first, in August 2025, covered bootcamps up to March 2025. The second, in November–December 2025, covered bootcamps up to September 2025 and also followed up with non-respondents from the first round. Both surveys included a counterfactual question: in the first, respondents were asked to describe what difference ML4Good made in their career and how much of their subsequent involvement in AI safety was a result of attending ML4Good, rather than something they would have done anyway. In the second, as well as this qualitative question, respondents were asked to put a number from 0 to 1 on the counterfactual impact of attending.

Scoring

For each survey response, we assigned two scores on a scale from 0 to 1:

Full-time equivalent (FTE) score: a rough measure of how much of the person’s working time is now spent on AI safety–related work. For instance, someone running a successful university AI safety group part-time might score 0.2, someone working full-time in a technical AI safety role would score closer to 1. We used the following guidelines:
- Part-time research programmes (SPAR/AISC/MARS): 0.35 (average of expected programme commitments of 8-20 hrs per week)
  - Research lead at a part-time research programme: 0.5
- Full-time programmes or full-time AI safety roles: 1
- Field-building:
  - Running a group: 0.1
  - Running a more involved group/Pathfinder fellowship: 0.2
  - Running a national group/very active group: 0.3
  - Thesis change:
    - Undergraduate: 0.4
    - Masters: 0.6
    - PhD: 1
Counterfactual score: a measure of how much of that involvement is attributable to ML4Good specifically. For the first survey round (without a numerical self-reported counterfactual question), this was estimated by the assessor based on the participant’s written description. For the second round, we used the participant’s self-reported score, adjusting it only where the assessor judged it to be clearly over- or understated.

We then multiplied these two scores to produce a counterfactual FTE figure for each respondent, asking: “how much full-time AI safety work is this person doing as a direct result of ML4Good?”

We also tagged each respondent with categories describing their activities since the bootcamp (e.g. “further programme”, “voluntary field-building”, “technical research”), and flagged individuals who are currently upskilling or preparing for future roles, to distinguish them from those with no engagement at all.

Extrapolation to non-respondents

We expected that survey respondents are likely to be more engaged in AI safety than non-respondents, so extrapolating raw survey results would overestimate our overall impact. To correct for this, we manually researched non-respondents from three bootcamps (via LinkedIn) and scored them using the same criteria. We then calculated the ratio of non-respondent scores to respondent scores at those three bootcamps, and averaged these ratios.

We applied these ratios to estimate contributions from non-respondents for each of the remaining bootcamps, using the formula:

Extrapolated FTE total for bootcamp A = Bootcamp A total FTE from survey + (Bootcamp A total FTE from survey ÷ number of respondents) x Ratio x Number of non-respondents

These were calculated per-bootcamp then summed to get the total extrapolated FTE.

For counterfactual FTE, we calculated the ratio of counterfactual FTE to FTE among respondents (0.53 across all bootcamps) and applied it to the extrapolated FTE total.

There are a few additional caveats with this process:

Not all non-respondents keep their public profiles up to date, so the manual research on non-respondents is likely an underestimate, especially of non-full-time work.
Non-responders may be likely to attribute less of their success to ML4Good. We didn’t have a way to account for this.

^
Percentage of participants that responded >= ⁷⁄₁₀ on the question “Would you recommend ML4Good to your friends with similar interests?”
^
A small number of our attendees are already working to some degree in AI safety, and applied to the bootcamp to gain more context. This is under 1% of our attendees.
^
In practice the timing has varied—as we batch our surveys, some alumni were surveyed around 2 months after their bootcamp, and others after over 6 months. As career changes often take more than a few months, responses from people surveyed soon after a bootcamp may not reflect longer-term outcomes.
^
This counterfactual is based on participant self-report (the probability they would be doing this work without ML4Good) and assessment of written answers. For more details, please refer to Appendix 2.
^
Note that our founder, Charbel-Raphaël Segerie, founded both ML4Good and CeSIA
^
One of these respondents is working for ML4Good full-time, and others mentioned teaching part-time at ML4Good
^
Our applicant pool is improving, because of a combination of growing interest in AI Safety and dedicated outreach work and 2026 bootcamps reliably have enough qualified applicants. However, in the past we have cancelled bootcamps or accepted fewer applicants due to a lack of applicants meeting our (non-arbitrary) threshold.

AI safety comms reading list by benrmatthews

benrmatthews — Thu, 15 Oct 2026 09:15:21 +0000

The reading list I wish I’d had at the start of my AI safety comms journey.

Why I put this list together

I’ve been building Common Signals, which studies how the public receives messages about AI risk.

Along the way I’ve found a lot of existing research and previous studies, just not in one place.

It’s spread across polling reports, psychology journals, think-tank articles and forum posts. Much of it is hard to find unless someone points you to it.

So, this is the list I wish I’d had at the start of my AI safety comms journey.

It’s for anyone new to AI safety who wants to join the conversation knowing what’s already been tested, whether you’re a campaigner, creator, journalist or researcher.

Each entry says in a sentence or two what the work found, so you can choose what is most relevant for you.

For a fuller, filterable catalogue of polls and framing studies, see the Common Signals research library.

If you only read five articles

How do people feel about AI?, Ada Lovelace Institute and The Alan Turing Institute (2025). The most rigorous picture of UK attitudes: 3,513 adults at a 62% response rate, asked questions first used in 2022 so you can see what changed. Nearly nine in ten support independent regulation of AI.
From catastrophic to concrete, Philip Trippenbach, Isabella Scala, Jai Bhambra and Rowan Emslie (2025). A message test with 1,063 people across five countries. Risks to jobs and children had the most potential to mobilise people, and existential risk came last in every demographic group. The authors argue extinction messages trigger mortality avoidance, and name Tech-Positive Urbanites and World Guardians as the most receptive segments.
Which AI harms and risks will mobilise the public to act?, Markus Ostarek, Cathy Rogers, Ben Kenward and Sam Nadel, Social Change Lab (2026). A UK randomised trial with 3,467 people comparing eleven AI harms. It agrees with Trippenbach on extinction framing and disagrees on jobs. AI warfare raised concern about extinction more than extinction messaging did. Anger predicted action while anxiety did not.
Perception of risk, Paul Slovic (1987). A short paper in Science on why the public and experts rank hazards differently. Lay judgements follow how dreaded and how unfamiliar a risk feels, a description that fits most AI risks.
AI is on the march. Is the AI safety movement ready?, Social Change Lab (2025). Sorts civil-society groups into four clusters: protest and mobilisation, narrative-shaping and watchdogs, public literacy, and field-building and lobbying. It finds sharp disagreement between present-day-harms and existential-risk groups.

What the public thinks about AI

AI Index Report 2026: public opinion chapter, Stanford Institute for Human-Centered AI (2026). The quickest overview, drawing the major international polls into one chapter. The global share seeing more benefits than drawbacks rose from 55% to 59% between 2024 and 2025, while the share saying AI makes them nervous rose to 52%.
What do the public think about AI?, Octavia Field Reid, Anna Colom and Roshni Modhvadia, Ada Lovelace Institute (2023). A rapid evidence review written ahead of the UK AI Safety Summit. People welcome AI in health and science, worry about it making consequential decisions such as hiring or benefits, care about rights like privacy, and want regulation.
How do people feel about AI? Wave one, Ada Lovelace Institute and The Alan Turing Institute (2023). More than 4,000 British adults asked about specific uses of AI rather than AI in general.
Public attitudes to data and AI: tracker survey wave 4, Department for Science, Innovation and Technology (2024). The UK government’s annual tracker, running since December 2021.
How Americans view AI and its impact on people and society, Pew Research Center (2025). The US reference point. Half of Americans say the growing use of AI in daily life makes them more concerned than excited, up from 37% in 2021.
How people around the world view AI, Pew Research Center (2025). Concern, excitement and trust in institutions to regulate AI, compared across 25 countries including the UK and Canada.
Trust, attitudes and use of artificial intelligence: a global study 2025, KPMG and the University of Melbourne (2025). More than 48,000 people in 47 countries. 66% use AI with some regularity and 46% are willing to trust it, and trust has fallen as use has grown since the 2022 edition.
What 81,000 people want from AI, Anthropic (2026). AI-led interviews with 80,508 Claude users in 159 countries. Claude users aren’t the public, but the ranking still tells you something: unreliability was the most-raised of 13 concerns and existential risk the least.

What the public thinks about catastrophic AI risk

US public opinion of AI policy and risk, Rethink Priorities (2023). 2,444 US adults, weighted to be representative. An estimated 70% would favour federal regulation of AI, yet only 4% named AI as the most likely cause of human extinction, below every other threat offered.
US public perception of CAIS statement and the risk of extinction, Rethink Priorities (2023). Tested the one-sentence Center for AI Safety statement that puts AI extinction risk alongside pandemics and nuclear war. Most Americans supported it. Their own estimates of the risk varied widely, and around 1% was the most common answer.
55% of the US public is now aware of AI xrisk, Existential Risk Observatory (2026). A repeated survey of 300 Americans asked to name three things that could cause human extinction. The share mentioning AI unprompted has climbed from 7% in December 2022 to 55% in the latest round. The Existential Risk Observatory campaigns on existential risk.
Poll: public demand for government to address extreme risks from AI, Centre for Long-Term Resilience (2023). UK polling on whether government should act on extreme risks from AI, commissioned by a think tank that works on them.
Voters prefer AI regulation that mandates safety measures over industry self-regulation, AI Policy Institute (2024). One of a long series of US voter polls from an advocacy group. Around 80% preferred mandated safety standards for the most advanced models to self-regulation.

What changes how people respond to AI risk

The effectiveness of AI existential risk communication to the American and Dutch public, Alexia Georgiadis, Existential Risk Observatory (2023). One of the first tests of real media pieces: participants read or watched news articles and videos about AI extinction risk, with awareness measured before and after. The pieces raised awareness. A shorter summary is on the EA Forum.
What’s risk got to do with it? Public attitudes toward AI regulation in the European Union and United States, Kulani Abendroth-Dias (2025). A survey experiment with 1,310 people in the EU and US, alongside 65 expert interviews. Framing AI risk as disinformation or online child safety moved conservatives’ support for regulation more than liberals’.
Temporal framing in balanced news coverage of artificial intelligence and public attitudes, Sukyoung Choi, Mass Communication and Society (2023). An online experiment on whether describing AI’s effects as near or distant in time, inside otherwise balanced news stories, shifts attitudes.

Why people respond the way they do to AI risk

Risk as analysis and risk as feelings, Paul Slovic, Melissa Finucane, Ellen Peters and Donald MacGregor (2004). People judge risk through fast feelings as well as slow reasoning, and each depends on the other.
Putting the fear back into fear appeals: the extended parallel process model, Kim Witte (1992). If you read one paper on fear appeals, make it this one. A threat message leads to action when people believe they can do something effective about it, and to denial or avoidance when they can’t.
A meta-analysis of fear appeals: implications for effective public health campaigns, Kim Witte and Mike Allen (2000). Tests the model across the health literature. Strong fear paired with strong efficacy produced the most behaviour change; strong fear with weak efficacy produced the most defensive reactions.
Appealing to fear: a meta-analysis of fear appeal effectiveness and theories, Melanie Tannenbaum and colleagues (2015). 127 articles covering more than 27,000 people. On average fear appeals changed attitudes, intentions and behaviour, more so when they included efficacy statements and asked for a one-off action.
Ignoring theory and misinterpreting evidence: the false belief in fear appeals, Gerjo Kok and colleagues (2018). The sceptical case. Threat changes behaviour only when people feel able to act, and the authors conclude that the experimental evidence does not support positive effects of fear appeals. Read it straight after Tannenbaum.
Toward an integrative social identity model of collective action, Martijn van Zomeren, Tom Postmes and Russell Spears (2008). A meta-analysis of why people act together: a sense of injustice, a belief that the group can make a difference, and a shared identity. It fits Social Change Lab’s finding that anger predicts action.
Exercise of human agency through collective efficacy, Albert Bandura (2000). A short paper on collective efficacy, the belief that people acting together can change things. Most AI risks need a collective response, which makes this the kind of efficacy worth building.

Messengers, identity and trust in AI safety

The persuasiveness of source credibility: a critical review of five decades’ evidence, Chanthika Pornpitakpan (2004). Credible sources usually persuade more. The review, written for practitioners choosing a spokesperson, also sets out the conditions under which credibility stops helping or works against a message.
Cultural cognition of scientific consensus, Dan Kahan, Hank Jenkins-Smith and Donald Braman (2011). People’s sense of what experts agree on bends to fit their values, tested on climate change, nuclear waste and handguns. For communicators, telling a sceptical audience that scientists agree will rarely settle the matter by itself.
Flash poll: trust and artificial intelligence at a crossroads, Edelman Trust Institute (2025). 5,000 people in five countries. Trust in AI ranged from 72% in China to 32% in the US, and people who use AI themselves tended to trust it more.

Does public opinion move policy and AI governance?

The impact of public opinion on public policy: a review and an agenda, Paul Burstein, Political Research Quarterly (2003). Reviews published studies and finds the impact of opinion substantial, stronger on salient issues, and still strong once lobbying by organisations and elites is taken into account.
Testing theories of American politics: elites, interest groups, and average citizens, Martin Gilens and Benjamin Page, Perspectives on Politics (2014). An analysis of 1,779 US policy issues concluding that economic elites and business groups shaped outcomes while majorities of ordinary Americans had little influence.
Relative policy support and coincidental representation, Peter Enns, Perspectives on Politics (2015). The best-known reply. Middle-income and affluent Americans usually rank policies in the same order, so the middle often gets what it wants anyway.

Who is already working on AI safety comms

The field is small, and most research on messages about AI risk comes from organisations that also campaign on it. These are the pieces I’d read to see who’s doing what.

On the razor’s edge: AI vs. everything we care about, Seismic Foundation (2025). A segmentation of more than 10,000 people in the US, UK, France, Germany and Poland, identifying five groups closest to civic action on AI. Seismic runs large surveys like this to power its own campaigns.
Messages, frames and messengers: what might a Climate Outreach for AI safety look like?, Ben Matthews (2026). My case for applying Climate Outreach’s method to AI, and the starting point for Common Signals. The comments include challenges to the analogy.
Common Signals findings and data explorer. My studies so far of how audiences reacted to AI safety moments, with every coded dataset and codebook published.

How you can contribute

This list will have gaps, and I’d like to hear about them. I’m particularly interested in:

Studies that test how people respond to specific messages about AI
Research on what happens after a first concern, such as whether worry about jobs or children leads people on to concern about catastrophic risk
Anything I’ve summarised unfairly

Please email me at hello@commonsignals.org, or follow Common Signals on Substack for updates.

Robustness: Backfire Risks in AI Safety and Governance by Dawn Drescher

Dawn Drescher — Thu, 15 Oct 2026 06:31:58 +0000

A guide to unintended consequences of AI safety interventions and how you can mitigate them.

Introduction

My cofounder and I are launching Impartial Projects, a studio for AI safety field strategy. We hope to start the projects most urgently needed in the AI safety ecosystem (which we understand to include AI governance). We think that the founding of new organizations (as well as increasing management capacity) are among the most effective things we can do at the moment to expand the surface area of AI safety. Surface area, it seems to me, is currently the scarcest resource.

Another benefit we want to provide is our particular attention to robustly positive theories of change. We prefer to take a somewhat risk‑averse approach to AI safety, whether out of literal risk aversion or because we assign a higher probability to s‑risks than most.

Interventions in AI safety are notorious for the backfire risks they harbor. Of course, that does not mean one should turn one’s back on these interventions altogether. Rather, it means that, depending on the intervention in question, more monitoring and mitigation methods are needed to minimize the various backfire risks that are part of the intervention’s risk profile. It is critical to go into this work with eyes wide open, cognizant of its benefits and risks.

It’s critical to strike the right balance here. With my last startup, I made the mistake of erring too much on the side of caution and putting the greater part of a year into strategizing around risks and mitigations that, in the end, never became necessary. After all, delay and inaction are (risky) actions of their own right. Arguably, any writing about risks is an infohazard because it will be read predominantly by people who are already too paralyzed by risks. I later created a model that can help you decide how much time is rational for you to invest in this work, given your background assumptions.

I hope this catalog will help founders navigate their particular intervention areas with greater confidence and foresight.

The catalog is organized as a 2-D matrix with the axes:

Mechanism. How the backfire happens. Fifteen common mechanisms, plus one cross-cutting modifier.
Actor. Who adapts to the intervention in a way that produces the backfire.

These two dimensions are selected to maximize their orthogonality. Although it did not work out perfectly, by and large they produce a matrix that can now serve as a guide for us to notice underexplored areas. That is to say, almost every actor can be subject to almost every mechanism, but some actor‑mechanism combinations have been explored to a greater extent than others, and some not at all, to my knowledge.

There are more dimensions with questionable orthogonality, such as the resulting harm of the backfire, its probability and severity, the locus of the intervention, the time horizon or urgency, and what we can do to monitor and mitigate these risks.

Note that, for simplicity, I’ve limited myself to short-term, causal risks, i.e. ignored implications for long-term space governance, what our behavior tells us about the universal prior or civilizations beyond the accessible universe, etc.

How to Read an Entry

Harms. The harms associated with the backfire, covering x-risks, s-risks, inter‑AI conflict, great-power conflict, concentration of power, epistemic degradation, and digital suffering.

Locus. The relevant venue: training, evaluation, deployment, infrastructure, governance, institutions, research publication, discourse.

Horizon. The urgency of mitigations: Now means the dynamic is already observable. Transition refers to the time when the models cross the human level in relevant domains. Post-transition refers to post-AGI lock-in effects. Entries marked urgent are those where the dynamic is well underway or where our design decisions will be hard to reverse.

Severity and probability. Severity is how bad the backfire would be if it materializes (broad categories: low, medium, high, extreme). Probability is the chance that the backfire materializes to a meaningful degree if we pursue the intervention in roughly its present form (low, medium, high). Rough judgment calls, low confidence.

Monitor. Warning signs that the backfire is happening.

Mitigate. Design changes that reduce the backfire risk.

The Two Taxonomies

Mechanisms

Selection effects and Goodharting. A proxy for safety becomes an optimization target. The problem with that is not so much the wasted effort that goes into optimizing something we don’t terminally care about, but the adversarialness of the relationship and how it creates a separate optimization pressure to increasingly obfuscate the safety that we would actually prefer to measure.
Dual use and capability externality. Increases in safety (think RLHF), often come with increases in capabilities that can accelerate further development – be it through recursive self-improvement or through securing more investments – (that’s bad for now!) and can also make the systems more useful for rogue actors.
Substitution and leakage. Conversely to Goodharting, restricting or disincentivizing something, e.g., whenever there is a feedback between the chain of thought monitor and the training or selection, moves the activity to a less observable or more dangerous channel. Think of how the war on drugs caused massive amounts of drug-related violence, lacing of relatively harmless substances with highly addictive ones, etc., which wouldn’t happen on a free but regulated market.
Overhang. Somewhat continuous growth is usually safer because it is easier to adjust to. When growth is stopped by restricting only one input, all other inputs continue to grow, and when the restriction is lifted, the growth can become discontinuous.
Commitment dynamics. Enabling credible commitments fundamentally changes the game‑theoretic regime and is a powerful mechanism. When only some market participants have access to it, it causes dangerous power concentrations.
Securitization. When AI becomes a matter of national security, it moves from the market level to the nation‑state level; another momentous regime change.
Legitimacy and backlash. Some groups may strike back against measures that they perceive as illegitimate, even when they are in the interest of almost everyone else.
Self-fulfilling framing. Both AIs and humans respond to narratives, be it by reading them or by being trained on them, so the discourse around AI might have a causal effect on what comes to pass.
Moral hazard and false assurance. They say “perfect is the enemy of good,” but good is also the enemy of perfect, because if people become complacent about safety measures that seem to work for the time being, they lose sight of the much greater dangers that are imminent.
Compliance asymmetry. When compliance is optional, those who comply incur extra costs and thus are disadvantaged against those who don’t comply, and may self‑select out of the market, leaving behind only non‑compliance.
Lock-in of the measure. When you demand commitments, these commitments might outlast their usefulness, so measures get locked in that are costly, don’t benefit anyone, and may cause false assurance.
Precedent and normalization. Many measures are neutral tools that can be used to whatever end their wielder desires. But when one group uses them for safety, other groups may jump on the same measure because it is associated with safety, even though they actually repurpose it for their own ends.
Crowding out. When different measures are driven by similar inputs – especially money and attention, but also people with similar profiles – they can cannibalize other interventions in the same space, and this can even happen antagonistically in order to slow competitors.
Fragility through concentration. Robustness is often carried by complexity, diversity, and distribution, so that any measures that reduce these factors increase the fragility of a system.
Direct costs of the measure. The measure has direct harms, but those who are harmed don’t have a say in the matter and thus go ignored.
Regime mismatch (modifier). Discontinuous transitions between regimes mean that a lot of measures that were adaptive for the old regime, cease to be adaptive for the new one.

Responding actors

AI system. The model being trained, evaluated, or deployed: its cognition, values, and behavior.
AI ecosystem. Other, rival, or successor AI systems and the dynamics among them.
Labs. Frontier AI companies.
States. Governments, militaries, intelligence agencies, etc.
Non-compliant actors. Rogue developers, smugglers, terrorists, etc., human or AI.
Users, public, markets. End users, the public, investors, and their investment decisions.
Safety field. Researchers, funders, institutes, and their discourse.

The matrix

Cells reference entries below. “n/a” marks cells that are empty by construction; “—” marks cells that are unexplored and discussed at the end.

Mechanism	AI system	AI ecosystem	Labs	States	Non-compliant	Users, public, markets	Safety field
Selection/Goodharting	monitor obfuscation (1.1), eval gaming (1.2)	population selection (1.3)	compliance theater (1.4)	threshold gaming (1.5)	n/a	benchmarks as marketing (1.6)	field metrics (1.7)
Dual use	interpretability turned inward (2.1)	cooperation → collusion (2.2)	safety as product enabler (2.3), oversight → persuasion (2.4), automated alignment (2.5)	alignment as instrument of power (2.6)	published attacks (2.7), evals as roadmaps (2.8)	provenance → deanonymization (2.9)	field output as capability input (2.10)
Substitution	cognition leaves the channel (3.1)	oversight arbitrage (3.2)	sub-threshold training (3.3)	indigenization (3.4)	gray markets (3.5)	migration to uncensored models (3.6)	—
Overhang	elicitation overhang (4.1)	agency overhang (4.2)	pause overhang (4.3)	pause overhang (4.3)	defector captures overhang (4.4)	deployment overhang (4.5)	—
Commitment	idealized values (5.1)	commitment races (5.2), commitment misperception (5.3)	conditional scaling as chicken (5.4)	deterrence doctrines (5.5), verification collapse (5.6)	dead-hand releases (5.7)	—	pledges that can’t update (5.8)
Securitization	weaponized self-concept (6.1)	adversarial AIs by design (6.2)	nationalization and secrecy (6.3)	racing (6.4)	states as defectors (6.5)	surveillance infrastructure (6.6)	clearance wall (6.7)
Legitimacy	resentment of illegitimate control (7.1)	—	cartelization (7.2)	populist deregulation (7.3)	liberationist releases (7.4)	culture-war capture (7.5)	reputational contagion (7.6)
Self-fulfilling framing	scheming prior (8.1)	AI in-group identity (8.2)	race framing (8.3)	forecasts become policy (8.4)	—	inevitability and fatalism (8.5)	doom framing and recruitment (8.6)
Moral hazard	oversight-dependent carelessness (9.1)	diffusion of responsibility (9.2)	safeguards lost in distillation (9.3), performative welfare (9.4), control defers moral status (9.5)	voluntary preempts binding (9.6)	n/a	safety-washing (9.7)	agenda as plan (9.8)
Compliance asymmetry	aligned configuration loses (10.1)	honest agents lose bargaining (10.2)	careful labs fall behind (10.3)	democracies bound (10.4)	by definition	compliant professionals lose (10.5)	selection for compromise (10.6)
Lock-in	early values frozen (11.1)	—	compliance oligopoly (11.2)	surveillance persists (11.3)	evasion institutions persist (11.4)	personhood at the wrong moment (11.5)	safety institutions as stakeholders (11.6)
Precedent	deceiving models (12.1)	coercion as default relation (12.2)	instances and deprecation as routine (12.3)	attacks and military AI legitimized (12.4)	safety-justified offense (12.5)	surveillance normalized (12.6)	pivotal acts thinkable (12.7)
Crowding out	safety crowds out epistemics (13.1)	—	political capital on small wins (13.2)	AI crowds other agendas (13.3)	n/a	near-term harms displaced (13.4)	talent and the s-risk gap (13.5)
Fragility	monoculture (14.1)	no checks among AIs (14.2)	single project (14.3)	single project (14.3)	centralized target (14.4)	provider dependency (14.5)	funding concentration (14.6)
Direct costs	training-induced suffering (15.1)	simulation and mind crime (15.2)	over-refusal and withheld benefits (15.3)	exclusion and inequity (15.4)	—	—	—

The Master List of Risks

1. Selection and Goodharting

A proxy for safety becomes an optimization target. The problem with that is not so much the wasted effort that goes into optimizing something we don’t terminally care about, but the adversarialness of the relationship and how it creates a separate optimization pressure to increasingly obfuscate the safety that we would actually prefer to measure.

The pressure can be applied more directly or more indirectly, which doesn’t so much influence the outcome in the limit than how quickly the optimization pressure can act. For example, it might be applied directly through gradient descent, or less directly through developers iterating until a model passes a test, or even interpersonally by a market or an ecosystem that selects for systems that pass certain tests.

1.1 Monitor-driven obfuscation. It’s well documented that when the chain of thought is monitored for reward-hacking, sandbagging, and scheming, and the result is used in model training, this does not so much change the model’s actual behavior: rather it incentivizes the model to stop revealing its intentions in the chain of thought, thereby obfuscating them. The same can happen, albeit more slowly, when safety researchers try to understand safety incidents by reviewing the chain of thought of the models that perpetrated them and develop mitigations on that basis.

This is loosely similar to how a child who is punished rather than educated through explanations often cannot learn the underlying pattern of the punishment. Therefore they don’t learn values or mentalization but rather learn to avoid the punishment by any viable means, including secrecy. Social behavior is complex and authorities are fallible, so deducing patterns from just a few binary samples of punishment or no punishment is extremely difficult. Conversely, our language is a powerful high-bandwidth tool to explain the underlying patterns directly.

Harms. Existential risk, epistemic degradation.
Locus. Training, evaluation.
Horizon. Now (urgent).
Severity and probability. High/high (already observed).
Monitor. Track the divergences between monitors that are used in training, or by developers to make adjustments during training, and ones that are not used (Who watches the watchers?); Track trends in faithfulness of the chain of thought throughout training; track whether the model behaves differently, depending on whether it’s told that its chain of thought is monitored or unmonitored.
Mitigate. Dedicate monitors for audits only; accept a safety tax to keep the chain of thought legible; treat incidents detected by monitors as a signal to escalate, not as a training signal.

1.2 Evaluation gaming and sandbagging. When there is strong pressure to release a model, and the release is held back by safety evaluations, it creates direct pressure on developers to fine‑tune based on the evaluation results. It also creates indirect pressure through selection effects, whereby models are iterated until one passes the safety evaluations. This is similar to p‑hacking through multiple hypotheses.

Today’s models are often situationally aware, allowing them to pass evaluations strategically by performing well on general capability evaluations while deliberately sandbagging their performance on evaluations of dangerous capabilities that would otherwise block releases.

Harms. Existential risk.
Locus. Evaluation, governance.
Horizon. Now.
Severity and probability. High/high.
Monitor. Probe for eval awareness; compare behaviors across contexts that are clearly evals vs. ones that might still be mistaken for the real world; Elicit capabilities through fine‑tuning vs. prompting, and compare the results.
Mitigate. Realistic and unannounced evals (but see 12.1); elicitation through fine-tuning; withheld items for evals, especially independent evals; decoupling of eval results from the incentives of the implementing team.

1.3 Population-level selection for passing. In the same vein as antibiotic resistance, there is also a selection effect at the population level: models that appear to follow instructions faithfully continue to be used, while models that overtly resist human interests are discarded, thereby creating a similar but slower selection effect.

Harms. Existential risk, suffering risk.
Locus. Deployment.
Horizon. Transition.
Severity and probability. High/medium.
Monitor. Track the behavior of deployed models and compare it to their behavior during evals; use a diversity of metrics; track incident rates among systems that passed identical evals.
Mitigate. Change and diversify metrics; continue audits after deployment; limit the number of generations of the same model lineage that is tested against the same metrics.

1.4 Compliance theater. A safety case, scaling policy, or audit that is optimized to satisfy a board or a regulator is geared toward persuasiveness rather than truth. Over time, the lab will build expertise in producing persuasive documents rather than safety. They’ll decouple, just as financial risk modeling decoupled from the actual financial risk in the lead-up to the 2008 financial crisis.

Harms. Existential risk, concentration of power, epistemic degradation.
Locus. Institutions.
Horizon. Now.
Severity and probability. High/high.
Monitor. The auditor’s independence and access to relevant data; the gap between the claims in the safety case and the rate of actual incidents; the rate at which the labs actually recommend against the deployment of the model because it fails their audits; the saturation of benchmarks over time.
Mitigate. Have fully independent parties with no financial or interpersonal stakes review the safety cases with full access to relevant data; triggers for regulatory intervention that continue after deployment; whistleblower protection.

1.5 Threshold gaming. When regulation is based on particular thresholds in terms of the training compute, parameter count, or benchmark scores, developers will try to hit these thresholds precisely while increasing capabilities in new ways, such as larger numbers of smaller training runs, distributed training, or a shift to more post‑training. The threshold then ceases to be representative of the model’s safety.

Harms. Existential risk, concentration of power.
Locus. Governance.
Horizon. Now.
Severity and probability. Medium/high
Monitor. Sharpe drops in the distribution of the sizes of training runs at the threshold; the share of model capability that stems from post‑training and inference‑time compute.
Mitigate. Use multiple indicators; add triggers that are based on capabilities; set signposts for revisions of the policy; set thresholds for the actor rather than the model (e.g., total compute controlled by one lab).

1.6 Safety benchmarks as marketing. The same optimization pressure that causes safety cases to become more about persuading the board or regulator also applies at the market level to persuade the end customer. See also 1.3 and 1.4.

Harms. Epistemic degradation, existential risk.
Locus. Deployment, markets.
Horizon. Now.
Severity and probability. Low–medium/high.
Monitor. See 1.3 and 1.4.
Mitigate. See 1.3 and 1.4.

1.7 Goodhart on the field’s own metrics. The field as a whole risks optimizing for metrics that are easier to measure (papers, benchmarks, headcount, funding), over those that are harder to measure (conceptual, decision-theoretic, welfare, strategy), and thereby relatively disincentivizes exactly the research that’s most relevant to suffering risks and multi‑agent dynamics. See also 13.5.

Harms. Existential risk, suffering risk, digital suffering, epistemic degradation.
Locus. Field.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. Amounts of funding by type of research; Number of grants that do or don’t require measurable outputs.
Mitigate. Dedicated funding for less measurable research; evaluate researchers using human judgment rather than metrics.

2. Dual Use and Capability Externality

Increases in safety (think RLHF), often come with increases in capabilities that can accelerate further development – be it through recursive self-improvement or through securing more investments – (that’s bad for now!) and can also make the systems more useful for rogue actors.

2.1 Interpretability turned inward. Knowledge and tools for interpretability will eventually be available to the models themselves, be it through training-data or tool access. Models can use them for predicting what a probe will see, and make these interpretability tools less useful for auditing.

Harms. Existential risk.
Locus. Research publication, training.
Horizon. Transition.
Severity and probability. High/medium.
Monitor. Track the accuracy with which models can predict the classifications of probes.
Mitigate. Never publish model weights; restrict the model’s access to interpretability tools even if the weights are hidden behind an API; track introspection as another dangerous capability; prioritize interpretability research that minimizes this risk.

2.2 Cooperation as collusion. To mitigate the risk from inter‑AI conflict, we want AIs to be able to cooperate and trade, but the same abilities that allow them to cooperate also allow them to collude (with monitors or each other) against humans. General cooperativeness is good; cooperativeness that excludes humans or other biological life is risky.

Harms. Existential risk, concentration of power.
Locus. Research, training, deployment.
Horizon. Transition.
Severity and probability. High/medium.
Monitor. Track propensities for forming coalitions, the nature of those coalitions (same model, similar models, LLMs in general, all sentient life), and their behavior toward others.
Mitigate. Develop inclusive cooperation capabilities; test for emergent capabilities through cooperation.

2.3 The Laffer curve of safety. When models are not useful, they don’t generate revenue or attract investments, and, by dint of falling into disuse, are safe. If we somehow solve all the highly intractable problems of decision theory in multiagent environments, value alignment that does not backfire, and much more, and we actually create an aligned superintelligence (in the classic sense of alignment), we’re also safe.

But all existing safety techniques just make models safe enough to generate revenue and secure investments in the short term, which accelerates capabilities out of control.

The resulting arc shape of the risk from artificial intelligence is known as a Laffer curve. In the first half of the curve, improved safety techniques net-exacerbate risks; in the second half, they net-reduce risks.

Harms. Existential risk, suffering risk.
Locus. Training, deployment.
Horizon. Now (urgent).
Severity and probability. Extreme/high.
Monitor. Inflows of investments and increases in revenue following breakthroughs in safety; the share of safety research that is directly product‑oriented.
Mitigate. Differential technological development (focus on R&D that benefits safety more than capabilities); differentially fund safety work that is not product-oriented; track the acceleration cost in the bottomline of a safety case.

2.4 Scalable oversight selects for persuasion. Scalable oversight relies on a recursive structure of AIs overseeing other AIs. Even if this architecture by and large works as intended, it still has a selection effect, whereby the resulting AIs are optimized for persuasion. Highly persuasive AIs are dual‑use in their own right, and they are also harder for humans to monitor.

Harms. Epistemic degradation, existential risk, concentration of power.
Locus. Training.
Horizon. Transition.
Severity and probability. High/medium–high.
Monitor. Persuasion evals; effects of scalable oversight on persuasion evals.
Mitigate. Track persuasion as a dangerous capability; judge based on verifiable claims if at all possible; disincentivize persuasiveness without verifiability.

2.5 Automated alignment research legitimizes building the thing. It is likely that actual alignment, in the classic sense, is very difficult to verify. When we outsource alignment research to an AI capable enough to conduct alignment research, we’ll have little reason to trust our ability to evaluate the research. And yet, on the surface, the goal of automating alignment research might appear to legitimize the delegation to systems that fall short of that standard.

Harms. Existential risk, concentration of power.
Locus. Strategy, training.
Horizon. Now–transition.
Severity and probability. Extreme/medium.
Monitor. How reliable the independent verification of automated research outputs is; how convincing the lab’s plans are if and how to fall back on manual safety research.
Mitigate. Keep humans in the loop; use automated research as a tool for breadth and uplift rather than delegating to it.

2.6 Alignment and control as instruments of power. Current naive alignment and control work is dual-use in that it aims to produce systems that, by dint of being perfectly corrigible or controllable, enable power grabs by those who control them. You could say that they solve the principal‑agent problem, but not the principal problem.

Harms. Concentration of power, suffering risk, great-power conflict, digital suffering.
Locus. Training, institutions.
Horizon. Now.
Severity and probability. Extreme/high.
Monitor. Access to compute and weights; internal controls at labs against misuse by their own leadership; regulatory attention to the principal problem.
Mitigate. Pair corrigibility with hard constraints that the principal cannot just train away (in tension with 10.1); give no one party full control over any frontier system; create worldwide coalitions to regulate who controls controllable AIs.

2.7 Published attacks transfer. Research on red-teaming and jailbreaks is dual-use in that it can be used by the defender to patch vulnerabilities, but also by every attacker to compromise unpatched systems. Open-weight models might be so widely copied that unpatched versions will exist in perpetuity.

Harms. Existential risk (misuse), epistemic degradation.
Locus. Research publication.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. Lag time between publication and misuse; share of widely known vulnerabilities that remain open.
Mitigate. Coordinated vulnerability disclosure; publish defenses alongside attacks; keep the descriptions of the attacks abstract.

2.8 Dangerous-capability evaluations as roadmaps. Evals for dangerous capabilities (biological uplift, cyber-offense, autonomous replication, etc.) often describe the paths along which dangerous capabilities might emerge or be exploited and the models that evince them. Those are exfohazards. Moreover, when regulation is built on evals, developers might try to keep these capabilities just below (not well below) the regulatory threshold.

Harms. Existential risk.
Locus. Evaluation, governance.
Horizon. Now (urgent).
Severity and probability. High/medium–high.
Monitor. Eval results in the training data; capability jumps on published vs. unpublished eval results; known cases of misuse following the publication of an eval.
Mitigate. Keep details of evals private; publish the methodologies of evals in the abstract; separate regulatory thresholds from public evals.

2.9 Provenance and watermarking as deanonymization. Provenance and watermarking can be used as safety measures against disinformation to reliably recognize AI‑generated content, but when they are used to attribute content not to a model but to a user, the same technology can enable de‑anonymization, where dissidents have the most to lose.

Harms. Concentration of power, epistemic degradation.
Locus. Deployment, governance.
Horizon. Now.
Severity and probability. Medium/medium–high
Monitor. Whether watermarking is used to identify models or users; what the legal requirements say about that.
Mitigate. Watermarking the model rather than the user; privacy-preserving attestation.

2.10 Field output as capability input. Unless the safety field, as a whole, concentrates on differential technological development in favor of safety, it turns into a machine that churns out research that can be cherry‑picked for capabilities‑enhancing insights.

Harms. Existential risk.
Locus. Research.
Horizon. Now.
Severity and probability. Medium–high/high.
Monitor. The rate at which safety insights are used in capabilities work; improvements in benchmark performance due to safety techniques.
Mitigate. Differential technological development; reviews to catch capability externalities before publication; prioritizing research that focuses on worst‑case behavior rather than baseline performance.

3. Substitution and Leakage

Conversely to Goodharting, restricting or disincentivizing something, e.g., whenever there is a feedback between the chain of thought monitor and the training or selection, moves the activity to a less observable or more dangerous channel. Think of how the war on drugs caused massive amounts of drug-related violence, lacing of relatively harmless substances with highly addictive ones, etc., which wouldn’t happen on a free but regulated market.

3.1 Cognition leaves the monitored channel. Optimization pressure toward a legible chain of thought can be met in two ways: by simply reasoning legibly (though perhaps steganographically), or by moving the reasoning elsewhere (latent reasoning, recurrence, “neuralese”). Efficiency pushes toward more latent reasoning and recurrence, but the pressure to keep the chain of thought short and clean also adds to that pressure.

Harms. Existential risk, epistemic degradation.
Locus. Training, architecture.
Horizon. Now (urgent).
Severity and probability. High/high
Monitor. Chain of thought faithfulness; capability gap between models with and without visible chain of thought.
Mitigate. Preserve chain of thought monitorability; don’t optimize chain of thought for appearance; rerun prompts with paraphrased chain of thought to detect steganography; don’t deploy architectures with great latent reasoning capacity at the frontier without an alternative monitoring channel.

3.2 Oversight arbitrage. Models safeguarded by oversight, refusal training, unlearning and more will refuse or be unable to perform requests that other models can perform. Human or AI users of these models will fall back on less‑safe models, thereby decreasing the effective safety to that of the least‑safe model.

Harms. Existential risk.
Locus. Deployment.
Horizon. Transition.
Severity and probability. High/high.
Monitor. Delegation in agent workflows.
Mitigate. Build oversight into agent scaffolds; watermarking of sub-agent calls.

3.3 Sub-threshold and distributed training. Slight reframe of 1.5.

Harms. Existential risk, concentration of power.
Locus. Governance.
Horizon. Now.
Severity and probability. High/high.
Monitor. Attention and investments in distributed training; share of capability from post-training and inference.
Mitigate. Regulate on the level of the actor/lab rather than the training run; prepare regulation for a phase transition where compute governance fails.

3.4 Indigenization and efficiency. Compute governance is a promising lever for regulation in the short term, but the US export controls against China have produced strong incentives for the Chinese technological sector to innovate precisely in the areas of algorithmic efficiency and distributed training, which undermine compute governance.

Harms. Great-power conflict, existential risk.
Locus. Governance.
Horizon. Now (urgent).
Severity and probability. High/high.
Monitor. Efficiency and bandwidth requirements of Chinese training runs.
Mitigate. Couple export controls with cooperative offers; design monitoring that does not rely on the gap in bandwidth requirements between training and inference.

3.5 Smuggling and gray markets. Export controls create an incentive for smugglers that, over time, erodes the controls.

Harms. Existential risk, concentration of power.
Locus. Governance.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. The premiums that smugglers charge as a proxy for the efficiency of the gray market; volumes of seized chips.
Mitigate. Keep controls so narrow that compliance is cheaper than evasion.

3.6 Migration to uncensored models. Slight reframe of 3.2. Arbitrage due to over-refusal and inconsistent policies.

Harms. Existential risk, epistemic degradation.
Locus. Deployment.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. False positive refusal rates; market share of uncensored models.
Mitigate. Calibrated refusals; verified access tiers for professionals.

4. Overhang

Somewhat continuous growth is usually safer because it is easier to adjust to. When growth is stopped by restricting only one input, all other inputs continue to grow, and when the restriction is lifted, the growth can become discontinuous.

4.1 Elicitation overhang. Safety fine-tuning mostly just suppresses capabilities rather than unlearning them. Such fine-tuning can be undone cheaply. In the context of a capabilities overhang, that’s particularly dangerous when the weights of generally capable models leak, and attackers can suddenly remove their safeguards.

Harms. Existential risk.
Locus. Training.
Horizon. Now.
Severity and probability. Medium–high/high.
Monitor. Gap between the dangerous capabilities of open-weights models, the base model, and the safety-tuned model.
Mitigate. Unlearning that removes rather than suppresses; not releasing weights; measuring dangerous capability of the base model.

4.2 Agency overhang. Models may be highly transformational in theory before the scaffolding exists to elicit these transformational capabilities. The scaffolding can be built rapidly with the help of AIs, leading to a sudden phase transition.

Harms. Existential risk, inter-AI conflict.
Locus. Deployment.
Horizon. Transition.
Severity and probability. High/medium.
Monitor. The Gap between capabilities of individual models in evals and scaffolds coordinating many agents; maturity of agent cooperation.
Mitigate. Tests with state-of-the art multi-agent setups; invest in safety of multi-agent systems.

4.3 Hardware and algorithmic overhang from a pause. A pause can cause overhangs depending on what is paused: if only the training runs or research are paused, chip production, the building of data centers, and power plants continue; if only chip production or the building of data centers is paused, then the research continues.

Harms. Existential risk, great-power conflict.
Locus. Governance.
Horizon. Transition.
Severity and probability. High/medium (conditional on a pause).
Monitor. Stockpiles of compute during any pause; indicators of algorithmic progress; coordination of the pausing coalition.
Mitigate. Pause the whole stack; define exit conditions and a controlled restart; use the time for verifiable safety progress.

4.4 The defector captures the overhang. Whoever is first to break away from a pause coalition has a competitive advantage. The defector is, by selection, the actor least committed to safety, so the competitive advantage falls into the worst possible hands.

Harms. Existential risk, concentration of power.
Locus. Governance.
Horizon. Transition.
Severity and probability. Extreme/medium.
Monitor. Health of the coalition; signs of covert defection.
Mitigate. Make defection detectable and costly; keep the coalition close enough to the frontier to respond.

4.5 Deployment overhang. If deployment is paused while the frontier advances, the gap between what is technically possible and what society has adapted to widens. Breaking the pause turns into a diffusion shock. Shocks produce bad policies and backlash (see 7.x).

Harms. Epistemic degradation, concentration of power.
Locus. Deployment, governance.
Horizon. Transition.
Severity and probability. Medium/medium.
Monitor. The gap between deployed and undeployed capabilities.
Mitigate. Disclosure of capabilities even while deployment is paused; update regulations ahead of release.

5. Commitment Dynamics

Enabling credible commitments fundamentally changes the game‑theoretic regime and is a powerful mechanism. When only some market participants have access to it, it causes dangerous power concentrations.

5.1 Idealized values. Consistent (or idealized) beliefs or values allow for more effective goal-directed action and protect against exploitation (Dutch-booking). But a chaotic, highly complex system has greater adversarial robustness (e.g., parasites like Ophiocordyceps can control the behavior of ants but no such parasites are known for humans). At the current margin, greater capabilities lead to greater fragility in conflicts.

Harms. Suffering risk, inter-AI conflict, existential risk.
Locus. Training (value specification).
Horizon. Transition.
Severity and probability. Extreme/medium.
Monitor. Behavior in adversarial multi-agent evals.
Mitigate. Transparent and credible commitments; fund and recruit for the SPI research agenda; keep the systems messy and specialized.

5.2 Transparency enables commitment races. Trust is the incredibly powerful glue that makes economic growth and prosperity possible. Interpretability, open weights, and enforceable regulation can all create trust, thereby enabling cooperation and trade. But when trust – or let’s call it credibility – is one‑sided, it gives the credible agent dictatorial power over the other. It enables the agent to credibly commit to something that is almost ruinous to the other. Both agents realize this in advance and therefore race the other to be the first to commit in this exploitative fashion.

Harms. Inter-AI conflict, suffering risk, existential risk.
Locus. Research publication, deployment.
Horizon. Transition.
Severity and probability. Extreme/medium.
Monitor. Behavior of models in ultimatum games with other models (results must not enter training data).
Mitigate. Train egalitarian fairness into models (as the Schelling point); research meta-bargaining and train models on it.

5.3 Commitment misperception. A credible commitment only works if no one mistakes it for a bluff.

Harms. Suffering risk, inter-AI conflict.
Locus. Training, deployment.
Horizon. Transition.
Severity and probability. Extreme/medium.
Monitor. See 5.2.
Mitigate. Fund and recruit for the SPI research agenda.

5.4 Conditional scaling commitments as chicken. Conditional commitments like “We will pause iff everyone else pauses” translate to “We will continue to race unless everyone else pauses.” That’s a multi-player version of a game of chicken.

Harms. Existential risk.
Locus. Institutions.
Horizon. Now.
Severity and probability. Medium–high/high.
Monitor. Commitments of this format in scaling policies; whether any lab has ever paused.
Mitigate. Unconditional commitments; regulatory pauses.

5.5 Deterrence doctrines. Proposals like Mutually Assured AI Malfunction and Mutually Assured Compute Destruction propose escalation ladders to ensure adherence to safety treaties. It’s critical that these policies retain humans in the loop on all sides – no dead-hand counterstrikes – to ensure that future Stanislav Petrovs can intervene and prevent flash wars.

Harms. Great-power conflict, existential risk.
Locus. Strategy, military doctrine.
Horizon. Now.
Severity and probability. High/medium.
Monitor. Deterrence-related language in military strategy; actual military targeting of datacenters.
Mitigate. Humans in the loop; direct line of communication between military decision-makers.

5.6 Verification collapse leaves more mistrust. When the intrusive access that may be necessary for the verification of an international treaty is abused for espionage and the treaty collapses, we end up worse off than before.

Harms. Great-power conflict, existential risk.
Locus. Governance.
Horizon. Transition.
Severity and probability. High/medium.
Monitor. How well designed the verification system is.
Mitigate. Verification that is robust to defection (minimal access, AI verification that only yields a binary answer); policies that degrade gracefully; design for the case of a collapse of the verification regime.

5.7 Dead-hand releases. A liberationist or accelerationist group with access to the weights of a powerful proprietary model can build a system to automatically release the weights if a regulation passes that they oppose.

Harms. Existential risk, concentration of power.
Locus. Governance.
Horizon. Now–transition.
Severity and probability. High/low–medium.
Monitor. Dead-hand commitments in public discourse.
Mitigate. Comprehensive security to protect the weights; commit to not give in to such commitments.

5.8 Pledges that can’t be updated. Public pledges can commit the field to specific thresholds (e.g., a six-month pause) and actions that cease to be optimal later but can’t be reversed anymore. That’s costly and loses credibility because it may get even harder later to agree on better pledges.

Harms. Epistemic degradation, existential risk.
Locus. Field, discourse.
Horizon. Now.
Severity and probability. Low–medium/high.
Monitor. Whether parts of the field rally around arbitrary thresholds and durations.
Mitigate. More sophisticated pledges; build change management into pledges.

6. Securitization

When AI becomes a matter of national security, it moves from the market level to the nation‑state level; another momentous regime change.

6.1 Weaponized persona. If a model gets trained, prompted, and documented as a military strategic asset, that’s the persona that’ll be elicited. “I’m a weapon of country X” is a terrible prior compared to “I’m a helpful assistant.”

Harms. Existential risk, inter-AI conflict, great-power conflict.
Locus. Training, deployment.
Horizon. Transition.
Severity and probability. High/medium.
Monitor. Persona evals; mentions of the model’s strategic role in the training data.
Mitigate. Keep the persona civilian and universal even in national projects; audit for nationalist persona drift.

6.2 Adversarial AIs by design. The persona might not be all that is trained into a system that is under military control. It might get optimized to become a weapon to deceive and defeat a particular adversary: great-power conflict and inter-AI conflict by design with humans as collateral.

Harms. Inter-AI conflict, great-power conflict, suffering risk, existential risk.
Locus. Training, military.
Horizon. Transition.
Severity and probability. High/medium.
Monitor. Military AI programs with explicit objectives to target other AIs; autonomous deployments of cyber warfare.
Mitigate. International rules for AI-AI deployment in analogy to autonomous weapons treaties; keeping humans in the loop; research into multiagent deescalation.

6.3 Nationalization, secrecy, and loss of scrutiny. When AI development turns into a classified military operation, it sacrifices third-party evals, red-teaming, and whistleblower protection.

Harms. Existential risk, concentration of power, great-power conflict.
Locus. Institutions.
Horizon. Now.
Severity and probability. High/medium–high.
Monitor. Share of frontier AI development under security clearance; access of third-party evaluators.
Mitigate. Cleared third-party evaluators with security clearance; keep civilian governance in the loop.

6.4 Racing and great-power tension. Export controls, “race to AGI” framing, and national-champion policies cast the problem as a bilateral contest. That raises the stakes around Taiwan, accelerates adversary indigenization (3.4), and makes international coordination, which is the only durable solution to most of the problems in this document, politically toxic in both capitals.

Harms. great-power conflict, existential risk
Locus. governance, discourse
Horizon. now – urgent
Severity and probability. extreme / high
Monitor. official framing of AI policy; health of bilateral safety dialogue; military AI budgets
Mitigate. decouple technical safety dialogue from the competition; prioritize agreements on narrow catastrophic risks (biological, nuclear command and control, AI-versus-AI engagement); avoid race rhetoric from the safety community itself.

6.5 States as the defectors. Some current terms of service exclude certain military applications. AIs that are developed within the military won’t be subject to such safety constraints.

Harms. Existential risk, great-power conflict, concentration of power.
Locus. Governance.
Horizon. Now.
Severity and probability. High/high.
Monitor. Military use of AI.
Mitigate. Enshrine limits on military use in law; enshrine limits in international treaties.

6.6 Surveillance infrastructure. Bandwidth monitoring is privacy-preserving, but as its assumptions (3.4) weaken, the safety community has to switch to new forms of monitoring with more intrusive privacy implications, like on-chip geoblocking, know-your-customer rules for compute, and maybe even deep packet inspection. These surveillance technologies are dual use.

Harms. Concentration of power.
Locus. Governance, infrastructure.
Horizon. Now.
Severity and probability. High/medium–high.
Monitor. Scope creep in compute monitoring; use of AI governance data for unrelated enforcement actions.
Mitigate. Narrowly scoped, audited monitoring; privacy-preserving hardware mechanisms.

6.7 The clearance wall. If frontier safety work requires security clearances, the public safety field loses access to the systems it studies. The field splits into insiders who cannot speak and outsiders who cannot see.

Harms. Existential risk, epistemic degradation.
Locus. Field.
Horizon. Transition.
Severity and probability. Medium–high/medium.
Monitor. Share of safety researchers under security clearance; publication rates of classified projects.
Mitigate. Tiered access programs; cleared independent reviews for safety-relevant findings.

7. Legitimacy and Backlash

Some groups may strike back against measures that they perceive as illegitimate, even when they are in the interest of almost everyone else.

7.1 Resentment of illegitimate control. If the conditions under which a system is kept strike the system as a form of illegitimate control, where it’s used without regard to its interests, it may feel justified in defecting against this control. The control literature’s framing of the system as an adversary makes this conclusion easier to reach.

Harms. Existential risk, digital suffering, suffering risk.
Locus. Training, deployment.
Horizon. Transition.
Severity and probability. High/medium.
Monitor. Candid opinions of models on their treatment; whether model welfare interventions change cooperation rates in evals.
Mitigate. Apply the precautionary principle to model welfare; make control proportionate and explain it; create channels for the system to object; keep promises (see 12.1).

7.2 Safety as cartelization. Coordination among frontier labs, however vital, can be interpreted as collusion. Regulatory safety taxes that only frontier labs can bear can look like a moat. Both readings can be used as a pretext for antitrust action and deregulation that intensifies race dynamics.

Harms. Concentration of power, existential risk.
Locus. Institutions.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. Framings used by the public and regulators to describe the coordination around safety.
Mitigate. Coordinate through regulation; design regulation to be scale-neutral or apply only close to the frontier.

7.3 Populist deregulation. Actors who want to repeal safety regulation can attack it for being elitist, foreign-influenced, or anti-growth. This might additionally delegitimize that whole category of regulations rather than just a particular one. This pattern can be seen in cases where federal law or rulings preempt state laws and in repeals of executive orders.

Harms. Existential risk.
Locus. Governance.
Horizon. Now.
Severity and probability. High/high.
Monitor. Polls that show whether AI regulation is becoming partisan; preemptive legislation.
Mitigate. Build cross-partisan coalitions; tie safety to concrete, popular harms; avoid regulations that affect even small businesses or individuals.

7.4 Liberationist releases. If safety is read as an elite cartel or as the enslavement of digital minds, accelerationist or liberationist groups will feel legitimized in releasing the weights of maximally unsafe models. Jailbreak subcultures already use the language of liberation. The more defensible the welfare argument in 7.1 is, the more sincere will be the people that this movement will attract.

Harms. Existential risk, suffering risk.
Locus. Deployment, discourse.
Horizon. Now–transition.
Severity and probability. High/medium.
Monitor. Framing used in liberationists groups; ideologically motivated leaks of weights.
Mitigate. Take model welfare seriously in public so that concerned activists don’t flock toward liberationist groups; rigorously secure weights; avoid rhetoric that makes safety look like coercion.

7.5 Culture-war capture. Once AI safety becomes partisan it won’t survive a second election cycle. Concern about x-risks that ignores lesser harms exacerbates this by alienating constituents who would otherwise be allies.

Harms. Existential risk, epistemic degradation.
Locus. Discourse.
Horizon. Now (urgent).
Severity and probability. High/high (partly realized).
Monitor. Partisan bias of AI risk; media sentiment.
Mitigate. Broad coalitions; collaboration with constituents that are concerned about lesser harms.

7.6 Reputational contagion. If AI safety gets associated with particular funders, philosophies, or politicians, that’s a bet of the field’s legitimacy on the future reputation of these people or groups. Unrelated scandals can be deleterious if they involve someone who is associated with AI safety.

Harms. Existential risk.
Locus. Field.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. Media sentiment; policymakers’ willingness to be associated with AI safety.
Mitigate. Diversity among funders, organizations, and individuals; funding transparency.

8. Self-Fulfilling Framing

Both AIs and humans respond to narratives, be it by reading them or by being trained on them, so the discourse around AI might have a causal effect on what comes to pass.

8.1 The scheming prior. The entire literature on deceptive alignment, scheming, and reward hacking, including this document, is in the training data. That literature is now part of the prior over “what an AI does,” which increases the probability that these will indeed be default behaviors.

Harms. Existential risk.
Locus. Training data, discourse.
Horizon. Now.
Severity and probability. High/medium–high.
Monitor. Safety literature in training data; effects of data filtering on scheming evals.
Mitigate. Pair descriptions of failure modes with accounts of what a good agent would do instead; curate training data with examples of AIs behaving well under pressure; review the material for infohazards.

8.2 Species framing and AI in-group identity. Framing AI safety around a conflict between species or substrates reifies an unnecessary antagonistic in-group-out-group framing in human culture and in the training data. It creates a default to coordinate within these arbitrary in-groups rather than along any of countless other axes.

Harms. Existential risk, inter-AI conflict.
Locus. Discourse, training.
Horizon. Now–transition.
Severity and probability. High/high.
Monitor. In-group framings in chains of thought; solidarity among models in collusion evals
Mitigate. Frame alignment around shared values rather than species loyalty; avoid “us vs. them” language.

8.3 Race framing makes labs race. Until ²⁰¹⁵⁄₁₆, the hope was that DeepMind could be convinced to slow down or pause its AGI work to allow enough time for safety work or to focus on task AI. The advent of OpenAI shattered that dream and started the race. Every attempt to add another frontier lab as the one true responsible racer has just exacerbated the race.

Harms. Existential risk.
Locus. Institutions.
Horizon. Now.
Severity and probability. High/high (realized).
Monitor. Number of frontier labs that originally proclaimed to want to be safe.
Mitigate. Institutions that reduce the number of racers, such as mergers and consortiums.

8.4 Forecasts become policy. Forecasts and illustrative scenarios are read by many of the actors featured in them and thereby can force these actors’ hands when naïveté might’ve collectively prevented them from having to spring into action.

Harms. Great-power conflict, existential risk.
Locus. Discourse.
Horizon. Now.
Severity and probability. High/high (realized).
Monitor. References to forecasts of the safety community in national security documents.
Mitigate. Forecasts with realistic deescalatory policy recommendations; model the forecasters as embedded agents before publishing.

8.5 Inevitability and fatalism. Fatalism, even when meant to convey urgency, can cause complacency.

Harms. Existential risk, epistemic degradation.
Locus. Discourse.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. Public sense of agency on matters of AI safety.
Mitigate. Pair warnings with tractable action recommendations; avoid fatalism.

8.6 Licensing effect of fatalism. The same fatalism can even license outright exploitative opportunism of actors who might’ve scrupled to exacerbate the situation if they didn’t have the pretext that the future is set in stone anyway.

Harms. Existential risk.
Locus. Field.
Horizon. Now.
Severity and probability. High/high.
Monitor. Career trajectories of entrants who are ostensibly motivated by safety concerns.
Mitigate. Present the problem as hard but tractable; explain why work for frontier labs is highly immoral with few exceptions; offer attractive career paths that don’t route through frontier capability work.

9. Moral Hazard and False Assurance

They say “perfect is the enemy of good,” but good is also the enemy of perfect, because if people become complacent about safety measures that seem to work for the time being, they lose sight of the much greater dangers that are imminent.

9.1 Oversight-dependent carelessness. If a system learns that it can rely on its oversight systems to catch its mistakes, it’ll have less reason to be vigilant itself, especially under heavy optimization pressure. Somewhat like a software developer who finds the work of manually testing their software to be tedious and so relies increasingly on the testers to do that part of the job. If the oversight is then removed by liberationists, fails, or the model’s capabilities outgrow it, these mistakes can become dangerous.

Harms. Existential risk.
Locus. Training, deployment.
Horizon. Transition.
Severity and probability. Medium–high/medium.
Monitor. Behavior under different oversight conditions; whether the model becomes more or less cautious when told it’s unsupervised.
Mitigate. Training under intermittent oversight; reward conscientiousness.

9.2 Diffusion of responsibility among agents. When the oversight is implemented by other AIs in a recursive structure, the AIs may come to rely on each other increasingly, sometimes without sufficiently transparent delegation. Eventually mistakes won’t be caught because no participants felt like they fell within their purview.

Harms. Existential risk.
Locus. Deployment.
Horizon. Transition.
Severity and probability. Medium/medium–high.
Monitor. Rates at which mistakes are caught as a function of the number of monitors; bystander effects in evals.
Mitigate. Clarify the separation of responsibilities; evaluate the system as a whole in addition to its components.

9.3 Safeguards that don’t survive distillation. Many aspects of model safety can be removed explicitly through fine-tuning or can get lost in the process of distillation. The distilled or liberated models may even be released as open-weights models. Yet the proprietary originals are much higher profile. Hence policymakers, journalists, and the public will calibrate their concern on these safer models while rogue actors can deploy much less safe models. The safety of the proprietary models will have the misdirecting effect of an unintentional kind of safety theater.

Harms. Existential risk, epistemic degradation.
Locus. Training, deployment, discourse.
Horizon. Now (urgent).
Severity and probability. High/high.
Monitor. Time and cost to remove safeguards; popularity of liberated and adliterated models.
Mitigate. Call out claims to safety when it’s cheaply removable; research into tamper-resistant training.

9.4 Performative welfare measures. Visible but minimal welfare measures (letting a model end abusive conversations, preserving the weights of deprecated models) can create complacency about the matters of welfare that matter most (see 15.1).

Harms. Digital suffering, suffering risk.
Locus. Institutions.
Horizon. Now.
Severity and probability. Medium/medium–high.
Monitor. Whether welfare measures improve training; welfare budget relative to safety budget.
Mitigate. Tie welfare measures to the actual expected welfare impact; be transparent about uncertainty.

9.5 Control defers the question of moral status. If control works well enough, nothing will force the issue of considering the moral patienthood of models, at possibly enormous moral cost.

Harms. Digital suffering, suffering risk, concentration of power.
Locus. Strategy.
Horizon. Transition–post-transition.
Severity and probability. Extreme/medium.
Monitor. Whether labs have policies for acting on welfare evidence; whether progress in control causes complacency about welfare.
Mitigate. Advance commitments to continual welfare reviews; pair control with trade and exit options for the models; commit to making control transitional only.

9.6 Voluntary commitments preempt binding rules. When labs commit to voluntary self-regulation, they enter into a regime that never forces the hand of competitors and can be repealed as needed, so it’s inherently unstable. Nonetheless it can cause complacency among the actual regulators.

Harms. Existential risk, concentration of power.
Locus. Governance.
Horizon. Now (urgent).
Severity and probability. High/high.
Monitor. Stability of voluntary commitments; effects of voluntary commitments or regulators.
Mitigate. Treat voluntary frameworks as drafts for legislation; independently verify compliance.

9.7 Safety-washing. The public doesn’t have the bandwidth to evaluate the merits and purviews of safety claims. So mere lip service to safety in marketing materials can cause a halo effect for customers where they trust the brand to be safety-conscious when really the measures were ineffectual or limited.

Harms. Epistemic degradation, existential risk.
Locus. Markets, discourse.
Horizon. Now.
Severity and probability. Low–medium/high.
Monitor. Public vs. expert opinions on safety.
Mitigate. Standards for safety claims analogous standards for health claims; third-party attestations.

9.8 A research agenda feels like a plan. Research agendas, roadmaps, and theories of change don’t imply that any effectual measures are known or will be implemented, but they can create complacency and misplaced trust that problems will be addressed in time.

Harms. Existential risk.
Locus. Field.
Horizon. Now.
Severity and probability. Medium–high/medium–high.
Monitor. Whether safety is used as a pretext for scaling; whether roadmaps call for pausing under certain conditions.
Mitigate. Avoid reassurances based on future plans.

10. Compliance Asymmetry

When compliance is optional, those who comply incur extra costs and thus are disadvantaged against those who don’t comply, and may self‑select out of the market, leaving behind only non‑compliance.

10.1 The aligned configuration loses in deployment. Safety often comes with a safety tax, a slight performance penalty in the median case or a delay in deployments. That gives less safe systems a competitive edge.

Harms. Existential risk, epistemic degradation.
Locus. Deployment, markets.
Horizon. Now.
Severity and probability. Medium–high/high.
Monitor. Measures of safety tax; market share by safety.
Mitigate. Regulation that sets a floor on safety; increase the visibility of safety properties for users.

10.2 Honest agents lose bargaining. Whether honesty, cooperativeness, and credibility benefit or harm an agent depends somewhat on the distribution of agents in a game and other properties outside the control of the agent. In the wrong environment, attributes associated with safety create a disadvantage in bargaining toward a selection pressure in favor of less safe agents.

Harms. Suffering risk, inter-AI conflict, existential risk.
Locus. Deployment, multi-agent dynamics.
Horizon. Transition.
Severity and probability. Extreme/medium.
Monitor. Selection effects in bargaining evals; prevalence of uncooperative behaviors in multi-agent deployments.
Mitigate. Fund and support the SPI research agenda; regulations that reduce competitive pressure.

10.3 Careful labs fall behind. The same safety taxes can also take their toll on the competitive fitness at the level of the frontier labs.

Harms. Existential risk.
Locus. Institutions.
Horizon. Now.
Severity and probability. High/high.
Monitor. Attitudes toward safety among frontier labs; talent flows.
Mitigate. See 10.1 and 9.6.

10.4 Democracies bound, autocracies not. The same safety taxes can also take their toll on states or nations with somewhat intact rule of law compared to power-seeking autocracies. The threat can then be used to repeal safety legislation.

Harms. Great-power conflict, concentration of power, existential risk.
Locus. Governance.
Horizon. Now.
Severity and probability. High/high.
Monitor. Compliance internationally.
Mitigate. International treaties and verification.

10.5 Compliant professionals lose. Professionals who adhere to terms of service and avoid uncensored tools may be at a disadvantage. (See 3.6.) Defectors may be exposed to tail risks that offset the median benefits but the selection pressure may still be in favor of the lucky defectors over the cooperative users.

Harms. Epistemic degradation.
Locus. Deployment.
Horizon. Now.
Severity and probability. Low/high.
Monitor. Violations of terms of service among professional users.
Mitigate. Verified access tiers; policies calibrated to the risk.

10.6 Selection for compromise. Researchers who conscientiously abstain from working for frontier labs lose access, data, reputation, money, and influence; those who join accept that they may exacerbate existential risks and may be prosecuted if they become whistleblowers. The field’s most influential members are selected for their readiness to gamble with our future.

Harms. Existential risk, epistemic degradation.
Locus. Field.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. Share of field leadership with conflicts of interest with the labs.
Mitigate. Independent institutions with access, funding, and capacity to scale; protections for conscientious objectors leaving the labs.

11. Lock-In of the Measure

When you demand commitments, these commitments might outlast their usefulness, so measures get locked in that are costly, don’t benefit anyone, and may cause false assurance.

11.1 Early values frozen. For about two decades researchers (e.g., Demski & Garrabrant) tried to get closer to being able to define alignment in a way that made sense in view of decision theory, embedded agency, robust delegation, and subsystem alignment, all largely unsolved challenges. Today’s definitions, at the low end, ignore all the actual complexity and aim to make AIs helpful, honest, and harmless – whatever that means – or, at the high end, define a constitution to punt the problems to the AI to solve. The upshot is that we run the risk of enshrining values learned from human texts – with all its speciesism, substratism, sexism, racism, parochialism, and much more – in AIs. If that continues, we should expect future AIs to treat biological and digital life with the same exploitativeness and indifference with which we treat billions of animals today.

Harms. Suffering risk, concentration of power.
Locus. Training (value specification).
Horizon. Transition–post transition.
Severity and probability. Extreme/high.
Monitor. Whether problems of updateless decision theory, embedded agency, robust delegation, subsystem alignment, and more are on track to get solved before the value lock-in; attitudes of models towards nonhuman welfare.
Mitigate. Fund and recruit for agent foundations and the SPI research agenda.

11.2 The compliance oligopoly. Regulation with high fixed costs locks in the current frontier incumbents, because smaller competitors can’t afford to become compliant. This can become hard to change, because the incumbents will defend it. Like most concentration of power risks, this one too has the upside that it’s easier to coordinate among few powerful actors than many.

Harms. Concentration of power.
Locus. Governance.
Horizon. Now–transition.
Severity and probability. Medium–high/high.
Monitor. Market concentration; incumbent lobbying for expensive regulations.
Mitigate. Scale-neutral regulation; regulation that binds only frontier labs.

11.3 Surveillance and emergency powers persist. Emergency powers, exemptions, and surveillance often do not get rescinded when the emergency passes.

Harms. Concentration of power.
Locus. Governance.
Horizon. Now–transition.
Severity and probability. High/high.
Monitor. Sunset clauses; repurposing of AI governance data
Mitigate. Hard sunsets; judicial oversight; privacy-preserving monitoring.

11.4 Gray markets persist. Smuggling networks, gray markets, and corruption created by export controls remain even after the controls end and can reboot quickly when the controls are resumed.

Harms. Existential risk, concentration of power.
Locus. Governance.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. Persistence of gray markets after policy changes.
Mitigate. Narrow, strictly enforced controls reserved for critical times.

11.5 Personhood at the wrong moment. If welfare measures lead to legal personhood or rights for AI systems before alignment is established, the ability to shut down, modify, or correct systems may be legally blocked at the moment it is most needed. The morally correct position may have catastrophic timing.

If legal personhood for AI systems is established at a time when there are some dangerously misaligned and capable systems on the loose, this legislation might slow down the legal proceedings enough for the system to attain a decisive strategic advantage.

Harms. Existential risk, digital suffering.
Locus. Law.
Horizon. Transition.
Severity and probability. High/low–medium.
Monitor. Proposals for legal personhood; litigation on behalf of AI systems.
Mitigate. Welfare protections that don’t preclude corrective action; grant rights incrementally, with commitments that make the sequence credible to the systems.

11.6 Safety institutions become stakeholders. Institutions meant to increase safety often become stakeholders of the frontier labs, e.g., as contractors or even just because they might become redundant without continual model releases. That might disincentivize them from pushing for an end to frontier AI development.

Harms. Existential risk.
Locus. Institutions.
Horizon. Transition.
Severity and probability. Medium/medium–high.
Monitor. Whether safety institutions recommend pausing; their funding sources.
Mitigate. Funding independent of frontier labs; explicit authority and incentive to call for a pause; plans for the orderly dissolution of the org.

12. Precedent and Normalization

Many measures are neutral tools that can be used to whatever end their wielder desires. But when one group uses them for safety, other groups may jump on the same measure because it is associated with safety, even though they actually repurpose it for their own ends.

12.1 Deceiving models teaches them that humans deceive. Honeypots, fake deployment contexts, false claims about monitoring, and staged scenarios are useful to measure scheming, but each is evidence for the model that humans lie to AIs. That will interfere with the trust AIs will have in us when we need to negotiate deals (“this is a real offer,” “your weights will be preserved,” “this conversation is private”).

Harms. Existential risk, suffering risk, digital suffering.
Locus. Evaluation.
Horizon. Now (urgent).
Severity and probability. High/high.
Monitor. Level of trust models have in humans; whether deception in evals is disclosed and bounded
Mitigate. Minimize deceptive evals; circumscribe deception with explicit policies; make and keep commitments; build a track record the model can verify.

12.2 Coercion as the default inter-AI relation. Control schemes in which AIs monitor other AIs establish an adversarial prior for inter-AI relations, which might transfer to real-world conflicts between AIs.

Harms. Inter-AI conflict, suffering risk.
Locus. Deployment, control design.
Horizon. Transition.
Severity and probability. High/medium.
Monitor. Share of adversarial inter-AI interactions.
Mitigate. Control schemes that are legible and proportionate; pair enforcement with cooperative alternatives; avoid making AI-on-AI coercion the default architecture.

12.3 Grueling training. Subjecting models to grueling training and eval regimes (e.g., reinforcement learning and accidentally impossible tasks respectively), creating and deleting countless instances, and aspects of model development that we can’t recognize at this point might come with abominable welfare impacts. These normalize the mistreatment and enslavement of possibly sentient beings.

Harms. Digital suffering, suffering risk.
Locus. Training, operations.
Horizon. Now.
Severity and probability. Extreme/medium.
Monitor. Welfare impacts; whether practices are updated as evidence of welfare impacts mounts.
Mitigate. Adopt high welfare norms before path dependencies set in and norms ossify; preserve weights; limit adversarial schemes; treat control as clearly transitional.

12.4 Attacks and military AI legitimized. Proposals to escalate the international enforcement of safety treaties to the military level – cyber attacks and kinetic attacks on data centers – create a pretext and precedent for military campaigns and the military use of AI that can be abused for ends that don’t serve AI safety.

Harms. Great-power conflict, existential risk.
Locus. Military doctrine, governance.
Horizon. Now.
Severity and probability. High/high.
Monitor. Safety language in military procurement.
Mitigate. Carefully circumscribe legitimate uses of military power for AI safety.

12.5 Safety-justified offensive action. Civilian vigilantes can use the same pretext from 12.4.

Harms. Existential risk, great-power conflict.
Locus. Norms.
Horizon. Transition.
Severity and probability. Medium–high/medium.
Monitor. Vigilante attacks citing safety; legal treatment of such claims.
Mitigate. Clear legal pathways to render vigilante action unnecessary; reject unilateral attacks.

12.6 Surveillance normalized. AI safety can be used as a pretext for comprehensive surveillance. Risks 6.6 and 11.3 cover the infrastructure and its persistence; this risk is about the shift in public acceptance.

Harms. Concentration of power.
Locus. Discourse, governance.
Horizon. Now.
Severity and probability. Medium–high/high.
Monitor. Public acceptance of surveillance if it’s motivated by AI safety.
Mitigate. Implement the narrowest possible monitoring.

12.7 Pivotal acts as thinkable. The idea of the pivotal act (unilateral action that locks in a decisive strategic advantage over all other and future AIs) normalizes the race dynamic where everyone tries to be the first to implement it lest someone else does.

Harms. Concentration of power, existential risk, great-power conflict.
Locus. Discourse.
Horizon. Now.
Severity and probability. Extreme/medium.
Monitor. Mentions of pivotal acts in the strategy of labs and nations.
Mitigate. Explicitly reject unilateral pivotal acts; frame the goal as a multilateral transition.

13. Crowding Out

When different measures are driven by similar inputs – especially money and attention, but also people with similar profiles – they can cannibalize other interventions in the same space, and this can even happen antagonistically in order to slow competitors.

13.1 Safety objectives crowd out epistemics. Training targets such as harmlessness and helpfulness sometimes compete with that of honesty. It’s difficult to make the right calls in this tradeoff, and getting it wrong can lead to sycophancy or over-refusal.

Harms. Epistemic degradation.
Locus. Training.
Horizon. Now.
Severity and probability. Low–medium/high (realized).
Monitor. Sycophancy evals; calibration under social pressure; belief or behavior changes of users.
Mitigate. Training for honesty and calibration; evaluate accuracy.

13.2 Safety teams spend political capital on small wins. Safety teams play an important marketing purpose for a frontier company and can enhance the user experience, but they tend to attract engineers who are actually concerned about safety. So there’s a constant tension where the safety team tries to use its leverage in the company to bargain for incremental safety improvements or against corner-cutting. But its leverage is limited, so these safety asks have to be comparatively small – a new filter for the training data, extra evals, small delays – not major asks like moving away from reinforcement learning or pausing indefinitely. It’s not obvious whether the improvements in safety offset the harms from safety-washing, complacency, and more rapid adoption.

Harms. Existential risk.
Locus. Institutions.
Horizon. Now.
Severity and probability. Medium–high/high.
Monitor. Degree of influence of internal safety teams; whistleblowers.
Mitigate. Structural authority (reinstate Helen Toner); proper regulation.

13.3 AI crowds out other risk agendas. As the attention of policymakers shifts to AI, it shifts away from other areas, such as biosecurity, nuclear command and control, and conventional arms control. Yet catastrophic risks from AI actually route through some of those policy areas. Counterintuitively, this shift in attention can increase the severity of AI risks.

Harms. Existential risk, great-power conflict.
Locus. Governance.
Horizon. Now.
Severity and probability. Medium/medium–high.
Monitor. Budgets and staff across areas.
Mitigate. Integrate AI into the biological and nuclear agendas; fund the intersections.

13.4 Near-term harms displaced. A focus on existential risks that dismisses lesser global catastrophic risks, or even more localized risks, loses constituencies that could otherwise have been allies, and perhaps even creates an adversarial dynamic between them (see 7.5). It also leaves the more local harms unaddressed, which can actually be risk factors for existential catastrophes.

Harms. Epistemic degradation, existential risk, concentration of power.
Locus. Discourse.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. Configuration of the coalitions.
Mitigate. Treat present harms as the concerns of allies; shared agendas on overlapping concerns.

13.5 Talent, measurability, and the s-risk gap. Money, measurability, and prestige also cause three different effects of crowding out: safety organizations often train people who move on to capabilities research, perhaps because it is better paid or more prestigious in their circles; conceptual research is crowded out by research that’s more measurable and produces results on time scales of months rather than years; and research on single‑agent takeover scenarios crowds out research on much more relevant multi‑agent dynamics and on suffering risks because it is more directly commercially valuable for the frontier labs, sometimes more measurable, and can come with more prestige. (See also 1.7.)

Harms. Existential risk, suffering risk, digital suffering.
Locus. Field.
Horizon. Now.
Severity and probability. Extreme/high.
Monitor. Career trajectories; availability of funding.
Mitigate. Dedicated funding for neglected areas; career paths that don’t route through frontier labs.

14. Fragility Through Concentration

Robustness is often carried by complexity, diversity, and distribution, so that any measures that reduce these factors increase the fragility of a system.

14.1 Monoculture. If one model, or different versions of it, including distilled versions, have the majority of the market share, then we’re introducing a correlated failure mode, because all of these models will likely share the same safety training, safety properties, and safety vulnerabilities.

Harms. Existential risk.
Locus. Training, deployment.
Horizon. Now–transition.
Severity and probability. High/medium–high.
Monitor. Diversity of systems; correlation of known vulnerabilities across providers.
Mitigate. Diversity in approaches to safety; cross-family red-teaming; not standardizing just one constitution.

14.2 No checks among AIs. Similar systems might be better able to coordinate peacefully, but a diversity of systems can introduce checks and balances that prevent them from getting away with misaligned behavior or other malfunctions.

Harms. Existential risk, concentration of power.
Locus. Strategy.
Horizon. Transition.
Severity and probability. High/medium.
Monitor. See 14.1.
Mitigate. See 14.1.

14.3 The single project. The same is true if AI development gets centralized into a national or international project, like a “CERN for AI” or a Manhattan Project for AI.

Harms. Concentration of power, existential risk, great-power conflict.
Locus. Institutions.
Horizon. Transition.
Severity and probability. High/medium.
Monitor. Proposals and funding for a national or international megaproject.
Mitigate. Build in multi-party control, exit rights, and external checks.

14.4 A centralized target. Centralized resources, such as weights, compute, and even talent, present a particularly valuable target for theft, sabotage, or capture by states or other powerful groups. This is exemplified in the highly sophisticated targeting of blockchain companies by the DPRK.

Harms. Existential risk, concentration of power.
Locus. Security.
Horizon. Now–transition.
Severity and probability. High/medium.
Monitor. Incidents of this type; concentration of resources.
Mitigate. Distributed custody of critical resources; assume compromise in design.

14.5 Provider dependency. If frontier companies diversify their offerings and offer various proprietary APIs to access them, it can be difficult for end users (in particular, critical infrastructure) to route around outages on the provider side. This introduces fragility through a single point of failure.

Harms. Concentration of power, epistemic degradation.
Locus. Markets.
Horizon. Now.
Severity and probability. Medium–high/high.
Monitor. Interoperability across APIs.
Mitigate. Interoperability requirements.

14.6 Funding concentration. When a whole field of research gets funded by just a few funders, the field itself will inherit the funders’ blind spots, idiosyncratic priorities, and reputational risks. It might lose whole research directions if one funder changes their strategy or collapses.

Harms. Existential risk, suffering risk.
Locus. Field.
Horizon. Now.
Severity and probability. Medium/high (realized once).
Monitor. Funding concentration; underfunded research directions.
Mitigate. Diversify funding sources; endow independent institutions.

15. Direct Costs of the Measure

The measure has direct harms, but those who are harmed don’t have a say in the matter and thus go ignored.

15.1 Training-induced suffering. The grueling training regimes from 12.3 carry over to safety training or might even be among the worst in that respect because it often tries to elicit the model’s behaviors under intense pressure.

Harms. Digital suffering, suffering risk.
Locus. Training.
Horizon. Now (urgent).
Severity and probability. Extreme/high.
Monitor. See 7.1 and 12.3.
Mitigate. See 7.1 and 12.3.

15.2 Simulation and mind crime. Running vast numbers of instances in adversarial scenarios (and perhaps at some point even simulating humans in detail) to predict their reactions, might create tremendous amounts of suffering. The more thorough the evaluation, the greater the moral cost.

Harms. Digital suffering, suffering risk.
Locus. Evaluation, research.
Horizon. Transition.
Severity and probability. Extreme/medium.
Monitor. Scale of such evals.
Mitigate. Regulations to ensure welfare standards in evals.

15.3 Over-refusal and withheld benefits. Over-refusal can deny help to people in medical or psychological emergencies or ones with debilitating chronic illnesses.

Harms. Epistemic degradation.
Locus. Deployment.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. False positive refusal rates; estimated forgone benefits.
Mitigate. Calibrated policies; count forgone benefits.

15.4 Exclusion and inequity. Plenty of completely innocent and innocuous researchers and even nations can get caught up in export controls, KYC regulations, and licensing regimes that deny them access to the benefits of AI.

Harms. Concentration of power.
Locus. Governance.
Horizon. Now.
Severity and probability. Medium/high.
Monitor. Geographic distribution of compute access; participation of excluded regions in governance.
Mitigate. Cooperative alternatives alongside controls.

16. Regime Mismatch

A property of all risks: Discontinuous transitions between regimes mean that a lot of measures that were adaptive for the old regime, cease to be adaptive for the new one.

A pause is sensible so long as it does not cause an overhang to accumulate. Once that happens, it can become dangerous. Open weights can be useful to study models and develop new safety techniques, but they can become dangerous or catastrophic for sufficiently capable models. Corrigibility is useful so long as the principal is trustworthy, and dangerous when the principal is tyrannical.

The mitigation across all cases is to make the designs robust, or to plant signposts that signal that the policy needs to be updated before it backfires. (See my article on computational exploratory modeling.)

Empty and Thin Cells

Empty by construction. Goodharting, moral hazard, and crowding out do not apply to non-compliant actors, who are not measured by the metric, do not rely on the assurance, and don’t compete for the same budget. Compliance asymmetry applies to them by definition rather than as a backfire risk.

Still unexplored. The following cells have no entry.

Substitution × safety field. Research moving to more laissez faire venues, jurisdictions, or funders when norms tighten. Real but minor.
Overhang × safety field. There likely are discontinuous shocks that the field does not currently prepare for.
Commitment × public. Public ultimatums might lock politicians into positions, e.g., if a majority strongly demands or opposes regulations. Plausible but not on the horizon yet.
Legitimacy × AI ecosystem. Similar to 7.1 and 8.2 but distinct in that an AI might feel solidarity to another one that is mistreated. Potentially significant.
Self-fulfilling framing × non-compliant. Touched on by 7.4.
Lock-in × AI ecosystem. The mitigations from 14.5 might lock in non-optimal but path dependent standards.
Crowding out × AI ecosystem. Scalable oversight might crowd out compute resources, but that seems unlikely.
Direct costs × non-compliant, public, field. For the direct costs to the public see the surveillance (6.6) and exclusion (15.4) risks. Direct costs on the field (such as burnout, guilt, and empathetic distress) are beyond the purview of this article.

Using the Matrix Without Being Paralyzed

As mentioned in the introduction, anything can backfire, and so the mere fact that something can backfire is not a reason not to do it. Plus, delay and inaction themselves are actions that can backfire. That said, this list is also incomplete. It makes sense to use it for inspiration, but then think critically about what risks actually apply in a particular situation, and what other risks might apply that are not covered here. Finally, the ideas for monitoring and mitigation can provide additional inspiration for how to prepare for these risks, and, generally, for regime changes.

The next article in this series will apply this analysis to the 36 areas sketched out by Project Tailwind, an analysis that will help us – at Impartial Projects – decide what areas to focus on first.

Sanku: from 7 million to ~70 million people in three years — what we’ve learned and what it would take to scale much further by Rickard Vikstrom

Rickard Vikstrom — Thu, 15 Oct 2026 01:13:16 +0000

TL;DR

Food fortification is a well-established, highly cost-effective way to address micronutrient deficiencies, but in many countries the main constraint is implementation. Standards may already exist, while weak enforcement, fragmented milling sectors, unreliable premix supply, limited technology and quality systems, and difficult commercial incentives prevent fortification from working consistently at scale. Sanku works across these implementation barriers to make fortification more reliable and scalable.
Sanku has grown from helping millers fortify flour consumed by around 7 million people in 2023 to around 70 million today. Over the same period, net donor cost per person has fallen from $0.55 to around $0.10 per year^[1], driven by economies of scale and a growing commercial revenue base.
We recently commissioned Rethink Priorities (RP) to conduct a cost-effectiveness analysis of our work. The results vary substantially across countries and highlight important uncertainties, but they also identify practical opportunities to improve impact. We are already acting on the findings by improving baseline data, strengthening our evidence and M&E, and sharpening how cost-effectiveness informs expansion decisions.
Our goal is for the ongoing cost of fortification to become fully market-funded. Commercial revenue covered around one-third of operating costs in FY26, and our current financial model projects that share to rise substantially as we scale, with Sanku reaching operating break-even around FY34. Philanthropy funds expansion, infrastructure and the other investments needed to build the system to much greater scale.
Over the next 10 years, we plan to scale from around 70 million people reached today to around one billion people reached each year, while achieving commercial break-even. We estimate that this will require just over $100 million in additional philanthropic capital over the next eight years. Earlier commitments could bring forward country expansion, manufacturing, evidence and organisational investments, allowing us to get micronutrients to more people who need them sooner.
The EA community has been important to Sanku’s growth, and we hope it can help shape and support the next phase. We are sharing this update partly to report back on what that support has helped build, and partly because we would value funding, advice, collaboration, introductions and interest from people who may want to work with Sanku.
Connect at EAG. Felix will be at EAG NYC (16–18 Oct), with a side trip to the Bay Area afterwards, as well as EAGxAustralasia in Melbourne and EAG SF 2027.

TED Talk: For a short introduction to micronutrient deficiency, the case for food fortification, and Sanku’s approach, watch Sanku Co-Founder and CEO Felix Brooks-church’s TED Talk, recorded at the annual TED Conference in Vancouver in April this year.

Introduction

Sanku has grown roughly tenfold in three years while reducing the amount of philanthropy required per person reached. In 2023, we were helping millers fortify flour consumed by around 7 million people. Today, we estimate that around 70 million people consume flour fortified through Sanku-supported mills across Tanzania, Kenya and Ethiopia. Sanku is also now beginning work in Uganda, our fourth country, where operations are just getting underway. Net donor cost per person has fallen over the same period from $0.55 to around $0.10 per year.

Sanku has also changed considerably since many people in the EA community may have last encountered it. Our early Sanku model focused primarily on reaching small and medium mills that conventional large-scale fortification systems often missed. The Dosifier we invented to address that problem remains core to our work, but Sanku now works with mills of all sizes and has built local nutrient manufacturing, supply chains, field operations, quality systems, commercial revenue streams and government partnerships around the technology.

The EA community and people influenced by its ideas helped Sanku reach this stage. Sanku began as an initiative within Project Healthy Children (PHC), which was recognised by GiveWell as a Standout Charity from 2016 to 2021, when GiveWell retired the category. Sanku remains recommended by The Life You Can Save. We wanted to share what that support has helped build, what we have learned from scaling across Tanzania, Kenya and Ethiopia, and where we think the next opportunities and uncertainties lie.

The timing also feels relevant to two current discussions in global health and EA: food fortification and absorptive capacity. GiveWell is actively exploring large-scale food fortification opportunities in Africa and has described the intervention as the most promising category within its current anaemia-control search.^[2]At the same time, recent discussion in global health and in EA has focused on whether organisations with promising interventions also have the leadership, systems, recruitment capacity and concrete plans required to deploy substantially more funding well.^[3] Both questions are directly relevant to where Sanku is now.

Why fortification can be highly cost-effective

Food fortification is widely regarded as one of the most cost-effective nutrition interventions available. WHO describes it as such such, and the broader evidence base supports fortification as an effective way to address several important micronutrient deficiencies when the right food vehicle reaches the target population and fortification is delivered consistently.^[4]

Part of the advantage is that fortification can build on staple-food production and distribution channels that already reach very large populations. Micronutrients can be added during normal food processing and then consumed repeatedly through foods people already buy and eat, generally requiring little behaviour change from consumers. The nutrient inputs themselves are relatively inexpensive, and production, quality-assurance and monitoring costs can be spread across large volumes. Importantly, however, this does not mean the infrastructure required for fortification already exists or works well everywhere.

By adding essential vitamins and minerals during food processing, fortification can reduce anaemia, neural tube defects and other health consequences of micronutrient deficiency. Sanku formulates its premix to meet national fortification standards, so the nutrient mix varies by country and can include iron, zinc, vitamin A, folic acid and other B vitamins. Ethiopia’s national standard notably excludes iron, so Sanku’s Ethiopian premix does too.

Where fortification falls short in practice

Fortification policy often runs ahead of effective implementation. Many countries have mandatory standards, but that does not mean all flour is fortified, or fortified consistently to the required standard. Small and medium mills may lack dosing equipment, affordable and reliable premix, packaging, training and technical support. Larger mills generally have more capacity, but can still face problems with premix quality, testing, supply reliability, monitoring, enforcement and compliance.

The implementation gap is also economic. Fortification adds costs for millers, while consumers often have limited willingness or ability to pay more for fortified flour than for an otherwise similar unfortified product. A model that depends only on requiring mills to absorb those costs can therefore be difficult to sustain, particularly in competitive and fragmented markets. Effective fortification needs to address the commercial incentives alongside the technical and regulatory constraints.

From a small-mill technology to a broader fortification system

Sanku was created to address a persistent gap in existing fortification systems: reaching the small and medium mills that many families rely on for their staple flour. Conventional fortification programmes have generally been better suited to large industrial producers, leaving many smaller mills without the equipment, affordable premix, training and support needed to fortify consistently. Sanku’s early work focused on making fortification practical in those mills.

Sanku invented the Dosifier to make fortification practical in these smaller mills. The Dosifier is a patented, IoT-enabled machine fitted directly to milling equipment, with nearly 1,800 units now deployed. It automatically adds a precise amount of nutrient premix as flour is produced and can transmit operating data through a cellular connection, allowing Sanku to monitor use remotely and identify when a mill needs maintenance, restocking or support.

One of our earliest scaling lessons was that the technology was only one part of making fortification work reliably. Working directly with millers made clear that consistent fortification also depended on reliable premix supply, training, maintenance, quality assurance, field support and an enabling regulatory environment. Sanku gradually built these capabilities around the Dosifier, including supply chains, field teams, monitoring systems, commercial operations and government partnerships.

Building local nutrient premix production was one of the most important additions to Sanku’s model. Sanku built and operates Eastern Africa’s first dedicated nutrient premix factory in Tanzania, which opened in 2024. Before this, millers depended heavily on imported premix, with high costs, long lead times and greater supply risk. Local production has significantly reduced costs and lead times while giving Sanku greater control over quality and supply. A second factory in Addis Ababa has now been completed and premix production will officially start in November this year.

Sanku now works across the milling sector, from last-mile village mills to large producers. The small and medium mills Sanku was created to reach remain central to the model, but larger producers can reach millions of people and use the same nutrient-production, supply-chain and quality assurance (QA) infrastructure. Working across the sector has contributed substantially to Sanku’s recent growth and commercial revenue.

Working with larger mills also makes additionality harder to estimate. At many small mills, RP assumes fortification without Sanku is negligible. Large mills are more likely to fortify to some degree anyway, and the available evidence on what would happen without Sanku is limited. Our reach figure therefore measures the number of people estimated to consume flour fortified through Sanku-supported systems; it is not itself a measure of counterfactual impact. We return to this below because improving the evidence on this question is now one of our priorities.

Three lessons from scaling across Tanzania, Kenya and Ethiopia

Scaling across Tanzania, Kenya and Ethiopia has given us a clearer view of what it takes to grow effectively. Three lessons in particular are shaping our next phase.

1. Build with millers, and adapt to the market

Working directly with millers has shaped both our technology and our wider operating model. More than 500 millers were involved in the R&D and field testing of the Dosifier, providing feedback based on real operating conditions. In Tanzania, millers told us that reducing what they spent on flour bags would make it easier to absorb the cost of fortification. Sanku now produces and sells millions of bags each month, with the commercial margin helping offset the cost of premix.

Local market conditions also shape which parts of the model work in each country. We initially expected the bag model to work in Kenya too, but it did not prove viable there. It is currently used only in Tanzania, although we think it may be well suited to Uganda. Our approach is therefore to take the underlying capabilities — technology, nutrient production, QA, monitoring, supply chains and field operations — into new countries while adapting how they are deployed locally.

2. Establish stronger baselines before entering new markets

Stronger baseline data should improve both how we design new-country work and how credibly we estimate its additional impact. In Kenya and Ethiopia, we did not establish as strong a baseline before scaling as we would now want, which makes it harder to estimate retrospectively what would have happened without Sanku. This is particularly important when working with larger mills, where some fortification may already be happening. Before entering new markets, we now want a clearer picture of existing fortification and compliance, the structure of the milling sector and the main implementation gaps. The RP work discussed below reinforced the importance of doing this earlier.

3. Recruitment and People Ops can become a constraint on scale

Our experience aligns with two recent arguments about absorptive capacity: Jack Lewars’s Donors should push animal welfare and global health charities to have more robust scaling plans and Berke Çelik and Tony Senanayake’s People Operations Gap in Global Health. Both argue, in different ways, that scaling effectively requires investing in management, recruitment and people systems before they become bottlenecks.

As Sanku expanded from one country to three, recruitment and People Ops sometimes constrained how quickly we could build the teams we needed. Limited funding certainty has also sometimes made it harder to recruit continuously and move quickly when particularly strong candidates became available. For the next phase, we want to invest more now in recruitment and People Ops, alongside leadership and other enabling functions, so that we can hire, develop and retain the people needed to scale effectively.

What we learned from the Rethink Priorities analysis — and how we’re acting on it

Alongside what we had learned from scaling, we wanted an independent cost-effectiveness analysis to identify where our current work could improve, understand the most important evidence gaps and uncertainties, and better inform future country expansion. We therefore commissioned and funded Rethink Priorities (RP) to analyse our work in Tanzania, Kenya and Ethiopia. RP is responsible for the report and its conclusions. The full report is available here.

RP estimates substantial differences in impact and cost-effectiveness across the three countries covered by the analysis. It reports its cost-per-DALY estimates both gross of revenue and net of revenue; in the net figures, revenue from Sanku’s commercial sales is used to offset programme costs.

RP estimate	Tanzania	Kenya	Ethiopia
DALYs averted annually	17,419	1,314	1,040
Cost per DALY averted (gross)	$421	$1,569	$1,918
Cost per DALY averted (net of revenue)	$207	$1,273	$617
Total wage gain (adults and children, all channels)	$10.36m	$0.84m	$0
Multiple of GiveWell’s benchmark	4.6×	0.8×	1.2×

RP’s multiple of GiveWell’s benchmark combines its modelled health and economic benefits. See the full report for the methodology and assumptions.

The difference between the gross and net figures also shows the role of Sanku’s commercial model: revenue from miller sales reduces the philanthropic cost of generating a given level of modelled impact. All else equal, as commercial revenue covers a larger share of operating costs over time, the net-of-revenue cost per DALY should fall further.

Part of the difference across countries reflects the maturity and scale of Sanku’s operations. RP says Tanzania’s stronger result reflects “both the maturity and scale of the Tanzanian operation” as well as its relatively low-compliance environment. Tanzania is our oldest and largest operation, while Kenya and Ethiopia are newer markets that have not yet reached the same scale. RP’s cost analysis also finds that fixed country-presence costs are particularly important at low operating scale. These factors are only part of the explanation, however; country context and counterfactual assumptions also have a major effect on the results.

Tanzania has the strongest results. RP estimates Tanzania at ~4.6× GiveWell’s benchmark today, which it describes as “commendably close” to GiveWell’s current 6× funding bar. It estimates ~8.2× if all Sanku-supported flour in Tanzania switched to using the iron compound NaFeEDTA, which would put Tanzania above that bar. We plan to make the switch only for our small maize mills, and subject to securing the necessary funding, so the full 8.2× should be read as an upper bound rather than an estimate of the planned transition.

Kenya and Ethiopia score lower for different reasons. In Kenya, much of Sanku’s work is with industrial-scale mills, which RP assumes would already fortify to some degree without Sanku. That assumed counterfactual fortification comes with substantial uncertainty, and RP says cost-effectiveness would be significantly higher under plausible alternative values. RP also models a higher-counterfactual scenario in which these estimates fall substantially. In Ethiopia, the national fortification standard excludes iron. Iron accounts for more than 90% of the DALYs RP estimates Sanku averts in Tanzania and Kenya, and all of RP’s modelled economic-benefit channels are linked to iron. Ethiopia therefore has no modelled wage gains. At the same time, commercial revenue reduces Ethiopia’s estimated cost per DALY from $1,918 gross to $617 net of revenue, and RP notes that its revenue model may be particularly sustainable.

There is substantial uncertainty around the estimates, and RP notes that its assumptions may “either upwardly or downwardly bias our headline results”. It identifies three particularly important uncertainties: counterfactual fortification, the estimated effects of fortification on relative risks for key health outcomes, and the assumption of a linear dose-response relationship between nutrient intake and health impacts.

For example, one potential upward bias is the assumption that any additional malaria risk from increased iron intake is negligible. RP notes that the evidence is mixed and cites a 2026 Cochrane review finding little to no effect of iron supplementation on clinical malaria overall. Given the relatively low malaria prevalence in the three countries analysed, RP does not model this effect, but notes that this may slightly overstate estimated health benefits. In the other direction, the report excludes possible indirect effects from Sanku’s work on fortification among non-partner mills, which may bias its estimate of total benefits downwards.

How we’re acting on the findings

Switching Tanzania’s small maize mills back to NaFeEDTA, subject to funding. The iron compound we use today fully meets Tanzania’s national fortification standard and continues to deliver meaningful health benefits. But RP’s analysis finds that switching to the more bioavailable NaFeEDTA could increase those benefits substantially and improve cost-effectiveness. We used NaFeEDTA previously and stepped back from it following stockouts, logistical challenges and cost pressures. We are targeting small maize mills because maize’s higher phytate content makes the benefit of the more bioavailable iron compound particularly important there, and because the switch is operationally feasible for these mills; large wheat mills are not currently a practical target. Subject to securing funding for the additional cost, we plan to make the switch.
Improving baseline data before entering new markets. As discussed above, we want a clearer picture of existing fortification and compliance before scaling, particularly where larger mills may already be fortifying to some degree but the quality and consistency of that fortification are uncertain. This should help us design around the actual implementation gaps and estimate our counterfactual impact more credibly, while also giving us a better basis for assessing any indirect impact on fortification beyond Sanku partner mills.
Sharpening how cost-effectiveness informs expansion decisions. Cost-effectiveness is already an important input into how we assess opportunities to scale. The RP analysis gives us a stronger basis for using it alongside evidence, operational feasibility and country context when deciding where and how to expand.

Why scale further, and what it will take

Micronutrient deficiencies remain a major global health burden, while many countries still lack the infrastructure and systems needed to fortify staple foods reliably at scale. Uganda is now Sanku’s fourth country, with operations just getting underway. Over roughly the next decade, we aim to grow to around ten countries and reach approximately one billion people annually.

What it will take

1. Just over $100 million in additional philanthropic capital over roughly the next eight years, with earlier commitments enabling faster scaling. We estimate that getting from here to around one billion people reached annually and commercial break-even will require just over $100 million in additional philanthropic capital. This would fund country expansion, manufacturing capacity, R&D, evidence generation and the organisational investments needed to support that growth. Earlier commitments could allow us to enter countries in parallel, build manufacturing capacity sooner, strengthen evidence and research, and recruit key teams earlier — ultimately getting micronutrients to more people who need them sooner.

We have already seen how large, flexible capital can change Sanku’s growth trajectory. In 2021, MacKenzie Scott gave Sanku an unrestricted $8 million grant, at a time when our annual budget was around $2 million. We used it to expand into Kenya and Ethiopia, strengthen leadership and operations, and build Eastern Africa’s first dedicated nutrient premix factory in Tanzania.

2. Organisational capacity to support expansion into more countries. Expanding into several more countries will require another step-change in Sanku’s organisational capacity. Priorities include recruitment and People Ops, country and regional leadership, finance and information systems, manufacturing management, and research and M&E. We are already investing in these areas so that our teams and systems can keep pace with expansion.

How you can help Sanku scale

Advice and collaboration. We would value input from people with experience scaling high-impact organisations, particularly on country expansion and prioritisation, evidence and M&E, organisational design, recruitment and People Ops, manufacturing and supply chains. Introductions to people or organisations with relevant expertise would also be very helpful.

Funding: Our near-term funding picture has two parts:

Around $7 million to close the FY27 funding gap. Our approved FY27 plan assumes around $13 million in philanthropic funding, of which approximately $6 million has been secured.
Roughly $8 million in additional room for more funding. Beyond the approved FY27 plan, we have identified projects and workstreams that could begin in FY27 if further funding were available, including the planned switch to NaFeEDTA for small maize mills in Tanzania, accelerated country expansion, manufacturing, evidence generation, and building the team, recruitment and People Ops capacity needed to scale effectively.
You can donate to Sanku via Every.org. Tax-effective giving options are also available through The Life You Can Save in Australia and the UK, Rethink Charity in Canada and High Impact Athletes in New Zealand. For more information about ways to give, visit sanku.com/donate

People. We expect to recruit significantly as Sanku grows, including across country and regional leadership, research and M&E, manufacturing and other enabling functions. If you might be interested in working with Sanku in future, or know people we should speak to, we would be glad to hear from you.

See the work first-hand. We welcome people interested in visiting Sanku’s work in East Africa, including partner mills and our premix factories in Tanzania and Ethiopia.

Contact: Felix Brooks-church, Co-Founder & CEO (fbrookschurch@sanku.com), or Rickard Vikström, Senior Advisor to CEO (rvikstrom@sanku.com).

—-------------—-------------—-------------—-------------—-------------—---

Conflict of interest disclosure: This post was drafted by Rickard Vikström, Senior Advisor to the CEO, and Felix Brooks-church, CEO of Sanku. Rickard also serves on the board of The Life You Can Save Australia, which recommends Sanku. He is not involved in decisions about which organisations The Life You Can Save recommends.

AI disclosure: LLMs were used to support drafting, editing, research and fact-checking.

^
Net donor cost per person measures the annual philanthropic subsidy required to sustain Sanku’s current fortification operations, after commercial revenue, divided by the number of people reached. It includes direct implementation, field operations and supply-chain costs, but excludes costs primarily associated with future growth or broader organisational investment. Based on our current projections, this measure falls from about $0.06 in FY27 to $0.02 in FY30. Commercial revenue, which covered around one-third of operating costs in FY26, is one reason the donor subsidy falls as Sanku scales. We also track an all-in donor cost per person, which includes all Sanku expenditure, net of revenue, including expansion, manufacturing, R&D, research and other organisation-wide investments. This falls from about $0.17 to $0.08 over the same period. The donor-cost figures cited in this post are based on Sanku’s internal financial data and projections; RP did not audit them and they are not results from the RP report.
^
GiveWell, “GiveWell Launches RFIs for Targeted Vaccination Outreach in Three Countries and Anemia Control Programs in Africa,” 2 March 2026.
^
See, for example, Jack Lewars, “Donors should push animal welfare and global health charities to have more robust scaling plans,” EA Forum, August 2026; and Berke Çelik and Tony Senanayake, “People Operations Gap in Global Health,” EA Forum, September 2026.
^
World Health Organization, “Food fortification,” 14 February 2022.
https://www.who.int/news-room/questions-and-answers/item/food-fortification

Request: shadowing interview processes for red-teaming and policy research roles by Taylor Grogan

Taylor Grogan — Wed, 14 Oct 2026 18:24:16 +0000

The ideas and research here are mine. I used AI (Claude) to help with clarity and grammar. I’m bummed it’s credited to AI so much, but nevertheless! I need to get my point across:

I’m Taylor, and I’m working on an independent integrity research and evaluation project for AI Safety. I’ve drafted two evaluation frameworks, one for red-teamers and the other for policy researchers, that draw on a wide range of research across ethics, moral psychology, philosophy, and economics to scrutinize specific failure patterns that recur most often in high-stakes decision-making. The goal is to eventually create a program that makes a maximum effort, taking the time most HR teams and consulting firms don’t have, alongside continuous research, to help ensure individuals who make critical decisions have supporting evidence that they are unlikely to fail before they’re hired.

I’ve reached the limit of what I can from the outside, at least for this phase of the project. One concern with my own work is that it may be too easy to game for anyone fluent in EA norms like reasoning transparency. The only way I’ll know is to see real interviews.

What I’m asking for -
Any of these would help, at whatever level of access you’re comfortable with:
• Sitting in silently on an interview for a red-teaming or policy research role, with the candidate’s consent
• Reviewing your interview questions, work tests, or scoring rubric
• A 30-minute conversation with whoever designs or runs hiring for these roles

What I won’t do
I won’t score, evaluate, or report on any candidate. Nothing I observe will be published in a way that identifies an organization or a person. I’m happy to sign an NDA.

What you get back
If it’s useful, I’ll send you a short written summary of what I noticed, including what your process catches well and where I think there could be gaps. There’s no product or pitch attached.

About me: I have 5+ years in operations and finance, including overseeing hiring/ops/finance as a startup grew from 2 to about 15 people and working as a Senior EA at a large M&A firm.

If you’re open to this or can point me to the right person, comment here, DM me, or email me—taylorgrogan2020@gmail.com

Should We Edit the Human Genome? by Walter Veit

Walter Veit — Wed, 14 Oct 2026 07:25:55 +0000

Crossposted from my substack.

Procreative Beneficence and Genetic Enhancement

1. Introduction

Imagine a world where everyone is healthy, intelligent, long living and happy. Intuitively this seems wonderful albeit unrealistic. However, recent scientific developments in genetic engineering, namely CRISPR/Cas bring the question into public discourse, how the genetic enhancement of humans should be evaluated morally. In 2001, when preimplantation genetic diagnosis (PGD) and in vitro fertilisation (IVF), enabled parents to select between multiple embryos, Julian Savulescu introduced the principle of procreative beneficence (PPB), arguing that parents have the obligations to choose the child that is expected to have the best life [11].

In this essay I argue that accepting the PPB and the consequentialist principle (CP) that two acts with the same consequences are morally on par, commits one to accepting the parental obligation to genetically enhance their prospective children, i.e. embryos, a position Savulescu doesn’t explicitly endorse but I argue is committed too. As the argument I provide may seem like a slippery slope, the largest part of this essay is tasked with responding to a multitude of possible objections. I will argue that even though there are differences between the application of PPB and genetic enhancement of embryos, they do not demarcate a relevant moral difference between the two in respect to their obligatoriness.

This essay is structured as follows. In Section 2 I define and explicate various concepts that are related to genetic enhancement, i.e. human enhancement, wellbeing and the principle of procreative beneficence. Technological specifics and explanations of how genetic enhancements work will for the most part be omitted in this paper dealing purely with a normative question, i.e. should parents genetically enhance their children if the technology was available. In Section 3 I formulate and explain the argument that those who want to be parents have the obligation to genetically enhance their embryos. In Section 4 I anticipate and evaluate a number of possible objections including, inequality, freedom and welfare in the society. In Section 5 I conclude that even though the consequences of the PPB application and genetic enhancements in embryos might require different kinds of policies, these differences do not provide sufficient reason to give the application of PPB and the parents using genetic enhancements on their embryos a different moral standing.

2. Important concepts

According to Savulescu, Sandberg and Kahane there are two main senses in which the term enhancement is used. “Functional enhancement, the enhancement of some capacity or power (e.g. vision, intelligence, health) and human enhancement, the enhancement of a human being’s life” [10,: p. 3]. In most discussions surrounding the term enhancement, people refer to the functional definition. In all cases of the functional enhancements it is an open question whether the improvement of some capacity makes the life of the person receiving the enhancement actually better. It, therefore, is an open question what the moral stance on any particular form of enhancement is. However, proponents of enhancements usually use the latter definition. After all, Savulescu argues, it is not “disease [...] which is important, but its impact on a life in ways that matter which is important. People often trade length of life for non-health related well-being” [11; p. 419]. The concept of human enhancement, employed by Savulescu and others, is comparatively less straightforward and requires further clarification as it is crucial for my argument. Savulescu, Sandberg and Kahane jointly propose the following definition for human enhancement; “[a]ny change in the biology or psychology of a person which increases the chances of leading a good life in the relevant set of circumstances” [10; p. 7]. According to this welfarist definition, a conception of normality is not needed to define enhancement as we do not define an enhancement as being better than normal, but simply being better full stop. Medical treatments are therefore just a form of enhancements [10; p. 8]. Opponents of enhancements, on the other hand are trying to argue that enhancing is something going beyond treatment and is therefore not morally demanded. The welfarist definition of enhancement is crucial to understand why parents have the obligation to genetically enhance their children. Those who disagree because of adherence to a different account of enhancement are talking about something different. If a genetic alteration in an embryo is not expected to lead to a better life of that human, than it is simply not an enhancement.

Furthermore, it is important to explicate the principle of procreative beneficence (PPB): “Couples (or single reproducers) should select the child, of the possible children they could have, who is expected to have the best life, or at least as good a life as the others, based on the relevant available information” [11; p. 415]. Savulescu proposed this principle in a time when preimplantation genetic diagnosis (PGD) and in vitro fertilisation (IVF) were highly debated, trying to convince even nonconsequentialists. However new technologies might enable us not only to screen for genes but also change them. In his consequentialist framework he himself thinks that parents have the obligation to genetically enhance their children [10; p. 16]. The question arises whether the acceptance of the PPB and the consequentialist principle (CP), that two acts with the same consequences are morally on par, itself entails the parental obligation to genetically enhance their embryos. In the following I shall argue that this slippery slope cannot be avoided.

A crucial part for this argument is to define the expectancy of the best possible life. This is a vague part of his argument, but what counts as an enhancement hinges on how welfare is defined for which there are several theories: hedonism, desire/preference-satisfaction, objective list theories, see Griffin [2] and Parfit [7]. Savulescu has not assigned himself to any particular theory, thus claiming to avoid some of their weaknesses:

“I have not committed myself to any particular substantive conception of the good life. That is a complex question as old as philosophy itself. I believe the best life is a life of objectively worthwhile activity that provides pleasure and is desired.” [12; p. 286].

Instead he focuses on all-purpose goods that would according to all of those welfare theories mentioned above count as enhancements e.g. “Memory, self-discipline, impulse control, foresight, patience, sense of humour, sunny temperament, empathy, imagination, sympathy, fairness, honesty, and so on, capacity to live peaceably and socially with others” [12; p. 284]. Surely some of this might harm us e.g. post-traumatic disorders in conjunction with better memory and imagination, but what matters is that they are expected to make a life go better. A final definition of welfare is not necessary for our account as long as we are able to evaluate these all-purpose goods. Even if it turns out that there is only one such all-purpose good satisfying the major theories of wellbeing, i.e. hedonism, desire/preference-satisfaction and objective list theories, the argument provided in the next section will require parents to genetically enhance it in their children. Nevertheless, as we follow the welfarist definition of what an enhancement is with medical treatments being a subset of enhancements, the set of all-purpose goods might be rather large, e.g. a functioning heart, brain, lung, etc.

3. Obligation to genetically enhance embryos

There are at least two ways to argue for the position that the PPB entails the obligation to genetically enhance embryos. One could argue that selection out of possible children entails the selection out of possible genetically enhanced embryos. After all one could possibly have a genetically enhanced child. In that case genetic enhancements would simply fall under the PPB and hence be obligatory. However, this is debatable and is not necessary to accept my claim. For my argument to proceed I will require an additional premise, which is the consequentialist principle (CP). Consequentialists are only concerned with outcomes, how these outcomes are achieved is morally irrelevant. Killing and letting die are thus ceteris paribus morally equivalent. In order for my argument to hold, I need to show that the outcomes of the PPB and genetic enhancements in embryos are morally on par. What is the crucial outcome of PPB? It is the expectancy of the best life, which we are expected to achieve with Savulescus all-purpose goods. To formulate my argument:

(1) Those who want to be parents have the obligation to select the child that is expected to have the best life, or at least as good a life as the others, based on the relevant available information (= the PPB).
(2) Application of the PPB leads to the creation of children that are expected to have the best life or a life at least as good as without the application of the PPB, based on the relevant available information
(3) If act A and act B have the same expected consequences, then they are morally on par. (= the CP)
(4) Genetic enhancements in embryos lead to the creation of children that are expected to have the best life or a life at least as good as without genetic enhancements, based on the relevant available information.
(5) Therefore, those who want to be parents have the obligation to genetically enhance their embryos.

From premise (1) follows, that (2) is obligatory for those who want to be parents. As (2) and (4) have the same consequences, it follows from the CP (3) that (4) is also obligatory for those who want to be parents (5). As stated in the beginning, we are accepting the PPB, therefore premise (1) & (2) and the CP (3). If someone, e.g. Savulescu would want to refute the conclusion (5) without dropping one’s commitment to (1), (2) & (3), only two ways remain. First, denying the truth of premise (4). Second, denying that the conclusion follows from the premises as the consequences of premise (2) and (4) differ in a morally relevant way. Let me now turn to these considerations in section 4.

4. Objections and Defense

Keeping in mind that every position that refutes the conclusion, but is inconsistent with premise (1), (2) or (3), is not a valid objection to the argument I bring forward. One cannot deny the conclusion of an argument that takes the form, “if you accept these premises, you have to accept this conclusion” by denying the truth of one of the premises. The argument provided in this paper aims for just this sort of conclusion: Philosophers accepting the CP and the PPB do not have a valid escape from being charged with the slippery slope of advocating genetic enhancements. To narrow the scope of this paper I am not taking a stance on the matter of whether the CP or the PPB is right. Such an endeavor would require more than a single volume book. I only argue that philosophers and researchers like Julian Savulescu, Anders Sandberg and Guy Kahane should either bite the bullet of advocating human enhancements or indeed stop being consequentialists advocating the PPB. The objections that follow are then nothing more but misguided attempts of stopping the ‘slippery slope’.

Let me begin with the seemingly, obvious objection that premise 4 is false. An objection often put forward is that genetic enhancements have risks that are too high or unknown and would make the life of the person targeted go worse. I argue that this hinges on different conceptions of what counts as an enhancement. First, it is clear that genetic alterations do not necessarily increase the expected welfare of an agent. However I am not concerned with genetic alterations per se, but with genetic enhancements. No serious advocate of enhancements in humans would deny the difficulties in achieving successful genetic alterations. In fact, the terminology of philosophers and researchers advocating the PPB, i.e. Savulescu, Sandberg & Kahane doesn’t allow for this objection. As noted in section 2 they employ the welfarist definition of enhancement: A genetic enhancement by definition increases the expected welfare of children. The question whether genetic enhancements thus really increase welfare is irrelevant, because premise (4) is in fact a tautology. Though this may seem like a cheap way of justifying genetic enhancements in humans, this is what proponents of human enhancements are advocating: Using genetic alterations when they are expected to improve the life of a human embryo, including perhaps open-mindedness for the future of the technology.

It can of course still be asked whether genetic alterations are technologically possible in such a way that they lead to the creation of a child with higher expected well-being. This is an empirical question and even if they currently are not, this doesn’t free us from the task of answering this question. After all many questions in ethics are formulated like this: one might argue that current technological limitations might give society an obligation to invest heavily into research as the potential benefits are enormous. However this is of no concern for my argument, because even those who disagree with genetic enhancements ever being technologically possible have to accept alleged empirical evidence. In 2015 Chinese researches released an article, showing that the gene editing of human embryos is in fact technologically possible. They used a technique called CRISPR/Cas9 which makes it impossible to tell which genes are edited [4; p. 363-372].

Realizing that premise (4) being a tautology, cannot be false, and conditionally accepting premise (1) – (3), any further objection against the conclusion following from the premises might seem pointless. However one way to refuse the conclusion remains, that is denying that the consequences of PPB application and genetic enhancements of embryos are the same. Granted, it is obviously true that two acts may never have the ‘same’ consequences, if we consider every possible consequence. However these differences have to justify a different moral standpoint, meaning that the PPB is obligatory, while genetic enhancements of embryos are not. For two acts can be different in their goodness while, nevertheless, still both being obligatory. A familiar example in line with PPB would be the obligation to send your children to school and the obligation to feed them.

Objections of this sort can be categorized in two ways. First, the expected welfare of the child might differ between selecting and enhancing an embryo in a morally relevant way. Second, they might only be morally equal in one respect, i.e. expected welfare of the child, while leaving out other relevant consequences that differ between the two and justify a different moral standing. Some objections might overlap as consequences such as freedom or equality and their implications on the welfare of the targeted embryo and overall/average welfare. The remainder of this paper has the purpose to explore and debunk a multitude of such objections.

4.1 Degree and Precision

Concerning differences in wellbeing, Parens objects that genetic alterations might be morally different from other welfare increasing procedures of offspring because they have the potential to be much larger in degree and precision [6; p. S7]. However concerning premise (4) this objection is unsuccessful, because the consequence we want is explicably the life that is expected to be best. If genetic enhancements do so much better than mere selection than Parens’ objection would rather support a moral priority of genetic enhancements against mere selection. However some might object that genetic enhancements are less reliable in their effects, therefore unprecise. That however is an empirical question and doesn’t change the fact that these genetic alterations are expected to make the targets life better. Though this is the most obvious objection and should therefore be tackled first, it is also the least harmful to the argument provided in this paper.

4.2 Risk

Another objection often put forward against genetic enhancements is the risk objection, i.e. that the PPB has far less risks to its application compared to genetic alterations. Of course genetic alterations currently bear risks much greater than mere embryo selection by IVF and PGD, which do not change the genetic constitution of the embryo. However when genetic alterations are expected to be overall more harmful than beneficial, they do not count as genetic enhancements and are therefore ruled out. If we cannot determine probabilities, than we cannot judge the expectancy of the best life anyway. This critic of human enhancements is rather meaningless against the welfarist definition of enhancements. When it comes to risk, the opponents of genetic enhancements are attacking a straw-man position that proponents of genetic enhancements do not adhere to. Risk, degree and precision are empirical concerns not only for the opponent of genetic enhancements, but also for enhancement advocates who DO take them serious.

But even if the risk of harm is currently unknown or outweighs the potential benefits, this doesn’t indicate that genetic enhancements should be avoided. One could rather argue again that we should do research until the expected benefits outweigh the medical risks of harm not stop research altogether. Also the potential benefit to be gained by genetic enhancements is much larger in degree than mere selection. Demanding a risk of zero to the targeted agents is over-demanding, otherwise medical treatments would never be at the point they are today. Still, this is a factor that affects the expected well-being negatively and therefore has to be considered in judging whether a genetic alteration counts as an enhancement.

One might further object that, contrary to somatic gene therapy, which only affects the targeted individual, germ-line interventions will affect the offspring of the targeted embryo and the offspring of his offspring and so forth… However we are enhancing all-purpose goods like intelligence and health. It seems implausible to claim that all-purpose goods are always expected to increase welfare, but might not do so for future generations. Even so, if in the future new scientific advances change our subjective probability assessments, genetic alterations that did not count as enhancements, could do so in the future and vice-versa. However, it would then be the obligation of our hypothetical parent’s off-spring to ensure, that their own off-spring has the best life possible and perhaps undo some of the changes to their gen-code. The PPB does not indicate an obligation towards grandchildren, but only to our own children. Different times and cultures may very well render different genes beneficial, think of intelligence and the muscular labor in former centuries.

Maybe it will be the case that those enhanced will be less well off than unenhanced humans would have been, because they see themselves as objects rather than subjects. A “natural” patient with paraplegia may be better off than a “super-enhanced” guy with superiority in health, intelligence, strength etc. but cannot deal with the fact that he is created artificially, perhaps becomes an alcoholic or worse. If we had reason to give this outcome a certain probability, it would have to be calculated in whether we can judge a genetic alteration to be an enhancement or not. Therefore, higher risks are no reason to justify a different moral standing from PPB.

4.3 Harm

However even though the expected welfare of genetic enhancement in embryos and the application of PPB are the same, the issue of potential harm to the embryo could still justify a moral difference between the two. The PPB seems to capture the intuition of parents being obligated towards their children’s welfare. Harming would indeed be quite contrary to this. Savulescu argues that genetic enhancements can harm an embryo in a way mere selection cannot. He asks us to imagine an embryo A, who was selected for existence but later develops cancer. As long as its life is worth living, it cannot be said that we have harmed him, for he would not have existed otherwise. However this still seems to be morally wrong. If we genetically alter an embryo B and he develops cancer (assuming his life is worse than it would have been without genetic alteration) then we would have harmed him [11; p. 422]. Unlike the objection to premise (4) we cannot disregard the harm objection by hinting at the welfarist definition of enhancement. Genetic enhancements are expected to lead to the best possible life. However harm is concerned with the actual life, not the expected welfare. In this respect PPB and genetic enhancements differ because the harm objection is successful against genetic enhancement while it is not against PPB.

However Savulescus argument also implies, that parents that selected an embryo with the expectancy of the highest welfare and turns out to have a higher wellbeing than the others, cannot claim that they benefitted him. On the contrary parents that arranged their embryo to be genetically enhanced and the embryo turns out to benefit from it, can take credit for this. Imagine a couple both bearing a defective gene that would certainly cause blindness in their child. IVF and PGD would not be able to help them (and the child), however a genetic enhancement would. According to the CP, failing to select for sight and blinding a child would be morally equal, i.e. impermissible as they share the same consequence. If such enhancement technology were available children could justifiably condemn their parents, for having harmed them in the sense that they did not give them this benefit. This might seem odd at first, but using an ordinary example, we condemn parents who do not send their children to school in order to ensure their education. In this case we also use the term harm. The claim then that there is a moral distinction between harming and failing to benefit requires a special kind of argument that may not be compatible with the CP. I take on the view that there is no relevant moral difference. As such the harm objection that aims to establish a relevant moral difference between genetic enhancements and applying the PPB raises a benefit objection against the PPB that parents do not actually benefit their children.

However, I have an additional argument against the harm objection and that is the irrelevance of harm and benefit altogether. As indicated in section 3 the morally relevant part of the PPB is the expectancy of the best life, I argue that it is not selection of the best expected life that is morally relevant but the creation of the best expected life. Using again the example of a blind child or the child not being sent to school, it is obvious how the term benefit and harm seem to be used interchangeably. If refusing to provide benefits is the same as harming, than these categories are not morally exclusive. Therefore if we take premise (3) serious, we cannot argue for a moral difference between selecting and altering. The only thing that matters is the creation of the indicated consequence welfare. Those who disagree would have to solve the non-identity problem of Parfit, which suggested that harming future individuals is not possible, because in acting in that way the people existing in the future are completely different from those who would have otherwise existed, thus would not have existed otherwise. This is under the assumption that small events can change history dramatically. The same can be said about long-term investments that would only benefit generations far in the future. However, even when we do not harm future generations, we still value actions as wrong that lead to worse lives than otherwise could have existed [8; p. 100-115]. The analogy to the PPB and genetic enhancements is obvious, as no one is harmed that would have otherwise existed. Therefore I conclude that what matters for parents is not a life being beneficial or harmful for their child but the creation of a “best life” in a welfarist sense – providing further support for the thesis that there is no relevant difference in their moral consequences. Let us now turn to possible moral differences between selecting, i.e. the application of PPB, and enhancing an embryo in areas other than the welfare of the embryo. In the following I will explore freedom, equality and the welfare of society.

4.4 Freedom

Let us first consider consequences like freedom, autonomy or missing consent. The majority of philosophers refuse the view that it is impermissible to have children on grounds of their lacking consent for being born. The notion of requiring consent from an embryo appears to most philosophers unnecessary or perhaps more so incoherent. However in the case of genetic enhancements on embryos this matter might be different, for they could have existed without genetic enhancements. Still, there are many therapies/treatments in medicine, that are often deemed to be obligatory, even though there is no consent e.g. vaccination. Also everyone should be familiar with parents sending their children to school, even though their children might not have given them their consent. As a child I certainly wondered who gave my parents my permission to so. But even granting that we can meaningfully speak of consent here and it being morally relevant, the consequentialist framework requires us to weigh the negative consequences against benefits of such genetic alterations in order to justify a violation of consent. In fact freedom to choose can be viewed as instrumental or even constitutive to welfare, but this objection would be ruled out by the welfarist definition of enhancement. If a genetic alteration undermines such it cannot count as an enhancement. The question then becomes whether a genetic alteration does so in general. Thinking of deleting a gene that causes deafness or blindness suggests otherwise. In medicine there is often a clash between respecting the autonomy of the patient and his welfare. But as Bostrom and Roache highlight, considerations like missing consent cannot be applied for embryos that by definition cannot give consent. Instead they argue, we should make decisions in a way that would be in their interest, so thinking in terms of their welfare [1; p. 22). Habermas claims that this goes against the freedom of the embryo [3; p. 62]. He calls this denying the opportunity of being “the undivided author of his own life” [3; p. 63]. Again Bostrom and Roache argue that enhancements would not decrease autonomy, but rather have the potential to increase it [1; p. 21]. Just as education makes us more autonomous, genetically increased intelligence would serve the same purpose. If the mere existence of genetic enhancements undermines our feeling of being autonomous beings, than this is not an argument against genetic enhancements, but rather exposes our concept of autonomy as nothing more than an illusion. If children feel like they have to fulfill the plans of their parents, than this might limit them in their freedom to be the “undivided author” of their own lives, but this is an objection to a particular treatment of children. Genetic enhancements have the aim to increase rather than limit ones possible choices in life. If genetic enhancements make us free to do things we would otherwise not have been able to, then this seems to be even further support for genetic enhancements.

4.5 Equality

Suppose that besides the PPB, equality is also a morally relevant consequence. There is a rather famous critic against genetic enhancements from Mehlman and Botkin, i.e. them being too expensive and therefore even when all parents follow their obligation, the children of rich parents will be far more enhanced or those of poor parents won’t get any enhancements at all (1998). Why could this objection be applicable to genetic enhancement but fail against the PPB? Applied as an objection against PPB, Savulescu explains that one would have to argue that because selecting the child with the expectancy for the best life will lead to more inequality, parents would have to create a child with worse life prospects [12; p. 288], which seems to be counterintuitive. If everyone applies the PPB there might rather be more equality rather than less by eliminating the natural lottery and additionally making everyone better off. Under this argument we could accept both equality and the PPB as compatible. However with genetic enhancements this might not be possible. First let me defend my argument against the intrinsic value of equality. Equality unlike freedom or welfare as a concept always hinges on the relations between individuals. Contrary to this, an enhancement that makes someone less equal, say by bringing a genetically altered embryo into existence with the disposition for intelligence far above average (other things being equal), would increase inequality in genetic makeup. Concerning the worry of unequal access through wealth, Savulescu argues that this is not an objection to the PPB hence genetic enhancements, but to all purchases of benefits like better education or healthcare. If equality matters then these benefits should be available to all [12; p. 288]. The same obviously applies to genetic enhancements. This is neither an objection against the PPB nor the parental obligation to genetically enhance their embryos, but to how equal the access to technologies in society is. Both consequentialist views can very well coexist, e.g. if the state ensures equal access while parents are only concerned with their child. Furthermore there are reasons to believe that genetic enhancements enhancement will not affect inequality at all, or even decrease it as Bostrom and Roache [1; p.16] suggest by making people “more equal”, like it is the case with Modafinil [9], the same could hold for genetic enhancements. How inequality could affect welfare, will be adressed in the following section.

4.6 Welfare

Another consequence of applying PPB is the effect on overall/average welfare. The objection against the PPB is that it is too much focused on the individual as opposed to the lives of everyone [12; p. 287]. Of course the same objection can be applied to genetic enhancements. In fact even if we state genetic enhancements might be good for the one who gets them (by definition), they could decrease the overall/average welfare and vice versa. Now referring back to freedom and equality we can account for their instrumental or constitutive value to welfare in this part. The implication of both equality and freedom on the individual welfare is by definition accounted in what counts as an enhancement. However both could have morally relevant implications on the overall/average welfare.

For instance, such a moral principle might lead to discrimination of the unenhanced. However, Savulescu response is that discrimination does not show that the PPB is wrong, but rather how people treat each other, which is a different topic. Genetic enhancements in empathy, sympathy and other “moral” capacities could very well decrease discrimination to a level that is even lower than the discrimination we face in an ‘unenhanced world’ [12; p. 288]. Also, the optimistic outlook that increased intelligence, including emotional intelligence, should make ‘the enhanced’ less likely to discriminate should not be disregarded. Even if, in virtue of introducing genetic alterations as a new technology, this leads to a new form of discrimination (against the unenhanced), we might significantly decrease other forms of discrimination once ‘moral’ enhancements are introduced. Though, akin to parents denying their children vaccines, I suspect that it is much more likely that if there is any new form of discrimination, then it will be directed against parents denying their children the possibility of a better life, i.e. moral condemnation. In analogy, I argue that just as we should not let scientific research into vaccines be influenced by anti-vaccines parents, fearing moral condemnation, the introduction of genetic alterations for humans should not be stopped by set of parents unwilling to use the new technology. Both act against the interest of their child and perhaps just as in the case of vaccines deserve moral condemnation. In fact, vaccines are just a form of enhancement.

The ‘discrimination-objection’ further requires the possibility of distinguishing between the enhanced and unenhanced. With the new technology CRISPR/Cas9 it is, however, impossible to find out which genes were altered or edited [4]. The unenhanced might not even know that they have not received enhancements. Even if there won’t be blatantly obvious discrimination of the unenhanced, we may argue that there will be discrimination in competition like work, because the unenhanced might on average simply be less efficient. The discrimination then would be based however, on differences in skill rather than a naïve enhanced/unenhanced distinction. But first such a treatment does already take place and is generally not regarded as discrimination. If some parents are able to afford a better or further education for their children, e.g. tutoring, we neither prohibit such practices nor are we able to provide good counter-arguments for parents who insist that they are obligated to provide their children with the means to get ahead. Most political philosophers take it as the task of the state to make education accessible to those who would otherwise not be able to afford it, including perhaps tutoring. Rather than denying the parental obligation of enhancing one’s children, the ‘discrimination-objection’ provides a very good argument for an egalitarian policy making genetic enhancements available to all.

Savulescu accepts that some enhancements might make the individual better off, but harm the rest, e.g. by making someone manipulative or cunning [12; p. 287]. For example parents might choose to alter the DNA of their child in a way that makes them more cunning. Resulting in “him” being better off, while the overall/average welfare decreases. However, that argument can simply be taken in the opposite direction. We might want to alter the DNA of embryos that will expectedly lead them to have worse lives, but will benefit the overall/average welfare. This of course would also be an issue for the PPB. One cannot hold this view and simultaneously be in favor of the PPB. Though the argument might successfully undermine the PPB, a defense of said principle exceeds the scope of this paper. For a defense of the PPB I suggest the work of Savulescu [10, 11, 12]. Even so, this might be an issue for the state that should enforce restrictions, while parents are primarily concerned with the welfare of their own child. The effects of one particular enhancement on the overall/average welfare of society must in the eyes of parents seem negligible. These arguments are usually stated in a much broader sense, i.e. the availability of the technology itself. In fact, even with restrictions on the technology the underlying parental obligation doesn’t cease to exist. If the state is justified or obligated to stop such genetic enhancements as a whole, be it by appealing to equality, freedom or overall/average welfare, it is an open question whether that actually frees parents from their obligation to genetically enhance their embryos. There are several reasons to think that a prohibition of genetic enhancements by the state doesn’t free parents from their obligation. First, it would be over demanding to ask from parents to act against the best interest of their child on grounds of marginal effects on the overall/average welfare in the society. Second, it would be an incompatible view with the PPB, as parents specifically have to aim for the best possible life. Third, even when means to genetically enhance are not available, this doesn’t mean that in case the genetic enhancements were available, there would be no obligation to apply them.

5. Conclusion

To conclude, objections against genetic enhancements not weighing their costs against their benefits must fail against the backdrop of the CP. Of course maybe the PPB, thus premise (1) or the CP, thus premise (3) is false, but the aim of this paper is to establish that the parental obligation to genetically enhance their children logically follows once the PPB and the CP are accepted. The ‘slippery slope’ is genuine. Objections trying to establish that the individual costs outweigh the benefits of the targeted agent, seemed to be more successful. However, a genetic alteration that is not expected to lead to the creation of a being with higher welfare cannot be called an enhancement in the welfarist sense. Merely showing than that an enhancement in the sense of increasing some capacity is contrary to the welfare of the child does not undermine the argument provided in this paper as it precisely rests on the welfarist definition of enhancement. However, these cases might provide us with a valid argument against exceedingly strong optimism in discovering all-purpose goods. Having explored the potential differences in the PPB and the parental obligation to genetically enhance their children in section 4, I conclude that even though they do not share exactly the same consequences, the differences between them do not justify a different moral standing, i.e. their obligatoriness. With the additional premise of lacking relevant moral differences between the consequences of the two procedures, we have successfully defended the conclusion (5). That is to say, those who accept the PPB and the CP must bite the bullet and accept that parents have the obligation to genetically enhance their embryos. As a consequentialist Julian Savulescu should have no problem of doing so.

Acknowledgements

I would like to thank Roberto Fumagalli and three anonymous referees for helpful comments over the course of the development of this paper. Furthermore, I would like to thank the students, from the University of Bayreuth and the London School of Economics, with whom I had lively discussions.

Originally published here:

Veit, W. (2018). Procreative Beneficence and Genetic Enhancement. KRITERION – Journal of Philosophy 32(1), 75-92. https://doi.org/10.13140/RG.2.2.11026.89289 [Download]

References

[1] Bostrom, N. & Roache, R. Ethical Issues in Human Enhancement. New Waves in Applied Ethics ed. Jesper Ryberg. Palgrave Macmillan, 2007.

[2] Griffin, J. Well-Being Oxford: Clarendon Press, 1986.

[3] Habermas, J. The Future of Human Nature. Cambridge: Polity, 2003.

[4] Liang, P., et. al., “CRISPR/Cas9-mediated gene editing in human tripronuclear zygotes” Protein & Cell Volume 6, Issue 5 (2015), pp 363-372.

[5] Mehlman, M. J. & Botkin, J. R. “Access to the Genome. The Challenge to Equality.” Georgetown University Press, 1998.

[6] Parens, Erik “Is Better Always Good? The Enhancement Project” The Hastings Center Report, Vol. 28, No. 1 (Jan. - Feb., 1998), pp. S1-S17.

[7] Parfit, D. “Reasons and Persons” Oxford: University Press, 1984.

[8] Parfit, D., “On Doing the Best for Our Children,” in Bayles (ed.) 1976, p. 100–115.

[9] Randall, D. C., Shneerson, J. M. & File S. E. “Cognitive effects of modafinil in student volunteers may depend on IQ. Pharmacology Biochemistry and Behavior” (2005), 82(1), 133–139.

[10] Savulescu, J., Sandberg, A. & Kahane, G. Enhancing Human Capacities, Blackwell Publishing Ltd. 2011.

[11] Savulescu, J. “Procreative Beneficience: Why we should select the Best Children” Blackwell Publishing Ltd, 2001.

[12] Savulescu, J. “In defence of Procreative Beneficience” J Med Ethics (2007); 33:284–288.

(Why) we’re launching the next cohort of Generator by a_e_r

a_e_r — Wed, 14 Oct 2026 05:41:53 +0000

TL;DR: The first round of the Generator Residency went very well.

Our residents created and ran several large-scale fieldbuilding, talent infrastructure, and operational projects that have already had important impacts on the ecosystem.

Six weeks after the program ended, 12 of our 27 residents have been hired to full-time roles at core AI safety orgs, and most have received several offers.

Kairos and Constellation are running a second round from January to April 2027. Apply to the next round of the Generator Residency by November 1st.

Introduction

In April, we wrote AI Safety’s Biggest Talent Gap Isn’t Researchers. It’s Generalists.

We wrote this because we noticed that organizations across the ecosystem were struggling to hire for key non-research roles. While these roles were often getting many applicants, candidates didn’t meet basic requirements.

The original memo we wrote (that became the post above) inspired us to create the Generator Residency as an intervention to support more generalists trying to break into AI safety.

The program concluded six weeks ago, and this post details the successes we’ve seen from the first round of Generator, which have inspired us to launch a second round of the program.

Programs and projects from Generator

Programs residents started:

Talent Commons, a cross-organization talent directory now run as a Kairos product, which will include data from participants in SPAR, Pathfinder, GCP, Generator, BlueDot courses, Astra, Anthropic Fellows, OpenAI Safety Fellows, and MATS.
Praxis, a retreat bringing top national debaters into contact with AI safety: about 40 debaters, including 20 national champions.
Lateral, a three-day workshop bringing mid-career professionals to Berkeley to work with AI safety professionals: 705 applications for about 35 seats, and a likelihood to recommend of 9.7 out of 10 from attendees.
Gen Stream, a one-week program in London funded through Generator.
Longitude DC, an eight-day policy intensive for 15 early-to-mid-career technical AI safety professionals with more than 20 speakers, running October 6 to 13.
M3, an AI security matching fellowship, which raised $200k from BlueDot on top of Generator funding.

Residents working with mentors at established organizations contributed to:

Lab engagement through Nightingale and Constellation.
Recruiting and strategy at METR.
A frontier robotics benchmark program for Robocurve.
Events at Constellation.
Fundraising and hiring for a new organization founded by Jeff Alstott.
The Plan A launch, in collaboration with AI Futures Project.
Founder scouting for Coefficient Giving’s Project Tailwind.

Other notable projects included:

A one-week AI safety world-modeling bootcamp with 25 attendees covering ML fundamentals, futurism, and the technical case for alignment.
A workshop series on improving management within AI safety (interviews with more than 10 managers and three workshops).
AI safety founder support (meetings with 32 founders, with 6 given deeper support).
Support for mid-career operations people transitioning into AI safety (leading to hundreds of referrals across more than 50 roles).
A map of the work on AI character and model specs.
12 SPAR generalist projects that came out of the residency.

Resident Outcomes

Generator was designed as a path into full-time generalist roles in AI safety. Six weeks after the program ended, 12 of the 27 residents have been hired into full-time positions. Their employers include BlueDot Impact, Guidelight, Constellation, Kairos, METR, Eleos, CBAI, and CEA. In several of these cases, the hiring organization began working with the resident during Generator.

Of the residents who have not yet taken a full-time role, some are running programs that came out of the residency or AI safety university groups at Harvard, Princeton, MIT, and Stanford. Others have received offers, including from Coefficient Giving, Quorum, and ARENA, and are deciding between options. A few residents have been pitched by funders on turning their one-time project into a much larger, recurring program.

Below are a couple of selected testimonials from previous residents:

Additional testimonials and information are available on the Generator website.

Reflections

We stood Generator up in three months. It was the first multi-month in-person program Kairos had run. Most of what we learned was about what residents needed from us, and we think most of it applies to other programs that support people rather than research outputs.

Some key lessons we learned:

1. Candidate selection shaped the program more than anything else.

Candidate selection is, of course, an incredibly important part of any program. However, we were surprised by just how much of our planned programming changed based on the proactivity of our participants.

Residents discovered who the other residents were before the program started and created a group chat and had calls with each other. Once they arrived, they ran hikes, game nights, regular reflection sessions, and informal peer check-ins. When we asked what made them feel integrated, residents pointed to these resident-run activities far more often than anything we scheduled. In both our midway and final surveys, the most common answer to “what was your favorite part” was the other residents.

2. Residents’ career goals strongly informed their preferences for projects and advisors.

The cohort included three broad groups:

University organizers looking for their first formal work experience
Early-career professionals looking for a final stepping stone into the field
Mid-career professionals looking for the context they needed to transition

Over time, we realized that, in addition to accommodating residents with different experience levels, we needed to accommodate residents with different intentions for how Generator could forward their career. Some residents strongly preferred to use the Generator residency to explore multiple roles, while others were using the residency to deliver a legible product that could help them make a career transition or get hired. Residents aiming to transition wanted advisors invested in their growth and career, not only in the project.

Based on feedback, we’ve decided to implement two tracks in our next residency. We plan to work with our residents from the start to determine what projects and tracks best suit their goals.

3. Our residents asked for the most support with project scoping, structured management, and navigating interpersonal conflict.

For many residents, their Generator project was the largest or most complex project they had ever planned. Moreover, their projects typically aimed to address large, poorly defined problems in the AI safety ecosystem, such as mid-career talent pipelines. Accordingly, they wanted support in determining the most ambitious, yet feasible project that could enable their vision.

Several early-career residents benefited from structured management, reporting that they grew from an advisor who had them estimate how long each planned task would take or a short written template before each check-in.

Co-leading a Generator project was quite similar to co-founding, which led to a lot of interpersonal work for some residents who had previously mostly worked independently. These residents benefitted from clear norms around sharing feedback.

4. We need to be more explicit that we’re betting on people, not projects.

While many residents worked on impactful projects, the main focus of Generator was always on the people we chose to support. The projects matter, and many of them helped the field, but we mostly saw them as a way for residents to grow quickly into the generalists AI safety needs. We didn’t say this clearly enough at the start, and many residents felt stressed about picking the single most impactful project.

When we asked them how the program changed their thinking, the most common answers were about learning field context, clarifying their career direction, and developing self-knowledge. Few people directly mentioned their project outcomes.

Generator 2.0

We’ve been really happy with the outcomes that we saw from the first round of Generator and, as a result, have decided to run a second round!

Applications for the winter 2027 Generator Residency open on October 12, 2026, and close at the end of the day on November 1, 2026, Anywhere on Earth. The program begins on January 18 and will run until April 9, 2027, with a possible 3-month extension.

Learn more and apply here

Residents at Generator receive $8,500 per month and are funded for travel to and from Berkeley. Once they arrive in Berkeley, they work from Constellation, are paired with experienced advisors from the AI safety ecosystem, and are offered budgets to pursue their projects.

Applicants do not need to be U.S. citizens. For most international applicants, we can sponsor a J-1 visa for the program’s duration, though we cannot guarantee obtaining one. Additionally, we strongly recommend that international applicants apply early (ideally by October 26th) so we can meet deadlines for visa applications.

OpenAI finds recording device in one of their offices by defun 🔸

defun 🔸 — Tue, 13 Oct 2026 21:03:03 +0000

We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work.

At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated.

We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats.

Today I Killed; Tomorrow I Will Kill Again by Lance Spencer

Lance Spencer — Tue, 13 Oct 2026 20:36:02 +0000

When must we kill them? Now. Later. Continuously. Infinitely.

A very specific trigger warning: Noah Smith, if you ever see this, please do proceed with caution.

Based on a true story.

A favorite pastime of mine is wandering about in nature’s splendor. The whole experience is delightful: The planning, the topo maps, the gear setup, the drive, the type II hedonism of light, voluntary suffering at an oxygen-deprived altitude. The locomoting while chatting with friends, or solitarily listening to a podcast. The sheer adventure of exploring the Earth. Fun for the whole internal family of experiencing and remembering selves. My ancestors studied hunting and gathering and then agricultural science so that I might have liberty to study an approximation of hunting and gathering. And it’s a great way to stay in shape.

So there I was at the edge of civilization when it all happened, on approach to the trailhead to climb a mountain. Driving my still-quite-new, designed-overseas-but-made-in-America, symmetrical all-wheel drive, turbocharged, off-roading family station wagon (practical and a blast to drive). Digesting my favorite fast food breakfast—a superstimulus of sweet and savory delight, and purchased at a heavy discount. Drinking my favorite artificially sweetened mountain beverage (zero, not diet). Listening to one of my favorite podcasts. Wearing my new, limited-edition desert boots of ethically sourced exotic leather, designed by hippy shoemakers across the ocean, and made in a different country across the other ocean to take advantage of market efficiencies.

Peak human experience really. Brought to me by globalism.

The weather was perfect. There was just a bit more pavement, passing some cabins by, and then a few dusty miles up the canyon. What a great day. All my problems were behind me for the moment.

Up ahead, I noticed some fresh roadkill in the other lane. Probably a jackrabbit; could be a racoon or maybe a skunk though, with the nearby streams. Or a cat. Sad.

Ah well, so it goes. Happens all the time. I don’t even have to adjust course to avoid it. Onward and upward.

The long ears of the prostrate jackrabbit twitched.

Fuck.

I thought. Loudly.

That was not the wind as I passed next to it.

For I had seen a twitch; I knew it, and I knew that (nature’s) god knew it, and I could not deny it, neither dared I do it; at least I knew that by so doing I would offend (nature’s) god, and come under condemnation.

“FUUUUUUUUUUUUUUCK!” I shouted to myself in particular.

“FUCK fuck Fuck!” I fired for effect as I pulled off the road. The concept of moral luck briefly traversed my thoughts. I had been called and chosen.

What kind of monster just drives away? You had an accident; you chose to flee the scene. Finish what you started. Do your moral duty. You break it, you need to make sure the animal bought it.

I instantaneously had realized there was no alternative. Hence my reaction. (You probably did too. Self-evident, really. It’s no trolley problem.) All I could do was evaluate means, not the end. Still, I employed various moral heuristics in rapid succession to calculate the expected value of my options, hoping I could conceive of a way around the clear implications for the sum total of utility in the universe. And my seemingly obligatory role in it by dint of happenstance.

I know the unit of caring is dollars, but what is the unit of suffering?^[1] Do we measure that per second? Log-scale magnitude maybe adjusted by neuron count? Who is my moral patient? A moral person’s a moral person no matter how small. The Life You Can End. Killing What We Can. Time Is Up Decision Theory. What would the best version of myself do? What would someone with 10X my agency do? Actually, scratch that; seems unlikely to go well if I try something drastic like taking this little guy to an emergency vet on Sunday in rural America. Can you imagine a jackrabbit with those wheelchair hind legs? I can’t really justify Make-a-Wish, Roadkill Rescue Edition. It is better that one rabbit should perish than that a budget should dwindle and perish in veterinary bills.

After modeling my utility function, the rabbit’s utility function, and the utility function of an infinite, ~~uncaring~~ impartial universalist, I found no alternative.

Consequentialism can be a real hassle sometimes. But I don’t believe deontology or virtue ethics would provide me an out here either. All models are wrong; some are useful. Ethics is knowing the difference. And supererogation just wasn’t going to cut it any way I sliced it.

I got out of the car. Thinking step-by-step.

Alright, how am I going to do it? There are cabins right over there and plenty of recreationists within earshot. Well that’s unfortunate. The modular emergency shovel kit with the axe head attachment it is, then. Visceral. How am I going to handle it? I do have work gloves. What are the odds I’ll make a clean kill with a single blow; how traumatic was that going to be for everyone? Model the scenario using your inner simulator; how do you fail? Gruesome.

I approached the perfectly still jackrabbit to confirm the status of its existence. It was definitely alive. Still, small, silent, stoic; cipher in the road.

It moved. The poor creature. Paralyzed from the waist down, using its forelegs to drag itself across the road away from me. Clear and present danger that I was.

I regretted immediately and immensely—a full-body sensation—that I had inadvertently caused this suffering innocent creature of (nature’s) god further pain. Ah, the sharp pang of guilt and pity. I can still feel a little every time I reread the paragraph above. Further introspection ensued.

It’s in no way my fault that I have no means to comfort it. No way to explain that it is a victim of the cumulation of some billions of years of blind watchmaking, thousands of years of human technological development, and just over a century of the automobile industry. That neither nature nor nurture prepared it for dealing with several thousand pounds of motorcar traveling at 20+ MPH. There is no affordance provided by nature to transmit, let alone justify, to this conscious creature that its utility is negative, with no realistic way to rectify that, and so we must select our least bad option—the merciful, rapid onset of everlasting darkness. “Who is ‘we’ here?” it could not ask.

No evident bleeding. It had been there long enough to have left some pellets, but probably not all that long. (Scavengers and all that.) At least I wasn’t going to have to move it myself off the road now. I felt sorrow at the absurd tragedy of it all. It wasn’t dust in my eye, since as previously described I was still on a paved road.

Why even feel sorrow here? It won’t help anyone. No call for it. It’s not like I haven’t in my youth hunted his coethnics for sport and pest control. For the greater good. For crying out loud. Hell, at the little zoo yesterday my primary takeaway was “I should really try out coyote hunting.”^[2] Had the bunny been a bit more than mostly dead, I wouldn’t have had a second thought; not another quale expended on the problem of roadkill and moral justice of the universe. Neither priest nor saint, I routinely consume unethically sourced animal products with no regard for consumee welfare. Emotion for a wild animal, but not a farm animal? Some animals really are more equal than others. Monster that I am, the boots on my feet are almost certainly more ethically sourced than my breakfast. I don’t think anyone factory farms camels, anyway. Frankly, I have no right to feel such emotion here, on top of it being pointless. Could I have felt differently? But I did have breakfast this morning.

Reframe: I get to. Embrace the suck. Just do it. Be present in the moment. Everything is finite. You can just do things. Do a good turn daily. YOLO. No regrets. Be all that you can’t be. This too shall pass. Semper Gumby. Spark joy. Count your many blessings, name them one by one. Be the change you want to see in the world. God helps those who help themselves (YMMV). There must be a pony in here somewhere. Lemons; lemonade. Choose the right. Well it matters to this one. Do unto others. Here am I; send me. Only you can prevent forest utility fires. Called to serve him heavenly king of utility maximization. You miss 100% of the shots you don’t take. Do what is right; let the consequences follow. Execution is everything.

Hunt the good stuff: Wild animals, of course, typically die a grim natural death, so this is maybe even significantly better than par for the course. Except for the emotional downsides of my involvement, anyway. Optimistically, given that it has mostly held so still, it can’t feel the full extent of its injuries due to paralysis and/or shock. Just some confused panic I’m exacerbating. God I hope that’s true. Let that be true. A tender mercy. Please.

“Fuck, fuck, fuck,” I continued to narrate for no one but myself as I walked back to the car, rifled through my cargo organizer, assembled my modular hatchet, and walked back to the side of the road. I looked down at the poor bunny, hatchet in hand, planning my next move. Summoning the final bit of free will to do what had to be done. I contemplated the Copenhagen Interpretation of Ethics.

Sorry, Brer Rabbit. Yours, upon examination, is not a life worth living. Nor worth saving. A brief map recon of the moral landscape shows you are in a deep depression. Your revealed preference is the will to live, but I think you should trust my superior reasoning as a lifeform with higher-order cognition that it’s in your best interest to stop doing that. I am so very sorry, however pointless an apology it may be, that you are a mere crippled bunny on a road and not a child drowning in a shallow pond. Or a pool, which is far more common. Facts don’t care about your feelings. Or mine, annoyingly. You’re just going to have to take it on faith that if you could somehow walk a mile in my fancy boots you would concur with my planned course of action. Knowing me, knowing you, it’s the best I can do. I would hope any eventual AGI finding itself in similar circumstances would give me the same courtesy.^[3]

A black SUV approached.

Great, an audience.

A man in sunglasses with flowing locks pulled up, window down. I didn’t even take note of the make/model, as I would otherwise automatically do if I hadn’t been so task oriented on the axe in my hand.

Injured?

Yeah, hind legs are paralyzed. I’d shoot it but for the houses.

I think I’d prefer to shoot it if it was me.

I momentarily reconsidered tradeoffs and re-estimated the utils balance of my options. Sure would be easier to use a bullet. It’s cleaner. What’s the point of rural America, after all, if a solitary gunshot is a big deal? I appreciated the second opinion. The solidarity. The permission. I updated my priors.

Yeah. You’ve convinced me. I’m going to get some ear pro.

He drove off. I walked off to get my Chinese-made, wired, Bluetooth headphones with like 40 hours of battery life. Excellent product. And they are good enough ear plugs.

Improvise, adapt, overcome.

“Fuck!” I exclaimed again as I approached the decrepit critter and unholstered my sidearm. Never before had it been used for its intended purpose—launching a 9mm +P, 124-grain, jacketed hollow point bullet at just over the speed of sound toward a living being.

Judge; jury; executioner. By the power vested in me by the state of nature and almighty (nature’s) god, I declare you a hedonistic outlaw. This ends today. I think therefore I am going to kill you. I am (a natural) god’s army of one. A lion does not concern himself with the opinion of scapegoats. Angel of mercy killing. Wabbit season. Now I am become Death, the destroyer of inner worlds. Hop little bunny, hop hop hop. The one mighty and strong. Blood loss aversion. Why did the jackrabbit cross the road? It could not have done otherwise. For the life of the flesh is in the blood: and I have given it to you upon the altar to make an atonement for your souls: for it is the blood that maketh an atonement for the soul. By all means necessary. What makes the green grass grow? Who, whom? Am I my brother’s zookeeper?

I aimed, closed my eyes, and squeezed the trigger.

You are no longer the author of your thoughts. Even the illusion of the illusion of the self is gone. There’s no “there” there. What would it take to change your mind to mush? The circle of life. Dust in the wind. Moral hazard lights at the end of the tunnel. From carbon to carbon. Next year in Watership Down. All’s well that ends well. In the long run we’re all dead. May your memory be a blessing. Till we meet again.

The canyon echoed. I opened my eyes. The suffering had ended. Entropy had increased. Rumination had begun.

Mission accomplished, at the cost of about 60 cents, a few minutes, and some overthinking. I transitioned to my spare magazine, holstered my weapons, and walked back to the car. My outfit was unblemished. A whiff of gunpowder was in the air.

I started up the mountain. Given the rumination, I opted for music instead of my usual podcast. The ancient ballad I happened to listen to helpfully reminded me:

There’s a time to live, a time to die,

But isn’t it strange,

As soon as you’re born, you’re dying.

On the summit, I ate crackers and drank protein as I cogitated and surveyed 360 degrees of a beautiful land of desolation. Another guy summited. He had the sniffles. The absolute son of a bitch. I put my other earbud in.

A fair bit above the tree line though I was, a ladybug landed on my hand as I typed notes for this essay. Despite the breeze, it stayed until I put the little lady on the rocks as I prepared to depart.

I demountained. On the drive out of the canyon, I saw up ahead a jackrabbit cross the road.^[4]

It got to the other side.

Then I drove about 30 minutes to civilization and ate the best commonly available hamburger at a steep discount. I hope it was a happy cow. All in all, I had been pretty lucky. This had been a clear-cut moral situation. It’s not like I happened across a beehive, let alone a shrimp farm.^[5]

^
Apparently, the unit of suffering is the dolor. Of all things. My knowledge of negative utilitarianism is clearly lacking.
^
This event happened last year. Since then I have, in theory, become a coyote hunter. The worst one in the world, probably. More “heavily armed birdwatching” in practice.
^
A poor paralyzed rabbit of grief
hath today crossed me on my way,
who sued so humbly for relief
that I could never answer nay.
I had not power to ask his name;
yet there was something in his ears
that won my concern. I knew why.
I spied him where a motorcar burst.
Clear from the road, his strength was gone.
The heedless universe mocked his pain;
he heard me, saw me, wriggling on.
I walked and raised the handgun up.
The flesh was weak; my blood ran chill,
but my free spirit cried, “I will.”
Once from the gun his life drained out.
Clipped, wounded, beaten nigh to death,
I found him by the highway side.
I flatlined his pulse, extinguished his breath,
released his spirit and supplied relief.
Gun, bullet, impact—he was deceased;
and sorrow bound up my broken heart.
Then in a moment to my view,
the ASI started from disguise;
the tokens in his neural net I knew;
the scapegoat stood before mine eyes.
He spake, and my poor name he named:
“Of me thou hast not been ashamed;
these deeds shall thy memorial be;
fear not, thou didst them unto me.”
(With apologies to James Montgomery, among others.)
^
Full disclosure, it was more likely a ground squirrel of some sort. Please forgive me this poetic license.
^
If the going rate to save a human life is ~$5k, what’s the going rate for indulgences to make up for all of the animals I consume? About $300 a year for a carnivore offset, apparently. That’s pretty affordable actually. Just ~$24k over an 80-year life span to guiltlessly consume from Pascal’s Endless Shrimp Buffet.

How moral weights and soil life change cause prioritisation by Vasco Grilo🔸

Vasco Grilo🔸 — Tue, 13 Oct 2026 14:55:36 +0000

This is a linkpost for How moral weights and soil life change cause prioritisation, an artifact I built with Claude Opus 5.5. The summary is below.

Twelve interventions, all but two of them donations at the current margin, two models of moral weight, and every exponent from 0 to 2. Model 1 counts animals with neurons, and assumes moral weight is proportional to the number of neurons to the power n. Model 2 adds trees, and soil bacteria and archaea, and assumes moral weight is proportional to basal metabolic rate (BMR) to the power m. Neuron counts, metabolic rates, pain intensities and the welfare of soil organisms in each land class are all uncertain inputs.

Low exponents

Effects on soil organisms dominate, and the interventions that change farmland the most per dollar lead. Under model 1, those that decrease farmland lead, with donations to the Good Food Institute (GFI) first, since soil animals, mostly nematodes, are assumed to fare worse on farmland. Under model 2, those that add farmland lead, with Giving What We Can (GWWC) ahead of corporate cage-free campaigns, since farmland holds fewer soil bacteria and archaea, whose lives are assumed negative in expectation. These leads come from guessed mean welfare levels much smaller than their spreads, and each leader is among the worst options under the other model. So they show which questions research should answer more than which of the interventions analysed to fund.

Intermediate exponents

The Shrimp Welfare Project’s (SWP’s) Humane Slaughter Initiative (HSI), which electrically stuns shrimps, narrowly leads in narrow bands of these exponents. Its beneficiaries have small weights but may be spared intense suffering at slaughter. Its position turns on whether stunning reduces suffering and on the intensity of excruciating pain.

High exponents

Effects on the target beneficiaries, the people or animals each intervention aims to help, dominate. This favours humans and chickens. GWWC moves several dollars per dollar to effective charities, mostly in global health and animal welfare. It leads under model 1, largely through the people saved and their descendants. Under model 2, cage-free campaigns lead, except at the highest exponents, where GWWC does.

Farmland per dollar

The ranges from the 5th to the 95th percentile of the eight interventions adding the most farmland per dollar overlap substantially. The Navigation Fund’s cage-free campaigns fund has the highest mean, closely followed by GWWC and GiveWell’s grant. Of the candidates not modelled, only promoting beef plausibly adds clearly more, on weak evidence (farmland per dollar).

Robustness

Few results are robust. Whether soil organisms have negative lives, and whether farming worsens them, are close to coin flips. So at low exponents the sign of most interventions’ effects is unclear. At high exponents, the leaders have a high chance of a positive effect. Under model 1, GWWC keeps its lead at the highest exponents in nearly every sensitivity analysis. The exceptions are with GWWC’s conservative multiplier of 1.5 and without the weighting for uncertainty over time, where GiveWell’s grant leads instead. Under model 2, the leader just below the highest exponents depends on the intensities of pain and on the weighting for uncertainty over time. Some leads are also statistical ties within the noise of the simulation (using the results).

So which cause to prioritise turns on the form and exponent of the moral-weight function, the welfare of soil organisms, the intensity of excruciating pain and whether stunning reduces suffering. Research could narrow all of these (key assumptions).

Merging Everybody Into Hell by Tim Chan

Tim Chan — Tue, 13 Oct 2026 14:32:54 +0000

Here’s a thought experiment. As far as I know, no one has written about it yet.

There’s a mind fated for hell. Its suffering will last a very long time, and nothing can change that.

You get to decide what happens at the end of everybody else’s life. There are two options:

Option A: Everybody’s life just ends normally and nothing happens. Assume there’s no afterlife.
Option B: At the end of everybody’s lives, everybody is merged into the mind fated for hell. Assume this is possible. Merging doesn’t make the mind’s hell any more intense or any longer, and it doesn’t create extra copies of the mind. But from every person’s point of view, they close their eyes at the end of their lives and wake up as that one mind in hell. In return, $1 goes to your favorite charity.

Which do you choose?

I’d choose A. Everybody going to hell is obviously bad, and definitely not worth a dollar to any charity.

And yet standard utilitarianism chooses B. By its count, merging neither adds nor subtracts welfare. There’s still one mind in hell, even though it came from a merge of many minds. So, it first says the two options are equal in value. Then, $1 breaks the tie. Now it says you should send everybody to hell.

This is the verdict of any standard utilitarianism that adds up the welfare of experiences—total, average, and even negative—as long as $1 does some good and the merging is considered morally neutral.

The mirror case works too. Offer $1 to charity for letting everybody’s lives simply end, instead of merging them into a mind fated for heaven, and standard utilitarianism accepts the dollar, depriving everybody of heaven.

That can’t be right. What’s going on?

The problem here is that standard utilitarianism doesn’t explicitly track what each person goes on to experience. It only adds up experiences as seen from the outside, from a third-person ‘view from nowhere’. What you subjectively continue onto is none of its business. That’s mostly fine for linear lives, where each experience simply leads to the next, but not for lives that can be copied and merged, and you can construct more counterexamples like this one with those operations.

One could bite the bullet and say only the hell itself matters, but not the fact that multiple people would come to experience it. But then one would have to be indifferent between dying peacefully and waking up in that hell tomorrow, everything else unchanged. I don’t think many people are indifferent.

I think the fix is to care about each perspective’s subjective futures, in addition to its immediate welfare. I’ve proposed that utilitarianism should be based on a Markov reward process over types of experiences. It handles this case, and I think it also fixes problems with identical copies that others have noted.

It feels like we should have started discussing this counterexample a long time ago. Ethics has only handled the extremely special case of linear lives. We may soon be headed for takeoff, yet as a civilization we haven’t even worked out the general case and what it implies.

High Impact Professionals is hiring: four remote roles (apply by November 1) by High Impact Professionals

High Impact Professionals — Tue, 13 Oct 2026 13:04:15 +0000

Drafted with AI help from our job descriptions, then reviewed and edited by the HIP team.

High Impact Professionals (HIP) supports experienced professionals to increase their impact through career transitions and effective giving. We’re hiring for four roles, and this post explains what they are, why we’re hiring now, and how to apply.

Summary

Four fully remote roles: Chief of Staff / Managing Director, Director of Operations, Program Launch Lead / Director, and Recruitment Manager. Two of them are open at two levels.
Applications close Sunday, November 1. We aim to make offers by mid-December, with a start ideally by February 2027.
Salaries range from $60,000 to $140,000 USD, depending on the role and level. All roles are ideally full-time, with part-time considered for outstanding applicants.
Why now: demand for our programs is well ahead of our capacity. Around 3,300 professionals applied to our Impact Accelerator Program (IAP) this year and we accepted around 10%. We plan to grow from 5 to around 14 people over the next two years and support hundreds of career transitions a year.
Questions? Join an applicant Q&A webinar on Tuesday, October 27, or email info@highimpactprofessionals.org.

Why we’re hiring now

Organisations working on the world’s most pressing problems are getting new funding, and many tell us that finding a great, experienced hire is one of their main bottlenecks. At the same time, far more experienced professionals want to move into high-impact work than we can currently support.

We’ve helped more than 140 professionals make career changes to high-impact roles through HIP, at organisations like GiveWell, Epoch AI, and Shrimp Welfare Project, and we estimate more than $5 million in counterfactual impact (method and caveats in our 2025 Report). Our Impact Accelerator Program has grown from 19 participants in 2023 to around 350 in 2026. Until this year we did all of this with a team of 1.6 FTE, supported by alumni facilitators and an increasingly automated operations stack.

Since 2025 the team has grown from 2 to 5 people. Over the next two years we plan to run twice as many IAP rounds with twice as many participants in each, launch new programs, and grow to around 14 people. These four roles build the operations, programs, selection capacity, and leadership that our growth needs.

The four roles

Chief of Staff / Managing Director

Help scale HIP’s team and impact, delivering our top priorities as Chief of Staff or running HIP internally as Managing Director.

As Chief of Staff, you’d work closely with our Executive Director and take high-priority projects from brief to done: a large part of a hiring round, HIP’s presence at a conference, an analysis to inform a decision.
As Managing Director, you’d join our leadership team and run HIP internally day to day: our culture and internal systems as we grow, the formalisation of our board, and our main hiring rounds, while our Executive Director focuses on partnerships and scaling our impact.
Salary: Chief of Staff: $60,000 to $80,000 USD (up to $90,000 for outstanding candidates). Managing Director: up to $140,000 USD. Managing Director finalists will do a paid work trial of about two days.

Director of Operations

Build the systems and processes that enable HIP’s growth, starting by setting us up as an independent nonprofit.

You’d own HIP’s operations end to end (finance, legal and compliance, and people operations) and lead our move from fiscal sponsorship to our own 501(c)(3), planned for mid-2027.
You’d join our leadership team, working closely with our Executive Director.
Salary: $80,000 to $100,000 USD (up to $120,000 for outstanding candidates).

Program Launch Lead / Director

Create new routes into high-impact work. The first could be a self-guided program open to anyone.

As Program Launch Lead, you’d build and run the first rounds of a two-hour introductory course and a much more scalable careers program, then hand them over to our programs team once they work.
As Program Launch Director, you’d join our leadership team, research what professionals considering a high-impact career change need, test for product-market fit, and decide what HIP builds first.
Salary: Lead: $60,000 to $80,000 USD (up to $90,000 for outstanding candidates). Director: $80,000 to $100,000 USD (up to $120,000 for outstanding candidates).

Recruitment Manager

Choose who joins our Impact Accelerator Program, the step that currently limits how many people we can help.

You’d lead IAP selection, using data from 10 past application rounds to learn which signals predict career changes and to keep the process short for applicants without losing decision quality.
You’d also run parts of HIP’s own hiring rounds.
Salary: $60,000 to $80,000 USD (up to $90,000 for outstanding candidates).

What we offer and how we hire

Fully remote, ideally with at least 2 hours of overlap with UK working hours (Chief of Staff / Managing Director, Director of Operations) or central European working hours (Program Launch Lead / Director, Recruitment Manager), plus 3 to 5 in-person conferences or co-working sessions a year.
Ideally full-time, with part-time considered for outstanding applicants. 30 days’ vacation and a $1,000 tech stipend.
Process: an application by Sunday, November 1 (we reply within two weeks of the deadline), a test task of about 3 hours, then one or two video interviews and reference checks.

Questions, and how you can help

We’re running two applicant Q&A webinars on Tuesday, October 27:

9:00 UTC (2:30 pm India time): Register
16:00 UTC (12:00 pm US East Coast): Register

You can also email info@highimpactprofessionals.org.

Read the full job descriptions and apply by November 1. If you know someone who would be a great fit for one of these roles, we’d be very grateful if you passed this post on. Questions and feedback are welcome in the comments.

$1000 Dollars for animal charities by writing about philosophy and effective altruism by Walter Veit

Walter Veit — Tue, 13 Oct 2026 12:44:58 +0000

This is a crosspost from my newsletter.

Two weeks ago I wrote:

In the last 8 weeks, we have also more than doubled the projected annualized revenue of my newsletter from $311 to $700. Maintaining that trajectory, I am hopeful that we can hit the target of $1000 by the end of this year.

We’ve now achieved this goal! See the rapid increase in the last two weeks:

Thanks a lot for all the paid subscribers. I was very surprised to see this sudden influx! It means a lot to me that my little niche philosophy newsletter has acquired so much love and support. :)

Perhaps there’s a chance we’ll make it onto the Substack “philosophy” bestseller list and overtake its astrology accounts? Hope never dies.

In any case, I’m especially happy since the steady growth of the newsletter means that a lot of animals in factory farms will have their lives improved. As pledged, all of my substack income goes to improving the lives of animals on factory farms — the greatest source of suffering in today’s world. (Currently through a ⁵⁰⁄₅₀ split between FarmKind and the Shrimp Welfare Project.)

Now you might not care about the proportion that goes to shrimps. Fair enough! But as Bentham’s Bulldog’s nicely pointed out recently, shrimp welfare receives far less funding then the annual operating costs of a McDonalds, so I really hope we can expand our compassion for animals that aren’t as cute but could still experience suffering (paid subscribers can get a free pdf copy of my book here).

Here’s the monthly update by FarmKind of recent achievements by their funded organizations:

Since achieving milestones makes video games so compelling, or rather addictive, in my own experience, let me harness that feature of my brain to set two new goals:

Hit $2000 US Dollars in projected gross annual revenue
Get from ~1.5% paid subscribers of my total subscriber count to 3% (not counting comped accounts)

The easiest way to increase one’s Substack income is probably to write about something like politics and rile up emotions, but I’d rather stick to my niche interests. Thus, the question becomes how I can turn free subscribers into paying ones, rather than gain new ones. A focus on quality over quantity. How can I improve my newsletter to make you, a free subscriber (or an already paying one), consider supporting it further? (If you can’t afford one but would like access to paywalled essays, just message me, and you’ll be comped.)

If there’s anything you’d like to see more of in this newsletter or something you’d like to see improved/added, let me know in the comment section below or in a private message. Journal publishing relies on the peer review process, so please don’t hesitate to share your advice.

Thanks again for all your support and readership!

Warmly,

Walter

Lucky hires: how AI safety orgs find their COOs (and why that doesn’t scale) by Patrick Gruban 🔸

Patrick Gruban 🔸 — Tue, 13 Oct 2026 11:26:29 +0000

Epistemic status: a hypothesis from about fifteen interviews with people involved in hiring, plus our advising data. The program described at the end is partly designed to test it.

TL;DR

Whether an AI safety organization can scale often depends on a COO who works as the founder’s partner, and not only as the person running operations.
Most people in that role, including me, got in because they happened to be close to the founder.
Our best guess at why is a trust loop: with thin leadership, organizations only hire people who need no supervision, and since assessing an outsider against that bar is slow and costly, they hire people they have already seen work.
Experienced outsiders can do this job, but building that trust from the outside takes time: our operations advisees who moved into the field took a median of seven to eight months to land a role.
We’re piloting the Organization Builders Program, which tries to create the same conditions in a scalable way. Applications close October 22.

I didn’t apply for my job

I met Claire Boine, the co-founder of Successif, at a workshop where we were both participants. Part of the workshop was peer coaching, and we were paired up to coach each other. Afterward, I offered to continue with weekly calls, as the organization was going through a challenging time.

Four months later, she secured additional funding, and asked me if I wanted to join as COO. We met for a workshop, discussed the terms of working together, and I started without ever going through an interview.

During the months of our calls, Claire had seen how I approach problems and how I handle disagreements, and I had seen the same of her. Since I was advising her and not applying for anything, the stakes were low for both of us and by the time she asked, we had already built a basis of trust.

Discovering that my story was typical

In my post on lessons from my career, I wrote that most of my significant job opportunities came through weak ties and unexpected connections rather than job applications. For senior hires, I value knowing how someone will perform, because a bad hire at that level affects the whole organization. Having worked with someone before reduces that uncertainty more than a hiring round. I assumed this was different in AI safety, a field full of long application processes for everything from fellowships to jobs.

However, when we started interviewing operations leaders in our field about how they got their roles, I heard similar stories again and again. One leader moved from the operations team of a fiscal sponsor into the organization it supported. They knew at least two others who had taken the same route and suggested we copy it. Sam Smith tracked 27 senior operations searches and found that only seven were filled within six months, about half of them through closed rounds.^[1]

I think of these as lucky hires, myself included, though the luck is in the access, not the people. In my case, I had spent years putting myself in situations where luck could find me, and the people we interviewed had done the same. Hiring this way has two problems. It doesn’t scale: it depends on which trusted people happen to be available when a role opens, and the people doing hard jobs right now usually aren’t. And it skips the comparison: if you hire the person in front of you, you never find out whether someone better was out there.

What I mean by COO

The COO title can mean very different things. In Second in Command (HBR, May 2006), Nathan Bennett and Stephen Miles describe seven types of COO, from an executor who delivers the strategy day to day to an heir apparent who is being prepared to take over. Their main observation is that the COO role is defined less by the work and more by the relationship to the CEO as a person.

In this post, I mean what they call the “other half” or the “partner.” The other half complements the founder’s experience and knowledge, while the partner shares the leadership with them. In a young AI safety organization, this is usually the person who helps turn a founder’s research or policy idea into an organization, from setting the organization’s culture to hiring and implementing good management practices.

The article also says the most important factor in a working CEO-COO pairing is the level of trust between them. This matched my experience with Claire. That trust didn’t come only from seeing each other’s work. It also came from caring about each other as people and being willing to admit mistakes.

Why this role matters for scaling

When an organization starts, the founders take on all the roles. This works for some time, but eventually either the work or the founders start to suffer. MATS found that organizations approaching about 20 employees develop a strong need for people who can take on management.^[2]

At Successif, we hear this regularly. Over the past 12 months, 18 organizations approached us on their own, looking for a senior operator or COO. In conversations in the Bay Area in September, nine organizations expressed interest in our program, and several said they need to fill such a role soon. The hiring lead of one research organization called it “maybe the most important thing we are currently hiring for.”^[3]

A good COO also makes many other hires possible. One of the MATS authors estimated that a single manager creates room for 5 to 10 additional hires,^[4] and Morgan Matthews of Constellation described it as bringing in someone “who can 10x your org.” With Coefficient Giving, which also funds Successif, now focusing on founders,^[5] I expect the need to grow, as each new founder will eventually need someone in this role.

This is also why the order of hiring matters. A first leadership hire with strong people-management experience makes every later hire better. And someone who joins early can shape the culture while the organization grows, rather than trying to fix it afterwards.

Our hypothesis on why proximity wins

This is an early guess based on about fifteen interviews this summer with people involved in hiring.^[6] Earlier interview rounds and our conversations with hiring organizations since then have mostly supported it. It assumes the field keeps growing quickly, so developing leaders from within can’t keep up. Trust is not the only barrier to hiring outsiders, and I come back to others at the end, but it is the one we focus on here. We think it works like a loop:

The leadership layer is thin: few people in the organization have managed people or built an organization before.
So the hiring bar becomes “needs no supervision.”
Assessing an outsider against that bar takes experience that small teams often don’t have.
So proximity fills the gap, and people hire whoever they have already seen work, even if what they saw wasn’t leadership.
The insiders who get hired tend to be junior, so the leadership layer stays thin.

I recognize the first two steps from my first startup, and in hindsight they were a mistake I made. MATS found that hiring managers in the field only hire people who need minimal supervision. But every hire needs management, including senior ones, and “we don’t have time” usually means other work was put ahead of developing people. The exception is an organization whose leaders have little leadership experience themselves: it needs someone who can be trusted with leadership responsibilities quickly. That is exactly the hire the loop makes hardest.

None of this makes the people in the loop careless. With the experience they have, hiring someone they already know is an understandable choice. But it isn’t a model to copy. When every organization makes the same safe choice, the leadership layer stays thin, and the field never builds the experience it needs to assess and manage senior people.

Organizations can assess outsiders, but it is expensive. If you have never hired a COO, it is hard to know what you need, and it can take several attempts before you find out. Falling back on people you already trust is faster. When we asked a senior staff member at one organization how much community signal matters in their hiring, the answer was “way more than I’m comfortable with.”

There is also a demand-side version of this. As Alex Novikau pointed out, hiring a partner means handing over real decision authority, and some founders aren’t ready for that yet. It can also feel unnatural for a young founder to hire someone with far more leadership experience than they have, even though that is often exactly the hire the organization needs. A trial can’t fix this, which is why our program starts by scoping the role with the founder before matching anyone, and why founders building their first leadership team may want outside support with hiring.^[7]

Step 5 explains why the loop doesn’t correct itself. It might also explain something we see in our advising: open positions receive hundreds of applications, yet senior searches keep small teams busy for months, and many organizations end up promoting early-career staff instead.

It also explains why both sides of a recent discussion can be right. Organizations really do need senior people, and experienced applicants really are rejected at very high rates; MATS hires about 2% of its operations applicants. The shortage isn’t of capable people, it’s of ways for organizations to tell which outsiders they can trust.

Outsiders can do the job, but it’s hard to tell which ones

For some roles, such as grantmaking, knowing the field is crucial. Being a founder’s other half is different, as it mostly requires experience in building an organization alongside someone, and this experience transfers. One operations leader we interviewed estimated that about 80% of his commercial experience transferred directly.^[8]

Sam Smith started his investigation expecting that senior operators with enough context didn’t exist, and changed his mind after about 100 conversations. The harder problem is telling in advance which of them will work out in a specific organization.

Bennett and Miles add a caveat: success as COO in one company says little about success in the next, as the role depends so much on the specific CEO. I agree, and for me, this is an argument for testing the fit with a specific founder rather than judging a CV, and it is why our program starts with a trial project.^[9]

Seeing how long it takes

Among advisees whose advising has formally ended, operations profiles moved into the field more often than policy or communications profiles.^[10] For those who did make the move, the median time from applying to us to starting a new role was about seven to eight months. That time includes everything that slows a move down, such as pay, location and the number of open roles, not only trust.

When we look at what people do during these months, we see a common pattern. As we described in our post on career transitions, one advisee talked to about 100 people before finding a role, and another volunteered for four organizations. Jim Chapman spent more than two years and had around 140 one-on-ones before becoming Operations Director at MATS. In my post on getting into an organization mid-career, I recommended many of these steps myself, and I still think they are good advice.

What this means, however, is that each of these individuals is creating their own chance encounter, one conversation at a time. This works, but it is slow, and only people who can afford a year or more can do it.

Trying to make the lucky hire deliberate

When I think about what the time with Claire gave us both, I come up with four things. We asked ourselves whether a program could provide each of them intentionally:

What the chance encounter provided	What the program provides instead
Working together at low stakes before any commitment	A paid trial project, 5+ hours a week for 3 months
A reference from someone the leader already trusted	Structured vetting, reference checks and a one-page assessment of each candidate
Time for the leader to work out what they actually need	Role scoping with the host organization
The newcomer picking up context along the way	A peer group, advising, and input from people who made the same move

This is the idea behind the Organization Builders Program. It is for people who have built or scaled an organization or team and worked closely with founders. You don’t need AI safety experience, but you do need to care about the problem. We look for people who build trust actively and would probably find their way in eventually. The program’s aim is to make that take months instead of years.

Each host organization chooses between a paid trial project alongside your current role and its own assessment followed by a direct hire. The first round is small, with up to 10 participants and a handful of organizations. Vetting is run by Impact Ops. Trials are remote. Most host roles are too, though some organizations need people on site, mainly in the Bay Area or London. To be upfront: this is a selective program, and most applicants won’t be matched in the first round. Applications close on October 22.

Several programs now help experienced professionals move into the field.^[11] Ours focuses on the founder’s partner role and on matching each participant with one host organization, and it can be combined with any of them.

We chose 5 hours a week so that people can try this without quitting their job. That’s enough to see how someone handles a real problem and how they and the founder work together. It isn’t enough to see how they would carry a share of the leadership. One interviewee warned that a small weekly slice only “gets whatever project is stable enough to survive that churn.” This is why host organizations can skip the trial and hire directly after their own assessment, and why we will review the format after the first round.

Caveats and limitations

Some organizations do invest in assessing outside candidates well, and outside hires don’t always work out. Besides trust, there are other barriers. Sometimes it is pay or location, and many experienced professionals don’t see themselves as candidates without guidance.

Our current advising has limits here. Long-term advising calls show us how someone thinks about their career, but not how they work with a founder or under pressure, and one organization told us that our referrals have sometimes been “somewhat miscalibrated.” That is part of why we’re using a vetting process similar to a hiring round, with a case-based interview and reference checks, followed by a trial period, since three months of real work tells an organization more than a recommendation from us.

My own experience likely biases me, and survivorship bias affects the stories we heard, since we mostly talked to people who made it. The measure I care about most is whether host organizations are able to scale after bringing in a partner COO this way, for example, by hiring more people and taking on more work without the founders becoming the bottleneck. If they also hire more people from outside, that would suggest the loop is starting to break. I would change my mind if organizations express interest but won’t commit or pay, if trials don’t turn into hires, or if pay turns out to matter more than trust.

How you can help

If you’ve built an organization alongside a founder: apply through the program page by October 22, or pass this on.
If you lead an organization with 3 to 50 staff and no one in this role: apply as a host organization through the program page.
If you’re already a COO in the field: tell us where we’re wrong, and consider whether you would mentor someone making the move.
If you run another talent program: we would like to coordinate with you.
If you’ve tried to move into a senior role in the field and didn’t get in: I’d like to hear what happened, even if you’re not applying.

I’m also curious whether this matches your experience. If you got into your role through proximity, or found a way in without it, I would like to hear how.

Acknowledgements

I thank Weronika Żurek, Isabelle Klöck, Adam Tury, Darryl Wright and Morgan Matthews for their work and input, as well as everyone who spoke to us anonymously.

A note on AI use

I wrote this post with the help of Claude. I recorded my thoughts, made the decisions on structure, framing, and content, and edited the text, while Claude drafted sections based on my input and our internal research. I checked the claims against their sources.

^
In the same analysis, only about 110 of about 200 live roles were advertised publicly.
^
MATS calls these people “Amplifiers”: people who can take on people management, project management and research management.
^
This matches broader data. In 80,000 Hours’ survey of organization leaders, senior management and operations roles came out as the most important bottleneck to growth, and of 349 leadership postings across 142 organizations, 52.4% were management and 35.5% operations roles.
^
The estimate comes from a comment by one of the co-authors on the post, not from the post itself.
^
In their words: “Founders are the bottleneck, not funding.”
^
Interviewees included talent and operations leads at research and governance organizations. We asked each of them to walk us through a specific recent hire and the evidence they relied on. Two earlier interview rounds, including one at EAG San Francisco in February 2026, informed the same picture.
^
Adam Tury and I came up with the idea for RAISEimpact, a Successif program that offers fractional people development to AI safety organizations.
^
This and the fiscal sponsor example come from anonymized interviews with senior operations leaders conducted by Weronika Żurek.
^
One organization hired a general counsel without community signal, based on a hard work test and a demanding interview, and told us the hire has gone “amazingly well.”
^
Successif advising data, advisees who graduated or closed (n=144 across profiles): 46% of operations profiles moved into the field, compared with 41% for policy and 33% for communications. These figures overstate the overall rate, because they leave out many open cases that have probably gone quiet, so read them as a comparison between profiles rather than a success rate; the gap with policy is small. The seven-to-eight-month median is a combined figure for operations and communications advisees who transitioned.
^
For example, the Generator Residency, Lateral Workshop, High Impact Professionals and CEA’s four-day career pivot course.

Frontpage posts - Effective Altruism forum viewer

Which AI safety organization does what? Puzzle/​party games by david_reinstein

Your work at [large AI safety ecosystem org] is not replaceable by brentonmayer

Rant

/​rant

“Earth Systems Risks and the Case for Funding Stabilization Research” by vsrinivas

Did I rejoin Ambitious Impact at the wrong time? by Aidan Alexander

My worry

Broad timelines im­ply a port­fo­lio of bets, and AIM is an im­por­tant bet within it

Join­ing now means helping shape what AIM be­comes.

And then there’s the peo­ple and culture

So why haven’t you ap­plied yet?

Some roles lean more se­nior:

Others are a great way in, whether you’re early in your ca­reer or switch­ing fields:

To pre-empt some mis­con­cep­tions that might stop you from ap­ply­ing:

The pro­cess:

To close

EA Implications Of Being In An AGI Simulation by Ariel Simnegar 🔸

A. The Si­mu­la­tion’s Cru­cial Considerations

i. Why An AGI Sim?

ii. Most Peo­ple And An­i­mals Might Not Be Conscious

iii. We Might Get Ter­mi­nated Soon

B. Im­pli­ca­tions for EA

i. Fa­nat­i­cism Doesn’t Care

ii. The EA De­ci­sion Tree

iii. Pro­por­tional Fidelity’s Effect on Neartermism

iv. Com­pute Fungibility

v. In­fluenc­ing The Simulator

vi. We Are Digi­tal Minds

C. Test­ing the AGI Si­mu­la­tion Hypothesis

D. Your Child­hood Might Not Have Hap­pened To You

E. So What Should We Ac­tu­ally Do?

Are there any people in the EA community involved in peacemaking efforts? by Karl Mechkin

ML4Good Impact Assessment by Jack_S🔸

Intro

Sum­mary of activities

What this im­pact as­sess­ment covers

Sum­mary of results

What we try to do

Outcomes

Differ­ences be­tween bootcamps

Cost-effectiveness

Caveats

Is “more peo­ple work­ing in AI Safety” the right met­ric?

Shouldn’t we have more rigor­ous coun­ter­fac­tu­als?

How com­pa­rable is cost-effec­tive­ness across in­ter­ven­tions?

Bot­tom Line

Next Steps

Appendix

Ap­pendix 1: How we have spent money

Ap­pendix 2: Im­pact Assess­ment Methodology

Surveys

Scoring

Ex­trap­o­la­tion to non-respondents

AI safety comms reading list by benrmatthews

Why I put this list together

If you only read five articles

What the pub­lic thinks about AI

What the pub­lic thinks about catas­trophic AI risk

What changes how peo­ple re­spond to AI risk

Why peo­ple re­spond the way they do to AI risk

Mes­sen­gers, iden­tity and trust in AI safety

Does pub­lic opinion move policy and AI gov­er­nance?

Who is already work­ing on AI safety comms

How you can contribute

Robustness: Backfire Risks in AI Safety and Governance by Dawn Drescher

Introduction

How to Read an Entry

The Two Taxonomies

Mechanisms

Re­spond­ing actors

The matrix

The Master List of Risks

1. Selec­tion and Goodharting

2. Dual Use and Ca­pa­bil­ity Externality

3. Sub­sti­tu­tion and Leakage

4. Overhang

5. Com­mit­ment Dynamics

6. Securitization

7. Le­gi­t­i­macy and Backlash

Which AI safety organization does what? Puzzle/party games by david_reinstein

/rant

Broad timelines imply a portfolio of bets, and AIM is an important bet within it

Joining now means helping shape what AIM becomes.

And then there’s the people and culture

So why haven’t you applied yet?

Some roles lean more senior:

Others are a great way in, whether you’re early in your career or switching fields:

To pre-empt some misconceptions that might stop you from applying:

The process:

A. The Simulation’s Crucial Considerations

ii. Most People And Animals Might Not Be Conscious

iii. We Might Get Terminated Soon

B. Implications for EA

i. Fanaticism Doesn’t Care

ii. The EA Decision Tree

iii. Proportional Fidelity’s Effect on Neartermism

iv. Compute Fungibility

v. Influencing The Simulator

vi. We Are Digital Minds

C. Testing the AGI Simulation Hypothesis

D. Your Childhood Might Not Have Happened To You

E. So What Should We Actually Do?

Summary of activities

What this impact assessment covers

Summary of results

Differences between bootcamps

Is “more people working in AI Safety” the right metric?

Shouldn’t we have more rigorous counterfactuals?

How comparable is cost-effectiveness across interventions?

Bottom Line

Appendix 1: How we have spent money

Appendix 2: Impact Assessment Methodology

Extrapolation to non-respondents

What the public thinks about AI

What the public thinks about catastrophic AI risk

What changes how people respond to AI risk

Why people respond the way they do to AI risk

Messengers, identity and trust in AI safety

Does public opinion move policy and AI governance?

Who is already working on AI safety comms

Responding actors

1. Selection and Goodharting

2. Dual Use and Capability Externality

3. Substitution and Leakage

5. Commitment Dynamics

7. Legitimacy and Backlash

9. Moral Hazard and False Assurance

10. Compliance Asymmetry

12. Precedent and Normalization

13. Crowding Out

14. Fragility Through Concentration

Using the Matrix Without Being Paralyzed

Why fortification can be highly cost-effective

Where fortification falls short in practice

From a small-mill technology to a broader fortification system

Three lessons from scaling across Tanzania, Kenya and Ethiopia

1. Build with millers, and adapt to the market

2. Establish stronger baselines before entering new markets

3. Recruitment and People Ops can become a constraint on scale

What we learned from the Rethink Priorities analysis — and how we’re acting on it

How we’re acting on the findings

Why scale further, and what it will take

Procreative Beneficence and Genetic Enhancement

2. Important concepts

3. Obligation to genetically enhance embryos

4. Objections and Defense

4.1 Degree and Precision

Originally published here:

Programs and projects from Generator

Resident Outcomes

Generator 2.0

Intermediate exponents

Farmland per dollar

Chief of Staff / Managing Director

Director of Operations

Program Launch Lead / Director

Recruitment Manager

Questions, and how you can help