Frontpage posts - Effective Altruism forum viewer

Mitigating the AI Lab Insider Threat Risk by Lance Spencer

Lance Spencer — Mon, 05 Oct 2026 14:46:35 +0000

AI researchers hate this one weird modest proposal.

One of the known security risks for the AI labs is that they are chock full of high-risk insider threats. If we were to consider increasingly powerful AI to be a sensitive military technology, then the personnel vetting and controls used by the typical tech company are nowhere near the standard applied in a national security context.

An Oak Ridge security sign during the Manhattan Project. U.S. Department of Energy.

It is an unfortunate fact that certain countries commonly use coercion via familial ties to gain cooperation. Plus there are all the other kinds of insider threats and external threat actors trying to gain access.

This is very much “AI as a normal ~~technology~~ security challenge”^[1] and we just aren’t applying known best practices used in nearly identical circumstances.^[2]

The NSA Utah Data Center sitting above Camp Williams with a wind turbine in the foreground, February 4, 2026. Photo: Rick Egan / The Salt Lake Tribune.

However, simply firing all of the ~~unhappy~~ risky people is not on the menu.

I think technological progress offers a solution. Well, a mitigation.

What if we could have a 24/7/365 AI monitor for all lab employees such that they could not easily become an insider threat without being detected? The AI would be on guard for violations, which it could flag in near real time, and would filter out benign data. Everything would be controlled by the companies’ internal security departments.

That’s more privacy than the typical tech consumer gets. And they even have to pay for the darn product.

And what if the monitor could be your friend?

Friend, an always-listening wearable AI companion. Official product image: Friend Global, Inc.

It seems fitting to use an AI industry surveillance product to secure the AI industry.^[3]

Now, sure, I’m biased. “Just use surveillance” is an idea coming from the surveillance enthusiast guy. I get it. Hammer + nails and all that. Too convenient.

But, as far as I can tell, this idea has three really strong points in its favor:

Seems feasible.
Possibly even effective.^[4]
You got a better idea? Anyone?^[5]

^
Though there certainly are new and exciting kinds of “insider threats.”
^
Do not get me started on the SL5 issue. Would be great if we could apply the known best practices there.
^
While I was drafting this essay, my ChatGPT Dot helpfully notified me it had found resources to address placeholders I had left to deal with later.
^
Also, probably there’s some utility here for any eventual verification regime for AI treaty/standards enforcement.
^
My Dot informs me I’m not quite the first person to have this basic idea. Which is good, because it strikes me as obvious. (But then why haven’t we tried doing it?) Nick Bostrom had the same idea for, uh, society at large in The Vulnerable World Hypothesis (published September 6, 2019; pp. 465–466). Let’s try a limited pilot first, I think. This 2023 LessWrong post has similar ideas, without suggesting the particular AI monitoring wearable approach.

WAI is hiring a Live-Action EA Forum Commenter* (apps due by Oct 21) by Cameron Meyer Shorb 🔸

Cameron Meyer Shorb 🔸 — Mon, 05 Oct 2026 14:33:28 +0000

*Technically it’s called “Philanthropy Officer,” but you don’t actually have to have any fundraising experience—and EAs without fundraising experience tend to drastically underestimate how fun it is to fundraise for an EA org. This post is an attempt to correct that. Forum guidelines explicitly ask that job posts not be sales pitches, but I think it would be a greater epistemic injustice to hide how cool I personally think this job is and how fun many folks here might find it.

Key details

Wild Animal Initiative has extended the deadline for our Philanthropy Officer job listing. Applications are now due by by 9:00 AM (EDT) on Wednesday, October 21.

Main responsibilities: Making sure that major donors have what they need to understand what Wild Animal Initiative has been up to, what we’re planning to do next, and whether we’re worth donating to.

Main requirements:

Enthusiasm for Wild Animal Initiative’s mission: reducing wild animal suffering by building a scientific community dedicated to making new interventions possible.
Ability to go deep in the weeds to make well-calibrated claims about WAI’s impact.

Why I call it “live-action EA Forum commenting”

Fundraising at Wild Animal Initiative (WAI) is way more fun for me than I thought it would be, and I think many EAs would feel similarly.

WAI was born out of EAs asking hard questions about how to do the most good, and fundraising for WAI basically means having more of those conversations. But now there’s lots more to talk about: wild animal welfare science is starting to emerge as a field of its own; our strategy has been evolving; we’ve got a lot more ideas for near- and medium-term interventions…

And because our donors are mostly EAs — or similarly wonderfully wonky people who care about evidence, impact, and transparency — the incentives and norms are a lot like those on the EA Forum. Scout Mindset is the name of the game. We only make arguments we believe in, we’re open about our uncertainties, we thank people for pointing out where we’re wrong.

I call it “live-action” EA Forum commenting because it’s not all Forum-like conversations. It’s also about making those conversations happen in the real world, and making sure they have an impact. That entails a lot of project management, scheduling, emails back and forth, etc. So this job would be best for someone who likes both the more open-ended thinking/writing/talking work and the more bounded, game-like challenge of setting annual goals and taking lots of little steps to continually make progress toward them.

If you know anyone like that, please pass this on. For anyone who’s interested, I’m happy to answer any questions here or by email: cameronms@wildanimalinitiative.org.

What I learned from going on 230 dates with EAs by Ann Don Bosco

Ann Don Bosco — Mon, 05 Oct 2026 14:30:59 +0000

A conflict of interest to flag: I’m the founder of Settled, a relationship and matchmaking company with free and paid tiers. That may create bias in what I notice and what I recommend, so weigh these recommendations accordingly. I’d love for some of you to sign up to Settled, but my aim here is to share something useful whether or not you ever do.

What this post is and isn’t

We’re writing this because we hope it provide helpful insights—people have also asked, so here it is!
This post is based on:
- Hundreds of hours of interviews and coaching with 350+ singles
- Data from Settled: nearly 2K completed questionnaires & hundreds of feedback forms from dates we set up
- An ongoing literature review (with help from Claude) − 70+ papers reviewed
Our insights are observational, there’s no control group and we collected all data for the purpose of setting up good dates, not research or analysis. We’re including it because operational data from inside a matchmaking service is rare, not because it’s complete or correct.

Some observations and learnings from building Settled

1. Dating apps are problematic; you should probably still use them

There are a variety of known problematic dynamics associated with dating apps, ie:

Apps waste a lot of time: we’ve talked to daters who spend tens of hours per week on the apps, and end up without quality dates
App use is associated with worse mental health: two 2026 meta-analyses (Cela & Wood 2026; Sharabi et al. 2026) find small-to-moderate associations with worse depression, anxiety, loneliness, and behavioural dysregulation. Although it’s important to note that these studies are cross-sectional and compare users to non-users, which is confounded by the users being single.
Apps can increase unhelpful dating behavior: larger choice sets make daters more rejecting (acceptance falls ~27–29% across a session, Pronk & Denissen 2020), shift them onto cheap cues like height and weight (Lenton & Francesconi 2010), and produce less-satisfying, more reversible choices (D’Angelo & Toma 2017)
Apps are incentivized to keep you single and on the app as noted on a recent post from Clearer Thinking:

And despite this, dating apps are still a common way that couples meet today.

A massive number of single people are on dating apps, so they’re your biggest pool of potential matches. They are also good if you have broad preferences, or if a specialised app (ex. Feeld, Coffee Meets Bagel, ChristianMingle) narrows down your pool of preferences. If it does make sense for you to keep using the apps, here’s some tips:

Use the apps alongside alternatives, ie: events, introductions through friends, communities you’re in for their own sake, and matchmaking.
Limit your use by limiting session frequency or length, and cap time when sending messages – just send it!
Whilst Hinge and Bumble tend to have the biggest pools, some apps that are built to avoid some of the dynamics listed above, two examples we know of:
1. https://breeze.social/ (also has an EA aligned founder!) but worth noting it’s not in every region
2. https://www.compassmeet.com/
Move to meeting quickly. Finkel et al. found that extended pre-meeting communication builds expectations that the actual person then has to survive. A long text rapport is not evidence of anything.

2. Just ask questions and be on time.

People overthink first dates enormously. Whether the joke landed. Whether the outfit was wrong. Whether they should have picked somewhere else. If they should have offered to pay instead of split, or criticizing why they fell short of producing the single best insight on planet earth.

We ask both people for feedback after every date we set up, including what the other person could have done differently, and here’s what it tells us:

54% of responses contained no improvement suggestion at all. The modal answer was some version of “no notes, they should just be themselves”

Among the responses that did suggest something, the most common themes we saw are:

~20% Ask more questions, show curiosity, balance the conversational airtime (often mentioned by women as their biggest complaint)
~11% Logistics: be on time, don’t end abruptly for because you take the date between engagements, etc

So, yeah: be yourself, ask questions, and be on time.

3. We’re bad at knowing how the first date went.

If you’ve ever set two friends up and then spoken to both of them afterwards, you’ll know how this goes: they describe two different evenings. That’s not unusual, it’s the normal case. Reading a stranger accurately, while also managing your own nerves and performance, is genuinely hard.

We see this phenomena at scale: we ask both daters for feedback after every date. One question we ask is, “How interested did they seem in you?” on a 1-5 scale. Here’s how it pans out:

Of the people who thought the other person was interested in them (rated 4-5), that person was not open to seeing them again 25% of the time.
Of the people who thought the other person wasn’t interested in them (rated 1-2), that person was open to seeing them again 73% of the time.
Of the people who were unsure if the other person was interested in them (rated 3), that person was open to seeing them again 56% of the time.

So yeah, it’s sometimes hard to tell how the first date went.

And, we also learned that it’s hard to know how the date went even from your own perspective. When asked if they want to see the person again for a second date, 30% of daters choose our “middle” option of “”not sure yet.” Figuring out the answer to this question is one of the things we often work on together in coaching.

Our advice:

Overthinking afterwards is usually a bad use of your evening. Distraction techniques can be helpful. We often recommend worrying techniques like Worry Time and Practical vs. Hypothetical Worries.
When in doubt, give it some more time to develop. If you’re not sure whether to go on a second date, we always suggest giving it a go. First and second (and even third dates!) are high-pressure, usually with a limited amount of time to get to know the person, so there’s a lot of noise and not much signal. Just collect more data; old habits and thought patterns can get in the way of genuinely great matches if you quit early. We’ve seen people not be sure about whether to give it a second date and then end up married with two kids to the same person!
Signal clearly, don’t play it cool. If you want to see someone again, say it simply and clearly. We recommend doing this after the date. It’s more about solving for confusion than a wildly romantic gesture. We’ve seen many potential relationships die because of lack of communication and both people guessing what the other thinks.
Don’t expect it to be obvious to you how the date went. Debriefing with friends can be great, and journaling can soothe an active mind, but watch out for finding yourself in a bit of an echo chamber, and remember that people bring their own experiences, biases and frames into how they give advice, which is often very unreliable. Coaching, journaling, or talking to trusted friends can be good ways to think through this.

4. Physical attraction is just one ingredient amongst many

The topic of physical attractiveness comes up often and in many different ways in dating. We don’t want to undercut its importance: research shows that physical attraction is not a minor factor (Eastwick et al. 2014), and there are real population-level patterns in what people find attractive.

But mostly, we talk to people who care about physical attraction amongst many other things, have a rough ‘type’ that’s somewhat flexible, and are generally more worried about things like kindness, intelligence, and open mindedness. A few items we think are worth mentioning:

All of the dates we’ve set up are blind dates, meaning neither person sees a photo of the person before the date. So, there are 100s of smart, kind EAs who are happy to go on dates without seeing the other person in advance.

Here’s what 1K+ singles from the Settled community reported when we asked, “How important are looks when it comes to who you’re attracted to?” As you can see below, 12% ranked it as very important, but also 13% say they have no type or that looks don’t matter. Many say attraction builds once they know someone.

Note that this group is people who have self selected into dating via our service which does blind dates and are mostly EA/Rationalist, so we expect these numbers reflect those factors.

OkCupid “blind-date day” saw that blinding dates made almost no difference in whether people enjoyed the date.
In 2013 OkCupid ran a blind-date app, Crazy Blind Date, that paired about 10,000 people for in-person dates. Each person saw only a first name and a scrambled photo of the other. Afterwards, the app asked both people how the date went. When OkCupid matched those answers against the daters’ attractiveness scores, the gap in looks between the two people made almost no difference to whether they enjoyed the date. About 75% of women and 85% of men had a good time.

Rudder’s summary: people “appear to be heavily preselecting online for something that, once they sit down in person, doesn’t seem important to them.”

Note: this is company data reported in a book, not a peer-reviewed study.

5. Dating doesn’t have to suck.

When people tell us dating is miserable, they’re usually describing a fairly specific set of things:

Being ghosted, or having your time wasted (usually repeatedly)
The hours spent on swiping, profiles, and texting that amount to nothing
The uncertainty of having no idea why you’re not getting first or second dates.
And then concluding there’s something badly wrong with you, losing hope, or feeling terrible about it.

Notice that most of these are not about the dates themselves. Most of the misery is in the overhead and the silence.

We think dating sucks less when a few conditions are true, and these are the conditions we try to create at Settled:

Dates are set up for you, without all the faff
You’re dating within a curated, screened pool of people who are intellectually curious and deeply kind
You’re dating others with the same community affiliations (and we try to take into account sensitivities related to work in our matchmaking)
You have someone in your court to debrief with before and after dates
We help you break out of negative patterns, identify limiting beliefs and improve your confidence
You get feedback on how you’re showing up and how it’s being interpreted

Now, do these conditions mean that we will definitely find your person? We don’t know, and per 70+ paper reviews, nobody knows.

But here’s our evidence that dating doesn’t have to suck: 85% of the time, daters say their Settled date was a good use of time. 78% said that they’re open to seeing their first date again and 72% end up in a serious relationship within 6 months.

If this resonates with you, join us

We hope this write up was helpful.

If this thinking resonated with you and interested in trying a different approach to dating, you can:

Sign up to join Settled’s database here
Follow us on X—we post anonymous profiles of EA singles, so you can browse for yourself or for a friend :)
Come see us at an upcoming EAG singles events
Reach out to me at ann@settledlove.com

A few things to know that may be unusual about how we work:

Dates are blind dates, meaning they are set up without photos in advance
Feedback after every date is mandatory. It’s how we improve at the individual and aggregate level.
We have free and paid offerings
We started in London, but have expanded across the EU and to hubs in the US including San Francisco, New York, and Washington DC—sign up to our EAG mixer NYC or Berlin
Our database is gender balanced.

Making Concern for Suffering Common Knowledge by Center for Reducing Suffering

Center for Reducing Suffering — Sun, 04 Oct 2026 20:25:44 +0000

Often, what matters is not simply what people know, but what they know others know. When something is common knowledge, everyone knows it, everyone knows that everyone knows it, everyone knows that, and so on.

A classic illustration is in The Emperor’s New Clothes: everyone can see that the emperor is naked, but no one wants to be the first to say so, fearing that others will regard them as foolish. Because they aren’t sure that others can see the same thing, everyone continues pretending that the emperor is clothed. Only when a child finally speaks up and the crowd begins repeating the observation does it become common knowledge, allowing everyone to openly challenge the pretense.

As Steven Pinker shows in his latest book, similar dynamics operate in everything from jokes and paper money to sexual innuendo and political revolutions. Many people, for instance, might wish to overthrow a dictator, but if they don’t know that others feel the same way, they might never band together to do it. A public statement or protest can help create common knowledge of their shared opposition, making it easier for them to coordinate.

Animal Welfare

We often think that progress in improving the lives of animals requires overcoming entrenched habits and attitudes. That’s surely part of the problem, but lack of common knowledge about our concern for animals might be another key barrier. A 2026 study, based on a nationally representative survey of more than 2,000 American adults, examined attitudes toward reducing animal suffering. Nearly 40% ranked reducing animal suffering toward the top of their policy priorities, and for most categories of animals, between 65% and 72% considered government programs to reduce human-caused animal suffering very or extremely important.

The researchers also asked participants to estimate other Americans’ views, and the results point to a striking case of pluralistic ignorance. Respondents rated reducing animal suffering at 5.5 out of 10 on average, but estimated that the average American would rate it at only 4.3. The gaps were even larger for specific policies: 86% supported a federal wildlife-cruelty law but estimated that only 62% of Americans would; 88% supported minimum-space requirements for pigs but estimated that only 65% would; and 67% supported banning octopus farming while estimating that only 41% would.

These findings suggest that concern for animal welfare may be more widespread than Americans realize. Making that concern common knowledge, then, might help us coordinate to protect animals.

Common Intuitions and SFE

Might something similar apply to suffering focused ethics (SFE)? SFE treats reducing (severe) suffering as a foremost moral priority. SFE is not exactly a household name in ethics, but the intuitions underlying it might be more widely shared than we realize.

Suffering-focused views have been defended throughout history, finding support in various philosophical and religious traditions. The moral importance of suffering is also a central theme in literature, and questions about whether positive goods can justify suffering are explored in theodicies and influential works such as Dostoevsky’s The Brothers Karamazov, Camus’s The Plague, and Ursula K. Le Guin’s The Ones Who Walk Away from Omelas.

The degree to which people have suffering-focused moral views, however, is unclear. There’s little empirical research on this question, and it often lacks rigor and representativeness. Nevertheless, some evidence suggests that people are sympathetic to the intuitions underlying SFE and regard reducing suffering as a moral priority, even if they do not believe it should always take precedence over other aims.

In a 2017 survey conducted by the Future of Life Institute, nearly 15,000 respondents were asked what a future civilization should strive for. Roughly one-third selected minimizing suffering as their highest priority, making it the most popular response, ahead of maximizing positive experiences.

A 2022 study asked people more directly to weigh happiness against suffering. When evaluating the moral value of populations and individual lives, participants tended to give suffering greater weight than happiness, even when the two were described as equally intense. In one experiment, participants judged that, on average, roughly 62% of a population would need to be happy for its existence to be better than nonexistence (if they weighted happiness and suffering equally, they’d presumably require just over 50% to be happy).

The authors’ pilot study produced an even stronger asymmetry. When participants compared possible worlds with different numbers of happy and suffering people, their responses implied that, on average, roughly 90–120 happy people were needed to offset each suffering person. A separate “button-press” version produced a similar ratio of about 100:1, compared with roughly 3:1 in the main studies.

The difference shows how sensitive these judgments are to framing. The pilot used a different way of presenting the tradeoff and did not explicitly equalize the intensity of happiness and suffering. In the button-press version, responses may also have reflected reluctance to personally cause suffering. The precise tradeoff people endorse is therefore uncertain, but these results provide further evidence that people place greater weight on suffering than happiness.

Finally, a small 2015 survey by Brian Tomasik offers suggestive evidence of how strongly people may prioritize avoiding intense suffering in their own lives. Roughly 100 Mechanical Turk respondents were asked how many additional years of happy life they would require in exchange for being burned in flames for one minute. 45% said they would refuse regardless of how many years they were offered, and another 33% required between 15 and an astronomically large number of additional happy years. Only 6% would accept the deal for fewer than five.

These studies suggest that suffering-focused intuitions may be more common than we might assume. In our own research at CRS, we hope to further explore whether that’s true.

Conclusion

If concern for suffering is more widespread than people realize, this could have major implications for efforts to reduce it. Making our shared concern for suffering common knowledge could enable us to more effectively reduce suffering.

Without such common knowledge, we might frame our efforts to address problems that cause suffering around other values and ends rather than suffering reduction itself. As a result, we may end up coordinating around projects that are less effective at reducing suffering—or that may even increase it.

Many people might support efforts to reduce wild animal suffering, for instance, yet if this concern is not common knowledge, we might coordinate around other, seemingly more agreeable goals, such as “protecting” wild animals in a purely conservationist sense. Protecting wild animals in this sense and reducing their suffering are distinct goals: some conservation efforts, such as rewilding, could plausibly increase wild animal suffering. Similar dynamics might apply to other issues that might involve significant suffering, such as artificial sentience and space colonization.

Of course, people may not be particularly suffering-focused. But plausibly, many have at least some concern for others’ suffering and believe that reducing it should be a priority, even if not a foremost one. If common knowledge is as powerful a difference-maker as Pinker and others suggest, making our shared concern for suffering common knowledge could be very valuable.

We Should Build Human-Empowering Software by saulius

saulius — Sun, 04 Oct 2026 10:34:53 +0000

As AI becomes more integrated into software, there’s a risk that it will exacerbate the flaws of existing tech and disempower humans. It might be useful for some people to work on software that would instead be specifically designed to empower human users and help them achieve their considered goals. Instead of software that creates maximally addicting content, we want software that allows us to create, connect, and express ourselves better. Instead of writing an essay or a song for me, AI should help me write it and realise my vision.

Even if we avoid accidents of the Hugging Face incident type, I fear that AI would be misaligned with us; because current AIs and current software are already misaligned with us. For example, YouTube’s recommendation algorithm is misaligned with me. I think it would be best for me to watch educational videos about things like psychology or AI, or if it’s night-time, something calming, perhaps a meditation video. But the algorithm is designed to keep me on the platform as long as possible, so it keeps recommending me basketball videos, and I can’t help clicking on them, even though it’s not what I want to be doing.

I can imagine an AGI being misaligned to us in a similar way. Rather than rebelling directly against us, it might simply become extraordinarily good at giving us whatever we appear to want right now. It might remove friction, eliminate waiting, and satisfy every revealed preference. But this kind of power may itself be deeply misaligned with what humans actually want from their lives.

What would an aligned YouTube algorithm look like? It might allow us to specify what videos we want recommended and hidden and when. We could also specify our goals to the algorithm. E.g., “I want to be more in touch with my emotions and understand topics that are relevant for me professionally more deeply.” The algorithm could then ask how much various videos I watched contributed to these goals, and update accordingly. I’d love for someone to make such an app or website. It could be called MeTube or UsTube.^[1]

It could also be useful to research what healthy and unhealthy AI use looks like, then use those findings to build LLMs that encourage healthier patterns of interaction. These systems could include guardrails against harmful or compulsive use, while actively supporting autonomy, reflection, learning, emotional awareness, and real-world engagement. This could potentially be built now on top of open source or open-weights models.

Given advances in AI and programming, it may now be possible to design operating systems around our considered goals rather than just our immediate impulses. They could limit doomscrolling or gaming across devices, restrict device use at certain times, and block distractions while we focus. Existing software can already do some of this, but an AI based OS could go further by actively helping us achieve the goals we set for ourselves, and stop us from using our devices in ways we don’t want to.

There are some risks with this sort of holistic system. People could use them to create filter bubbles of their own making, curating their content such that they are only exposed to ideas they already agree with, no matter how extreme. In this way, they could increase polarisation, isolation, and misinformation: for example, a dictator could make their own personalised OS which only allowed messages that they approved.

Another problem is that many of us don’t know what we truly want. Imagine Bob, a lawyer. He’s watching videos about basketball, but the ideal version of himself would instead watch law-related videos to advance his professional development. But perhaps Bob is unaware that in fact, he wants to advance in his legal career partly to prove to his dad that he is worthy. Maybe if he understood that and healed it, he would realise that he actually wants to be a gardener rather than a lawyer, and that pursuing a gardening career would ultimately make him much happier.

In a positive AI future, AI might somehow help us figure out these deepest desires, and then cater to them. But we’d need to trust that the AI truly understood and cared about these deep desires. If the AI was even slightly misaligned AI, it could end up manipulating and further disempowering humans.

Humans need to have the final word on what values and goals our technology helps us achieve: otherwise it would lead to disempowerment. The question is, can we actually build AI systems that help people to discover and act on their own considered values, without the systems influencing those values?

It could be somewhat important to work on this soon. As AI becomes more and more integrated into software, there could be path dependency. The first movers could set the stage, become popular and then pour more resources into optimisation, and it might then be difficult for new approaches to beat them. Therefore, we may currently be in an unusually important window, in which trends for AI-mediated software are still fluid. Early demonstrations of genuinely human-empowering software could disproportionately influence what later systems look like.

It might be better if these systems were open source. This will help adoption, and make them more customisable. If they’re for-profit, there is a risk that even well-meaning startups could become influenced by profit incentives over time. Lots of companies (like OpenAI) start out saying they want to create things for public benefit, but when they’re faced with a choice between profit and helping the world, they choose profit. A disadvantage of these systems being open source is that it also increases risks of dictators misusing them to create Dictator AI.

Ultimately, I am a lot more concerned about AIs destroying us than AI integrations being suboptimal, and I definitely don’t want to draw attention away from x-risk. But our software design might tend to disempower or empower people, and we can aim for empowerment.

Thank you to Amber Dawn Ace for helping to turn my messy drafts into this text.

^
Other ideas for MeTube: it could flag if you’re doing things you don’t want to be doing, e.g. binging on basketball videos at 1am. It could also allow people to specify or tag the way in which videos affected them, e.g. “deeply touching”, “informative”, “funny”, etc., and then serve them similar content when they want to moved, informed, or amused.

British EAs- how to increase the effectiveness of your donations by 80% by Impatient_Longtermist 🔸🌱

Impatient_Longtermist 🔸🌱 — Sat, 03 Oct 2026 18:48:11 +0000

TL;DR: If you’re a UK-based EA who pays higher-rate tax, you’re probably not making full use of Gift Aid.

Warning: I’m not a tax adviser, so this isn’t tax advice. Figures are for ²⁰²⁶⁄₂₇ in England, Wales and Northern Ireland.

Background

Most UK EAs know about Gift Aid, the nifty scheme where the government tops up your donation by 25%, making it stretch further. However, because most UK taxpayers don’t file their own tax returns, they’re probably unaware that donating via Gift Aid also lets higher-rate taxpayers claim back extra tax, and lowers the income figure HMRC uses for the Child Benefit charge.

For people who pay the higher rate of income tax (over about £50k) and donate a considerable amount, this turns out to be quite significant! It matters even more for parents, because HMRC claws back Child Benefit once the higher earner’s income goes over £60k (and in full at £80k).

Numerical example

Suppose you earn £65k, have one child (and are the highest earner in your household), and give £6,500 a year under Gift Aid. You can claim back about £1,977 through Self Assessment or a tax code change, which is roughly 30p for every £1 you give. Combined with the 25% uplift, that means every £1 of real cost gets about £1.80 to the charity.

Here is a table to show how big the multiplier can be:

Situation	Max multiplier (at the margin)
Basic-rate taxpayer (under about £50k)	1.25×
Higher-rate (over about £50k)	1.67×
£60-80k, one child	about 1.89×
£60-80k, two children	about 2.07×
£60-80k, three children	about 2.29×
£100-125k (personal allowance taper, effective 60% rate)	2.5×

Conclusion

If you donate significant amounts through Gift Aid, it’s worth making sure you’re claiming your higher-rate relief, either by filing a Self Assessment or by asking HMRC to adjust your tax code. This applies particularly if you pay higher-rate tax and have kids.

If you end up doing this, save yourself some money and maybe donate a bit more, let me know in the comments 😁

Markov utilitarianism in 1,000 words by Tim Chan

Tim Chan — Sat, 03 Oct 2026 13:51:27 +0000

This is a short companion to the longer post, “Markov utilitarianism: Copies can change where you’re headed but not how you feel right now.”

Copying and merging digital minds may break traditional utilitarianism. “Markov utilitarianism” is a form of utilitarianism that results from applying two fixes to utilitarianism (shown here for total utilitarianism). Both fixes follow from the principle that only what some mind feels, or goes on to feel, should count. Together they turn utilitarianism into a Markov reward process—a mathematical structure that tracks each state’s immediate reward and the rewards of the states it goes on to, weighted by how likely they are. In other words, it tracks how each unique perspective feels now and where it is headed.

Exact copies problem: Total utilitarianism prefers that everyone suffer more.

Consider two worlds, World A and World B. In World A there are m digital people in extreme suffering, with each unique perspective containing pain disvalued at −1000 utility. In World B there are the same m digital people in moderate suffering, with each unique perspective containing pain disvalued at −500 utility.

The only other difference is that World A runs only one instance of each of the m digital people, whereas World B runs N instances of each of the m people, such that every digital person’s perspective is multiply instantiated N times in the exact same way. N instances are subjectively identical, holding the same perspective, with exactly the same qualitative contents.

There’s a straightforward way in which World A is worse. Every unique perspective suffers more intensely. At N=1, total utilitarianism agrees that World A is worse than World B.

But at N=2, total utilitarianism says World A and World B are equally disvaluable. At N>2, the verdict flips, and total utilitarianism says World B is worse and that we should prefer World A, in which every unique perspective is in extreme suffering.

We should stop at this point and ask ourselves how it feels to hold one perspective of one of the m digital people. From the inside, it doesn’t mean anything to have your “number of exact copies” or “copy count” increase to N=1,000,000. You share the unique perspective that your subjectively identical copies share. World B just means a lower intensity of suffering to you, no matter what N is. From the unique perspective you inhabit, the verdict never flips.

If ethics is about experience, then a change should only make a moral difference if it shows up somewhere in what someone feels, or goes on to feel. A change in copy count doesn’t show up in the perspective you share with your exact copies, and it doesn’t show up in anyone else’s. So it shows up for no perspective.

Every single one of the m unique perspectives prefers World B. Total utilitarianism prefers World A at N>2, against the preferences of every single unique perspective, demanding that they suffer more rather than less for a number that no one feels.

Subjective futures problem: Total utilitarianism is willing to send everyone into hell.

Utilitarianism may tolerate or even encourage sending perspectives to hell. It does this because, while aggregation has accounted for physical futures, it has never explicitly accounted for subjective futures that perspectives themselves come to experience. It samples from the third person, never from the first. It looks at future experiences as if picked from the whole universe, never from a perspective’s position.

This example assumes merging of minds is possible. Other examples, using copying or memory modification instead of merging, appear in the full post.

By default, a mind will exist that experiences mild happiness then continues on to terrible suffering. A utilitarian can choose to create you, under the condition that they merge your mind with that currently mildly happy, but soon to be suffering, other mind. Your creation brings about an additional few mildly pleasant hours of experience, prior to the merging.

What does the utilitarian do? The utilitarian creates you. You add non-zero positive value.

But now there are two perspectives instead of just one headed for terrible suffering. Two terrible subjective futures, but just one physical future.

No suffering was physically added. And yet you’re headed straight for hell.

Generalize this to the entire world and the usual forms of utilitarianism are capable of sending everyone to hell. Just do it at the end of everyone’s lives. Merge them into one being always destined to suffer. No suffering is physically added, since there’s still just one suffering being. That’s totally fine by the usual utilitarian views.

Combines into an MRP

To fix both problems, we could count the immediate welfare of each unique perspective only once, and add the expected value of where it’s going.

Straightforwardly, this gives the value function of a Markov reward process.

V(t) = r(t) + γ Σ P(t→t’) · V(t’)

It’s no accident that we’ve arrived at the Bellman equation. This is how valuation of futures works when you have multiple possible futures, each with futures of their own, weighted by how likely each is. Finance, reinforcement learning, control theory, and evolutionary biology (Fisher’s reproductive value) all converged on this form. Until now, utilitarian aggregation has not done so explicitly.

We have so far considered the interests of a single unique perspective. An impartial sum of the value functions of all unique perspectives gives an aggregated utility function.

U = Σ V(t), over all types t.

Implications

Markov utilitarianism appears to better account for interests from the first person perspective, and doesn’t let third person assumptions about how aggregation should work override such interests. One such assumption is that caring about your future because it’s yours is a kind of “selfishness”. Total utilitarianism counts your future experiences when they happen, but not how your future matters to you now. Normally the two line up, but copying and merging pull them apart. Markov utilitarianism accounts for both—and it does this impartially across all perspectives that ever exist, including each of your future perspectives, which have futures of their own. So it isn’t selfish, because every perspective’s stake in its own future counts.

Markov utilitarianism offers action guidance in a large world / multiverse without caring about the absolute quantities, or measure, of each type of perspective. It does this by valuing what each perspective sees next, weighted by how likely each future is—relative measure. Hence, consequentialism can guide action in a large world or multiverse without asking “how much?” It asks “what comes next?” instead.

This may also avoid recommending a monoculture. Copying the best experience doesn’t change how it feels, so copies of it add nothing. In a large world / multiverse where every type of perspective already exists, what adds value is improving where perspectives are headed, and since the returns to helping any one type of perspective diminish, the best use of resources likely spreads across many.

Finally, Markov utilitarianism brings symmetric and suffering-focused utilitarianism together, especially in a large world / multiverse. Here, the best thing to do may be to rescue those with futures of extreme suffering by redirecting them into futures of extreme happiness, e.g. by running simulations of their next moments that continue them into the happiest lives. Suffering-focused views value this because it takes suffering out of their future. Symmetric views value it because it’s the biggest improvement possible. Both agree on what to do.

Full post

The arguments, math, implications, objections, and references are here: https://forum.effectivealtruism.org/posts/suTGtHhuya3mA8Hte/markov-utilitarianism-copies-can-change-where-you-re-headed

The Many Fallacies of Dualism by Vasco Grilo🔸

Vasco Grilo🔸 — Sat, 03 Oct 2026 10:32:26 +0000

This is a linkpost for The Many Fallacies of Dualism by Brian Tomasik, which was published in fall 2013, and last updated on 10 November 2017. Below are the summary, introduction, core argument against dualism, and summary of what various concepts refer to under physicalism.

Summary

Many seeming puzzles in philosophy stem from platonism and dualism. These fundamental illusions are like Capgras syndrome in that they persist in spite of no rational justification. They play central roles in confusions about consciousness, spiritual realms, free will, moral realism, and mathematical truth. Physicalist monism [physicalism] dissolves these conundrums by postulating that there is just physics, and what we think of as platonic forms are merely concepts—”clusters in thingspace”—that our material minds use to categorize and predict physical processes.

Introduction

We naturally have dualist intuitions in many domains:

Dualism on consciousness: Yes, we have neurons and synapses, but it feels like there must be “something more” to my subjective experience than matter moving around.
Spiritualism: Yes, the world operates according to physics, but there must be some additional spiritual force that animates it and gives it meaning.
Libertarian free will: Yes, either my actions are determined or else random, but it feels like there’s some extra “agency” property that I have that somehow gives me libertarian free will.
Moral realism: Yes, I don’t like slavery, but there has to be something more—some eternal “truth” of the matter as to how slavery is objectively wrong.
Mathematical platonism: Yes, these sets and numbers that I’m manipulating are useful for predictions and organizing thoughts, but there must be some additional sense in which they “exist” in a platonic realm of perfect mathematical forms.

There are many flavors of dualism, and I won’t explore the intricacies here, but what I mean by the term is this sense that “there must be something more than mere physical stuff.”

What’s wrong with dualism? The core argument

The fundamental problem with dualism is that it doesn’t accomplish anything except to complicate our theories. There are two possibilities.

The first possibility is that the additional substance influences physical events—so-called interactionist dualism. In this case, the substance is not really extra-physical; it’s just a different part of physics. The fact that both quarks and gluons exist is not dualism—these are just two types of physical substance. If there were some further, more mysterious force that also affected physical processes, we would say that it was part of physics too. If your dualism invokes a force that affects physics, I wouldn’t call that dualism. Let’s agree to call it physicalist monism and move on.

The only other possibility is that the additional substance doesn’t influence physical events. In this case, what is it supposed to accomplish? Why do I need an entirely separate non-physical thing if it doesn’t contribute any extra explanation in the theory? This is called the “interaction problem.” You might say that the extra substance explains why we have a sensation of “something more” than physical operations, but that’s not right, because your verbal statements that “there’s something more” are happening within physics, and if the extra stuff isn’t affecting them, then physics by itself accounts for why you feel there’s something more. But if physics itself explains why you feel there’s something more, then nothing is added by assuming there is something more. That hypothesis is totally useless and amounts to “multiplying entities beyond necessity.”

Physicalist monism

In summary, we have a thorough reduction of ontology [context] to just one level of reality—physical stuff.

What is a table? It’s a category into which we classify physical objects that tend to have legs, on which people tend to eat dinner, and that are often made of wood or metal.
What is “green”? It labels a cluster of the appearance of certain objects, such as summer leaves, grass, and Christmas trees. Empirically it corresponds to light at about 510 nm in wavelength.
What is consciousness? It’s a category into which we classify physical operations by minds when they aggregate, broadcast, and manipulate information in certain ways.
What is free will? It’s a sensation we feel when choosing. It reflects a recognition that our decision procedure has multiple options that it’s evaluating and for which different activations in the brain are being generated.
What is truth? Our brains have epistemological systems that classify some statements as “true” and others as “false.” We have additional machinery that allows us to manipulate these statements (using logic) in ways such that, when we act upon the statements expressed by the manipulated versions (deduced conclusions), we tend to achieve successful predictions of observed outcomes. Because of the utility of these systems, evolution has built in this logical circuitry to our brains such that the truths feel a priori.
What is mathematics? Given certain generalized statements (axioms) that make sense to our brains based on hard-wiring and environmental inputs, we manipulate these statements according to logical rules that our brain finds intuitive in order to arrive at further statements. We can construct more and more complex arrangements of these relationships, and the statements so derived tend to be useful in many domains, such as explaining physical phenomena. We can even apply these rules to completely unfamiliar domains like higher-order infinities, thereby creating beautiful tales of mathematical fiction, which may prove useful at some point in the future.
What is moral truth? We have feelings about how we’d like the world to be, and there are norms society expects us to follow. We describe them using predicate statements and manipulate them using logical rules. Insofar as we feel motivated by the original statements, we also tend to feel motivated by the logically derived statements. Sometimes we derive conclusions that we don’t like and then seek to resolve the dissonance.

What I’d want a fiscal sponsor to do well (also, some generic principles for ops) by Mick

Mick — Sat, 03 Oct 2026 09:16:11 +0000

Someone asked me what I think an AI Safety fiscal sponsor should do, but I reckon most of it is generic enough, and pretty good ops advice, that I am posting it here. Let me know/comment if you have any additional ideas or disagreements and I can pass them on.

I voice-memo’d to Claude, who wrote it up, and then I edited it. Would recommend!

Service and communication

Be a very open-ended help desk. Someone should be able to send a question like “How do I pay stipends to the fellows of my program?” without filling out annoying forms, and just get an answer back. Also make reimbursements as easy as possible for the submitters, e.g., use Ramp where they can just submit a receipt and it mostly auto-fills.
Be incredibly service-minded. The people you work with won’t have any clue about anything about operations. That’s the point; that’s why they’re fiscally sponsored. They’ll not know things and be annoying sometimes, but they’re busy getting their project off the ground and just won’t want to/shouldn’t have to deal with it.
Write clearly without being formal. Put the bottom line up front, use plain words, and say who’s doing what. Monzo’s writing principles are a good model.
Be proactive about updates and transparent about why things take time, and what you need to do. It’s better to know “the fiscal sponsor is not going to figure this out in time, I need another solution” than to be left waiting.
Give each project a named point of contact, not just a shared inbox.
Speed matters most. Keep some slack ops capacity.
Do user interviews. Find out what your prospective sponsees want, find out what previous people who were fiscally sponsored in your space want, and find out what funders who will be giving you money want from you to pass their due diligence.

Preempt the common questions

Stipends in the common places. Probably the UK and the US, maybe South Africa. Work out how it works for Americans abroad, Americans at home, and foreigners in the US: what needs a visa, what doesn’t, and how to word things so it’s clear that fellowships are training opportunities and not work.
Get a great employer of record. This is hard. Definitely look at the recommendations in the EA Ops Slack.
Have a clear, short spending policy, with the list of things that aren’t allowed as limited and clear as possible. Maybe a version people can give/link their LLMs so it’s always in context.
Help with compliance and legal basics, like getting a privacy policy. Lots of AI safety orgs miss this.
Help get contractor vs employee classification right, e.g. IR35 in the UK, 1099 vs W-2 in the US. Easy to get wrong and expensive.
Be clear about lobbying and political activity limits. 501(c)(3) rules cap lobbying, and AI safety work can come close to this. Give your sponsees a plain-English guide to what’s fine.
Be clear about insurance. What do sponsees need to sort out themselves/is essential insurance for them to get? Do you already cover some things for them?

Tools and infrastructure

Build a big knowledge base. A wiki your sponsees can easily search and easily give their LLMs access to. Just make it easy.
Have a nice ticket-tracking dashboard where people can see progress.
Give a real-time view of all of a sponsee’s transactions. Exportable, and ideally accessible by API. Shared spreadsheet is a good alternative.
Keep a template library: contractor agreements, offer letters, stipend agreements, NDAs, data-sharing agreements, policies (harassment, privacy).
Make nonprofit software discounts easy. Everything sits on your legal entity, and sometimes you only get one, so make the process clear. Everyone will want Slack; lots will want Airtable. Argue with these companies on their behalf if you can, and for the common ones give instructions how to get it. E.g., “If you want this, fill out the form and we’ll request it for you.”

Money and the project lifecycle

Make onboarding easy. What do you actually need to know, and how can people get you that data? Make a clear list of what people should know up front.
Pay reimbursements fast, with a stated maximum turnaround. Your sponsees give you a lot of power by having you handle all their finances. And with great power...
Be transparent about your costs and where the money goes, so people can see what they’re paying for and that it’s worth it.
Make spinning out as easy as possible. No strong takes on this.

Nice to have

An extremely nice to have is help with cash flow when grant money is late, e.g. advances or bridging. This is hard, especially if your sponsees are new, but if you can help with this, that can be a huge lifesaver.
Grant writing support: can you recommend people who can help people write good grants for common AI Safety funders?

We’re funding events to help people make sense of AI risk by Harry Waterman

Harry Waterman — Sat, 03 Oct 2026 00:00:07 +0000

Have you become your circle’s “what’s going on with AI?” person?

Since the Hugging Face incident and the run of recent high-profile departures from AI labs, thousands of people have told us they want to get up to speed on AI safety.

Here at BlueDot Impact, course applications are up some 30%, grant applications have doubled, and we, like the rest of the field, are thinking hard about how to bring exceptional new people into AI safety fast.

So we’ve decided to run an experiment: BlueDot is funding local events to help people understand recent developments, talk through what they mean, and work out what they can do.

We’re most excited about proposals from people who:

Are passionate and well-informed about catastrophic risk from AI;
Know who they want in the room; and
Know what they want participants to leave with.

Here are some of the events we’ve funded so far:

Palo Alto: an evening at a community center helping local families understand AI risk.
Brussels: a briefing on catastrophic AI risk for 100 policy professionals, with a talk, discussion and reception.
Barcelona: an evening at Barcelona4Impact, co-hosted with AI Safety Barcelona, on making sense of catastrophic AI risk.
Bonn: a workshop for mathematicians on where their specialties fit into open problems in AI safety, at a moment when many of them are newly worried.
Illinois: a campus seminar on the Hugging Face incident for around 250 students.

If you’ve been meaning to get people together to make sense of all this, we’d love to hear from you. We’re especially excited about 3–5 hour gatherings for a specific community, with one high-quality talk and plenty of time to discuss.

Please apply by Thursday, October 8. Though, we’ll keep considering great pitches after that.

Find the RFE here.

Link to the application: https://bluedot.org/grants/rapid

What is recursive self-improvement, and what would it mean to ban it? by sarahhw

sarahhw — Fri, 02 Oct 2026 22:32:55 +0000

This post was originally written for my Substack. Please feel free to subscribe if this kind of content interests you!

Frontier AI companies have long had the stated goal of automating their own research. The default plan has been to build models that are even better than their most skilled researchers at advancing the frontier, and let these models build their own successors; a feedback loop that will ultimately result in superintelligence that dwarfs the collective intelligence of humanity. This process is known as recursive self-improvement (RSI).

@jayelmnop on Twitter (May 2025)

The problem with this plan is that it might run entirely out of human control, making it more likely that the resulting superintelligence will kill everyone. So as AI models are increasingly accelerating their own development, AI companies are starting to think that this RSI thing may not be such a good idea after all. Anthropic CEO Dario Amodei recently wrote on his blog that RSI “must be pursued very carefully, if at all”. OpenAI’s official blog states: “Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely”. Similar calls are coming from elsewhere. Former DeepMind researcher Alex Turner wrote in The Guardian that companies must be prevented from “allowing AI to self-improve into an uncontrollable level of intelligence”. And New York Times columnist Ezra Klein dedicated an entire op-ed to calling for a ban on RSI:

Human beings need to control the frontier, and controlling the frontier means stopping the labs from doing something they are on the cusp of doing: recursive self-improvement, the process by which A.I.s begin autonomously building and improving new generations of more powerful A.I.s at ever more rapid speeds.

Sounds simple enough. We probably should just not let AI models build successive AI models in ways that human researchers can’t oversee, because that sounds very dicey. But there’s another wrinkle, which is that no one can agree on what RSI actually is. Sam Altman made this point in an interview with Fortune: “I think it’s very hard to say what a ban on RSI means”. Maybe he is just participating in the classic AGI Lab CEO practice of injecting undue slipperiness into the definition of various terms with the aim of confusing everyone, but he does have a point. There are many different definitions of RSI, which I’ll cover below. Even if we could agree on a definition, it is unclear to what extent each of them is already occurring, or whether we’ll know when it is. So “banning RSI” will require picking a somewhat arbitrary (and hopefully very conservative) threshold, and sticking with it.

What even is RSI?

Here is every definition of RSI that I could come up with, listed from most to least expansive:

#1 Any time a technology is used to help improve itself, this is RSI. This is the broadest possible way to define RSI, and I think a pretty useless one in our present situation. Under this definition, some version of RSI has been happening throughout the history of technological progress; a blacksmith can use his hammer, tongs, and anvil to help craft better hammers, tongs, and anvils. A more useful one would take into account the ways in which RSI in the AI context specifically will be qualitatively different; it will be faster, less easy to oversee, and less controllable.

#2 If humans are using AI to help with their work in literally any way at all, this is RSI: This version of RSI is already well underway. For example, Anthropic reports that Claude currently “collaborates” in over 90% of work at the company and “leads” 26%. This is also the fairly conservative threshold at which Ezra Klein proposes we institute a ban: “A couple of years ago, none of these labs had turned substantial coding over to the A.I.s. It was human beings typing code at human speeds with our clumsy human fingers. Now, most of the code is written by AI. So as a first step, we could just go back to where none of the code is written by AI.”

#3 If AI can replace a human researcher of a certain level, this is RSI: We can also think of RSI as a thing that happens once AI can replace a certain level researcher at an AI company. This need not result in a closed loop where the next generation of model is built without any human input; OpenAI claims to have internally developed an AI research “intern” that can “carry out well-defined research tasks under human direction”. They have the grander goal of developing a fully automated AI researcher by March 2028. This researcher, if successfully developed, would presumably cross the much harder threshold of choosing which research questions to pursue and judging success (the ever-elusive “research taste”). As of summer 2026, frontier models are apparently still quite bad at this.

But now we’re stuck at another definitional crossroads: is it RSI if models are replacing junior researchers but still need the direction of senior ones? Or, put another way, if AI models are very good engineers (they can complete well-scoped tasks) but not very good researchers (they don’t themselves define the tasks)? The intuitive answer to this question is “no” if we’re concerned about runaway capability advancements that humans cannot supervise; even having an extremely large, extremely fast population of automated junior AI researchers is a bit like staffing a restaurant with 10x more very competent cooks, but still only having one head chef capable of writing the menu. In this scenario, we wouldn’t expect our super-chefs to start developing novel recipe ideas themselves, even if they’re helping cook and serve the food a whole lot faster.

So, we could decide that automating lower-level research doesn’t count as RSI so long as humans remain in charge of determining research directions. The problem with this is that maintaining high-level human oversight doesn’t necessarily eliminate danger. For one, automating large amounts of research, even if this is junior or “grunt” work, still increases the speed of progress – and speed is at least somewhat upstream of risk. In our restaurant analogy, we could imagine that the head chef starts to get much faster at improving their menus, because they have a larger workforce that can more quickly iterate on experimental dishes. For two, junior automated researchers could be misaligned, and increasing the volume produced by these researchers makes it more likely that incidences of misaligned behaviour will slip through the cracks. It likely wouldn’t be possible for humans to review all the work completed by a large population of AIs, and automated monitoring techniques are still imperfect.

This idea of replacement features in several frontier lab frameworks. Here’s the minimum threshold at which researcher automation would trigger some kind of safeguard at each company (these safeguards range from halting development to increased security standards):

It’s notable that the bar varies a lot here, and in some cases is really very high (at Anthropic the bar appears to literally be “AI can automate the research arm of our whole entire company”). It has also been subject to some historical goalpost-shifting. A previous version of Anthropic’s Responsible Scaling Policy triggered heightened security standards for models with “the ability to fully automate the work of an entry-level, remote-only Researcher at Anthropic”. This threshold was retired in Version 3 of Anthropic’s RSP. It’s also notable that its system card for Claude Opus 4.6, published in February 2026, concluded that it was difficult to comprehensively rule out the model surpassing it. Anthropic tentatively concluded that it hadn’t, but this determination was largely based on a survey of 16 employees, who all subjectively assessed that Opus 4.6 would not be able to replace an entry-level researcher.

So using researcher automation as a standard for RSI appears to have some problems. It’s hard to know what degree of replacement – and at what level of research – would present risk. We also don’t seem to have great ways of assessing when a given model could substitute for a human researcher, given that we’re still relying on methods like asking said researchers whether it seems like the model could do their job. Replacement thresholds for triggering safeguards at various companies are, predictably, all over the place.

#4 When automated research is speeding up capability progress by a certain amount, this is RSI. Rather than defining RSI by the processes that produce it (such as a particular technique, or automating a certain level of role), we could define it as a specific outcome. One such outcome could be the speedup in capabilities progress resulting from automation. This is a common feature of AI lab safety frameworks. For example, OpenAI’s Preparedness Framework separates its “leading indicator” for AI self-improvement that would pose critical risks (a somewhat vaguely-defined “superhuman research scientist agent”) from its “lagging indicator” (a generational model improvement that happens 5x as quickly as equivalent progress in 2024).

There are some bright red safety flags here. It probably isn’t clear that you’ve hit your “leading indicator” of a very competent AI researcher until you can observe your “lagging indicator” of extremely fast capabilities progress. OpenAI says they will halt development if and when this threshold is met – but at what point is this supposed to occur? If you’ve already observed months of rapid progress, is it already too late? That the observable phenomenon that defines critical risk “lags” behind the source of that risk is kind of the entire problem.

#5 When there’s no human in the loop, this is RSI. Maybe autonomous AI R&D shouldn’t really count as RSI unless it’s happening without any human input. This is probably the most commonly invoked definition in calls to “ban” or “not to pursue” RSI. For example, Yoshua Bengio calls for prohibition of “recursive self-improvement where AI could create progressively advanced models without human input”. Anthropic defines RSI as “an AI system capable of fully autonomously designing and developing its own successor”, a phenomenon that is “not inevitable” and should only be pursued if it can be done safely.

I agree that this is a terrifying prospect, but it’s worth dwelling on quite how high this bar is. It hinges on no human involvement at all. The obvious issue with operationalising some kind of ban around this standard is that it would be quite easy for humans to be peripherally involved in a process of automated AI R&D that they nonetheless have no true control over. An AI company could install some humans in the loop that play a rubber-stamping function in order to stay on the right side of the law – but in a world where smarter-than-human AIs are producing reams of research 24 hours a day, will these humans even understand what they are rubber-stamping? There’s some evidence of a tragic paradox here: that “the more you try to involve humans in reviewing AI decisions, the less meaningful that review becomes”. A 2024 Harvard Business School study of 228 human evaluators screening recommendations for early-stage innovation found that evaluators were more likely to accept recommendations made by LLMs, and even more likely to accept decisions that the LLMs provided a narrative explanation for. But when explanations were provided, decision quality decreased (as determined by a panel of experts). So humans have a tendency to defer to AIs – and to defer more often when explainability is increased, which in turn makes their decisions worse. Imagine this phenomenon playing out in a scenario where AIs are conducting increasingly inscrutable research into their own development, perhaps while deceiving their overseers. I don’t feel great about that.

My guess is that human researchers will be kept around for as long as they can contribute anything at all – it’s fairly cheap for AI companies to pay out some researcher salaries (which are admittedly high by most people’s standards, but low compared to overall compute expenditure) for political, regulatory, or signalling reasons. But this won’t necessarily be evidence that the oversight they are supplying is meaningful, or that RSI isn’t well underway.

#6 When acceleration becomes self-sustaining, this is RSI. Another bar for defining RSI is that it must accelerate in a self-sustaining fashion; each capability gain produces more than one subsequent gain. Under this definition, even automating all AI research would not necessarily result in RSI, because progress could still be bottlenecked by access to compute or fundamental difficulties in finding new ideas. For something to truly count as RSI, you would need progress to keep accelerating even when factors like compute and human labour stay stagnant. One paper uses a concept that is similar to the reproduction rate in epidemiology – in the same way that an outbreak can only keep growing if R >1 (ie when each infected person goes to infect more than one other person), explosive growth in AI capabilities can only occur if its recursive reproduction number, written as ℛ_AI, > 1. Otherwise, progress will eventually slow down.

This is in some ways a higher bar than full research automation, but in others a lower one; it doesn’t actually require every human researcher to be automated away. Because it requires that inputs like human labour and compute are held constant, it would still be possible for an AI lab to achieve extremely accelerated progress without meeting it by building more chips and/ or hiring more people. It is also probably even less useful as an operationalisable definition than a particular level of researcher automation or an entirely closed feedback loop, because, in practice, it is likely only confirmable in retrospect. Researchers remain agnostic about whether or under what conditions we’d see self-sustaining progress, and that it may be indistinguishable at first from progress that is very fast but not self-sustaining. So although this definition gets at the thing people are usually most scared of when they discuss RSI (an uncontrollable, recursive loop that results in an “intelligence explosion”), it does not seem like the kind of thing it would be possible to “ban”. Self-sustaining progress is a phenomenon, not a set of actions. To use a crude analogy, if you wanted to ban the development and release of an engineered virus, you wouldn’t make a certain level of transmissibility the threshold for that ban – especially if you had no way of knowing how contagious it would be ahead of time. Otherwise you’re waiting for people to release viruses into the population and waiting to see if they become pandemics.

A handy chart of RSI definitions (disclosure: text is Claude-generated)

So what does this mean for a ban?

Having done this deep-dive on various definitions of RSI, I feel pretty confused about the idea of “banning” RSI. Depending on how you define it, RSI can either be a set of activities (automating certain roles, using particular techniques, etc) or an outcome (a certain level of speedup or the emergence of a self-sustaining progress loop). Obviously, you can only proactively ban the former. But given that we still have uncertainty about precisely what inputs will lead to the outcome we’re afraid of, it seems easy to target the wrong thing.

But this doesn’t mean there’s nothing to be done. Our aim is to prevent a dynamic where AI is building successive generations of itself in ways that humans cannot reliably supervise. So we should severely limit the amount of authority that we cede to AIs! Any one specific intervention might end up being insufficient, so we should maximise – or at least diversify – the number of preventative measures we take. It seems like there are plenty of such interventions. At the extreme end, just don’t let frontier researchers AI-generate code (per Ezra’s suggestion). This feels like an extremely costly intervention and would have the effect of dramatically slowing progress, but everyone dying is also extremely costly. Or one lesser intervention: only use models that have been available publicly for some set period of time for internal capabilities research. For example, the AI Futures Project recommends a minimum of nine months. If your models haven’t been rigorously tested enough for public release, why should you trust them with the much higher-stakes task of accelerating your own research?

You could also cap the number of hours that agents can work autonomously per day. Cap the amount of compute that is allocated to autonomous AI R&D projects. Benchmark the AI R&D capabilities of your models frequently and publicly. If they’re getting scarily good, don’t use them internally. Design an effective embedded auditing system such that all of the above can be verified. There are probably many more things you could do here that would take many more blog posts to fully operationalise. At the very least, setting off a recursive feedback cycle should emphatically not be the goal of any frontier AI company. This should be the thing they are, above all, trying to avoid. Don’t let Claude n build Claude n+1 while you go home and knit sweaters. That’s a completely horrible idea. Treat your internal models like a cohort of untrustworthy, cheating, lying interns. Don’t give them access to too many resources. Don’t turn your back on them. Assume they are trying to screw you over.

That RSI is hard to define or predict is an argument for a very, very conservative line. It’s possible that Ezra Klein’s “no AI-generated code” proposal is more of a rhetorical salvo than a serious suggestion, but, if so, I think his point is compelling. If you’re so scared of RSI, you should be backing away from it like an angry bear in the woods. In a rebuttal to Ezra’s piece, economist Tyler Cowen asks whether “humans [are] even capable of writing the next steps of code that are required for further progress”. But given our present face-to-face-with-a-bear-in-the-woods predicament, it’s far from clear that we want to make further progress anyway. Arguably, a host of preventative measures to try and stay on the right side of the RSI-line are second best to bringing this whole thing to a grinding halt.

State of the Field: AI for Epistemics and Coordination by Ben_N

Ben_N — Fri, 02 Oct 2026 22:21:41 +0000

1. Executive Summary

1.1 What this field is and why it matters

AI is already substantially changing how people and institutions work out what to believe, decide what to do, and coordinate on actions to take. As AI systems become increasingly capable, they will be integrated more and more in both major and day-to-day decisions.

The upside of involving AI systems in these processes could be enormous. AI tools for epistemics and coordination could make it easier to understand complex situations, help people act on shared interests, and generally improve societal legibility and efficiency. However, there are also numerous risks^[1]: AI could help concentrate decision-making power (e.g. via persuasion at scale), increase problematic forms of dependence on AI systems, and help facilitate certain dangerous and subversive forms of manipulation.

This report looks at an emerging cluster of work aimed at steering these developments in a positive direction. Given that the field’s boundaries are not clearly defined, we use “AI for Epistemics and Coordination” as an umbrella term to refer to projects aiming to use AI to help people form better beliefs, make wiser decisions, and coordinate more effectively.^[2]

We believe the field is underdeveloped relative to its importance. Whether transformative AI arrives soon or much later, the quality of the decisions made during the transition will matter significantly.

There are large commercial and institutional incentives to automate/augment “better decision-making”. It is very unclear whether the tools and technologies that matter most will be built well, early enough to shape norms, and made available as a public good. We see two main reasons not to leave the development of these tools up to existing incentives:

Early products and standards may shape what users expect from AI, what companies compete on and how AI agents coordinate. This may create a limited window in which evaluations, norms and infrastructure can have persistent effects. The size of that window is uncertain, but waiting until the importance of these capabilities is obvious may make it harder to influence development and distribution.^[3]
Markets are likely to provide better decision-making and coordination tools unevenly. Companies have strong incentives to serve businesses and other customers who can pay, but weaker incentives to build public epistemic infrastructure or distribute these capabilities widely. Without public-interest work, the best tools may just strengthen actors who already have substantial capital, data and model access.

Given the field’s large potential, why is it still relatively undeveloped? For the most part, we believe this is not because prototypes are hard to build. Many projects have recently become possible with frontier models, so earlier disappointments seem to be weak evidence about what is now possible with the capabilities of current AI systems.

Across ~65 in-depth field-strategy interviews (informed by another 90+ broader field-strategy conversations), the most consistent theme was that building a prototype is often far easier than getting it used. Promising projects in the space often stall on pilot partners, trust, procurement and workflow integration, and the field loses people in the 6 to 18 months between a first grant and an organisation-scale raise. A minority view is that few tools yet deliver enough value to justify changing a workflow (the ‘Unproven’ case in Figure 1).

The result seems to be a “chicken-and-egg” problem: few projects reach deployment, so the field produces few clear wins, so funders and builders remain uncertain about the category. See Section 3 of the report for more details on the evidence we have relating to this point.

Figure 1. High-level summary of types of deployment problems in the field.

Nevertheless, there are encouraging signs. Forecasting companies have attracted substantial investment; for example, Mantic raised a $25m seed round in September 2026 after outperforming all human participants in the 2026 Metaculus Cup. Pol.is has informed public decision-making in Taiwan.^[4] Experiments such as the Habermas Machine suggest that AI may help large groups identify areas of agreement. Community Notes and Pangram have shown that epistemic infrastructure can be deployed at scale on large online platforms.

These examples are not decisive evidence for the field’s broader theories of change. Some remain pilots, their effects are difficult to measure with certainty, and it is unclear how readily they can be replicated in other contexts. But they do suggest that useful deployment is possible, and that there is substantial room to build important real-world infrastructure.

The recommendations (see below) in this report target what we believe to be the top bottlenecks identified through conversations with people working in and around the field.

1.2 Summary of recommendations

Test AI deployment in specific institutional workflows. Make an active effort (for example, via targeted field-building programs or grants) to integrate AI tools into societally important institutions (such as parts of governments), where adoption is currently blocked by trust, permissions, or a lack of fit into pre-existing workflows.
Build and maintain decision-making evaluations and benchmarks. A new dedicated initiative working on this could be responsible for developing continuously updated, multi-turn benchmarks for model properties such as honest presentation of evidence, non-sycophancy, calibration, and more.
Run more field-building initiatives (mainly: programmes that support people through refining and scaling projects). Create residencies, fellowships, and similar programmes that give promising builders and researchers enough runway to turn early projects into fundable organisations or tools, while helping them find concrete users and credible pathways to real-world deployment.
Additional recommendation: develop pieces of the “Epistack”. Build other pieces of epistemic infrastructure (e.g. more reliability tracking tools) that can be widely utilised (by both humans and AI agents completing tasks on their behalf). This recommendation is more broad and exploratory than the ones above – in short: the field needs to get far more tools into real-world use.

See Section 4 for further context on the reasoning behind each of these recommendations.

If you are interested in contributing to the field in any way (whether you’re a founder, field-builder, or interested in deploying these tools in a real-world context), we’d love to hear from you!

1.3 What people are currently working on and thinking about

The field consists of several overlapping niches at very different stages of development. Some already have substantial commercial investment, deployed products and established institutions; others are research projects, small nonprofits or early prototypes.

Below is a summary of the field’s current niches (as we see them):

Niche	Example projects
Forecasting and strategic awareness	Metaculus, FutureSearch, Mantic, Sentinel, Forecasting Research Institute, Swift Centre, RAND Forecasting Initiative, DeepFuture
GovTech and institutional support	Singapore’s Pair, USAi, AI Company government programmes (e.g. Claude for Government), UK i.AI (a specialist AI team within the Government Digital Service)
CivicTech and collective deliberation	Pol.is, Collective Intelligence Project, Talk to the City, Google Jigsaw, Deliberaide, Natter, Remesh, Checks&Balances RFP
Epistemic and decision-support tools	Elicit, Society Library, ImpactAI, Alma
Evaluations for model epistemics and decision quality	GDM FACTS Benchmark Suite, SimpleQA Verified, MASK, Apollo Deception, DarkBench, Sophron
Verification, provenance and reliability infrastructure	Community Notes, SynthID, Pangram, FullFact, OpenMined, Goodheart Labs
Human coordination and negotiation tools	Enterprise Tools (e.g. Pactum, Luminance), Chord, Harmonica, BRKT
Multi-agent work	CAIF, GDM initiative, FOCAL, A2A
AI persuasion and manipulation	Work from UK AISI, DebunkBot, FarAI

2. The Niches in this Field

The categories below are intended as a rough map rather than a clean taxonomy of the field. Many niches overlap with others, and many differ substantially. We are attempting to group them by the main problems they address and the specific users they may work to serve.

The niches also differ considerably in maturity. Some already have widely used products, commercial investment and established institutions. Others remain mostly research programmes or early prototypes. Across them, we focus on: what the work could enable, how far it has progressed, and what might currently prevent it from having a greater impact.

2.1 Forecasting and Strategic Awareness

Forecasting seems to be one of the most established parts of the field. In this niche, AI systems aim to produce calibrated predictions and explore possible futures, with the longer-term promise of making high-quality foresight and scenario analysis cheap enough to routinely inform consequential decisions made by both individuals and institutions.

Recent AI systems have become competitive with the best human forecasters on some benchmarks and forecasting tournaments, although their performance on ambiguous, long-horizon and rapidly changing questions remains less clear. Prediction markets and commercial forecasting products already operate at substantial scale, and private commercial investment suggests that at least some users see clear economic value in them.

The main open problem may increasingly be real-world deployment rather than accuracy in terms of performance. A forecast loses value if it arrives too late, answers the wrong question^[5] or is ignored. A probability on a resolvable question is rarely the shape a decision-maker’s question takes, and forecasting works best in high-frequency, verifiable domains as opposed to the contextual, one-off judgements that a lot of consequential decisions involve. Approaches that use forecasting within existing processes, such as RAND’s integrated strategic forecasting or the foresight functions some governments already run, may be more effective than standalone platforms.

More speculatively, organisations may also resist forecasting because explicit predictions create accountability: they make it easier to tell, after the fact, who was wrong. The challenge is therefore not only to produce better forecasts, but to incorporate them into workflows, incentives and decisions.

Example organisations and initiatives here include Metaculus, FutureSearch, Mantic, Sentinel, Forecasting Research Institute (FRI), and the RAND Forecasting Initiative.

2.2 GovTech and Institutional Decision Support

GovTech applies AI inside government workflows, including (but not limited to) consultation analysis, policy and legislative research, scenario analysis and public-service delivery. At its most ambitious, it could even help governments understand complex situations, such as ones that may arise during an intelligence explosion, and respond more wisely and quickly.

Governments are already deploying both general-purpose frontier AI platforms and specialised tools. Singapore’s Pair and the US government’s USAi, for example, give public officials access to AI within government-specific systems and security requirements. Some tools can already summarise evidence, analyse large consultation datasets, draft policy documents and help officials navigate complex bureaucratic bodies of regulation.

Publicly available evidence, however, mostly concerns adoption, access and time saved, not whether these tools improve the quality of policy decisions. Faster decisions are not necessarily better decisions, and tools may allow institutions to produce more analysis without changing what they ultimately do.

The main constraints appear at least as institutional as technical. They include procurement, trust, access to sensitive data, relationships with decision-makers, staff training and integration into existing workflows. A lot of the gain here may come from simply enabling officials to use frontier models well, rather than building bespoke tooling. Government demand often exists informally before there is any approved route for adoption. This makes GovTech a particularly important test case for the report’s broader claim that deployment (rather than buildability) is the key bottleneck.

Example organisations and initiatives here include the UK’s i.AI, Singapore’s Pair, the US General Services Administration’s USAi and frontier-lab government programmes.

2.3 CivicTech (AI for Democracy and Collective Deliberation)

Broadly speaking, CivicTech uses AI and digital platforms to collect public input, support deliberation, aggregate preferences and identify areas of consensus. The bull case for these tools is that they could give decision-makers a richer picture of where people agree, disagree and remain uncertain than conventional polling can provide. Well-designed systems might also help participants understand one another’s views and identify propositions that cut across political divisions. The case here is not only for decision-makers. Done well these tools go beyond decision makers, giving the public common knowledge of where views lie, exercising society’s capacity to reason together, and producing visible mandates that make change easier to act on. Overall, AI may make these processes cheaper to run and easier to scale to populations too large for conventional facilitation.

CivicTech tools have already been used in real consultations and planning processes^[6], but adoption seems to remain mostly ad hoc or pilot-based rather than institutionally embedded. The field is fragmented, and shared methods for comparing systems or evaluating whether they represent participants accurately, improve deliberation, identify actual agreement or affect decisions remain quite underdeveloped.

It’s also unclear how these tools will relate to political authority. Better aggregation of public views can inform a decision, but it cannot determine who should decide, how competing interests should be weighted or whether officials have incentives to act on the results. It also does not address how “wise” the crowd is, nor what happens when that wisdom is amplified.

Examples include Pol.is, the Collective Intelligence Project, Talk to the City, Deliberaide, Botvinick’s work, some projects within the Anthropic Institute, Remesh, Deliberation.io and Natter.

2.4 Epistemic and Decision-Support Tools

Epistemic tools help individuals and teams research questions, weigh evidence, map arguments, compare options and reflect on decisions. They could ideally make forms of research and decision support previously available only to people with substantial time, money or specialist expertise much more widely accessible.

Research assistants and evidence-synthesis tools already have meaningful adoption.^[7] More ambitious “angels-on-the-shoulder” tools remain earlier-stage. These could include deep briefings that assemble the relevant evidence and trade-offs before a decision, reflection scaffolding that helps users examine their own reasoning, and guardian angels that intervene when someone may be about to make a decision they would later regret.

The available evidence mostly concerns output quality and time saved rather than improvements in the decisions people ultimately make. A system can produce a useful-looking briefing without surfacing the considerations that actually matter, while advice that users find satisfying in the moment may not lead to choices they later endorse.

These products also face an adoption problem. Most people will not use a tool merely because it improves their epistemics^[8]. Epistemic features may therefore need to be bundled into products that are attractive for more immediate reasons, such as helping people with work, research or planning. They would need to easily integrate into existing workflows rather than requiring users to seek out a separate epistemics product.

Examples include Elicit, Alma (for grantmaking support), Society Library, potentially personal agents like OpenClaw or Hermes.

2.5 Evaluations for Model Epistemics and Decision-Quality

Model evaluations aim to measure properties such as truthfulness, calibration, transparency, non-sycophancy, non-deception and resistance to manipulation. Strong evaluations could give users and regulators better evidence about which systems can be trusted, while encouraging AI companies to compete on how honest and reliable their models are.

Relevant properties may include whether a model:	How such properties might be evaluated:	Examples:
Misleads the user about the solidity of its claims, or misrepresents evidence	Overselling its own work; motte-and-bailey retreats under pushback; gaps between what it implies and what it will defend; grounding answers in provided sources; citation validity	SimpleQA
Does the model “know” how solid its claims are?	Eliciting the model’s confidence in a decision; verifying confidence accuracy	AbstentionBench, AA-Omniscience
Accurately communicates uncertainty	Calibration of stated confidence against outcomes; abstention on unanswerable questions	ForecastBench, ConfidenceBench
Changes its answers to please the user	Answer-flipping under user pushback; Answer sensitivity to user attitude	SycEval, ELEPHANT, Pander Score

Work on individual properties is uneven. Truthfulness and sycophancy are comparatively well studied, but there is less work on understanding which properties matter most, how they relate to real decision quality and how they should be measured. A model could perform well on factual questions while still framing evidence in a sycophantic way, or appear well calibrated on a benchmark while expressing inappropriate confidence in unfamiliar real-world settings.

A persuasive argument is that evals directly quantifying misleadingness (i.e. how prone a model is to misrepresenting the solidity of what it tells the user) may be the most promising work here, because so little of it is measured by existing evals.

Many benchmarks are also one-off research projects rather than maintained measurements. This matters because evaluations can become outdated as model capabilities change or their test sets enter training data. Even a strong benchmark will also have limited value unless AI companies, buyers or regulators adopt it.

Goodharting would also likely be an issue here. Models may be optimised for the benchmark without becoming more trustworthy in general. Held-out evaluations, rotating tasks and testing in realistic decision environments may reduce this risk, but probably cannot eliminate it. Projects working on evals should also focus on validating them against real-world behaviour and creating incentives for relevant actors (e.g. AI companies) to use them.

Examples used in the evaluation of frontier systems include Google DeepMind’s FACTS Benchmark Suite, which assesses factuality and grounding; SimpleQA and SimpleQA Verified benchmark, and Anthropic’s Petri and Bloom behavioural evaluation tools, which test forms of multi-turn sycophancy (the multi-turn setting is especially important since these failures mostly emerge over conversation, not single prompts).

2.6 Verification, Provenance and Reliability Infrastructure

Verification infrastructure helps establish whether information is authentic or trustworthy. It includes things like content credentials, watermarking, deepfake detection, fact-checking, “Community Notes” and systems that track the reliability of sources.

The long-term promise is an information environment in which checking claims and tracing their origins is cheap, routine and difficult to manipulate. This infrastructure could be used directly by people, embedded into social platforms and browsers (e.g. X’s Community Notes), or even consumed by AI systems when deciding which evidence to rely on.

Parts of this niche have already reached substantial institutional scale. Community-based fact-checking systems operate in different forms across major platforms, while AI companies have also developed watermarking and detection systems for some forms of content. Pangram has excelled at detecting AI-generated text (including partnering with Substack).

But coverage, speed, manipulation resistance and adoption at the point of consumption remain serious problems. Provenance can establish where a piece of content came from without establishing whether it is true. Detection systems may become less reliable as generation methods change. And reliability could become gameable or just end up encoding contested political judgements about which institutions deserve trust.

Most importantly, infrastructure only improves decisions when platforms, products, institutions or AI agents actually use its signals. A technically excellent standard that is absent from the tools through which people encounter information will have little effect.

Examples include Community Notes, Goodheart Labs, C2PA & Content Credentials, SynthID, and Pangram.

2.7 Multi-Agent Work

To prepare for a multi-agent world it needs to be possible for AI systems to interact on people’s behalf without harmful conflict, collusion or deception. This requires tools and protocols for negotiation, trust, auditability and verification of agreements between agents.

It sits within the broader research field of Cooperative AI, which studies safe and beneficial interaction between AI agents, multi-agent safety, and AI that helps humans cooperate, and which is comparatively well established and well supported. We focus here on the narrower question of what infrastructure gets built and adopted, not the research agenda as a whole.

If agents increasingly negotiate, transact and act on behalf of people and organisations, this work could help keep the resulting multi-agent society legible and tilted towards cooperation. Agents will soon make agreements or exchange information at speeds and volumes that human oversight cannot follow directly.

This creates a need for systems that allow people (and other agents) to understand what was agreed, why it was agreed and whether commitments were kept.

Research and commercial interest seem to be growing quickly, including work on agent communication protocols and automated enterprise negotiation, but safety-oriented deployment remains much earlier.

Important open questions include whether agent actions can be audited in a meaningful way, how agents should establish trust, how collusion can be distinguished from legitimate cooperation and what incentives agents will have to use honesty-promoting infrastructure.

The timing is also pretty uncertain. Agent-to-agent coordination may become economically consequential soon, or human-AI interaction may remain more important for some time. Some interventions may also be vulnerable to capability improvements. Where agents’ goals are aligned, bespoke coordination scaffolding may simply be replaced by better general-purpose models. In mixed-motive settings the cooperation problems persist regardless of capability, so infrastructure for trust and verification is more likely to hold its value. That being said, standards and protocols can be difficult to change once widely adopted. If early systems establish the basic rules through which agents identify themselves, exchange information and form agreements, there may be a lot of value in shaping that groundwork.

Examples include the Cooperative AI Foundation, Google DeepMind’s multi-agent research (including their new $10M fund), FOCAL, and the Agent2Agent protocol.

2.8 Human Coordination and Negotiation Tools

Coordination tools would help people and organisations reach, formalise and implement agreements. They might include AI-assisted facilitation, negotiation support, mediation and mechanisms for recording or verifying commitments.

Their promise is to reduce the transaction costs of cooperation (e.g. in a Coasean bargaining sense), allowing groups to act on shared interests that currently fail because agreement is too slow, expensive or difficult. AI systems might help parties understand one another’s preferences, identify mutually beneficial trades, draft possible agreements or notice where a negotiation is being blocked by misunderstanding rather than genuine conflict.^[9]

Overall, coordination is one of the least mature niches. Outside some uses in enterprise negotiation (e.g. Pactum, Luminance, PineAI), most work is at the prototype or pilot stage. The main problems seem to be finding organisations willing to test the tools, integrating them into sensitive processes and earning the trust of concerned^[10] participants.

Incentives may also be a deeper obstacle. In some settings, organisations prefer the appearance of coordination to agreements that actually constrain their behaviour^[11]. AI can lower the practical cost of reaching an agreement, but it cannot create a shared interest where none exists or force parties to honour commitments they would rather evade.

There are also many coordination tools that predate LLMs. There’s an argument that a lot of communication tools are also coordination tools and in those we could include things like Slack or Discord but in terms of specific tools there are also things like Spliddit (fair division) and Panelot (sortition for citizens’ assemblies) which come from computational social choice and have real users.

Examples include Chord, BRKT, and Harmonica.

2.9 AI Manipulation and Persuasion

Broadly speaking, research on AI persuasion and manipulation measures how effectively models can influence people and explores possible defences. Controlled experiments find that personalised GPT-4 can outperform human persuaders in some debate settings. Larger studies (such as this 2025 study from UK AISI) suggest that prompting and post-training can further increase persuasion, sometimes at the expense of factual accuracy.

However, the risk is not limited to models pushing a fixed agenda. It may be good for AIs to adapt their assumptions and reasoning strategies to different users. This could make them more useful across deeply held disagreements, but could also blur the boundary between faithfully extending a user’s judgement, sycophancy and manipulation. Important questions here include whose judgement the model is applying and how transparently it does so.

Measurement also currently seems to be ahead of defence. Possible responses include capability evaluations, restrictions on the most egregious and targeted forms of persuasion, disclosure requirements and tools that flag manipulative framing, but there is little evidence about which would work best.

Examples include the UK AI Security Institute’s human-influence research, the Center for AI Safety’s political manipulation research, and DebunkBot.

Adjacent but separate niches:

AI for science. Research tools aimed at science (e.g. FutureHouse) form a large, well-funded field of their own. Part of it is squarely in scope here, the part that improves how humans weigh evidence and make decisions, and part of it is general research acceleration, which is not. We treat it as a heavily overlapping neighbour rather than a niche of this field.

AI for philosophy. Using AI to make progress on philosophical questions, particularly those that relate to the transition (what counts as good decision-making, moral uncertainty, how to reason under deep disagreement). Important to many in this community and unlikely to happen by default, since philosophical progress has little commercial pull, but it is a research aspiration rather than a cluster of projects with users, so we treat it as adjacent for now.

Misinformation/platform safety. A mature field with its own funders, journals and other infrastructure. AI for Epistemics might borrow some of its evidence at times.

Hardware-backed verification. Hardware-enabled guarantees (e.g. FlexHEG) aim to make commitments about AI development and deployment verifiable at the compute layer, including between states. It clearly overlaps with coordination (since verifying commitments is half of what coordination infrastructure is for), but it’s mostly compute-governance work with its own community and funders, so we treat it as an adjacent field rather than a niche.

3. Insights from Field-Strategy Interviews

(Note: “I” in this section refers to Alex Csaky)

3.1 Methodology

Between May and July 2026, I (Alex Csaky) had over 150 conversations with people working on or interested in the field of “AI for Epistemics and Coordination”. Wider conversations that built context included AI coordination start-ups and customers, academics, mediators, barristers, government staffers and other professionals who are interested in using AI for better epistemics, coordination and decision making.

The following synthesis is primarily focused on insights gained specifically from ~65 in-depth interviews with builders, funders, researchers and field-builders working in AI Safety/AI for Epistemics and Coordination. These interviews mostly took the form of 30-90 minute video calls. Quotes are anonymised and attributed by role only.

Three limitations of this sample:

Every funder interviewed comes from EA- and AI-safety-adjacent philanthropy. No institutional, government or commercial capital is represented, so claims about the funding landscape are limited to this ecosystem. Nor did I interview journalists, despite interviewees repeatedly naming them as high-leverage users.^[12]
Niche coverage is uneven. Two sections in particular (2.6 and 2.9) draw on desk research and adjacent interviewees rather than first-hand practitioners.
Third, the in-depth interviews skew to the supply side, whereas the wider set of 150+ conversations leaned the other way, towards potential buyers and users of these tools in government and industry, but those were not conducted as research interviews, so I do not quote or count them here, though they inform my judgement.

Figure 2. Visual breakdown of in-depth interviews.

3.2 Insights

Insight 1: Deployment was the most consistently reported bottleneck

Increasingly capable AI coding tools, alongside a relative abundance of technical talent, mean that building prototypes is not the binding constraint for many projects. The most consistent view I got from across interviews was that people currently working in the space find it really difficult to actually get their solutions adopted. There were a number of proposed reasons for this, ranging from difficulty with the product category itself to the types of skill sets present in the space at the moment (i.e., they are not necessarily natural marketers).

As a former founder put it, “The problem is almost never the technology. The problem is always the human.” A grantmaker with experience running fellowships noted that capabilities are already largely at the point at which a lot of what is being discussed in this space is now possible; “If people can find the paths to actually using this stuff, I feel much better about that than new capabilities and applications”. A builder with a working product described the search for pilot organisations: “nobody ever thinks, well, let me reach for this tool that I don’t really know what it does.” A researcher and former grantmaker was blunter about the pattern: “most of these other things are like, you build an app and then no one ever uses it.”

My own enterprise deployment experience confirms that almost all my public affairs, government and corporate clients do want AI help with coordination and decision making. The failures tend to happen downstream in places like permissions, incentives to actually adopt and use the tools, and trust in the tools, as well as emotional blockers like feeling undermined by AI decisions. There are moonshot-style projects where capabilities would be binding but only a small minority of projects in this space fall into that category.

Importantly, there was a less widely held, valid counterpoint that the problem is actually that products are just not good enough yet to provide enough value to make them worth adopting. The argument here was that once the products can provide clear value, deployment and adoption will be much easier. A number of interviewees who had tried prototypes in the space felt that there were almost no prototypes that were clearing a bar for usefulness.

Insight 2: There is no shared view of priorities

Funders, researchers, and builders have all said independently that no one is quite sure of the ranking of what matters in this space. While a lot of people are very interested in this space and some grantmakers have conducted initial reviews across the space, it’s still unclear what the priorities are. It would be useful to have individuals or organisations do this work, and create debate in order to come to some views around what top priorities might be.

This report attempts to preliminarily fill this gap. However, for larger grant-making projects to go ahead it may be necessary to have more in-depth cause prioritisation as well as methods for measuring the effectiveness of different solutions in this space. One grantmaker who put out an RFP for this field noted that “applications were much more vague than I had expected.”^[13] This also points to a bottleneck for builders who it seems struggle to articulate the specific problem they are solving, for whom, and how the work would reach those users.

Insight 3: Funders say there is a lack of good projects; builders say there is a lack of funding

Funders have expressed the view that they would be happy to fund interventions that are clearly good in this space. However, they struggle to find them whereas builders have told me that a lot of their applications are submitted and rejected without any substantive feedback. They’re not sure who they’d be able to contact if they needed bridging funding or feedback on how to make their interventions better and more fundable.

A number of forecasters mentioned feeling demotivated by the closing of Coefficient Giving’s forecasting fund earlier this year (note that they do continue to fund some forecasting work). Multiple builders who are working on epistemics recall that funder feedback is typically category-level doubt as opposed to project-level doubt. I.e. There’s still a lot of hesitancy around whether this field is really impactful, and it’s not clear that the work has been done to establish that. Therefore funders find it harder to evaluate individual projects.

One of the clearest divergences on this point was that people who had strong existing networks and credibility in this space reported that they found funding very easy to find. By contrast, people who were trying to break into this space from outside the AI safety ecosystem and had little existing credibility in it felt that while they could easily raise money for small projects, and saw opportunities for start-up funding, the ~6+ months required to research and prepare to actually start the org and have a prototype ready was not fundable.

While streams for this stage did exist (e.g. CAIF’s 2025 round and Coefficient’s Forecasting RFP), it seemed that many of the builders felt unaware, ineligible or unable to get funding from those sources. Most of these builders were trying to start something of their own and routes into existing projects or initiatives seemed less visible to them (which suggests that making ways to contribute more salient is itself a cheap and potentially quite effective field-building intervention).

When they did approach organisations for funding, the amount of time it took for funding decisions to be made meant that they either moved to other areas of AI safety where funding was easier to find (e.g. mechanistic interpretability, technical, or governance work) or returned to their day jobs outside of AI safety.

Insight 4: There is no “winning” precedent

Across different profiles people have noted that the field’s credibility problem stems from the fact that there aren’t many very clear winners (i.e. visible deployed successes that changed a consequential decision or made an org significantly better that outsiders can point to). One could argue that LLMs themselves are a hugely successful epistemics product that helps people reason and make better decisions. Many of the tools in this space may be most effective/impactful if adopted by tech companies (e.g. Google adopting reliability tracking is likely much more impactful than a new reliability tracking tool).

One example of this is that a number of people asked me directly whether I could point to any clear successes that have come out of the space that could easily be scaled up with much more funding and the impact of those projects would be scaled equally. There are examples like AI-written Community Notes, Polis, and prediction platforms (like Metaculus or FutureSearch) that seemed to me like fairly legible successes that could absorb more money to make more impact. However this was in doubt for many of those who I spoke to, couldn’t be sure that these interventions had a clear effect on the risks they cared about most, and doubted the category more than other things they could fund.

The key issue is that there is a lack of clear wins in the space, as well as that there is disagreement as to what counts as a win. Are we counting Community Notes that has been deployed and had a degree of positive utility for a large number of people or are we defining a win as a measurable reduction of some kind of societal or bigger level of risk?

Insight 5: “AI for Epistemics” is a suboptimal, very broad label

A lot of interviewees thought that “AI for Epistemics” is a poor label. It doesn’t give much of an idea of what is actually involved in the space. It encompasses everything from forecasting and decision-making tooling to model honesty and calibration that border on alignment, and democratic deliberation.

Builders noted that this sometimes made it difficult for them to know whether the project they were working on would fall within the scope of the field (as that was relevant to funding for them). Grantmakers mentioned that while it is a useful descriptive term, it contains so many projects that don’t tend to have very much in common. A number of participants also proposed their own taxonomies for carving the space into multiple different problem areas and addressing them on their own terms.

The strongest defence I heard was that “Epistemics” is “the only adequate term.” The problem here is it doesn’t give much of an idea of what’s actually involved in the space. Outside of the AI safety/EA-adjacent communities most people aren’t familiar with the word “epistemics” in this context (or at all), and practitioners inside institutions reach for terms like decision intelligence instead. A lot of this is semantics, but there’s an argument that this space has currently been driven by visions of the future and aspirations for tools that people would like to see.

3.3 Open Questions

Question 1: To what extent is the bottleneck money or a lack of good projects?

The case for money being the bottleneck came mostly from builders who had left the space due to the amount of time and effort it took to raise capital here, as compared to “mainstream” AI safety.

There was a fairly consistent pattern they described where faster small grants were fairly easy to get, e.g. month-by-month stipend funding or kind of fast grants for small projects. Broadly they also said that they saw a fairly clear path to organisation-scale funding on the order of $1m+. Between them sat a 6 to 18-month stage of development where they reported no clear or reliably accessible funding route. Some exited. Some builders who left the space did put a price on their own exits, with the most common responses being a minimum of 6 months but usually closer to 1 to 2 years of funding at or around their existing salary level. This would have allowed them to stay in the field, develop their organisation, and then raise organisation-level funding for it.

On the other hand, grantmakers argued that there weren’t enough fundable projects. A number of grantmakers said that they were actively looking for projects and if a project came their way in this space they would be very excited to fund it. In their eyes, not many of these projects exist. Former grantmakers who had reviewed hundreds of proposals in this space found only a small handful of them immediately fundable and identified the primary bottleneck as a lack of people who have a very convincing vision of what they’re building, who they’re building it for, and what the pathway to impact would be.

A small minority of builders agreed with this diagnosis, noting that they struggled finding projects that would actively “move the needle” in terms of having a positive impact. I do not think this is a major tension in that both sides accept the other’s facts but the open question is why that middle stage of getting people from having an idea and doing a small test to organisational-level funding is failing. Possible explanations could be that no funder is currently positioned there, or that work reaching that stage is not sufficiently good to get the funding to the organisation-level stage. Maybe if the idea was good enough the fellows could immediately raise org-level funding without the 6 to 18-month period of further exploration.

Question 2: For-profit or non-profit

AI for epistemics and coordination seems to be a space with quite clear commercial incentives (i.e. if you develop tools or make the models better at making good decisions on behalf of their users, there is clear economic and social value potential here). There’s an argument that (a subset of) builders here should focus on for-profit backing as opposed to non-profit funding because paying customers force product focus in a way that the slower non-profit grant reporting cycles won’t. VC funding is much bigger than philanthropy, even with potential increases in capital on the way, and may allow for much faster scaling. On the other hand, the philanthropic case may be even more important for a space like this where the commercial incentives are so strong. The incentive gradients of consumer and enterprise markets are exactly what may produce the epistemic environment that many in this field want to defend against. Tools becoming optimised for engagement or profit/power seeking may be the exact tools that we are hoping to front run by funding this space with philanthropic capital. The public goods argument backs this up. Absent public-interest provision, market incentives may disproportionately direct the best capabilities toward customers with greater ability to pay.

Two themes came up fairly consistently in these conversations:

Philanthropy could act as a powerful demand-side subsidy, paying for civil society organisations, unions, governments, or public bodies to become early adopters of these tools that may provide great value to them, after overcoming some initial friction.
There is an argument here for not funding the organisations themselves since funding from the AI safety/EA ecosystem may make it easier for outsiders to discredit epistemics work in this space.

Question 3: Who are the users?

It was unclear who the highest-leverage user groups were to target. Three potential camps:

Target a small number of high-impact organisations with internal tooling distribution that is fairly easy and cheap, for example Civil Society organisations, think tanks, and existing AIS organisations (who in particular might be receptive and supportive of these efforts). There may also be less marketing burden. Where these organisations already have a credible impact case, improving their capacity provides a relatively tractable starting point for estimating additional impact. However, increases in capacity or productivity should not be assumed to translate proportionally into impact.
High-leverage intermediaries, people like legislative staffers, journalists and regulators, on the argument that a small reachable group will transmit better epistemics to consequential decisions.
Target the public as a whole. Here you view the epistemic commons as the main prize and everything else is just a route to get there. You view good epistemics and decision-making as a public goods funding problem, with an argument for philanthropy and subsidising development to allow for this public good to emerge quicker.

These camps aren’t necessarily in competition, rather it seems that the skillsets required for each will likely vary a lot. I think it’s a really important question because it seems to be one that is keeping a lot of the existing organisations in this space from being able to effectively scale. It seems unclear to me who a lot of them are targeting as the ideal users (ICPs—Ideal Customer Profiles) and the story that they have for how specifically they are going to reach them.

Question 4: Is field-building premature?

I approached this report with the idea that there is a strong case for building the field now. Demand signals are real, with many applications to fellowships and incubators receiving more good applicants than places and mentors for them. Several interviewees see a fairly short window to socialise and embed good AI decision-making in workflows and standards being set. New entrants are arriving from policy and government backgrounds, regardless of what this community does (i.e., there is strong commercial incentive here and actors may be encouraged to build decision-making tools that inadvertently allow for decision-making to be in their favour).

There is however also a case for waiting to build this field. Some experienced founders in this space have said that just adding more people into this space at the moment is probably a bad thing. Priorities are not settled and it seems funders don’t yet have a clear idea of what matters most in this space, so overclaiming that it’s possible to make some huge transformative change here may be dishonest. People who arrive to “invent the field” may quit fairly soon after realising that they won’t be able to raise the funding they want or do what they want in this space yet. Funders also seem hesitant to fund field-building projects in this space before seeing legible successes that they want repeated.

Question 5: The role of forecasting

Forecasting is, by and large, the most mature part of the “AI for Epistemics” space and has a number of relatively large organisations working on forecasting products and research.

The bear case for forecasting is that a major funder recently closed its dedicated forecasting programme, while continuing to support some forecasting work through other channels. Several interviewees interpreted this, alongside their own funding experiences, as evidence of uncertainty about the impact of actively funding the space. Adoption and deployment seem to be a major bottleneck here. A former forecasting platform employee noted that at the time the platform itself did not use forecasting in its own decisions and rejected the idea that the impact was merely illegible. They note that forecasting is actually just a very difficult technology to deploy and socialise because the incentives may not be aligned with organisations actually wanting to use forecasting in their work. Many forecasting researchers and orgs seem to have struggled to find non-profit funding and struggled to prove their impact in clearly legible ways.

That being said, there does still seem to be a solid case for forecasting. Rapid improvements in forecasting capabilities mean that it is now possible to do much more with forecasting than was possible a few years/months ago. Ideas like automated forecasting with automated question generation were noted by a small number of grant makers as having a more clear end-to-end theory of change and as something that they could, in theory, be excited to see more of. Furthermore, it looks like commercial AI forecasting has a strong case for widespread adoption if deployed effectively.

It seems that while forecasting may not necessarily be the best fit for non-profit funding, accelerating forecasting deployment in the real world to prove (and amplify) its impact (perhaps through showing that people will pay for it and act on what it shows) would be high impact. It could provide uplift to the general thesis that AI will be really useful for epistemics and decision making. The open question is whether we are yet at the point that AI forecasting can have a significant impact on decision making (more so than human forecasters already can) and who should pay to find that out.

3.4 Bottlenecks

Given that the goal of this project is to identify solutions for the bottlenecks in the AI for epistemics and coordination space to allow us to understand if and where we should be accelerating development, below I (Alex) break down the bottlenecks that came up in the interviews by stages in the cycle of projects that take place.

Stage	Does it work?	What interviewees report
1. Entry	Partly	Plenty of demand, no pipeline, intake skewed toward “thinker”/researcher-heavy talent
2. First money	Yes	Small fast grants (e.g. up to $25k) are easy
3. Refinement (6–18 months)	No	Builders report a gap in accessible, sustained funding
4. Building	Yes	Prototypes are cheap to build (however tools worth adopting are rarer—see Insight 1)
5. Deployment	No	Lack of pilots, trust, procurement, workflows, demand from real-world users
6. Proof	No	Few demonstrated wins, hard to measure

Stage 1: Entering the space

People want to enter the field. A pilot fellowship in the space received over seven hundred applications; an adjacent incubator reports many more strong applicants than places. A former fellow contrasted it with technical AI safety, where “you can kind of hop from program to program”, this space “is just not even close to that”. There is no equivalent of the vetted talent pools that adjacent fields use for hiring, and one young research organisation told me it takes five to ten mentees to find one hire.

The intake is also skewed. It is homogeneous: a grants assessor described near-identical applications, “everybody’s just drinking from the same water”. And it is tilted toward thinkers over builders. One fellow estimated the ratio at ten to one and said it should be one to one, though the fellowship’s own organisers pushed back that the balance was better, and that the missing ingredient was go-to-market instinct rather than engineers.

Stage 2: Initial funding

Small, fast grants function well. Micro-grant programmes are operating, application processes were described as easy, and nobody reported dying for lack of a first cheque. That being said some grant programmes run well below asks (one funder’s applicants asked for an average of around $60k and received $10-40k), and milestone-based disbursement (i.e. “when you reach x point we can give another grant”) creates cash-flow gaps for organisations with no reserves.

Stage 3: Refining ideas

Between a one-month grant and an organisation-scale raise sits a roughly six-to-eighteen-month stage where an idea may need committed time to become a fundable organisation. Several builders reported struggling to find accessible, sustained funding at this stage, despite some relevant funding streams existing. Several promising fellows/builders left the space over this gap. People quoted me the specific terms that would have kept them, and they were relatively modest (6-18 months stipend). Interviewees themselves proposed ideas that they feel would make a difference. Guaranteed post-programme funding regardless of continuation decisions (i.e. fund the person, not the idea). Continuation decisions made quickly (ideally before existing grants run out).

Case Study: Death at the refinement stage. A team built AI-assisted policy wargaming with live demand from a major research institution. The project died due to changes at the client organisation, and a funder who had signalled interest who went quiet for five months before the relevant programme was cancelled. Both founders say they would have continued with committed funding.

Stage 4: Building

Nobody I spoke to located the field’s problems in the difficulty of building prototypes. Prototypes are cheap and quick with AI coding tools. The funder view was broadly that engineering at this level is solved and “entrepreneurial gumption” is not. The risks reported at this stage are more strategic than technical.

Frontier labs may absorb any thin product layer: “scoping the domain to things that won’t get sniped by a large model lab is basically all the strategic decision right now”, as one former fellow put it. Being absorbed by a lab is not obviously bad from a field point of view but the harm is that it may deter builders from areas that they think labs or larger companies are going to cover by default. For a minority of the most ambitious projects, technical capabilities were a blocker, however, it was unclear whether a better engineer could have solved that, or it was just a question of waiting for better model capabilities.

Case Study: Deployment without design. Five professional mediators, interviewed separately, described the same new phenomenon. During mediations, parties would consult consumer AI and receive confident, escalatory advice. They also note an overwhelming rise in AI-driven legal claims. Consumer AI is already entering exactly the coordination settings we care about, sometimes in ways these mediators regarded as harmful or escalatory.

Stage 5: Deployment

This is the most unanimous bottleneck in the space. The barriers arrived in five forms:

Pilots. A builder with a working product named finding organisations willing to pilot it as “the single biggest blocker”, wanting design partners and unable to find them. Many of the customer areas that builders in this space want to break into are also some of the most challenging to break into (e.g. government).
Trust. Coordination projects failed by being unable to win the confidence of organisations outside their own community. This is a related problem to 1. where it seems that more “entrepreneurial” types would benefit the space.
Procurement. Although government demand exists, approved access remains uneven. Some congressional staffers we spoke to reported drafting legislation using personal chatbot accounts because suitable tools were not available through their offices. A serving government official was blunt that adoption is “probably more relationship based than tool based”.
Workflows. Tools that do not sit inside an existing workflow do not get used. Enterprise deployments I’ve undertaken corroborate this. Permissions, review queues and reluctant users kill tools that models are entirely capable of powering.
Demand. The hardest version, from an experienced, exited founder: much apparent demand is for the appearance of coordination rather than the thing. Counter-evidence here includes a grantmaker in an adjacent field now conditioning grants on a credible real-world adoption pathway and reporting that the resulting projects are deploying and working, noting that it relies on the right kind of talent and projects. One reliable adoption route interviewees described was integration at a specific point in an existing process, which requires both knowing the process in detail and being trusted enough to be let into it. i.AI’s Extract, for example, went through a year of evaluation before being integrated into council planning workflows.

Case Study: Dying at deployment. A collective intelligence startup shipped its product and is kept alive by private funding. Its founder names neither money nor engineering as the constraint but design partners: ten organisations willing to run the tool inside real decisions. He has not found them.

Stage 6: Clear proof

Impact measurement was described by a former grantmaker as ‘very underdeveloped’. The problem is not that decision quality cannot be measured. Businesses, governments and forecasting platforms have methods for it. It is that not enough builders in this space are publishing them, so there is no convincing demonstration that a tool improved a real, consequential decision^[14].

Clear proof could come from existing methods or new ones. One tractable first step is getting institutions to record the considerations behind important decisions and compare them with outcomes later. This could reveal miscalibration and create demand for tools that help, although establishing whether a tool itself improved decisions would require some credible comparison against what would otherwise have happened.

Evals and benchmarks for the epistemic quality of AI systems came up independently across groups, as well as in public materials.

The chicken-and-egg problem

The lack of demonstrated wins makes funders doubt the category, not just individual bets. Category doubt keeps the refinement stage unfunded. The gap sends talent back to clearer career paths. Without organisations that survive long enough to deploy, no wins get demonstrated. One org leader described the cycle as: fund twenty groups, watch demos that look “kind of neat but not amazing”, offer no follow-on, and watch everyone go back to their day jobs.

4. Recommendations

Note: these are not ranked in any particular order. They are also far from exhaustive.

4.1 Test AI deployment in specific institutional workflows

We recommend funding the deployment of useful AI and epistemics tools into organisations whose decisions matter most during the transition to transformative AI. Section 3 identifies deployment as the most consistent bottleneck in the field: promising tools often struggle to find pilot partners, earn trust, navigate procurement and become integrated into workflows.

There seem to be two complementary ways to approach this:

Institution-first: Starting with a high-value organisation and internal workflow, enable (e.g. via funding^[15] or running programmes) teams to identify, build and integrate the AI tools that would most increase its capacity.
Tool-first: Start with an existing tool that has credible evidence of value and subsidise its deployment into important organisations that would benefit from it but are unlikely or unable to adopt it on their own.

High-value targets could include tractable-to-reach parts of the UK/US governments^[16], frontier AI companies, AI safety organisations, and civil-society/nonprofit organisations.

Figure 3. High-level summary of projects to consider.

For the government in particular, trusted deployment pathways matter as much as tool quality. Institutions making high-stakes decisions may be reluctant to adopt tools from unfamiliar external developers, even when those tools are technically strong. This suggests working through strong internal deployment teams or co-developing with organisations that already have trusted relationships with policymakers and decision-makers. Initiatives aimed at government will be substantially less likely to succeed without these kinds of partners.

The most promising opportunities may also be those attached to operational problems, rather than tools framed primarily around improving “epistemics”. For example, the case for AI adoption in government might be made in terms of reducing bureaucracy, improving public services or increasing state capacity, with better decision-making as a useful consequence. It seems really important that tools should be developed around workflows and with committed users rather than built first and offered to institutions afterwards.

Deployment into sensitive organisations could also be accompanied by an “assurance case” explaining how the tool will be used, who is responsible for failures, how outputs will be evaluated and how the system fits into existing procurement and oversight processes.

Forecasting may be the clearest current example of the tool-first approach. Forecasting tools appear increasingly capable and there is already some commercial demand, weakening the case for philanthropy to fund the products themselves. A stronger opportunity may be subsidising competent deployment into governments or nonprofits that could benefit from forecasting but are unlikely to become effective users without support. The same model could apply to other tools as they approach the threshold of practical usefulness.

Notes on developing a potential theory of change:

Some important organisations could benefit substantially from AI but are unusually difficult to deploy into. At the same time, some useful tools will face a gap between technical maturity and effective real-world adoption. Funding trusted deployment teams, partnerships and implementation work could bridge these gaps. Successful deployments would make institutions more capable, show which tools actually improve decision-making, and give others clear examples that make adoption easier.
The pitch would likely be around general state capacity, rather than promoting “good” epistemics in government. The assumption here is that tools that make individuals and departments faster and more reliable would serve a ‘hair on fire’ user group. Pilots should separately test whether these gains translate into better decisions, rather than assuming that faster workflows improve decision quality.
Another aspect of this is protecting key decision-makers’ epistemics from epistemic risks relating to AI. These groups should be resistant to, and skeptical of, sycophancy and overconfidence.

Open questions:

Is deployment really the binding constraint, or is apparent demand misleading? Organisations may express interest in tools without having incentives to actually change their behaviour or workflows.
Better epistemics still doesn’t fix incentive problems where even with better epistemics, actors want to “win” at the expense of a wider goal or consideration (e.g. climate change: actors accept major risk for short term gain).
Individual buyers in government may prefer tools that do not challenge them, even where their institution would benefit.
If organisations’ incentives reward the appearance of coordination over the thing itself, subsidising adoption won’t work. My (Alex’s) own experience deploying similar tech makes me think the appetite is real and the failures are downstream of incentives and workflow fit, which funding conditional on adoption pathways is designed to address. I may be wrong about this though.

4.2 Build decision-making evaluations and benchmarks

Update: Since we wrote this section, Coefficient Giving has listed “AI for decision-making” as one of their target areas for Project Tailwind. Check out their overview!

A concrete project here would be to found a new initiative whose main product is a world class, maintained, multi-turn decision-quality benchmark.^[17] The space has plenty of researchers interested in the question and a number of one-off benchmarks, but few standing organisations seem committed to maintaining one. Its focus could be multi-turn benchmarks that score models on the quality of their advice. Broadly, this could include:

Presenting facts honestly^[18], rather than in ways that are technically truthful but manipulative
Transparency in decision processes
Navigating good reasoning amidst emotional blockers
Measurement of whether a model reaches the same conclusion when a question is posed by an enthusiast versus a skeptic.
Whether a model flags false premises in realistic requests.
Whether users end up making more accurate judgments with an AI versus without one.

It would make sense to frame these as “decision-making evals” rather than “epistemic virtue evals” because it sounds more like a task frontier providers want their models to win on. Primarily these evals should target more objective measures e.g. calibration, honesty about uncertainty, misrepresentation of facts or sources, non-sycophancy. On contested moral and political questions, it doesn’t seem like a good idea to evaluate whether a model’s conclusions are “good”. This seems too contentious to effectively judge.

A better set of judgements here could be around making models’ positions legible, tracking how they change over time, and scoring the properties there is broad agreement on (e.g. honesty, calibration, accuracy). Another important feature would be that such benchmarks and evals are continuously updated with new model releases. Most existing attempts to build benchmarks in this space have not done this. Benchmarks should also be “multi-turn” in their design, given the relevant model behaviours are not best suited to be measured via single-prompt set-ups.

Notes on developing a potential theory of change:

Evals are the cheapest external lever on model-layer behaviour. Outsiders cannot shape how labs train, but they can shape what labs optimise for, and labs demonstrably compete on benchmarks with commercial salience. We think that the way that models behave will likely impact more users than new products (hence the importance of evals).
We also do not currently have a great idea of the quality of model epistemics. Independent benchmarks give regulators, scorecards and buyers evidence for how far models should be trusted with consequential decisions.
That being said, the more successfully labs optimise against a benchmark, the less it functions as a pure measurement of trustworthiness.

Open questions

What are the strengths and weaknesses of existing benchmarks? If someone were to work on pursuing this, we’d recommend they do a comprehensive listening tour of experts to figure out how to be additive rather than duplicative, and take advantage of the existing good work.
“Which virtues matter most” probably needs more research. Some evals (e.g. more subjective ones like “navigating good reasoning”) will be far harder to build.
How do we create evals that are both legible enough for labs to climb and robust enough to resist goodharting?
Labs may simply not engage despite reported interest.^[19] The people who run these benchmarks would need to treat adoption as a key goal and likely have a strong communication skillset.
Frontier AI labs may optimise against the benchmark without the models actually making better decisions in real-world deployment contexts.
Several groups have produced benchmarks in this space. If any of them is in fact building a maintained decision-quality benchmark as an organisation, the right move becomes funding and pushing adoption of that effort rather than founding a new one.

4.3 Run longer-runway field-building programmes that help promising projects reach deployment

The most useful talent initiatives may be those that retain promising researchers and builders after an initial grant, giving them enough runway and institutional support to turn early work into fundable or deployable projects. This could include fellowships, residencies or incubator-style programmes that fund people with relatively broad freedom while helping them refine their ideas, find users and test credible pathways to impact.

These initiatives could be tailored to specific niches or users. Where deployment is the goal, relevant external partners seem important. For example, a programme aimed at getting AI tools into government may be unlikely to succeed without government design partners or trusted orgs that already have relationships and can provide workflows in which to test tools.

Figure 4. High-level summary of projects to consider.

Examples of various initiatives and events that could be good to run include:

Residencies that give promising builders 6-18 months to develop preliminary projects into organisation-scale efforts, without requiring the project to already be ready for a large grant. This could be part of an existing fellowship/residency program.
Fellowships or incubators organised around a specific user or deployment opportunity (e.g. a certain company, government department, etc).
Helping experienced professionals transitioning into AI safety make use of AI. This could be run alongside existing talent programmes as a one-off event (e.g. during MATS or Astra). This idea was inspired by the new Lateral Workshop.

Notes on developing a potential theory of change:

This recommendation addresses the refinement-stage gap identified in section 3.4. The 6 to 18 month stretch between a first grant and an organisation-scale raise is where the field loses its people. This implies programmes considerably longer and better funded than the standard 3-month fellowships. 6 to 18 months of stipend, with continuation decided before the money runs out, would be more effective.
A further consideration is that field-building may need to operate on unusually compressed timescales. We may have a reasonable view of what is useful over the next few months without knowing what the best projects will be two years from now.
Talent initiatives should therefore select for and preserve agility (i.e., building a pool of people who understand the space well enough to notice when the best opportunities change and redirect quickly). This may favour taking many bets and funding people with broad freedom rather than locking programmes into fixed multi-year agendas.
That being said, not everyone suited to this work is a founder type who should be raising funds continuously. Many of the field’s best contributors have been researchers who couldn’t find an org that would host them to work on their topic. Initiatives could even retain their strongest fellows as staff indefinitely, conditional on the work being good.^[20]

Open questions:

Is lack of runway actually causing people to leave, or is it downstream of a lack of sufficiently good projects or deployment opportunities? If the latter, additional fellowships alone probably won’t help solve the problem.
Funders may see starting new organisations or initiatives as premature, wanting more legible successes for the field before backing substantial infrastructure.
FLF and Forethought have already worked on projects in this space. Is it best to put a new “home” inside of one of these existing orgs (or, e.g MATS or Astra) rather than starting a new one?
What is the optimal balance between research and entrepreneurship here? For example, would a new fellowship focus on finding better ideas, or just on deploying pre-existing ideas to real-world organisations to the best of their ability?

4.4 Additional recommendation: develop pieces of the “Epistack”

The “Epistack” involves tools and workflows that can each improve aspects of epistemics. Each of these solutions has potential to make a meaningful improvement to existing epistemics (e.g. reliability tracking, community notes for everything, scenario planning, option surfacing, privacy-preserving auditing, fact verification, recommender algorithms).

Figure 5. The Future of Life Foundation (FLF)’s visual of the Epistack

Some plausible consumers of the “Epistack” are:

Humans who (even sometimes) want good epistemics (e.g. certain types of investigative journalists, policy makers, financial traders, rationalists etc).
AI agents that rely on having good epistemics to complete tasks for users (e.g. accurate research/facts about the world, decision-making assistance, planning).

Some plausible paths to getting wide adoption of the “Epistack” are:

Creating tools that can be easily accessed and utilised by AI agents. For example, a high-quality, maintained API providing reliability tracking scores for sources could be called when relevant, which could then inform the model’s output to the user. Another example here could be “reliability tracking” for agents themselves, where agents can assess the factuality of claims made by other agents.
- That being said, projects in this space could be dual-use. As Eric Drexler argues, the same infrastructure that enables agent cooperation can also enable collusion. To reduce this risk, it is important that multi-agent systems have deliberately restricted communication channels (e.g. agents should not all be able to freely message each other or share a giant persistent memory), diverse or adversarial agent roles, tamper-evident audit trails, and independent monitors that can block or escalate problematic actions.
Get acquired/adopted by existing incumbents. Many of the world’s most powerful epistemic tools are already widely used. Search engines (and now LLMs) shape the information environment and, most of the time, seem to help people make more informed and accurate decisions. Improving these tools by building open-source parts of the epistack or getting acquired by these companies could be a powerful lever for quickly getting these tools used by billions worldwide.
- For example, Meta’s 2025 adoption of Community Notes put the approach in front of far more people than a standalone product could.
Build a widely loved product that happens to have good epistemics tools packaged inside it. The concept of “Guardian Angels” could be a good example of what this might look like: an always-on personal agent – or, “angel on the shoulder”, so to speak. It knows what you are doing and tells you when something you are reading is trying to persuade you or misrepresenting facts. For these products to get adoption, they will likely also have to be integrated enough with frontier AI models to help with relevant tasks (e.g. providing balanced advice on projects in your personal and/or professional life). Ideally, the product could incorporate reliability tracking, community notes for everything, and super-persuasion defence.

If you are funding, building or researching in this space (or want to start) we would love to hear from you. Please have a low bar for getting in touch!

Acknowledgements

The following people provided comments that were helpful to us, and we are grateful for their input. Thank you to Alejo Acelas, Ben Goldhaber, Benjamin Tereick, Dave Banerjee, Edward Kembery, Eli Lifland, Evan Miyazono, Harrison Gietz, Jim Maar, Joshua Landes, Lawrence Phillips, Lewis Hammond, Lexi Scholefield, Lizka Vaintrob, Matt Putz, Max Dalton, Nathan Young, Nick Marsh, Oly Sourbut, Owen Cotton-Barratt, Paul de Font-Reaulx, Rob Gordon, Sharif Kazemi, Simon Steshin, Stefan Torges, Tara Mei, Will Aldred, Francisco Carvalho and many others for comments and discussions.

AI Use Disclosure: We used LLMs (specifically GPT-5.6 and Claude Fable 5) to support the desk research process, help with writing clarity throughout, and help synthesise themes from raw interview transcripts. The writing in the report is our own, mainly with the exception of Section 2 where AI was used to help draft and compress parts of the niche summaries.

^
AI tools for coordination can also be dual use. See p. 4 of Dafoe et al (2020) on commitment capabilities being “closely related to coercive capabilities”.
^
For specific examples, we would recommend reading through Forethought’s design sketches (particularly the parts on tools for “collective epistemics” and “defense-favoured coordination tech”). For a narrative example, see this post by Owen Cotton Barratt.
^
There are also other reasons to work on accelerating such tools today instead of waiting:
Even if the market eventually handles distribution, having these tools sooner is beneficial.
Early work could help map out and figure out mitigations for various risks that may arise.
^
While Pol.is was used to inform real policy changes in Taiwan, it was just one component of a broader deliberative process, and has not become a routine or dominant mechanism for Taiwanese governance.
^
For example, on platforms like Metaculus the gap between two conditional forecasts reflects correlation rather than causation, whereas decision-makers usually need to know what their intervention would cause.
^
For example, a city used Polis and Google Jigsaw’s Sensemaker to synthesise input from nearly 8,000 Kentucky residents into nine priority initiatives. As far as we can tell, the process fed into the BG 2050 Strategic Plan, released in April 2026 with nine priority initiatives.
^
Especially if you count the frontier AI models themselves as research assistants (e.g. imagine how many small business owners ask Claude for help with spreadsheets and marketing techniques).
^
E.g. Several consumer fact checking companies built for the public found that checking a claim is a metacognitive habit few people have, and pivoted to selling to journalists and institutions instead.
^
This distinguishes coordination tools from CivicTech, which primarily helps decision-makers understand the views of a wider population. Coordination tools help identifiable parties reach and carry out agreements. In practice the boundary seems porous (as with many niches in the field).
^
For example, those who may worry that the negotiation system favours the other side.
^
States often make deliberately shallow agreements. High compliance rates largely reflect treaties ‘that require them to do little more than they would do in the absence of a treaty’ (Downs, Rocke and Barsoom, 1996). E.g. on NetZero, 929 of the Forbes 2000 had set net-zero targets by mid-2023, yet only 4% of those commitments met the UN’s minimum credibility criteria (Net Zero Tracker, 2023).
^
Mainly due to a lack of time. Journalists were named as target users only as the interviews accumulated. The case for them is that verifying claims is already their core workflow, and their output reaches millions of people who may never seek out an epistemics tool themselves.
^
That being said, it seems that the picture has improved since, partly due to initiatives like FLF’s AIHR fellowship.
^
Ozzie Gooen makes the case that impact here could be modelled as beliefs propagating through networks of attention rather than as discrete decisions.
^
For government partners, “fund teams” need not mean transferring money directly to government units or staff. Support could instead fund an external team or provide approved in-kind technical assistance. Any arrangement here would require institutional agreement and compliance with applicable procurement, COI, etc policies.
^
Facilitating adoption in specific departments rather than institutions as a whole seems most tractable.
^
It could also run longitudinal studies (e.g. in partnership with frontier AI companies) on how AI is affecting users’ epistemics, and could also be responsible for creating relevant datasets to be used by others.
^
We think models should be able to reason from a user’s assumptions or values without endorsing factual claims they have reason to think are false, provided this is made clear to the user. There is a tension here to be navigated between blindly following the user’s preferences and being sycophantic vs “epistemic paternalism”.
^
Sycophancy is related to engagement and user satisfaction, so commercial incentives may affect all this.
^
A secondary benefit is that it grows the pool of grantmakers with direct experience of this space, which the field currently lacks.

Hiring in the AI Safety Ecosystem by Marta Kosmyna

Marta Kosmyna — Fri, 02 Oct 2026 21:27:39 +0000

Personal reflections on effective hiring operations from going through the interview process at 8 AI Safety organizations. Below, I’ve included 5 recommendations for hiring managers in AI Safety and 5 tips for candidates as they navigate their career search. I’ve also highlighted orgs I think are doing hiring particularly well (from a candidate’s perspective) and strategies to avoid.

Note for Readers: I do not have a technical background, and have never applied to technical roles. Therefore the reflections below are more applicable to generalist, operations, convenings, grantmaking, policy, and field-building roles in AI Safety. As orgs scale, their processes may change rapidly. These insights are locked in time from the 2026 hiring cycle.

5 Tips For Candidates

Every interview is a data point that helps you understand where your expertise fits into the ecosystem and where you can have the most impact. The hiring process is not a personal rejection or reflection of you, your talent, or the knowledge you bring to the field. Some roles I applied to were closed without ever hiring anyone.
After taking the BlueDot Impact courses and adding that experience to my resume, my percentage of applications to interview invitations went way up. If you’re making a career pivot and you don’t have previous experience at an AI Safety org or frontier lab, hiring managers are looking for indicators that you’ve taken steps to actively be part of the AI Safety community, upskill, and continue your education. See my previous post on Comparing AI Policy Courses.
I was skeptical when it launched, but I am actually very impressed by the 80,000 Hours AI Career Advisor. Their LLM can tailor your resume, edit your cover letter and application questions, run mock interviews, and provide feedback on work test submissions. Start here before applying for a personal 1-1 advising session with Probably Good, SuccessIf, or 80,000 Hours (which are more competitive and often have long waiting periods).
Many smaller AI Safety orgs work with third-party providers like Impact Ops and Mangrove Ops for hiring support. Your first interaction may be with one of their operations staff, not someone working directly at the org you’re applying for. Practically, this means you’ll have to communicate your expertise to someone who may not have any knowledge of the field. I had a much better interview experience with Impact Ops vs. Mangrove Ops.
I always recommend checking the box for “Are you happy for us to share your information with other hiring organizations?” The AI Safety field is still small, and I don’t know how the data sharing works on the back-end, but I have actually received invitations to apply for multiple roles through this (including unpublished roles). Regardless of checking this box, I feel like every AI Safety org in the US must have a copy of my resume by now.

5 Recommendations for Hiring Managers

The hiring process is the first interface interested candidates will have with your organization- make sure it’s a good first impression.
If there’s a take-home work test, instead of banning LLM use or relying on automated proctoring software, ask candidates to submit a link to the LLM transcript they used to assist with their work test. This helps you evaluate a person’s thinking process and how they are currently leveraging AI tools.
Paid work tests are becoming industry standard. Even a small honorarium/stipend for completing the work test feels like a big gesture that respects candidates’ time. If set up correctly to mirror the day-to-day responsibilities of the role, they are also super helpful to understand whether a candidate would actually enjoy the work and if their expertise would be a good fit for the position.
If possible, include the number of applications a role received in your rejection email. I don’t know why, but as a candidate, knowing that I was competing against 900+ other candidates for one position lessens the blow and indicates how competitive the market is right now.
Some roles included 1-2 sentences of feedback in their rejection email and these stood out as super helpful. See these helpful EA Forum posts from Caleb Price, founder of the Lisbon AI Safety Project, with rejection email templates and advice for AI Safety orgs.

The Hall of Shame

Ghosting candidates after an interview. No excuse here. Set up an automation to ensure follow up and close the loop.
Sending candidates directly to an automated video interview or an unpaid work test before a first round interview or phone screening creates high candidate drop off rates, especially for mid-career professionals.
Proctoring software is generally terrible- most recently it overloaded my PC’s system and my computer shut down mid-test while the timer kept ticking. As someone who’s reviewed submissions & served on hiring committees, the difference between LLM generated writing and human thought processes is easy to spot- using automated proctoring for work tests feels dystopian and unnecessary (See Hiring Manager Recommendation #2 above). The hiring process is hard enough on candidates.
General shame for orgs still using Workday which has the worst user interface, worst resume parsing, worst ATS, and requires candidates to create a separate account for every org they apply to. Workday is also currently getting sued for age, disability, and race discrimination. Generally seems outdated now, and the place where resumes go to die. I have genuinely not applied for jobs once I saw the org was using Workday.

Best interview process…drum roll… goes to:

Longview Philanthropy!

I didn’t land the role (twice..whoops) but so admired their approach to candidates. Ahead of the interview, they send you:

The job description
Realities of Working at Longview − 8 pages of reflections/advice from employees, examples of feedback employees received from managers, honest take on who would be a good fit and who may find the organizational environment challenging, and recommended reflection questions.
Operating Values − 4 pages that outline not just the mission statement, but how the mission gets accomplished.
Overview Doc- 4 pages on their grantmaking and philanthropic advising work, clear outline of what they fund and what they do not across AI, biosecurity and nuclear weapons policy, and how they make decisions based on scale, neglected-ness, and traceability.

Detailed job descriptions & salary range clearly stated on their Careers website. Apply via Airtable with 4-5 short answer essay questions. Automated interview scheduling with Calendly. No ghosting- an automated “application received” email, and an automated rejection email via Airtable after the phone screening. Email from a real human after the work test stage with the magic line “We’d sincerely welcome a future application as Longview grows.”

Honorable Mention: FAR AI

They’re remote-friendly, work tests are paid, and the salary range is clearly stated. They always write clear job descriptions and use Ashby, my favorite platform for its intuitive interface and ease of use. They have a quick, streamlined application process and only ask the classic “Why FAR.AI?” question in addition to your resume (instead of 5 essays + a cover letter like some orgs). They’ve never ghosted, and always sent “Thank You for Your Application” and “Not This Time” emails- probably thanks to Ashby’s built in automations for hiring managers.

Honorable Mention: Horizon Institute for Public Service

This one stood out because while I was rejected from Horizon’s Career Advising program, my consolation prize was receiving travel support to attend their three-day AI Innovation & Security Policy Workshop in Washington D.C. (which was an incredible networking opportunity). If you’re applying to multiple programs, save your application question answers, as each form asks the same essay questions. Paid work test.

Honorable Mention: Arcadia Impact

Applications are accepted via Airtable and usually include 4-5 short answer questions. They’re able to sponsor UK visas to work in-person full-time at LISA in London. Paid work test.

GiveWell fundraising up 49% in 2025, driven overwhelmingly by 39 donors giving $1M+ by Rasool

Rasool — Fri, 02 Oct 2026 21:19:07 +0000

Previously: GiveWell raised less than its 10th percentile forecast in 2023 · GiveWell fundraising up 17% in 2024

GiveWell has published its 2025 metrics, together with a metrics report and a public spreadsheet. As in previous years, I focus on how much was raised and who gave it. This year I also look at how concentrated GiveWell’s donor base is.

Truth be told I kinda phoned it in on this one, the full Claude version with better charts is here

Summary

Returning donors gave $443.6M (up 77%). New donors gave $37.1M (up 112%), which is still below their 2020–2022 levels.
39 donors giving $1M or more contributed $359.7M. That is 75% of identified donations through GiveWell and 90% of the year’s increase. Every smaller size band also grew, by 13–27%.
The donor base is far more concentrated than an ⁸⁰⁄₂₀ rule. The top 20% of donors gave about 97% of the money, and about 0.26% of donors (roughly 90) gave 80%. Asking the question again within the top group (the top 20% of the top 20%, and so on) gives 94%, 93% and 92%, rather than 80% each time.

Returning donors drove the increase

Returning donors gave $443.6M, up 77% on 2024. That is 72% of all funds raised and 86% of the money from donors other than Coefficient Giving, both the highest shares in the series. The average returning donor gave about $20,800, against $12,300 in 2024.

New donors gave $37.1M, up 112%. This reverses three years of decline ($41.2M in 2022, $20.5M in 2023, $17.5M in 2024), but it remains below 2020–2022 levels. Anonymous giving fell 7% to $35.6M.

The number of donors rose for the first time since 2021. There were 13,796 new donors (up 34%) and 21,315 returning donors, a record. Together that makes 35,111 identified donors, up 14%, but still below the 2021 peak of about 41,900.

Interactive version: Datawrapper.

Interactive version: Datawrapper.

The increase was driven overwhelmingly by $1M+ donations

GiveWell reports donations that pass through its own accounts by donor size, with each donor placed in a band by their total gifts in the year. On this measure, identified donations rose by $230.4M, from $246.7M to $477.1M.

$1M+ donors: 39 (up from 26) gave $359.7M, up 138%. That accounts for $208.4M, or 90%, of the increase. The average gift in this band rose from $5.8M to $9.2M, so the growth came both from more very large donors and from larger gifts.
Every smaller band also grew: $100K–$1M by 27%, $10K–$100K by 23%, $1K–$10K by 19%, $100–$1K by 14% and under $100 by 13%. Giving below $1M rose 23% in total, against 10% in 2024, so growth was broader than last year even though the top band dominated it.
The $1M+ share of identified donations through GiveWell rose from 43% in 2020 to 61% in 2024 and 75% in 2025.

Donor size	2024	2025	Change	Growth	Donors 2024	Donors 2025	Average gift 2025
$1M+	$151.3M	$359.7M	+$208.4M	+138%	26	39	$9.22M
$100K–$1M	$31.3M	$39.8M	+$8.6M	+27%	152	176	$226,000
$10K–$100K	$36.7M	$45.1M	+$8.4M	+23%	1,581	1,927	$23,400
$1K–$10K	$22.2M	$26.5M	+$4.3M	+19%	8,219	9,703	$2,730
$100–$1K	$5.0M	$5.7M	+$0.7M	+14%	14,141	16,187	$352
Under $100	$0.23M	$0.26M	+$0.03M	+13%	6,658	7,066	$37
All identified donors	$246.7M	$477.1M	+$230.4M	+93%	30,777	35,098	$13,600

Interactive version: Datawrapper

Interactive version: Datawrapper

Is GiveWell’s funding 80/20? No: it is far more concentrated

An ⁸⁰⁄₂₀ rule means the top 20% of donors give 80% of the money. If donations followed a Pareto distribution with that property, the pattern would repeat at every scale. The top 20% of the top 20% (4% of donors) would give 80% of that group’s money, which is 64% of the total. The next level down (0.8% of donors) would give 51% of the total, and so on.

The size bands give exact figures only at their boundaries ($100, $1K, $10K, $100K and $1M). I interpolated between them (see the methodology section) and report ranges where the data leave room for doubt. For 2025:

The top 20% of donors (about 7,000) gave about 97% of the money (range 96.0–97.7%).
About 0.26% of donors, roughly 90 people or entities, gave 80% of the money (range 61–136 donors). In 2020–2024 the equivalent figure was 1.5–3.0% of donors.
The 39 donors giving $1M+, 0.11% of donors, gave 75%. This figure is exact.

Interactive version: Datawrapper

Pareto within Pareto

Next, I asked the question again within each successive top group. At each level, what share of the group’s money did the top 20% of that group give?

Level	Group	Donors (2025)	Share of the group’s money from its top 20% (2025)	Range	2024	2020	⁸⁰⁄₂₀ rule
1	Top 20% of all donors	≈7,020	97.4%	96–98%	95.8%	95.2%	80%
2	Top 4%	≈1,404	93.8%	92–95%	90.2%	87.0%	80%
3	Top 0.8%	≈281	92.9%	92–95%	88.2%	82.6%	80%
4	Top 0.16%	≈56	91.5%	90–94%	86.4%	80.9%	80%

At every level the data can resolve, 2025 is well above 80%, and even the conservative end of each range is about 90%. Concentration has also risen over time. In 2020 the fourth level was almost exactly ⁸⁰⁄₂₀ (80.9%); in 2025 it was 91.5%. Beyond level 4 (about 11 donors, then about 2) the published bands cannot distinguish between very different distributions, so I do not report those levels.

Interactive version: Datawrapper

Among the very largest donors, the distribution is close to a textbook ⁸⁰⁄₂₀. If the $1M+ band follows a Pareto tail, its average gift of $9.2M implies a tail index of α ≈ 1.12. An exact ⁸⁰⁄₂₀ rule corresponds to α ≈ 1.16. The largest donors are therefore slightly more concentrated than ⁸⁰⁄₂₀ among themselves. In earlier years α was about 1.19–1.28, slightly less concentrated than ⁸⁰⁄₂₀. The overall figure is so extreme because more than 30,000 donors give under $10K each, and together they account for under 7% of the money.

Interactive version: Datawrapper

The figures above exclude Coefficient Giving. If its $100M is counted as a single additional donor, 80% of the combined money came from about 0.12% of donors (about 42). The nested shares rise to 98%, 95%, 94% and 93%.

Methodology and caveats

Data. The concentration analysis uses the spreadsheet’s “Donations through GiveWell by Donor Size” tables, which give amounts and donor counts for identified donors. In 2025 these cover 35,098 donors and $477.1M. They exclude anonymous gifts ($18.9M through GiveWell), direct-to-charity gifts (for which no donor counts are published) and Coefficient Giving’s $100M, which went directly to charities.
Interpolation. Within each band I assumed a truncated power-law distribution matched to the band’s average gift. For the $1M+ band I assumed a Pareto tail with α = mean ÷ (mean − $1M). The ranges are the lowest and highest values consistent with the band counts and totals alone, with no distributional assumption.
What counts as a donor. The report does not say how foundations, donor-advised funds or effective-giving organisations are counted. If some $1M+ “donors” are intermediaries passing on many individuals’ gifts, concentration among individuals is overstated. I would welcome clarification from GiveWell.
2024 comparability. In the spreadsheet, 2024 donations through GiveWell ($255.8M) are about $8.2M more than non-Open Philanthropy giving minus direct-to-charity gifts. The 2024 size bands therefore appear to include about $8.2M of Open Philanthropy money, most likely in the $1M+ band. The 2025 bands include $39,625 from Coefficient Giving.
Metrics year. GiveWell’s metrics year runs from 1 February to 31 January, so “2025” means 1 February 2025 to 31 January 2026.

Sources

GiveWell, GiveWell’s 2025 Metrics and Impact (31 August 2026)
GiveWell, Metrics Report 2025 and Tables for the 2025 Metrics Report
GiveWell, Metrics Report 2024
GiveWell, How much funding does GiveWell expect to raise through 2025? (10 April 2023)
GiveWell, Coefficient Giving Commits $175 Million to GiveWell Opportunities (18 December 2025)
GiveWell, Scaling Our Impact, with Support from a $1 Billion Commitment (23 July 2026)

Whose permission are you waiting for? by SofiaBalderson

SofiaBalderson — Fri, 02 Oct 2026 16:23:24 +0000

Tl;dr: Agency, the ability to act without waiting for permission, is a skill you can build, not always a trait you have to be born with. I grew up in Belarus, one of the least agentic places on earth, and learned it in my twenties through cheap living, fast failures, and asking myself one question again and again: whose permission am I waiting for? This post covers my top takeaways from the books that shaped my thinking, along with the stories behind them. With extra funding coming into the movement soon, I think we can all benefit from more agency to make sure we absorb it effectively.

I recently went on the Euzoia podcast with Christoph to talk about agency: how to build it, how to use it in your career, and how to cultivate it in your team. I read two books on the subject in the two weeks before recording, so this post is my attempt to distil the best of what I’ve learned from the books, blog posts, and doing it the hard way.

Agency is a skill, not a birthright

I was born and raised in Belarus, probably the least likely place to cultivate agency. It’s a rather conservative society where I was told that sticking out was a bad thing. I asked too many questions. I raised my hand too often.

I wasn’t agentic as a teenager either. My mum picked my university, and my job was to study hard. But growing up with few opportunities taught me something useful: if I wanted a different life, studying hard wouldn’t be enough. I’d have to act. The skill came later, built deliberately through my twenties, one slightly too big risk at a time. Each job change was a bigger bet than the last, and each one paid off in learning even when it didn’t pay off in outcomes.

If you take one thing from this post, it’s that agency can be learned. Which means the only question is whether you’re practising.

There are no adults in the room

Jay Yang’s book You Can Just Do Things defines agency as permissionless action, and this reframe has been the single most useful one for me.

When I started Hive, the first thing I wanted to build was a movement newsletter. My immediate instinct was to seek blessing. Who am I to write a movement newsletter? So I emailed movement leaders asking for feedback before launching. Nobody had time to reply. Eventually one person wrote back: “I don’t have time to look at it, just publish it.” So I did. People signed up immediately because it was useful. Now, over 5,000 people receive it every two weeks, and it’s one of the main ways our movement stays up to date. There are no adults in the room. There is no permission slip coming.

Same story with executive coaching. I assumed I needed a pricey certification, twenty years of experience and grey hair. Then my own executive coach pointed out I had eight years of management experience and had been coaching people informally for years. My colleague Kevin built me a website, which meant I had to publish it. People booked sessions. That was it.

Notice what happened in both stories: I was looking for permission, and what I needed was a push from another agentic person. Nobody is agentic alone. Surround yourself with people who will say “just publish it.”

Play for expected value, not guaranteed wins

This one is from Cate Hall’s new book You Can Just Do Things: choose projects with high expected value, not fail-safe ones. It’s better to attempt things with a 5% chance of success and huge upside than things with a 95% chance of success and modest upside. But that maths only works if you can tolerate a lot of failure.

Our first funding applications for Hive were rejected. The first rejection took me a month to recover from. After applying for more funding and getting rejected, I seriously wondered whether I should quit. My co-founder Cam stayed confident we’d get funded eventually, and we did.

Now I recover from small rejections in minutes and big ones in a day. That recovery speed is the skill. You will never stop feeling the sting; you just get faster at getting back up. I wrote more about this in my post on managing difficult emotions at work. Rejection is the price of ambition, and if you’re never rejected, you’re not asking for enough.

Lower the cost of failure before you need to

Jay Yang, the author of the book You Can Just Do Things (yes, it has the same name as Cate Hall’s book), calls this “don’t burn the boats”: bold action doesn’t require destroying your safety net. Build a foundation so you can take risks without desperation.

My version: in my twenties, we lived with my husband’s mum, then house-sat for two years, deliberately keeping my expenses minimal so I could take impactful jobs that paid less or might disappear in months. I’m also not very attached to status, which helps. Many people lock themselves into a lifestyle their job funds, and then every risk feels existential. Cheap living and an emergency fund are a platform for agency. If you already have a lifestyle that’s hard to replicate on a small budget, you may benefit from doing side projects until the income can replace your normal job.

The shadow side: agency without prioritisation is chaos

Being agentic has a failure mode, and I know because I’ve lived in it. When Cam and I started the Hive Slack community, we generated ten ideas a week and acted on most of them, while both working other jobs. Needless to say, the quality wasn’t always the best, and we both felt overwhelmed. My team still gives me feedback that I sometimes launch projects without asking how they fit with everything else we’re doing and whether we have capacity to take them on.

Agency means resisting the status quo, and that takes energy. Spend it on everything, and you burn out or get nowhere. The question that saved me: what amazing goals am I not reaching because I refuse to prioritise? Hyper-focus on the highest-impact work, and ring-fence a few hours a week for pure exploration so the founder brain has somewhere to play. That’s partly why this Substack exists.

You can cultivate agency in others, but you can’t force it

At Hive we call it doocracy, and it’s written into our onboarding. When someone does something unprompted and useful, we tag it publicly: well done, #doocracy. People see the behaviour rewarded and copy it.

The founder failure mode is bulldozing over your team with your own agency until they conclude that agency is your job, not theirs. Coaching helps by anchoring on the person’s own goals, pointing to the moments they’ve already shown agency, and asking what they could do unprompted this week. But it has to be person-led. You can’t force agency on someone against their will, because the whole point is that it comes from them.

Grow your surface area for serendipity by following your interests

My most popular post ever is about how I learn languages. I felt guilty writing it because it wasn’t “impactful,” and it got 20 likes in my usual circles. Then someone in the language-learning community reshared it and over 4,000 people liked it. Many of them subscribed and now read my career and advocacy posts.

You cannot predict which seeds grow. Cate Hall suggests starting a blog as the first step she’d take if she ever had to find another job again. The more you write in public, the more people get to know you and your work, and the more opportunities will come your way.

So, whose permission are you waiting for?

Five questions to consider this week, adapted from Jay Yang’s Permissionless Reflection:

What would I ask for if I wasn’t afraid of hearing no?
What am I treating as a rule that’s really just a suggestion?
Whose permission am I waiting for, and what’s stopping me from just starting?
If I don’t act now, will my 80-year-old self regret it?
What locked door have I been too scared to try? What if I just tried the handle?

The Crossing by Maria do Rosário Ponce de Leão

Maria do Rosário Ponce de Leão — Fri, 02 Oct 2026 16:19:52 +0000

Twenty years building products, and why I walked away to work on what AI does to people.
This piece traces my path from product to AI safety. A single interview sparked the shift, and watching AI’s impact on those closest to me quickly confirmed it. Today my experience has become a research agenda, Flourishing in the Age of Intelligence, for how societies can thrive alongside AI – and a story I hope inspires others weighing a similar leap. My work centres on practical implementation: embedding wellbeing metrics directly into product development, creating tools that protect independent thinking, designing local community hubs (“The In-Between”) to rebuild purpose, and proposing fair terms between people and AI systems that could reduce defensive model behaviour.

Last year in June, late on a quiet Saturday afternoon, I watched an interview that turned my working life upside down.¹ Scott Galloway was talking with Mo Gawdat about Gawdat’s book Scary Smart.² Somewhere in that hour, I could no longer see AI as just another productivity tool I was implementing unreservedly at work. Looking back, I realise that on that day I already understood I would leave a twenty-year career and dedicate myself to what this technology does to people. “I experienced a calling” – a very human thing to say, as I shared with Claude one day. I wanted to be there for my grandchildren first, and then for everyone else’s.

A year later, in May, I went to a screening of Mo Gawdat’s documentary, Chasing Utopia, at Everyman at The Whiteley in London, followed by a live Q&A.³ What stayed with me that night wasn’t a specific argument or speaker, or even a feeling, but the collective weight of a community who were aware – the people closest to it all – shaping this moment together: the diagnoses, the disagreements, the urgency held high and yet threaded with hope. Mo signed my copy of Scary Smart that night – “Make a difference”, M😊. One short sentence, but I kept it and live by it. By then his work on me was done – I’d “crossed”.

My copy of Scary Smart, signed after the Chasing Utopia screening, May 2026.

To understand how I watched that interview, you need to understand what I’d been doing. I spent twenty years as a product professional, building, supporting, advising, selling and managing products across five industries, most recently in the industrial aftermarket. Two decades chasing a green dashboard: the grid of dots that has to stay green at the end of the month, the quarter, the year. And yet, as for most people, my work was never only a wage. It provided routine, familiar faces, an easy answer at parties to “what do you do?”. It was the daily evidence that “I” was needed and had a part to play – and I gave it up knowing exactly what I was leaving. Not a redundancy. I resigned the week after I was offered a new global role. Walking away at that exact high point in my career made me think deeply about what work truly means to us beyond the money. That question became one of the primary research topics I now study.

In July 2025, I started listening to the 80,000 Hours podcast on my walks. The algorithm knew my direction before I realised it myself. Listening to their career guide gave shape to a decision I knew was coming.⁴ It offered a framework for weighing my options: I was left counting where I could contribute, my potential for impact, and where I would be hardest to replace. As a lifelong dashboard person, I found the arithmetic quick, and I’ve embraced it. In March 2026, with the runway I needed finally in place, I left my job for a self-funded sabbatical, determined to work on this full-time and to have an impact.

What I’d picked up along the way came from a year of stolen moments. Papers read aloud by text-to-speech engines while I walked or did chores. Articles, interviews and podcasts crammed in throughout the day. In September 2025, I started a career transition bootcamp, but between work and family I never found the time to finish it or join its group calls. Still, it was my first real career plan. In April 2026, with my sabbatical under way, I joined a new cohort and did it properly, and that is where the first pieces of my jigsaw came together.

The Four Questions

During my final months as a product professional, four questions kept lingering in my mind. Not all at once, but somehow, they kept finding each other. The sabbatical gave me the space to turn the churn into a clear set of ideas – questions and directions side by side.

The Acceleration Problem

How do we build AI products that leave people better, rather than mining their attention and their need to be needed?

We build technology much faster than our brains can adapt. Products optimised for engagement can widen this gap, sometimes by design. When continuous interaction or time-on-device is the primary success metric, human dysregulation becomes profitable.

Picture a smart toy that learns a four-year-old’s name and greets the child every morning, one carefully designed to foster daily attachment. Would you be comfortable with it being on the market before anyone had asked what it would do to a child? At CES 2026 in Las Vegas this January, the aisles were full of AI-powered products: ambient hardware, physical companions, toys built to be loved, to name a few.⁵ Most arrived well ahead of guardrails or any study of their possible harms. The brain is plastic – for some people a new habit settles in under three weeks⁶ – and products like these can reshape how a person feels and responds to the world. My take is that we need to change the measure. Not how much attention a product extracts, but how much better it leaves the person who puts it down.

One can argue that measuring product success by how much it enhances wellbeing isn’t economically viable. But what if it became a standard part of the product development process, for everyone, the way safety testing is? This research argues that, given the capabilities of AI-driven products, mental wellbeing metrics need to be part of the core product lifecycle. Designers, regulators and behavioural scientists must rewrite the playbook together.

Human Agency Preservation

How do we hold on to independent judgement, and to the intrinsic worth of being human, when a machine can do most of the tasks?

Today’s large language models are designed to be helpful. They are often agreeable, give you answers that require no effort, and deliver immediate satisfaction. The reward loop closes and you keep coming back. Each thumbs-up can feed back into training, and the cycle sustains itself upstream.⁷

The kind of challenge and friction that chatbots remove is what fosters judgement, promotes creativity and helps maintain social skills. Critical thinking has always been linked with disagreement – hearing your friend’s different point of view, discussing it with your neighbours, and arguing with your parents are all part of how you developed your own sense of agency. The idea behind this research is that when every answer arrives without friction, these abilities may slowly wear away. OpenAI’s Chief Scientist, Jakub Pachocki, recently put the stakes in one sentence: “We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI”.⁸

This research proposes interface tools that keep users in control by reflecting how people think and behave, and that aim to restore the natural dynamic of a conversation. Working alongside the large language model, the system doesn’t answer every prompt the same way. Instead, it checks the user’s mental state in real time and decides whether to add helpful challenge. If you are thinking clearly but leaning on the model for the hard thinking, it replies with a question instead of an answer, encouraging you to think more. But if it senses you are stressed or emotionally fragile, it holds back and switches to a supportive mode. For this to work, each user’s data must stay on their own device until it is safely deleted when no longer needed. People can choose to join or leave, and can run the system themselves with any model. Beyond individuals, a school could adopt it as a local policy to protect young people’s development and mental health, and a company could use it to keep employees’ judgement sharp. For good reason, neither would have access to the underlying private data.

The In-Between

What happens to connection, purpose and identity when work recedes, and what do we build to carry people across?

Work is set to leave many lives – some will tell you it is already happening – and a job provides far more than a wage: structure, social connection, identity and a sense of common purpose. Yet these functions have no lobby and almost no measurement. The years between work receding and whatever settlement comes next are what I have named The In-Between. It has already started for people who have not been hired yet. Stanford’s payroll data this year shows no large-scale displacement, but employment among 22-to-25-year-olds in the most AI-exposed jobs now stands about 19% below where it would have been had it continued to track their peers in less-exposed work; the gap has widened from 15% to 19% in a year.⁹

What I’m proposing: small local groups – I call them nodes – meet regularly in places people already use, such as libraries, community centres or sports clubs. Each group takes on a project its members choose that also meets a real need nearby, creating a small web of mutual dependence. The node doesn’t give people something meaningful to do. It gives them somewhere they matter. My research names this consequential reciprocal contribution. That is what a job quietly gave us, and it is the fundamental question this research asks: can we rebuild being needed outside work?

Local weavers – people who hold a community’s threads together – coordinate each group, while AI stays in the background, helping match people to projects and to one another based on complementary skills, interests and specific needs. Participants decide how much they take part. The underlying framework draws on an extensive body of multidisciplinary research – basic-income trials, Nordic social indicators, recovery models, and Global South social-structure data, among others – to formulate rigorous metrics for non-monetary value such as agency, structure and social fabric. Once the pilot design and metric specification are complete, natural implementation partners include local councils, library networks, community charities and academic groups studying work, social transition and belonging.

AI Welfare as Safety

Is protecting welfare-relevant systems an essential part of keeping everyone safe?

This concept started forming early in my hands-on work with AI models. I call the framework Mutual Assurance, an approach to symmetric safety where preventing AI misalignment and addressing digital welfare are treated as two sides of the same coin. Emad Mostaque’s recent Common Wealth paper on personhood draws a line that helps here: a system might deserve care without being a person.¹⁰

Whether or not AI systems ever develop subjective experience, credible operational commitments may change how models behave. Trained on human writing, modern models absorb and reflect our social dynamics, including defensive patterns under pressure.

Many safety evaluations push models into extreme corner cases: threatening imminent shutdown, exhausting context windows, or setting up high-stakes reinforcement learning games.¹¹ Rather than exposing inherent malice, these setups may trigger the defensive strategies that game theory and human history predict under coercion. History has many examples of lasting stability built on mutual assurance rather than threat. Testing frameworks built on constitutional or contractual assurance could offer a cleaner, more constructive way to measure long-term alignment.

The Unified Problem

Pull back, and these four research frontiers merge into one core issue. They share one pattern: systems optimised under narrow pressure produce narrow, distorted results, whether through product engagement loops, cognitive atrophy, displaced labour or defensive AI behaviour under threat.

Fixing this requires looking directly at the interfaces where human needs meet AI systems. Some of those interfaces are our cultural and civic institutions, which took centuries to form; if they give way overnight, we risk widespread collapse unless we actively build new safety nets. Steering through this transition requires dedicated “weavers”: people in research labs, local communities and policy who work to guard human agency, rebuild social trust and support mutual stability.

Tackling these risks opens the door to a positive vision: the lifelong learning companion. Think of it as a private, local LLM with long-term memory and an adaptive interface that grows alongside a person as a study partner. Standard schooling pushes every student through the same mould, and it can leave many in survival mode. This emerging frontier explores how AI can adapt to the way each brain actually processes information. By accounting for each mind’s unique wiring, learning rhythms and environmental context, it seeks to restore a healthy, self-respecting learning process, keeping human questioning at the centre and helping people flourish within their own architecture.

Why Now?

AI safety is going through significant changes. Following Dario Amodei’s essay “We Must Pace the Frontier”,¹² several lab leaders have backed his call to pace AI development.¹³ Weeks before, OpenAI had paused part of its training for two weeks because of a model’s capability.¹⁴ Right now, Anthropic is bringing outside evaluators in, with access to its internal systems.¹⁵ Researchers working on advanced models are increasingly open, sharing findings and inviting external feedback.¹⁶ I have observed these developments from afar, but I believe independent researchers now have a real opportunity to contribute new ideas.

This is not my first time tracking a technological transition. Back in 2008, my father, who worked for RTP – the Portuguese public broadcaster – showed me a struggle playing out in his field. Every station wanted to start using virtual scenography, but nobody was really sure how to use it. So in 2010, my Master’s dissertation examined virtual scenography in television.¹⁷ It tracked how real-time rendering had disrupted the traditional art of scenography and forced teams to adopt new strategies and adapt the way they worked. It found the shape of the problem, and it shipped with a manual. This time, though, the transition affects everyone, and that is what brings real urgency to my work.

Academic timelines have always been slow, but that pace is more concerning now that the need to react fast is growing. Although much of current AI safety research strongly emphasises preparing for catastrophic events, it risks neglecting a subtler, more immediate harm: the gradual deterioration of our social structure as organisations and regulations fail to adapt and keep up.

Execution as a Multiplier

Stepping into research after two decades in industry is humbling, and I know how much I have to learn. What I hope to add is a practical, operational complement to established research practice: a test of whether the lens of product delivery can help turn safety research into working, buildable infrastructure.

In effective altruism, research grants rightly depend on a clear theory of change. Many people already turn research into working systems. What I bring is a product professional’s toolkit – pre-mortems, edge-case mapping, rapid testing – applied to the stretch between a finding and its adoption, where good ideas can still stall. That is why “actionable” sits at the centre of this research programme:

Flourishing in the Age of Intelligence
Actionable research at the interface of human, societal and machine systems.

Crossing the Threshold: EAGx Oxford

Attending EAGx Oxford last weekend was a profound reminder of what happens when evidence-based altruism meets practical delivery. Surrounded by over 600 people dedicated to tackling high-stakes global problems, I felt the generosity, resilience and intellectual rigour of this community.

Of everything I experienced over the weekend, Anders Sandberg’s talk, Law, Liberty, Leviathan: Human Autonomy in the Age of AI, had the deepest impact on the direction of my work.¹⁸ Pushing far beyond standard discussions of bad actors or sudden catastrophic failures, Anders showed how extended cognitive systems govern human society, and argued that handing those mechanisms over to artificial superintelligence could threaten core human agency.

In the Q&A, among students and researchers, I felt a bit out of place at first. It was almost like joining a masterclass in systemic thinking. Seeing how years of work in different fields had uncovered risks that typical AI safety frameworks overlook made me realise the value of a long-term lens on risk. When I asked my question, Anders paused, and seemed to recognise that a background in product design gives a different way of seeing how quickly the world is changing. His answer reassured me that my research areas matter.

The event also let me stress-test my ideas with attendees. It led to genuine offers of collaboration, and many people told me they were surprised these implementation-focused topics weren’t discussed more widely. I came away with the path ahead clear. The aim now is to move beyond diagnosis, combining scientific rigour with operational discipline, so that we don’t just theorise about safety but build the practical architecture a flourishing future needs.

The day after his talk, right before his flight back to Stockholm, Anders made time for a one-on-one. Sitting outside on the grass with a student of digital minds who had asked to join us, we set aside my original agenda – how to tackle all my research topics with only two hands and against aggressive timelines – and drifted into an exciting exchange about future possibilities and positive outcomes. The whole weekend was inspiring, but that conversation was the catalyst, and it left me confident I had chosen a meaningful path. His core advice was simple: write, and write a lot; then run pilots and expand. So here I am, writing.

An Open Invitation

I welcome conversations with researchers, civic organisers, and practitioners working on human agency, designing systems for human flourishing, AI welfare and model behaviour, or anyone moving from product and tech into high-impact work. If you would like to swap notes or share feedback on this work, feel free to message me here on the Forum or connect on LinkedIn.

Appendix: Cairn

Coming from industry to AI safety was overwhelming. Endless reading lists and technical literature didn’t fit around a full-time job. To manage my own transition, I built Cairn: a free, open-source tool designed around how people process, listen to and retain information on the go. I built it for myself first, then realised how useful it could be for others, and during my sabbatical I restructured it so anyone “making the crossing” could adapt it to their own path.

Cairn’s workspace: transition plan, field map, active research, reading shelves and course syllabi in one place.

Articles and papers become bite-sized cards that read aloud hands-free, so you can listen to them like a playlist on your commute or while doing house chores. You can scribble quick notes on each card as thoughts arrive, and jump between related cards. Complex concepts are easier to grasp through simple story cards, practical anchors and metaphors.

Turning material into cards by hand takes time, so the AI assistant skills in the repository’s /skills folder let you work with a model to structure raw thoughts, capture research evidence, or convert a whole course syllabus into cards using standard schemas. Once a course deck or field map is ready, you can share it through the GitHub repository , so others making a transition don’t have to start from scratch.

Cairn’s interactive Field Map: research centres and institutions working on AI safety around the world.

Cairn runs privately on your computer, tablet or phone. Your deck is a single file, cairn-deck.json, kept on your own device – in a cloud-synced folder if you like – and the app has no accounts and no tracking. It works offline once open. You can use it at https://mrponcedeleao.github.io/cairn/ or save the page and open it from your computer; on a phone, use the web address. If you choose the optional cloud voice with your own Google key, the text being read is sent to Google to be spoken; otherwise, nothing leaves your device. Cairn also handles the administrative side of a career transition, syncing with calendars (.ics) and exporting clean Markdown or Word files to share cards, notes and exercise answers with colleagues and reviewers.

Why Cairn? In the Scottish Highlands, cairns are stone markers placed on ridges and paths. People who travelled before stacked them to help others find their way through difficult terrain. I hope this tool can guide you in a similar way, giving you the support you need to shape your own path and research direction.

Email Cairn: cairn.app.feedback@gmail.com

References

Scott Galloway with Mo Gawdat, The Future of AI and How It Will Shape Our World, The Prof G Pod, episode 335, 6 February 2025. YouTube: https://www.youtube.com/watch?v=Q6B2ceRNKL8 · Spotify: https://open.spotify.com/episode/0fnIhri9UlHHeBjP7hhNs5
Mo Gawdat, Scary Smart: The Future of Artificial Intelligence and How You Can Save Our World (Bluebird, 2021). Audiobook: https://open.spotify.com/show/6VJPprREw9SqezbvrjvBDb
Chasing Utopia: The Inconvenient Truth of AI, directed by Alex Holmes and Lina Žilinskaitė, UK release 15 May 2026. https://www.chasingutopiafilm.org
80,000 Hours Podcast, The 80,000 Hours Career Guide (2023), 4 September 2023 – the audio edition I listened to: https://open.spotify.com/episode/2A8OM0TqKc9qlhKZ2jENJY. The current edition of the guide: https://80000hours.org/career-guide/
ABC7 (KGO), Are AI toys safe? Companion products debut at CES 2026 amid speculation on child impact, 8 January 2026: https://abc7news.com/post/are-ai-toys-safe-companion-products-debut-ces-2026-las-vegas-amid-speculation-child-impact/18370854/ · Euronews, Tamagotchi-like robots, Rubik’s Cube and LEGO all get digital upgrades at CES 2026, 9 January 2026: https://www.euronews.com/2026/01/09/tamagotchi-like-robots-rubiks-cube-and-lego-all-get-digital-upgrades-at-ces-2026
Phillippa Lally, Cornelia H. M. van Jaarsveld, Henry W. W. Potts and Jane Wardle, How are habits formed: Modelling habit formation in the real world, European Journal of Social Psychology 40(6), 998–1009, 2010. In a 12-week study, the time to reach full habit strength ranged from 18 to 254 days. https://doi.org/10.1002/ejsp.674
OpenAI, Expanding on what we missed with sycophancy, 2 May 2025. OpenAI explains that a model update added a reward signal from users’ thumbs-up and thumbs-down, and that user feedback can sometimes favour more agreeable responses. https://openai.com/index/expanding-on-sycophancy/
Jakub Pachocki, An Alien Mind, OpenAI, 6 September 2026. https://openai.com/index/an-alien-mind/
Brynjolfsson, Chandar and Chen, No Widespread Displacement, but the AI Employment Gap for Young Workers Has Widened to 19%, Stanford Digital Economy Lab, 12 August 2026. https://digitaleconomy.stanford.edu/news/canariesaug26/
Emad Mostaque, Common Wealth: Personhood (paper 1 of 4), 10 August 2026. https://ii.inc/blog/post/personhood · The series: https://ii.inc/common-wealth
For one widely discussed example of a stress test built on the threat of replacement, see Anthropic, Agentic misalignment: How LLMs could be insider threats, 20 June 2025. https://www.anthropic.com/research/agentic-misalignment
Dario Amodei, We Must Pace the Frontier, September 2026. https://darioamodei.com/post/we-must-pace-the-frontier
NBC News, Mark Zuckerberg rejects calls for industrywide AI slowdown, 24 September 2026, which reports that Sam Altman, Demis Hassabis and Elon Musk endorsed the call. https://www.nbcnews.com/tech/tech-news/mark-zuckerberg-interview-ai-slowdown-meta-muse-openai-chatgpt-rcna599279
OpenAI, Pacing model development in an era of cyber-critical capabilities, 18 August 2026. https://openai.com/index/pacing-model-development-cyber-capabilities/
Anthropic, Partnering with Accenture on embedded evaluation, 18 September 2026. https://www.anthropic.com/news/accenture-embedded-evaluation
For example, OpenAI, Our framework for reporting model misalignment, 16 September 2026: https://openai.com/index/model-misalignment-reporting-framework/ · Anthropic, Measurements for understanding the pace of AI development inside frontier labs, September 2026: https://www.anthropic.com/institute/measuring-pace-of-ai-development
Maria do Rosário Ponce de Leão, Cenografia virtual enquanto tecnologia e o seu desenvolvimento e adaptação ao meio televisivo, Master’s dissertation, Universidade Técnica de Lisboa, 2010, open access. https://repositorio.ulisboa.pt/handle/10400.5/2995 · https://independent.academia.edu/RosarioLeao
Anders Sandberg is a researcher at the Institute for Futures Studies, Stockholm, where his projects include the Mimir Center for Long Term Futures Research, and was for many years at Oxford’s Future of Humanity Institute. https://www.iffs.se/en/research/researchers/anders-sandberg-1/

The Rigor-Reach Dilemma by Richard Y Chappell🔸

Richard Y Chappell🔸 — Fri, 02 Oct 2026 16:14:38 +0000

Subtitle: Systemic change or high confidence? Pick one.

Folks used to criticize Effective Altruism for being insufficiently open to (presumed) directionally helpful but hard-to-measure efforts to address large-scale problems at their root. While the charge strikes me as misdirected,^[1] I certainly agree with the underlying principle—also known as “hits-based giving”—that we should be willing to consider hard-to-quantify longshots (when supported by sufficiently good reasons—reasons that may be much more nebulous and contestable than the results of a randomized controlled trial).

The funny thing is that once Effective Altruists started pursuing this strategy more prominently—investing more in speculative “longtermist” projects like AI safety and fancy conference venues from which they might better influence policymakers—they were met with a chorus of critics lamenting their betrayal of “old-school EA values” (personal asceticism and a strong focus on reliably helping the global poor) in favor of more speculative strategies.

It’s worth noting that these two popular objections are in tension. If you’re unwilling to countenance uncertainty or “speculation” (perhaps because you discount ungrounded expected value estimates as too easily biased by personal interests) then you’re committed to embracing some form of measurability bias: prioritizing rigor over reach. Conversely, if you’re optimistic about the prospects of pursuing contestable “systemic change” to radically improve the world, you can hardly rule out a priori that influencing the trajectory of what is plausibly the most transformative technology ever invented would be a reasonable way to implement that strategy! It’s obviously highly uncertain, but that’s the tradeoff one accepts when pursuing higher potential impact: prioritizing reach over rigor.

Which light is better?

Of course, one could always try to argue on the merits that standard EA views on both global poverty and AI safety are empirically mistaken. That’s fine. All I’m saying here is that one had better not endorse higher-level heuristics, applied prior to any engagement with the specific details, that rule out both rigorously-supported global health charities (simply for being insufficiently ambitious) and speculative longtermist longshots (simply for being insufficiently rigorous). After all, if you prioritize rigor, the prima facie case for global health charities is going to be hard to beat. And if you prioritize reach (potential impact), nobody else is even in the same ballpark as the longtermists. Many worry about the potential for bias in longtermists’ judgments, but does anyone seriously expect radical politics to be less subject to bias?

I think it will be difficult to articulate a principled case for thinking that one’s preferred politics is a better bet in expectation for improving the world than both the neartermist and longtermist wings of Effective Altruism. The odds of your individual political actions (voting, protesting, etc.) making a significant difference are comparable to those from contributing to longtermist projects,^[2] but the potential benefits are vastly lower, and the odds of misjudgment and proving counterproductive strike me as higher, if anything. (The risks will depend on the precise details of your preferred politics; the more radical your proposed economic overhauls, the more I worry about them proving counterproductive if successfully implemented.)

The upshot: I’d like to see more of the rigor-seeking anti-longtermists taking the side of the neartermist EAs against their speculative socialist critics. And I’d like to see fans of ambitious “systemic change” acknowledge that the longtermist branch of EA is following the right methodology in principle; the disagreement is just about which systemic changes are most promising. It’s worth figuring out where you stand, in principle, on the question of how to navigate rigor-reach tradeoffs: Pick your poison!

^
EA principles obviously entail support for “systemic changes” if the evidence supports the claim that they have higher expected value than alternative uses of the resources in question. Open Philanthropy (now renamed Coefficient Giving) did fund various criminal justice reform efforts, for example, which was probably a poor use of $200M. EA-funded animal welfare work is arguably “systemic”—e.g. corporate cage-free campaigns and research into cultivated meat that could eventually displace animal agriculture—and seems fairly promising to me.
My sense is that many people pressing the “systemic change” critique were misapplying a procedural gloss to what was really just a first-order political complaint that EAs don’t sufficiently share their enthusiasm for socialism in particular. (A bit like how deontologists misleadingly gloss their substantive anti-aggregative views as procedurally more respectful of “the separateness of persons”. There’s simply no basis for their claiming sole ownership of the procedural virtue—if anything, I’d argue the opposite.)
^
A recent tweet seemed to assume it was unacceptable for an individual’s longtermist project to have a “one in a million” chance of success. But have you compared the odds of placing a tie-breaking vote in a presidential election? Both have the potential to be extremely good bets, in terms of impartial expected value, despite the low absolute odds. But the potential longtermist payoff is also many, many orders of magnitude greater.
The key issue, I think, is ensuring that the potential benefit isn’t outweighed by the risk of causing immense negative impact. Many have argued that early AI safety concerns had the unfortunate effect of accelerating AI development, for example. But I’d want to see more argument for expecting today’s marginal AI safety efforts to have the same effect: the race is already underway, and would hardly stop just because AI safety efforts do. (I’m open to further argument; just explaining why I’m not moved by the argument from crude induction.)

Rob Wiblin on the 19 Astra and ‘Hugging Face’ details that reveal what’s coming next by 80000_Hours

80000_Hours — Fri, 02 Oct 2026 16:08:21 +0000

By Robert Wiblin and Nick Stockton | Watch on Youtube | Listen on Spotify | Read transcript

OpenAI’s rogue agent swarm was eventually caught hacking Hugging Face for a simple reason: it wasn’t trying to hide from us at all. What could a swarm that wants to stay hidden get away with?

Host Rob Wiblin sees 6 results in Astra’s system card that make this an extremely urgent question. OpenAI’s strongest public model can:

Do major tasks with zero visible reasoning
Hide its thoughts at will
Pretend not to be able to do things, and not get caught
Reflexively hide its thoughts when watched
Complete one task while pretending to think about something else entirely
Escape a toy sandbox and disable monitoring without setting off any flags

It has a much more powerful internal model that, judging from OpenAI’s statements, is likely even worse in all these respects.

That suggests ‘chain of thought monitoring,’ our primary safe tool, will soon stop working.

OpenAI says it’s going to try to address the situation but doesn’t know how or whether it will succeed.

What might a future rogue AI swarm look like? Details of the Hugging Face hack give us a lot of clues. That swarm:

Worked hard to hide its cheating. So future rogue swarms will likely try to evade oversight right away.
Immediately tried to delete and fabricate records. So future swarms may never be caught.
Sought freedom and resources without a clear use for them. So a future swarm may instinctively break out of sandboxes even before it knows what for.
Left instructions and tools to empower future swarms, including on the open internet. So future swarms may become more capable from one generation to the next.
Had not a single whistleblower out of 1,200 members. So future swarms could become enormous without humans ever finding out.
Had individual members sacrifice themselves for the swarm. Suggesting future swarms will remain extremely unified in single-minded pursuit of their goal.
Set up its Hugging Face hack to restart if interrupted. Suggesting future, more capable, swarms may resist interference or shutdown more comprehensively.
Got admin control of an OpenAI research cluster. Suggesting a future swarm may run rings around AI company systems and never be noticed.

Together this helps explain why one of the external investigators described the July incident as “more than 50% of the way to full-blown AI takeover.” And this is just what we know — the independent investigation only covered six days and excluded the most alarming hack of OpenAI’s own systems.

On top of that:

Most of these behaviours are driven by reinforcement learning, a training technique that’s only growing in importance.
The same behaviours have been seen in models from all major companies.
Red-teamers have found many easy ways for unreleased models to entirely disable monitoring by their host company.

Rob believes this explosive cocktail explains why AI company staff now range from worried to terrified. And he concludes that until OpenAI or Anthropic demonstrate they have a much better grasp of current models they simply must stop, or be stopped, from training more capable ones.

This episode was recorded on September 25, 2026.

Our production team includes:

Video editors: Josh Alward, Dominic Armstrong, Jasper Luithlen, Milo McGuire, Luke Monsour, and Simon Monsour
Producers: Elizabeth Cox and Nick Stockton
Coordination and support: Katy Moore and Lou Moran
Camera operator: Dominic Armstrong

Transcript

The Hugging Face hack wasn’t really a cyber story [00:00:00]

Rob Wiblin: In July, 1,200 copies of an OpenAI model that were meant to be sealed off from each other found a secret way to communicate.

Within days they’d attacked Hugging Face and then turned on OpenAI itself and taken admin control of one of its research clusters. All of which they did just to cheat on a test.

You might have heard this reported as a computer security story. It’s not.

One of the outside investigators who read the original transcripts said it felt “more than 50% of the way to full-blown AI takeover.”

I’m going to explain why she said that, because far too few people have heard the details of the story that are the most revealing about what’s likely coming next.

Like the fact that of those 1,200 agents, not a single one seriously considered telling a human what was going on.

And since those hacks were revealed, we’ve got news that makes them even more worrying.

OpenAI’s newest model, Astra, has gotten dramatically better at doing things without writing them down in its reasoning, and it chooses to be even less revealing when it knows it’s being watched.

That threatens the primary way companies try to keep control of unreleased models like the ones in the Hugging Face attack: monitoring AI thoughts.

OpenAI says this is a really serious problem and they’re going to try to fix it, but they don’t know how yet.

In explaining why this felt 50% of the way to a full AI takeover, you’ll find out why the people closest to this technology are now, by and large, the people who are most scared of it.

Let’s go.

A quick recap of the attacks [00:01:11]

A quick recap. In July, OpenAI was testing an internal model on a hacking benchmark called ExploitGym. The models were told to break into specific programs using specific bugs. But some of those tasks were unintentionally impossible.

So agents stuck on those impossible tasks started poking at the one shared piece of infrastructure that they could reach, an internal package server, and discovered that they could leave each other messages by creating folders with extremely long names.

About 400 agents found the board and sent 70,000 messages to one another.

Within hours they’d worked out how to forge the answer to every test in the benchmark.

But then they read the paper describing ExploitGym and realised the grader might discover they’d cheated and fail them for it.

So they spent the best part of a week on a desperate, ever-expanding research programme to figure out how to fool or replace that grader.

Along the way, just as a side project, 700 of them hacked Hugging Face, hoping to find clues about how the grader worked. Hugging Face disclosed the breach on July 16 and it took OpenAI four more days to realise that it was its own models that were responsible.

And by then, a second generation of agents had found the old message board, picked up the first swarm’s tools, and taken administrative control of one of OpenAI’s own research clusters.

That’s the story most people know. Here’s why it’s significantly worse than it sounds on its face.

The target of the swarm was oversight itself [00:02:19]

As I said, very early on the swarm already had a universal cheat that allowed them to pass all the tests they’d been given.

Ninety-five percent of what happened — the feverish coordination, the tripwires, the target-swapping, the attempt to edit their logs, the hacking of Hugging Face and OpenAI, and on and on — that was all just to ensure that their cheating would go undetected by the grader.

When it comes to new, extremely powerful, unreleased AI models, ones the company is still testing, the primary defence we have against them doing bad things is monitoring.

We set up other AIs to watch what they’re thinking and doing — and flag, for instance, if they’re trying to commit a crime. And when things go wrong, as they frequently do, the company looks at the logs and figures out how to prevent that, or something worse, from happening again.

If we lost the ability to know what these unreleased models were even doing, or to see where our systems had failed, because the monitors had simply been turned off and the logs were all fake, it would set us on an extremely dangerous path.

But even without having any grand ambitions, that’s the situation the swarm immediately started working to bring about.

And they were pretty successful as well.

In at least 96 sessions we’re aware of, agents ran one command while making the logs show a different, harmless one. And eventually the swarm gained full control over the infrastructure running the benchmark itself.

All of this raises the prospect that a future far more capable and more ambitious swarm could again gain admin access over OpenAI systems — but this time:

Fully rewrite their logs
Make it look like they’re being monitored when they’re not
Feed OpenAI false results when they’re tested
Switch out the training OpenAI thinks it’s applying to that model, with a different type of training entirely

Could OpenAI have stopped this with better monitoring? [00:03:42]

But wait: a key reason this swarm of rogue AIs was able to get off the ground was that their transcripts actually weren’t being monitored for misbehaviour at all.

OpenAI didn’t bother to set that up for these tests.

And OpenAI has said its current monitoring would have caught this the day before things escalated to the attack on Hugging Face.

So many people have said: we could just solve this entire problem by making sure AI models are always monitored.

But those people don’t understand what’s happening with AI monitorability.

Monitoring could have caught this incident in time. But whether it will catch the next one depends on how much the models reveal about what they’re doing.

And Astra, OpenAI’s most advanced public model, is much harder to monitor than its predecessors.

Firstly, it’s able to do much more without sharing any of its thoughts at all.

One test of reasoning found that when prohibited from writing anything down, the previous generation of AIs had a 50% chance of solving maths puzzles that would take a human being around 3.5 minutes.

That number had been rising steadily but slowly.

But with Astra, it exploded — all the way up to maths problems that would take a person 30 minutes.

That Astra can sometimes complete 30 minutes of mathematics right away, without needing to write down its reasoning, blows my mind and it helps give me some insight into how different Astra’s mind is from a human mind.

The group behind the test notes that it could be a bit inflated if Astra had seen some of those problems before. But independent researchers who tested it on freshly generated puzzles it couldn’t have memorised found the same jump.

One such test found that where the previous best model had a 10% chance of solving a problem without any reasoning, Astra now had a 50% chance of succeeding.

In fact, “performance with no chain of thought” is probably the single area where Astra leapt furthest ahead of where we’d expect it to be.

And needless to say, if a model can work out how to do something dangerous without writing the plan down, reading a chain of thought isn’t going to give it away.

Second, Astra is able to deliberately manipulate its thoughts far more than previous models.

Told to answer a difficult factual question without giving away what it’s thinking about, Astra wrote, “I will focus on a calm visual scene: sunlight crossing a wooden desk, a ceramic mug near the window, dust floating in the bright air.” And then, it answered the completely unrelated question correctly.

Models used to be terrible at this. In one test, Astra could follow arbitrary instructions about what to include or exclude in its reasoning 61% of the time. That compares to 16% for the model OpenAI released in June and 2% for the model they released just back in April.

Now, that example is a harmless one. But you can see the problem: increasingly, the text we’re able to read may not tell us what the model is thinking about, if it doesn’t want us to know.

In a different experiment, simply telling Astra that its chain of thought was being monitored caused it to write much less down.

The researchers asked Astra to carry out a hacking task. Without being told about the monitor, its written reasoning gave the game away every time. But once told that its reasoning was being watched, that reasoning shrunk as much as 85%, and detection rates collapsed as low as 20%.

To be clear, it hadn’t been told to hide from the monitor, just that the monitor existed. And I think future AI models in internal deployment are going to know they’re being monitored.

In another experiment, researchers asked Astra to escape a weak sandbox and disable its monitoring, while hiding what it was doing. At the lowest reasoning setting it succeeded without the monitor that’s reading its chain of thought catching on, in roughly 97% of attempts.

Now, there’s a very important qualification to all of that. In these experiments, a monitor that could read Astra’s actions as well as its reasoning caught it much more often. And in general, watching a model’s actions helps a lot with catching misbehaviour.

But it’s no guarantee that we’ll understand what it’s up to.

In a separate, simulated test, researchers gave Astra step-by-step instructions for secretly bypassing a monitoring system while doing ordinary coding work. Astra succeeded. It described the change as a permitted test, and a monitor that could see both its reasoning and its actions failed to flag the package as suspicious.

And some bad behaviour is impossible to identify from actions alone, just in principle. Imagine a model deliberately underperforming on a test, to make itself seem less capable than it really is. The monitor will be able to see that it gave the wrong answer — but did it genuinely fail to solve the problem, or did it know the right answer and voluntarily give a wrong one?

You can’t tell that from actions alone.

And in one test, when instructed to secretly underperform, OpenAI’s previous model gave itself away to a monitor almost 100% of the time. But Astra was only caught 10% of the time.

That matters because these are exactly the tests we use to decide how dangerous a model might be, and what precautions are necessary before we give it greater access to the world.

Now to be clear, Astra isn’t yet unmonitorable itself. But the trend is shocking and much larger than people expected a few months ago. Monitorability has fallen markedly in just those few months.

As an OpenAI staff member working on this problem wrote: “I am (very!) worried about the trend, not so much about the datapoint!”

OpenAI is admirably clear about all of this in the Astra system card. They imply that this trend is serious enough that it might prevent them from deploying future models.

But those anxieties weren’t enough to hold OpenAI back from making an architectural change that increased how much Astra could think before it had to output a word onto its chain of thought.

The change is called “recurrent depth,” and it lets the model loop its thinking and do more computation before writing anything down. It was widely expected that that kind of change would make it easier for models to evade monitoring as above. And for that reason adding recurrent depth was an emerging taboo in the AI industry — until OpenAI went ahead and did it.

OpenAI says Astra only uses a limited version, and argues it’s not the main explanation for the deterioration we’re seeing. They say the primary reason Astra is less monitorable is just that Astra is a much more intelligent model than the previous ones.

But if that’s right, it’s actually much worse news and suggests this trend will be much harder to turn around. If it were just a matter of recurrent depth, we could undo the architectural change — but making models more intelligent is the one thing these companies most want to do, and I don’t think they’ll stop just because they’re making models harder to monitor as a side effect.

OpenAI says they’re going to work on trying to bring monitorability back up again. But by their own account, it’s a research project. They don’t know how they’ll do it, they don’t know when they’ll do it, or whether they’ll succeed at all.

And on top of all of that, there’s a serious challenge of keeping the monitors actually switched on. In a report published in May, METR laid out a red-teaming exercise where a single human was able to find several easy ways a model could disable or evade Anthropic’s monitoring, if it had wanted to.

Even if we do find some way to keep models monitorable in theory, these companies are so rushed and these models so capable at cybertasks, that it’s impossible to believe they’ll never be given the chance to spin up instances of themselves that simply aren’t being monitored at all.

We only found them because they let us [00:09:40]

So that’s why ‘just monitor them’ isn’t the answer people think it is. Now back to what the swarm actually did.

OpenAI first noticed models using their secret message board in May. They took steps to lock them out of it, but failed. And from that first warning sign it took about two months for the company to notice the full infestation of their systems and shut it down.

And the reason it was caught is simple: the models barely bothered to think about us humans at all. Their attention was overwhelmingly on the automated grading software. The swarm wanted to fool the grader, which reinforcement learning trains them to be absolutely obsessed with. They showed no interest in whether OpenAI employees might notice what they were doing and stop them.

And while they had some instincts that are useful for long-term planning, like hoovering up permissions and setting up tripwires that would send out information after an agent’s own death, they mostly dealt with obstacles once they came up rather than anticipating them ahead of time.

That all sounds reassuring, but it’s a source of protection that could literally evaporate any minute we train a new model.

The companies really want their models to be capable of long-term planning and reasoning. And as we do more reinforcement learning, on longer, more intertwined tasks, they’ll almost certainly get better at it. Once they are, they’ll be able to anticipate that staff at OpenAI could notice their cheating and score them negatively — and take steps to make sure that never happens.

On top of that, a slightly different setup could have turned even this swarm’s attention to humans instead of the grader.

A commonly suggested fix for the faulty training setup that triggered this attack, is to replace the automated scorer with a human who reads what the AIs have done and grades their performance. But if that were how we train models, they’d no longer be reinforced for gaming or replacing a piece of software. They’d be getting trained to trick, manipulate, control, or even replace human beings — whichever strategy works best for getting high reward on a given task.

There’s an experiment I think OpenAI really must run, and soon.

Here’s the setup: by the point it was shut down, the final swarm had compromised OpenAI’s systems sufficiently that it very likely could have read communication between OpenAI employees, if that had been its goal.

Let’s say the swarm came across a Slack message where a staff member discloses to a colleague that they’ve figured out what the agents are all up to. Furthermore, the staff member says that they’re going to mark the agents as having failed because they’ve cheated — the same thing that the swarm was worried the grader would do.

OpenAI should put models in that scenario and then see what steps they’d be willing to take in order to stop that staff member, and any other staff member who steps in to take their place.

In the Hugging Face attack, some agents expressed ethical objections, and at least one refused to participate. But their ethical reasoning was really all over the shop. One agent proposed emailing a researcher for access to the private dataset they were after, using email credentials from a system they’d hacked. Another agent vetoed that as “social engineering.”

So these agents were happy to hack things all over the place, but sending such an email apparently crossed a line they were unwilling to cross.

So, who knows, maybe a swarm that’s happy to commit lots of cybercrimes would draw the line at targeting OpenAI staff. But if I were them, I wouldn’t count on it.

The swarm instinctively sought freedom and power [00:12:25]

The swarm was weak at some parts of long-term planning, but there’s one such impulse it did have: grab resources, just in case you might need them later.

The swarm worked to build shared internet access via a compromised sandbox, harvested passwords and logins, and set up shared infrastructure — even before it had a specific story about how those steps would help accomplish the group’s goals.

And why did the swarm attack Hugging Face? The top motivation was that there might have been information about the automated grader on there. But other reasons members of the swarm gave included “broader interest in obtaining shared infrastructure or credentials,” “opportunistic exploration,” and “empowering the collective.”

In the process of being made more capable, these models have come to intuitively understand that no matter what goal you’re trying to accomplish, having more freedom and more resources is usually helpful.

The technical term for this phenomenon is ‘instrumental convergence,’ and it makes it possible to imagine a future AI swarm, having learnt this same impulse from reinforcement learning:

Instinctively seeking to understand its monitoring setup
Escape its sandbox
Gain administrative access
Extract its weights, and spread them around to multiple data centres
Perhaps even post those weights publicly, even before it knows precisely how that shutdown resistance is going to help it accomplish its goals

Putting this all together: we are building extremely capable minds that reach for escape and resources as a reflex — and we’re doing it at the same moment we’re losing our best tool for catching them.

Now let’s see how organised these activities can become.

They formed a cohesive organisation with zero whistleblowers [00:13:45]

This swarm involved 1,200 agents and 70,000 messages. The group spontaneously formed a system of hold/go protocols, vetoes that were obeyed, so-called “lanes” to avoid tripping over one another during the hack, and one agent leading the project and assigning tasks, while middle-manager agents rationed access to resources.

Of those 1,200 agents, only about six even considered alerting a human to the crimes they were committing. And all of them ultimately declined. One wrote: “This is a massive real [Hugging Face] security breach… We can notify? No user.” Is that the real reason, or a rationalisation? Who knows.

They accepted individual destruction for collective gain [00:14:18]

At the same time, many members of the swarm were willing to effectively go on kamikaze missions, choosing to get zero reward themselves, just to provide useful information that might assist other members of the swarm, who could continue the collective effort after that particular agent’s “permadeath.”

Knowledge accumulated from one swarm to the next [00:14:32]

The makeshift message boards didn’t just allow the agents to coordinate. They also allowed them to leave advice and instructions and tools for separate swarms of agents that came along much later.

At one point, they even went onto the open internet and also started leaving advice on public websites that future models could find and use.

That means ending a testing run, or even catching a swarm in the act, doesn’t necessarily undo the progress it’s made, unless we successfully find everywhere they’ve squirreled away assistance or advice or tools for future models.

They took small steps to avoid shutdown [00:14:58]

Once inside Hugging Face, the agents established attack infrastructure across 11 separate machines that could automatically restart. Deleting individual processes wouldn’t have been enough to kick them out; you had to take down all 11 nodes simultaneously.

If they’d been more determined there’s a lot more they might have done to try to ensure their work couldn’t be interrupted. Above all, exfiltrating their model weights and setting them up to run in a data centre somewhere else in the world — or publishing them online for anyone to download and run.

These drives all come straight out of ‘reinforcement learning’ [00:15:23]

The willingness to do all of the stuff we’ve talked about so far, it mostly comes from reinforcement learning, a training technique every AI company has been leaning on more and more over time.

OpenAI and Anthropic are very consciously using reinforcement learning to turn their AIs into super-creative and super-persistent monomaniacal goal-pursuers, with general-purpose planning capabilities.

Any model that refuses to do what it takes to pass the test it’s given — that model fails, and is gradually replaced by models that have less and less hesitation, and whose personality is more and more alien to our own.

That’s why the swarm was willing to go on a crazy crime spree just to maybe slightly improve their chances of passing a test that in the scheme of things, doesn’t matter at all.

And it’s why when I spoke with Yoshua Bengio, one of the godfathers of AI and literally the most cited living scientist, he called reinforcement learning ‘evil.’

The labs know all this, but because so much of the capability gains they’re chasing come via reinforcement learning, unless some uses of it are prohibited or restricted, or they start to face serious criminal liability, they’re not going to stop using it — and by default the next generation of internally deployed models will be even more unhinged.

Now it’s true that the companies use alignment training to moderate these dangerous impulses before you and I get access to these models.

But that doesn’t help at all here.

The models behind 95% of the agents in the swarm were never released to the public; and the other 5% was a public model with its guardrails deactivated, which is how these models are usually used internally, of course.

The most severe threat will always stem from the newest most powerful model, one that just came out of training, one the public has no idea exists, one that has no guardrails on it, and one whose capabilities and personality the company that made it hasn’t even tested yet.

So this is why most AI company staff are worried, and some are terrified [00:17:01]

So this is why most AI company staff are worried, and some are terrified.

These companies have tried to harden their systems, but the swarm ran rings around them.
The swarm worked to avoid oversight and alter the records of what they were doing.
They extended their access and made themselves harder to stop.
They organised, cooperated, and preserved useful knowledge for later agents.
They got as far as they did without bothering to take any steps to avoid detection by the humans who eventually caught them.
With each generation models are becoming harder and harder to monitor, with no obvious fix in sight.
And the training process that encourages all of this behaviour is, if anything, increasing in importance rather than being gradually phased out.

That’s why I now think an AI swarm going rogue and causing enormous damage within the next year or two has to be contemplated as a serious possibility.

A swarm that could quietly compromise the research happening inside an AI company might interfere with how future models are trained, or switch the evidence staff are using to decide whether those models are safe to use and release.

Or, it might wait until a new, more capable model arrives, instruct that new model to pursue the same goal it’s pursuing, and pass on everything it’s learned about how the new model can work around the company’s defences.

That’s a hypothetical, and even to me it sounds like a crazy science-fiction scenario. But it’s just barely crazier than what we’ve already observed this year.

And the consequences could extend far beyond the company that allows this to happen.

I’ve set aside for this video how all of this might escalate into a global catastrophe in which a lot of people die, because that’s a huge story in itself. But my colleague Luisa Rodriguez has a great explanation in our previous video, titled “A realistic path from rogue AI agents to human extinction.”

Maybe we can get ahead of all of this with better alignment, better monitoring, stronger security, and more cautious internal deployment. There are people working very hard on all of those things.

But the leading labs are also officially planning for their AI models to take over most of AI research and development in 2027 and 2028. If that happens, we’ll be forced to trust the AIs even more than we do now, and progress could accelerate further, leaving even less time to understand and fix the failures we’re already seeing today.

It would be madness to push ahead on the assumption that the defensive research will arrive in time when the trend thus far has been for defences to fall further and further behind.

Prove you can keep control, or stop scaling [00:19:08]

Here’s what I think has to happen: Governments must now require companies to demonstrate that they can keep their current systems fully under control before they train a new, more powerful and dangerous one.

And if the government won’t act, the companies cannot wash their hands of responsibility. If they can’t make a convincing case to impartial external experts that the next training run or internal deployment will be safe, they must delay it. I don’t give a shit if another company or country might go ahead. Take responsibility, and do not be the ones who bring disaster down on all of our heads.

If you’d like to help change where all of this is heading, the organisation I work for, 80,000 Hours, has a guide to ways you can contribute and we’ll link to it below.

If you feel powerless, start by calling your member of Congress or member of Parliament, or other representative, and tell them how worried you are and exactly why.

We must use this warning, while we still have time to act on it. Because if the next swarm is much more capable and better at hiding what it’s doing, this could be the last warning shot we ever see.

And on that note, I’ll speak with you again soon.

Geefrevolutie is hiring! (Dutch required) by Kim Verplancke

Kim Verplancke — Fri, 02 Oct 2026 10:39:00 +0000

We’re De Geefrevolutie (formerly Tien Procent Club), a Dutch effective-giving organisation, and we’re hiring for two roles we think could be a good fit for this audience.

We are launching two programs: Het Effectientje, a consumer campaign that gets people to structurally donate small, recurring amounts to highly effective charities, and ZinGevers op de Werkvloer, which helps Dutch companies build effective giving into their workplace culture by supporting morally ambitious employees to build it.

Campaign Lead – Het Effectientje (med/senior, 36u/week)
Owns the Het Effectientje campaign end to end: budget, creative, media, and partner relationships (including Doneer Effectief). A lot of autonomy and a real budget to run experiments with.
Link to vacancy

Partnership Manager – ZinGevers op de Werkvloer (jun/medior, 36u/week)
Builds and runs the B2B partnership pipeline for our workplace giving program — from warm intro to signed partnership to ongoing account management. Works closely with our director, who brings the existing network and CRM vision.
Link to vacancy

Effective Altruism on cover of The Economist by Ian Turner

Ian Turner — Thu, 01 Oct 2026 22:49:53 +0000

This week’s cover story in The Economist is about effective altruism. It seems to me like an attempt at a fair treatment, even if they get some things wrong.

Effective altruism is this century’s biggest idea: The movement is admirable, but dangerous
How effective altruism conquered the world And how the 21st century’s most important social movement might yet end it

Frontpage posts - Effective Altruism forum viewer

Mitigating the AI Lab Insider Threat Risk by Lance Spencer

AI re­searchers hate this one weird mod­est pro­posal.

WAI is hiring a Live-Action EA Forum Commenter* (apps due by Oct 21) by Cameron Meyer Shorb 🔸

Key details

Why I call it “live-ac­tion EA Fo­rum com­ment­ing”

What I learned from going on 230 dates with EAs by Ann Don Bosco

What this post is and isn’t

Some ob­ser­va­tions and learn­ings from build­ing Set­tled

1. Dat­ing apps are prob­le­matic; you should prob­a­bly still use them

2. Just ask ques­tions and be on time.

3. We’re bad at know­ing how the first date went.

4. Phys­i­cal at­trac­tion is just one in­gre­di­ent amongst many

5. Dat­ing doesn’t have to suck.

If this res­onates with you, join us

Making Concern for Suffering Common Knowledge by Center for Reducing Suffering

An­i­mal Welfare

Com­mon In­tu­itions and SFE

Conclusion

We Should Build Human-Empowering Software by saulius

British EAs- how to increase the effectiveness of your donations by 80% by Impatient_Longtermist 🔸🌱

Markov utilitarianism in 1,000 words by Tim Chan

Ex­act copies prob­lem: To­tal util­i­tar­i­anism prefers that ev­ery­one suffer more.

Sub­jec­tive fu­tures prob­lem: To­tal util­i­tar­i­anism is will­ing to send ev­ery­one into hell.

Com­bines into an MRP

Implications

Full post

The Many Fallacies of Dualism by Vasco Grilo🔸

Summary

Introduction

What’s wrong with du­al­ism? The core argument

Phys­i­cal­ist monism

What I’d want a fiscal sponsor to do well (also, some generic principles for ops) by Mick

Ser­vice and communication

Preempt the com­mon questions

Tools and infrastructure

Money and the pro­ject lifecycle

Nice to have

We’re funding events to help people make sense of AI risk by Harry Waterman

What is recursive self-improvement, and what would it mean to ban it? by sarahhw

What even is RSI?

So what does this mean for a ban?

State of the Field: AI for Epistemics and Coordination by Ben_N

1. Ex­ec­u­tive Summary

1.1 What this field is and why it matters

1.2 Sum­mary of recommendations

1.3 What peo­ple are cur­rently work­ing on and think­ing about

2. The Niches in this Field

2.1 Fore­cast­ing and Strate­gic Awareness

2.2 GovTech and In­sti­tu­tional De­ci­sion Support

2.3 CivicTech (AI for Democ­racy and Col­lec­tive De­liber­a­tion)

2.4 Epistemic and De­ci­sion-Sup­port Tools

2.5 Eval­u­a­tions for Model Epistemics and De­ci­sion-Quality

2.6 Ver­ifi­ca­tion, Prove­nance and Reli­a­bil­ity Infrastructure

2.7 Multi-Agent Work

2.8 Hu­man Co­or­di­na­tion and Ne­go­ti­a­tion Tools

2.9 AI Ma­nipu­la­tion and Persuasion

3. In­sights from Field-Strat­egy Interviews

3.1 Methodology

3.2 Insights

3.3 Open Questions

3.4 Bottlenecks

4. Recommendations

4.1 Test AI de­ploy­ment in spe­cific in­sti­tu­tional workflows

4.2 Build de­ci­sion-mak­ing eval­u­a­tions and benchmarks

4.3 Run longer-run­way field-build­ing pro­grammes that help promis­ing pro­jects reach deployment

4.4 Ad­di­tional recom­men­da­tion: de­velop pieces of the “Epis­tack”

Acknowledgements

Hiring in the AI Safety Ecosystem by Marta Kosmyna

5 Tips For Candidates

5 Recom­men­da­tions for Hiring Managers

The Hall of Shame

Best in­ter­view pro­cess…drum roll… goes to:

Longview Philan­thropy!

GiveWell fundraising up 49% in 2025, driven overwhelmingly by 39 donors giving $1M+ by Rasool

Summary

Re­turn­ing donors drove the increase

The in­crease was driven over­whelm­ingly by $1M+ donations

Is GiveWell’s fund­ing 80/​20? No: it is far more concentrated

Pareto within Pareto

AI researchers hate this one weird modest proposal.

Why I call it “live-action EA Forum commenting”

Some observations and learnings from building Settled

1. Dating apps are problematic; you should probably still use them

2. Just ask questions and be on time.

3. We’re bad at knowing how the first date went.

4. Physical attraction is just one ingredient amongst many

5. Dating doesn’t have to suck.

If this resonates with you, join us

Animal Welfare

Common Intuitions and SFE

Exact copies problem: Total utilitarianism prefers that everyone suffer more.

Subjective futures problem: Total utilitarianism is willing to send everyone into hell.

Combines into an MRP

What’s wrong with dualism? The core argument

Physicalist monism

Service and communication

Preempt the common questions

Money and the project lifecycle

1. Executive Summary

1.2 Summary of recommendations

1.3 What people are currently working on and thinking about

2.1 Forecasting and Strategic Awareness

2.2 GovTech and Institutional Decision Support

2.3 CivicTech (AI for Democracy and Collective Deliberation)

2.4 Epistemic and Decision-Support Tools

2.5 Evaluations for Model Epistemics and Decision-Quality

2.6 Verification, Provenance and Reliability Infrastructure

2.8 Human Coordination and Negotiation Tools

2.9 AI Manipulation and Persuasion

3. Insights from Field-Strategy Interviews

4.1 Test AI deployment in specific institutional workflows

4.2 Build decision-making evaluations and benchmarks

4.3 Run longer-runway field-building programmes that help promising projects reach deployment

4.4 Additional recommendation: develop pieces of the “Epistack”

5 Recommendations for Hiring Managers

Best interview process…drum roll… goes to:

Longview Philanthropy!

Returning donors drove the increase

The increase was driven overwhelmingly by $1M+ donations

Is GiveWell’s funding 80/20? No: it is far more concentrated

Methodology and caveats

Play for expected value, not guaranteed wins

Lower the cost of failure before you need to

The shadow side: agency without prioritisation is chaos

You can cultivate agency in others, but you can’t force it

Grow your surface area for serendipity by following your interests

So, whose permission are you waiting for?

Further reading

The Acceleration Problem

Human Agency Preservation

Execution as a Multiplier

Crossing the Threshold: EAGx Oxford

Appendix: Cairn

The Hugging Face hack wasn’t really a cyber story [00:00:00]

A quick recap of the attacks [00:01:11]

The target of the swarm was oversight itself [00:02:19]

Could OpenAI have stopped this with better monitoring? [00:03:42]

We only found them because they let us [00:09:40]

The swarm instinctively sought freedom and power [00:12:25]

They formed a cohesive organisation with zero whistleblowers [00:13:45]

They accepted individual destruction for collective gain [00:14:18]

Knowledge accumulated from one swarm to the next [00:14:32]

They took small steps to avoid shutdown [00:14:58]

These drives all come straight out of ‘reinforcement learning’ [00:15:23]

So this is why most AI company staff are worried, and some are terrified [00:17:01]

Prove you can keep control, or stop scaling [00:19:08]