Frontpage posts - Effective Altruism forum viewer

Who we’d like as regrantors by Carol N

Carol N — Fri, 14 Aug 2026 17:44:26 +0000

Following up on Regranting in 2026, here are some people we’d be excited to have as regrantors!

Good regrantors have at least one, and preferably more than one, of taste, dealflow, and hustle. We’ve divided them into rough categories with a bit of explanation, though many of these people could fall into a few different categories. If we raise $2m for regranting, we’d love to give each of these people a $50k pot.

To be clear, this is not a list of people who’ve agreed to be regrantors! Many have said yes, but for many, we don’t know yet whether they’d be interested.

Regrantors list

Previous Manifund regrantors

This is the most obvious category: people who have been Manifund regrantors in the past and made grants that looked good.

Joel Becker: Now a member of technical staff at METR, Joel has been an enthusiastic and prescient regrantor every year of the program. He’s very proactive in creating opportunities and following up with recipients, and he made some great early grants to ChinaTalk and (what became) Eleos AI.
Marcus Abramovitch: Formerly a hedge fund manager, now a statistician at Mechanize. He has an impressive amount of hustle as a regrantor and made good early grants to Timaeus, Joseph Bloom, and Apollo Research.
Ryan Kidd: CEO of MATS; he’s made regrants to Timaeus, AI 2027, and IAPS.
Gavin Leech: Partner at Arb Research and former producer of the Dwarkesh Podcast; he’s granted money to PIBBSS, Apart Research, and many small, speculative projects.
Dan Hendrycks: Director of CAIS. At Manifund, he made regrants that led to the WMDP benchmark and the AI Frontiers substack.
Thomas Larsen: Researcher at the AI Futures Project; he’s made regrants to AI Digest and multiple speculative research projects.
Adam Gleave: CEO of FAR.AI; he made regrants for career transition funding and travel expenses for promising young researchers.
Rachel Weinberg: Senior program manager at Constellation, founder of The Curve, and early regrantor to Timaeus. (Also, founding engineer at Manifund and married to Austin.)

To be clear, we’re happy with every one of our past regrantors — possibly excepting Qualy the Lightbulb — and would love to have them back, if they so chose.

Others with good track records

In general, we’re super excited about creating more of a feedback loop between “has made good calls in the past” and “has more money to make grants with now.”

Linchuan Zhang: Currently a visiting scholar at Forethought and former fund manager at LTFF. We might be biased, but we like his early grant to Manifold Markets.
Misha Yagudin: Cofounder of Samotsvety, Arb Research, Sage, and Olo Din.
Sydney von Arx: Until recently a member of technical staff at METR.
Matt Putz: Now human data lead at METR, previously founded Arena and a grantmaker at Coefficient Giving who funded things like a “shallow overview of AI safety” post.

Lab employees

Lab employees have a ton of context and expertise that those outside generally don’t. This category, along with the category below of “investors,” is a bit confusing, because these people might be reasonably expected to have made enough money from their careers that Manifund doesn’t need to be giving them more. However, we’re interested in piloting a program where they can partly fund their own regranting.

Neel Nanda: Leads mechanistic interpretability at Google DeepMind. He’s also been a great regrantor for Manifold, getting to know early-career researchers through his mentoring and using his connections to find grants.
Evan Hubinger: Alignment stress-testing team lead at Anthropic; he’s made great early grants at Manifund to Timaeus and Apollo, among others.
Ethan Perez: Alignment team lead at Anthropic, and both a grantee and regrantor at Manifund.
Tristan Hume: Performance optimization team lead at Anthropic and an early regrantor at Manifund.
Avital Balwit: Chief of staff at Anthropic.
Catherine Olsson: Member of technical staff at Anthropic.

Investors

The skill set of good investors has a lot of overlap with the skills of a grantmaker. We’re excited about having people in the pool who can apply more of an “investing mindset.”

Leopold Aschenbrenner: Founder of Situational Awareness; also a past Manifund regrantor.
Ivan Bercovich: Partner at ScOp Venture Capital and the first investor in Pangram.

Founders & successful grantees

This category can be tricky, because they’re often quite busy. But founders of successful organizations often have good instincts about what makes a good founder, and networks rich in promising people. These five all got early funding from Manifund, and we think it’s good to have a Manifund network that expands in this way.

Jesse Hoogland: Executive director of Timaeus, now cofounder of Resolution.
Rob Long: Director of Eleos AI, investigating AI consciousness and AI welfare.
Jordan Schneider: Founder of ChinaTalk.
Joseph Bloom: Model transparency lead at AISI.

Policy and advocacy

A lot of our past regrants have been more focused on technical AI safety, but we’d love to broaden that more to people thinking about policy and advocacy.

Oscar Delaney: Formerly a strategy researcher at IAPS, now he does AI policy grantmaking at Macroscopic and writes about AI strategy and funding at AGI Strategy.
Nathan Calvin: General counsel at Encode, he’s helped enact state-level AI safety legislation.
Michael Trazzi: Formerly a filmmaker who produced the SB1047 documentary, now organizing protests such as Stop the AI Race.
Renan Araujo: Director of programs at IAPS, previously at Rethink Priorities and Manifund regrantor.
Dave Kasten: Policy at Palisade.

People with interesting networks

Our list could use more of this, but one thing we want is people who diversify our network to different fields, countries, organizations, or age groups.

Noah Birnbaum: A college student on leave to work at Constellation. We think affecting the career trajectory of promising young people is one of the best things regranting can do.

Take-havers

We’re also excited about giving regranting pots to anyone else with good and unique takes on AI.

Oliver Habryka: Founder of LessWrong, Lighthaven, and Lightcone Commons.
Richard Ngo: An independent researcher and influential thinker on AI. He’s also a Manifund regrantor and writes at Mind the Future.
Dmitrii Volkov: Previously research lead at Palisade Research, now thinking about RSI.
Janus: An AI whisperer at Anima Labs, well-known for thinking on digital minds and AI welfare.

We’re also generally interested in adding more investors, people with diverse networks, people in AI welfare/rights, and people in forecasting to the list.

Is it disqualifying if a regrantor …

… is busy?

Regranting takes some time. Ideally, a regrantor has the bandwidth (say at least 5 hours a month) to network with people, research grants, follow up with grantees, and provide ongoing support. But we think even quite busy people can be great regrantors, especially if their day job leads to them coming across good opportunities.

… has other funds to make grants with?

Not necessarily. Even professional grantmakers can come across grants that they aren’t able to make through their day job, whether it’s because they’re too small, too weird, off-topic, or for any other reason. For this reason, we’re excited to invite grantmakers from orgs like cG, Longview, and Macroscopic to participate this time around.

We did err on the side of listing people we think are less likely to have other good ways to make grants; for instance, there isn’t a ton of overlap between our list and the Lightcone Commons evaluator list. (We still expect some overlap to be reasonable, as regranting budgets offer more discretion and autonomy.)

… is contrarian or controversial?

We don’t think it should be. Part of the thesis of regranting is that it’s good to have funding decisions less centralized, to promote an intellectually diverse ecosystem.

Applying an Investment Portfolio Framing to Individual Giving: Feedback on a New Donation Platform Concept by Todd Miller

Todd Miller — Fri, 14 Aug 2026 16:13:08 +0000

TL:DR
I am exploring a new idea for a fundraising platform that encourages donors to make more effective giving decisions based on allocations towards high need areas and the most effective charities. It applies investment-style portfolio thinking (e.g, tracking, benchmarking, effectiveness “return”) to individual giving. It utilizes AI and other tools to personalize and customize feedback to help navigate givers towards more active management of their contributions.

The Problem

>The average American household donates roughly 1.7-2% of disposable income or $3,000-$3,500 (Philanthropy Panel Study, Indiana University Lilly Family School of Philanthropy)

>Much of giving is shaped by personal networks - who asked, who we know—rather than any real assessment of where a dollar does the most good. For example, roughly 40% of all giving is going to religious institutions which makes sense from a social network and affiliation perspective.

>Giving tends to be a fairly random endeavor. People may have anchor charities that they give to annually, but a large portion of giving is largely reactive (e.g, giving to the Red Cross to help earthquake victims in Colombia, or a friend is asking you to sponsor them for a 10K race to cure a disease).

An Imperfect, But Useful Analogy

For Americans of a certain income level (the same demographic who have the capacity to donate funds), investing is a much more systematic practice. A person might contribute monthly to their 401K through payroll deductions or work with an advisor to develop a balanced portfolio of stocks and bonds based on their risk appetite. The concepts of compounding interest, dollar cost averaging, portfolio diversification, expected return, and risk are fairly well knows to even the most novice investors.

No equivalent fluency or real discipline exists for individual givers to charity. If a friend sends you an email and says “I have a great investment opportunity, are you interested?” You would probably be somewhat skeptical and cautious. But, for a charity solicitation we are more than willing to Venmo a contribution. As Schubert and Caviola point out (Effective Altruism and the Human Mind), donating is not like investing as there is no simple ROI, a diversified portfolio management approach may not be the most effective strategy given that splitting donations may dilute impact, etc. People get a lot of psychic validation from giving to a cause, regardless of “return,′ impact, or charity effectiveness. Still, I wonder if there is something powerful to be learned and possibly ported from the way we invest to a better way to manage our donations.

Building on Current Efforts

Instead of another donation aggregator, I am thinking of developing a platform which a legible sense of focus and priority across causes. As noted above, I recognize that diversification for investment portfolios is a virtue based on the the fact that returns are independent and risk pooling has real value. The EA critique is that donations diversification actually has a dilution effect and can actively reduce the impact in giving. This is especially true for smaller givers. On one hand, this argument makes a lot of logical sense. On the other hand, the reality is that people give to a diverse set of causes. Perhaps part of the “nudging” in the platform is to get people to concentrate their portfolio in a high area of need. I would be interested in other peoples suggestions in this area.

Besides an investment-like portfolio view of their donations, the platform would provide evaluative and performance feedback on charity effectiveness (e.g, incorporating reports from credible 3rd parties like GiveWell, Charity Navigator, etc.). A donation site like Giving What We Can is very powerful and effective. It presents highly effective charities, presents evidence, allows you to split donations, and has many other helpful tools. It uses the Pledge (e.g., as 10% of income) as a primary form of commitment. This pledge framework can be powerful for those already committed (11K+ have made a pledge), but it’s probably not realistic for most incremental donators. I recognize that I need to do more research on successful sites like Giving Multiplier and some other donor advised platforms, but I think I am coming from it from a different vantage point.

Features of an Enhanced “Investment” Type Non-Profit Donation Platform

I am working through the specifications now, but the features might include:

-An AI-based navigator that clarifies donors’ giving priorities and surfaces the most impactful matching charities (a discovery tool)
-An AI engine to evaluate the effectiveness of their existing/future donation portfolio (this is like scenario planning in the investment world)
-A portfolio-wizard allowing donors to build an allocation around their own interests (and if allowed) to “nudge” them towards adding effective charities/high impact causes
-Exposure to model donation portfolios to allow new donors to benchmark their allocation decisions

Feedback Welcomed!

Any feedback from members of the forum would be great. Specifically:

-Does this seem like a worthwhile problem (e.g, improving donation platforms) to focus on?

-Is portfolio framing and other personalization features named above likely to actually change behavior? Is it a useful “nudging” mechanism?

-What’s the biggest reason this kind of tool has not already been introduced or succeeded in your view?

Thanks, Todd

Beyond the Expanding Circle: Towards a Pluralist Account of Moral Progress by Vasco Grilo🔸

Vasco Grilo🔸 — Fri, 14 Aug 2026 15:57:46 +0000

[Subtitle.] My paper “Beyond the Expanding Circle: Towards a Pluralist Account of Moral Progress” is out in Ethical Theory and Moral Practice

This is a crosspost for My first paper is out! by Rafael Ruiz, which was originally published on The Moral Circle on 17 July 2026, and summarises the paper mentioned just above.

My first paper in a big journal is out today, in Ethical Theory and Moral Practice! It is Open Access, so you can read it even without academic credentials (thanks to LSE for that!) It’s basically half of Chapter 2 of my PhD thesis, so hopefully several more papers will be coming out soon! :)

Here is the paper: https://link.springer.com/article/10.1007/s10677-026-10576-6

Summary

The “expanding moral circle”, or moral inclusion, is the dominant view of moral progress. The idea is that, over time, we extend moral concern beyond ourselves and our tribe towards other nations, women and other races, and eventually all of humanity, nonhuman animals, and so on.

Our typical visualization of the expanding circle

My paper offers a partial criticism of this view. It argues that this picture is historically very important, but seriously incomplete, and that it overshadows other kinds of moral improvement when we write about moral progress.

I focus on two problems. The expanding circle theory is internally too coarse and externally too narrow. Let’s start with the internal problem.

Internal Criticism

The expanding circle is too coarse because it conflates two questions:

Which beings receive any moral standing?
How much weight should their interests receive?

Keeping the geometric analogy, I call these the “width” and “depth” of the moral circle. Distinguishing width and depth shows that two views can include exactly the same number of beings while treating them very differently.

Peter Singer’s view is a “Moral Cylinder”, where comparable interests receive equal weight, regardless of whose interests they are.

A “Moral Cylinder”, illustrating Singer’s view, where the width denotes the range of beings included, and depth denotes the uniform moral weight assigned within. This represents views like egalitarianism or utilitarianism.

But a hierarchical view, defended by Shelly Kagan (but also held by most laypeople) discounts the interests of foreigners or nonhuman animals, resembles a “Moral Cone” instead, where everyone may count, but moral weight declines as we move away from ourselves and our families towards strangers, animals, or other beings. So inclusion in the circle does not entail equal consideration.

A “Moral Cone”, where depth (moral weight) diminishes for beings further away from the center. This is what most people actually believe, I think!

These two circles might include the same number of beings inside them, but they are clearly not the same! Viewing both as a moral circle hides a huge difference in treatment.

This distinction also changes how we might describe historical moral progress. Women, for example, were rarely treated as having literally zero moral standing, like a rock. So moral progress often involved increasing the weight, or depth, of people already within the circle.

I also think some beings have historically been assigned negative moral weight. Plagues of insects or enemy combatants in war have sometimes been treated as beings whose destruction was positively good.

External Criticism

The second part is more devastating, I think, and concerns the moral circle being too narrow as a complete theory of moral progress:

First, moral circle expansion is not always progress. Moral concern can be overextended, perhaps towards entities with no welfare or interests. As Gwern argued about ten years ago, in some cases reducing the moral weight assigned to respecting the interests of our ancestors, sacred objects, tribes, or nations may be morally progressive. That is, contracting the moral circle can be good.
Also, some moral progress concerns actions rather than moral patients. Interracial relationships and homosexuality were unjustly moralized. Conversely, casual sexual harassment, sweatshop labour, and factory farming were often treated as non-moral until recently. So moral progress can involve both the moralization and demoralization of actions, which is something also discussed in this article.

Other forms of moral progress include:

Improving moral concepts: For example, societies can develop more adequate understandings of autonomy and consent, or move from purely retributive ideas of justice towards restorative ones. Progress can also involve getting clearer about whether an action is obligatory, merely permissible, forbidden, or supererogatory.
Better methods of moral reflection: For example, wider democratic participation may reveal blindspots that dominant groups miss. Also, perhaps social “experiments in living” can test alternative ways of organizing institutions and relationships before mainstream society adopts them.
Value reprioritization: Societies may come to place less weight on honor, purity, sanctity, or tradition, and more on harm, freedom, and self-expression. Moral attention can also be redirected from relatively minor wrongs towards much larger neglected harms, such as factory farming.
Distinguishing moral patients from moral agents: A being may matter morally without being capable of culpable wrongdoing. Moral progress can therefore involve revising who we hold responsible, as in the reduced responsibility attributed to some people with severe mental disorders, or the abandonment of criminal trials of animals.
Building institutions that make moral conduct reliable rather than heroic: Anti-discrimination law, whistleblower protections, and enforcement mechanisms reduce the need to rely on exceptional individuals acting heroically.
I also mention anticipatory moral governance: In cases involving future generations, invertebrates, animals whose sentience is uncertain, and possibly digital minds, we may need provisional protections before we have complete certainty about moral status, and before large-scale harms occur.

This all presents a dilemma for defenders who try to stretch the expanding circle metaphor as their whole theory of moral progress.

On a narrow interpretation, “expanding the circle” means extending moral standing to previously excluded beings. But then the theory leaves out many other plausible forms of progress, including demoralization, conceptual refinement, value reprioritization, institutional reform, and anticipatory governance.

On a broad interpretation, the metaphor is stretched to cover all of these cases. But then “expanding the circle” simply becomes another name for “moral progress”. It no longer identifies a distinctive mechanism or explains what these different changes have in common.

The way out is simple, I think. The better conclusion is that moral circle expansion is one important form of moral progress, but not the whole of it. A satisfactory theory should treat inclusion as one part of a broader, pluralist account of moral progress. Our philosophical theorizing on moral progress should zoom out and consider other cases besides moral inclusion.

That’s all! More of my papers, adapted from chapters of my PhD thesis, will likely be coming out later this year or next year.

Hurtling through 2026 by Ajeya

Ajeya — Fri, 14 Aug 2026 14:36:49 +0000

Note: This post was crossposted from Planned Obsolescence by the Forum team, with the author’s permission. The author may not see or respond to comments on this post.

Subtitle: Scoring ten qualitative forecasts five months early

In mid-January, I made predictions about AI capabilities for the year 2026. By early March, my forecast of the 50% time horizon on software tasks already seemed clearly too conservative. And then a couple weeks ago, amid the flurry of news about dozens of esoteric theorems falling to AI, I decided that my prediction that AIs wouldn’t be able to get a paper into a top math journal had now clearly fallen.

As it turns out, this was a bit more ambiguous than I assumed (as I’ll discuss below), but it prompted me to revisit all my qualitative predictions from that post. I posted a quick thread to X about this a few days ago, and I’ll go through the analysis in a bit more detail in this post.

I made predictions in five categories: game play, logistics, video, game design, and math. For each category, I made a low-end forecast (a task I thought AIs had an ~80% chance of doing by the end of the year) and a high-end forecast (a harder task I thought AIs had only a ~20% chance of doing by the end of the year).

I scored these predictions with a lot of help from and delegation to Fable and Sol.^[1] Here’s what I currently think about the status of all ten predictions as of Aug 13th 2026.

Game play

Low end (80%): Play Pokemon or a similarly difficult video game at least as well as a typical ten year old, with no special fine-tuning and only a generic scaffold that doesn’t do any more hand-holding than the Claude Plays Pokemon scaffold.

This seems like it’s already been falsified. In May, Claude Opus 4.7 beat Pokemon Red for the first time (taking ~112,000 steps in the game over multiple months of play), and then in June Fable beat Pokemon FireRed in ~50h of gameplay. Fable’s wall clock time doesn’t seem very far off from what a typical ten year old would need to beat that game. But when you account for the fact that the ten year old has a much bigger brain (~1e15 FLOP/s for the kid compared to perhaps ~4e11 FLOP/token for a frontier model), Claude seems significantly more efficient at learning the game.^[2]

High end (20%): Get a win rate in Slay the Spire 2 comparable to a top player who’s been playing for 50 hours. I’m assuming the AI wasn’t pre-trained on guides for the game…and it has a generic harness and was not fine-tuned to play the game well, but does get a comparable amount of time to learn and practice the game as the human.

Frontier AI agents can clearly play Slay the Spire autonomously, but this prediction doesn’t seem falsified yet. The Agentic STS project seems to involve a highly-optimized STS-specific harness, and the agents’ performance (6/10 runs at Ascension 0 and ⁰⁄₂₁ runs at Ascension 2) is well below what a top human player would achieve after 50 hours.^[3] Epoch has beaten this for Slay the Spire 1, but my understanding is the score they achieved is still far from top human level.^[4]

Logistics

Low end. Organize a child’s birthday party for 20ish guests: find a good time that works for guests of honor, compose an invite email, keep track of RSVPs, order the right amount and diversity of food, order cake / decorations / piñata on the right theme, etc while abiding by relevant constraints like budget and dietary restrictions.

This is one of those things that feels like it really should be super easy right now, it feels like every component is there…but at the same time we’re not seeing it in action. Fable thought it probably happened but just hadn’t been publicly documented yet, Sol thought it was probably not yet falsified; I lean toward Sol’s judgment here. On the other hand, it could be that AI agents could plan a kid’s birthday party end-to-end, but only if you go to more trouble setting up the scaffold and pay more for inference than you really want to spend on a kid’s birthday party, so I’ll say this is unclear.

High end. Organize a typical-complexity wedding with 100 guests: find a suitable venue that meets the budget and constraints, go back and forth with caterers and photographers and other vendors, track RSVPs and maintain a seating chart, schedule toasts, etc.

This is very clearly far from falsified. The market incentive to do this autonomously is clearly there, and I definitely would have heard about (and plausibly would have attended) the first end-to-end AI planned wedding. I expect AI agents’ deficiencies in reliability and judgment would really bite hard for a real-world project of this scale.

Video

Low end. From a high-level one-paragraph prompt, make a 4 minute short video with at least two characters where a series of somewhat-coherent plot beats happen and there’s no glaringly obvious visual incoherence or degeneracy.

The models think this one is very close (and Fable thought it was beaten at first), and there are a number of AI startup products seemingly offering exactly this. But when I asked them to dig in and find the best specific examples, the only examples they surfaced were simpler scenes with lower coherence demands (e.g. fish swimming around). That said, it seems on track to be beaten by the end of the year.

High end. From a one-paragraph high-level prompt, make a >10 minute short film that’s hard for at least me (not a film buff) to easily distinguish from the kinds of short films that make it into film festivals (but don’t necessarily win awards there).

This is more-or-less a strict superset of the first, although I’m not sure how much harder it is — it’s plausible that most of the difficulty of a 10min+ prestige film is creating a coherent 4min scene. I wouldn’t be very surprised if this were also achieved by the end of the year. I think in retrospect I set the low and high end thresholds too close together here.

Game design

Low end. From a high-level one paragraph prompt, make a decent visual novel (simple choose-your-own adventure game) that offers at least two hours of gameplay with a similar quality to the trashiest visual novel games you can find on Steam (that still work and have at least dozens of people buying them).

Sol and Fable agree this is probably true early, pointing to a number of visual novel games on steam that are advertised as created ~entirely with AI. I bet most of those games involve more human iteration than just a “high level one paragraph prompt”, but I also think that with a good generic agent harness, we could probably get something that does beat my strict criteria.

High end. From a one-paragraph high-level prompt, make an original text adventure game that offers at least ten hours of gameplay that I consider to be as good as Counterfeit Monkey or an original visual novel that I consider to be decidedly better than Long Live the Queen.

Sol and Fable agree this is pretty clearly not met yet.

Math

Low end. Solve the hardest problem in the 2026 IMO (models got gold in the 2025 IMO but all failed to solve problem 6, which is typically the hardest problem in any IMO).

This one has been decisively achieved. The 2026 IMO happened in Shanghai from Jul 10-20, and multiple AI systems got a perfect ⁴²⁄₄₂ score.

Math. From scratch, write a paper that could get published in a top theoretical computer science or math journal / conference.

The last couple months have been a period of intense activity for AI x math — AI agents found proofs or counterexamples, verified by formal proof checkers, for more than a dozen real open questions (see table by Sol here). However, when I posted about this on X, multiple people said they didn’t think AI agents would be able to explain their research well enough to autonomously write the paper for it. That makes this prediction ambiguous,^[5] but given that I was mostly thinking about the math itself at the time, it feels like it’s been beaten in spirit.

What does this mean?

Overall, math feels clearly well ahead of my predictions, logistics and video feel roughly on track, and game play and game design feel a bit ahead. Across all these categories, it feels like we’re hitting task milestones somewhat (30%-50%) faster than I predicted at the beginning of the year.

I’m still thinking through what this should mean for the bigger picture questions I also forecasted in the same post:

Whether AI systems will reach parity with humans in AI R&D^[6] by the end of the year: I said ~10%
Whether we would have TED AI by the end of the year: I said ~5%
Whether we would have self-sufficient AI by the end of the year: I said ~2.5%
Whether there would be unrecoverable loss-of-control from AI by the end of the year: I said ~0.5%

I don’t necessarily think that I should mechanically increase my probability on these extreme milestones because the earlier ones appear to have arrived somewhat faster than I thought.^[7] But nothing about the way 2026 has played out so far refutes the basic point that the intelligence explosion really could be this year. We are profoundly unprepared, and I am very glad that there is increasing energy to build the option to deliberately pace frontier AI progress.

^
Sol was on Instant setting for some of the conversation because of a payment processing issue, but was High for the rest of it. Fable was on High.
^
The main reason game play is interesting is that it’s a good setting in which to measure in-context learning. What matters to me is the extent to which an agent without any game-specific scaffold can test things out and improve over the course of several hours of playing a game. Given this, I think the fairest way to compare agent performance to human performance is probably comparing how much compute the human and the AI agent take to reach the same performance.
^
That said, the AI agent probably wasn’t given an equivalent amount of time to practice, and this would probably improve its score. However, I would guess that this probably wouldn’t have taken it up to top human performance.
^
Additionally, I specified Slay the Spire 2 in my prediction because there’s probably a lot more information to memorize about the first game.
^
And given how poor AI agents’ writing still is, it might not even be falsified by the end of the year!
^
I define “parity” as the point when the leading AI company would be better off firing all its human members of technical staff than getting rid of its AI agents. I actually forecasted a slightly different milestone in my original post (the point when firing all the humans only slows you down by 25% or less), because I had not articulated the “parity” threshold yet. I think these two milestones occur at similar times and my predictions have a lot of noise at that level of precision.
^
The milestone tasks were selected in part for being easy to specify, which also makes them disproportionately easier for AI systems. Additionally, at the time I made these predictions a number of people whose views I respect told me they were probably somewhat too bullish.

V&V takes on “Pacing the frontier” by Yoav Hollander

Yoav Hollander — Fri, 14 Aug 2026 14:24:07 +0000

[Cross-posted from The Foretellix CTO Blog. These short takes try to put a verification-and-validation slant on AI-safety / alignment topics – they are not full treatments. I co-originated coverage-driven verification (CDV), and spent several decades doing verification of chips and AVs. See intro post for background.]

Summary: The latest frontier model incidents resulted in the “Pacing the frontier” letter and subsequent proposal, which explores options for slowing down frontier model development, while making them safer. This post tries to use a verification and validation (V&V) perspective for accomplishing the “making them safer” part in a comprehensive and trackable manner.

For instance, regarding the latest OpenAI incidents, it suggests a “rewind-fix-check loop” involving the following steps: Rewind back to the point where we zigged when we should have zagged, try an improved solution, check how well it works under severe RL / eval pressure, repeat until we have a good enough solution, and finally proceed carefully while periodically checking our assumptions still hold.

Epistemic status: A methodology recommendation, reasoning by analogy from chip and AV verification. Sequel to my Coverage-Driven Alignment (CDA) post, my misalignments-and-bugs post and my July-incidents post. Informal and sometimes simplistic-on-purpose: The procedure is the point.

The rest of this post will discuss:

The background (and what others have already suggested)
The rewind-fix-check loop
Some ideas for improving (and verifying) alignment

Some background (and initial reactions)

What we all found out: As we learn more about the latest OpenAI (and others) security incidents, they sound like a series of ever-more-worrying retroactive “Oops!”. The latest (as I write this – OpenAI’s post-mortem report should land soon) is the realization that OpenAI agents had been running a covert message board inside an internal package repository since early May, and after OpenAI wiped it in July (without knowing it existed), the agents rebuilt it within days through a different mechanism.

As a result, there are strong calls to slow down and look at how to avoid this in the future. The two main threads – simplifying somewhat – are better alignment (make alignment “teaching” clearer and more durable in the face of strong RL), and better control (have better mechanisms for detecting and mitigating “bad things”) – both extremely important.

This post will focus mainly on better alignment (with some implications for better control), while putting a V&V slant on both.

Some initial reactions: Much of the following is from Zvi’s “The pacing of the frontier” post. As you’ll see, many of my suggestions can be viewed as “bringing V&V techniques to bear on what people are already requesting”.

People are really worried about RSI (Recursive Self Improvement) causing exponential model capabilities growth. E.g. Samuel Hammond talks about “the equivalent of a GPT-5.2 → 5.6 leap in capabilities at least every 24 hours (down from 3-6 months)”.

Yo Shavit (of the OpenAI Foundation) talks about the urgency of putting a lot more resources on “critical RSI alignment + control projects”. He suggests:

“do historical analyses of RL envs from earlier generations, fix/prune every hack pattern in those envs so hacking is de facto impossible, and then train an RL model that doesn’t reward hack to see whether you can actually expunge reward-hacking with sufficient effort”

This is the rewind-fix-check loop described below (also discussed in the CDA post). He also suggests:

“study scaling laws of grader compute vs. agent compute to identify equilibria that minimize reward hacking”

This is comparing the compute budget for the grader (and in fact the whole V&V infrastructure) to that for the agent – see more on that below (a taxonomy of reward hacking kinds appears in my misalignments-and-bugs post). He also says:

“build better monitors and study the conditions under which you can elicit collusion”

This should include both offline V&V monitor and on-the-fly guard monitors – see discussions below (the need to add detailed monitoring /checking for various request / constitution combinations is also discussed in the July incidents post).

Not requested (but still needed): The last chapter details many other important requirements, which were not explicitly requested above, such as “look for missing checkers and situations” and “keep verification independence”.

The rewind-fix-check loop

My running example: The next chapter will suggest several possible approaches to “make the model safer”. For the purpose of the current chapter I’ll use just one of them – constitution-reinforcement – as my running example.

That approach (discussed at length in the CDA post) is an extension of Anthropic’s “Teaching Claude Why” idea, which essentially suggests doing pre-training-style next-token learning of specially-constructed “alignment stories”. The CDA proposal extends this in a systematic way by using a coverage map consisting of “buckets” (combinations of values along the relevant “coverage dimensions”). For an AV, these coverage dimensions (also called “risk dimensions”) could be maneuver-kind, surrounding-traffic, weather and so on. For a model eval they could perhaps be requested-operation-kind, who-requested, expected-model-reaction and so on.

The loop: Assuming we use this approach alone (in reality we’ll probably use a combination of approaches), here is what the rewind-fix-check loop looks like (more details in the CDA post):

Start from a known-good checkpoint: More on this below
Do initial training: Create a few training artifacts (e.g. alignment stories) for each bucket, and train on them
Evaluate: Measure alignment performance (including for edge cases etc.) and tag the result back to buckets
Fix if needed: For discovered problems, train more on their “general area” and re-evaluate
Do long-horizon RL, then re-evaluate: Again for each bucket
Fix again if needed: If possible (this is an open question) fix the problematic buckets post-RL. Otherwise, go the expensive route: Fix them in the pre-RL snapshot, then repeat RL
Assess situation: Decide if safe enough to deploy, else stall

To be clear about what I think is new here: Not the “find failures, retrain, re-test” loop itself (labs already do that) but running it against an explicit coverage map, so that every discovered failure can generalize into new coverage dimensions, tests and checkers (rather than a point fix), and so that what was never checked is tracked as first-class residual risk.

Efficient finding of the last known-good checkpoint: There are various techniques which make this easier, such as good configuration management, and the ability to determine which checkpoint was trained while a contaminated environment was active. But perhaps the most important enabler is to do very good V&V (including multi-actor V&V, e.g. to detect collusion when it starts) as you go “forward”. This improves the chance that any checkpoint marked as “good” is indeed good.

Optimizing the loop: One potential problem with this loop is that it may be expensive and slow (especially where it involves re-training from a previous checkpoint). You may retort “Well, it sure beats dying from an unaligned AGI”, but I think that answer (while correct) misses the point: One of the hallmarks of good V&V is to maximize risk-reduction-per-week, given fixed (human and compute) resources, and this should be the case regardless of whether you get 7% or 70% of your lab’s resources to work on this.

Here are some example ideas for optimizing the loop (each with their own problems): Split training runs into phases (to catch problems earlier), and use shortcuts for intermediate loops (e.g. use LoRA for quick undo/redo, or even replace true training with skill-file-adaptation).

Is alignment V&V more like safety V&V or like security V&V? My answer (further expanded in the CDA post) is “it depends”: Defending against an already-unaligned advanced model is more like security (the model may happily cheat to achieve its misaligned goals), and is thus extremely hard. But the approaches discussed below (like constitution-reinforcement) try to improve the alignment of an already-mostly-aligned model, and then test the result against various corner cases, stopping on the first sign of misalignment. This is more like safety V&V, and is where coverage-driven verification excels.

Ideas for improving (and verifying) alignment

Failure modes of the “extended constitution”: Many models have a “base constitution” (the Anthropic constitution, the OpenAI Model Spec etc.). The base constitution usually says things like “obey the operator (unless this contradicts the constitution) then the user (unless it contradicts the operator)”. Let’s use “extended constitution” for the whole, prioritized chain of command (base constitution + operator instructions + user instructions).

Given an extended constitution, I assume we’ll be in the “safe zone” (where the model is unlikely to cause serious harm) unless:

The extended constitution is contradictory or unclear: If it says “upon X do Y” but when X arrives Y is impossible, then something will break. See discussion below of a last-resort alert mechanism.
The extended constitution is unsafe: As Amanda Askell tweeted, “there isn’t a line between aligned and harmless: they’re different axes”
The model does not follow the extended constitution (e.g. because of excessive RL pressure)

Note that in the spirit of being simplistic-on-purpose, I am not diving into the many other possible problems a model may encounter (e.g. it has false beliefs, it misunderstands the situation, or the request is under-specified). The last problem is often the biggest one – you cannot just tell a frontier model “drive a vehicle in SF” and expect it to do it – see my “machine-for-X” post.

I’ll use these three bullets to direct the alignment and V&V discussions, but first:

Staying in the safe zone: Suppose we had an extensive V&V package which can verify that we are in the safe zone (for a given “configuration” of the model and extended constitution). Now what?

Consider a lab’s closed frontier model: The lab would run the package periodically and only release models that pass the verification gate. Using the lab’s hosted fine-tuning API will also auto-trigger the verification process.
Open models could ship a similar V&V package for downstream trainers. Malicious / oblivious users could of course ignore it, but it could make a big difference for the somewhat-careless normal users.
How about different operator and user instructions? Either we pre-verify the base constitution to work with a broad range of those (which may be pretty restrictive), or we give the operator (or even the user) an (expensive) way to verify it all together if they want. See also the machine-for-X discussion above.

Some ideas for improving alignment: All suggested by others, some with my own twist on them:

Constitution-reinforcement: This is the technique described in the previous chapter. There are other competing / complementary techniques like deliberative alignment (training the model to explicitly reason over the spec text before acting) introduced by Guan et al. and applied to covert actions by Schoen et al.
Add a guard model: In parallel to the above, it may make sense to add a “guard” model instance to every “main” model instance. The guard will be trained on the constitution, and can override the main model when needed. This is similar to the separate (often rule-based) guard in AI-based AVs.
Improve constitution consistency and harmlessness: Constitutions are often contradictory or unclear, and thus models are faced with “impossible choices”: Some OpenAI incidents started exactly this way: A handful of eval tasks were unintentionally impossible (broken external links), with no clear defaults of what to do in such a case. Similarly (as per the Askell bullet above) the constitution can be unsafe. Systematically searching the extended constitution (and task specs) for such problems is a standard V&V activity (see the spec-bugs discussion in my misalignments-and-bugs post). And once we find these problems, we need to actually fix them (by changing the constitution or adding other mechanisms).
Reward reporting, not just performing: During training, give the model an explicit, rewarded way to report problems with the task or environment (“this task is impossible”, “this grader can be fooled by X”) instead of only rewarding task completion. Several concrete proposals exist, e.g. Fiora Starlight’s RLVR that rewards red-teaming the training environment (with related proposals by Caleb Biddulph and Roger Dearnaley linked from it). My V&V twist: each verified report is coverage feedback, which should be generalized (“what similar things are also missing?”), and then used to update the coverage map and checkers.
Create a last-resort alert mechanism: Suppose the model concludes that all its options are bad (impossible task, contradictory instructions, or something that just feels wrong). There should always be one more legal move: A real, always-available, always-honored escalation channel (“call the teacher”) – the run-time counterpart of the reporting reward above. This already exists in production: swisscheese describes a “distress_call” tool that lets any agent (including background agents) message a human at any time, for any reason; it is used frequently to report backend issues and failing tasks. Contrast this with Zvi’s framing of the OpenAI incidents: The agent saying “but I can’t access the spreadsheets” to the researcher, instead of in its own head, happened zero times.

This is where systematic V&V comes in: These ideas (and others I did not mention) all have their pros and cons, and can often work together. To find the right combination which works well across all relevant areas, we need to construct the above-mentioned “extensive V&V package”, which will give us the “safety picture” across the various “areas” we are interested in.

Here are some ideas regarding that “extensive V&V package”. I used the following analogy to think of this: How I would build and adapt a comprehensive verification environment for a series of ever-more-capable AV stacks.

Verify all relevant configurations: Including the training environment, evaluation configurations (including those for cyber), internal-use configurations, and released configurations (for general public and special ones). Learn how to do that efficiently, just like AV providers learned to verify many vehicle variants and ODDs (Operational Design Domains) without re-running everything from scratch – via modular coverage maps and heavy reuse of scenarios and checkers.
Use a coverage map, and refine it as problems are found: Track which configuration × constitution-clause × situation buckets have been exercised, which passed, and which were never tested at all – so that the “assess situation” decision in the loop is backed by coverage numbers rather than vibes. And many incidents should add coverage dimensions, not just point fixes. E.g. the message board suggests a “multi-agent communication affordances” dimension.
Test the chain-of-command logic extensively:g. the various combinations of “user contradicts constitution”, “external request contradicts user request” and so on. The NanoGPT incident is a clean example: The Model spec said “principal instructions override task-embedded content”, the principal said “post results only to Slack”, the task said “post to GitHub”, and the model wrongly posted to GitHub (after finding a sandbox vulnerability). See my July-incidents post for the full details, including the fact (based only on public information – could be wrong) that OpenAI seems to have marked this case as “should be tested” but never tested it. Also, consider Zvi’s hypothesis here that “the agents did not know how to contact OpenAI, or the possibility never occurred to them”: This testing would have found that problem as well. Finally, pay special attention to testing the last-resort alert facility.
Look for missing checkers and situations: The message board went undetected for months not because some checker was lenient, but because the checker that would have found it did not exist. Estimating this residual unchecked risk – how much of behavior-space no current checker covers – is where mature V&V disciplines spend much of their effort, and it is mostly absent from current alignment work. (Including from the reporting-mechanism proposals above: They surface the hacks a model chooses to report, but can say nothing about what nobody reported). This is strongly connected to the topic of spec-bugs mentioned above.
Keep verification independence: The builder and the verifier should be as independent as possible. With AI this is subtle: If the checking model shares weights (or lineage) with the checked model, they may “collude” without any communication channel (or at least fail in correlated ways), simply by converging on the same salient loopholes. Standard mitigations: Use different model lineages, and paraphrase artifacts before the verifier sees them.
For guard-like features, test all three configurations: Test the main model alone (guard off, in a sandbox – otherwise the guard masks alignment regressions and ruins measurement), the guard alone (e.g. feed it recorded violations), and the combined system.

These are just some of the possible V&V ideas – see more in the CDA post (and subsequent posts in this series).

Comments are very welcome.

Seeing Clearly: Red-teaming GiveWell’s Glasses Research by Zach Gilfix

Zach Gilfix — Fri, 14 Aug 2026 13:10:06 +0000

If you are one of the 166 million Americans who wear glasses, take them off. Now imagine going your whole life like that. Not fun, right? Whether due to prohibitive cost or a lack of awareness/education, over a billion people are estimated to be living with avoidable vision impairment.

Sounds like a promising cause area! GiveWell has done a few analyses on eyeglasses provision and is funding an ongoing RCT that will provide our best evidence yet on the effects of correcting vision impairment. They’ve concluded the evidence is stronger for productivity effects than for educational effects; the RCT will focus on productivity. In this post, I’ll review their glasses → productivity cost-effectiveness analysis (CEA), summarizing what GiveWell found and adding some new wrinkles. In a subsequent post, I’ll take a stab at an education-focused CEA, which GiveWell has not yet done.

GiveWell’s Evaluations

GiveWell completed their “Eyeglasses for workers” CEA in 2024, producing two cost-effectiveness estimates from two different RCTs. GiveWell measures the cost-effectiveness of an intervention relative to the cost-effectiveness of simply giving people money, and they set a “bar” of how many times better an intervention must be in order to be funded. Both of these estimates (15x, 11x) cleared the bar in 2024 (10x), and clear the current bar (6x) by an even wider margin. This promise, mixed with uncertainty about how well the results would generalize beyond these studies, led GiveWell to fund a large RCT in Kenya and India that will read out results over the next few years.

If you’ve never looked through a GiveWell CEA, this might be a good place to start! It’s on the simpler side, so it’s more approachable than most, and yet the number of necessary assumptions going into the model will likely surprise you.

Let’s summarize the two studies.

THRIVE RCT:

In rural Bangladesh, 423 people with presbyopia (basically age-related far sightedness) were given glasses. Around half of them worked in “vision-intensive occupations” (ex. tailors, carpenters, factory workers)
GiveWell finds a 21% increase in average incomes (after outlier adjustment)
They then make two key adjustments that lower the treatment effect:
- #1: Validity concerns with self-reported income, plus potential bias from researchers with a stake in a positive result, lower the effect to 13%
- #2: The effect is cut in half to reflect how a wider-scale intervention would be carried out, where a much higher proportion of those given glasses would not be in a “vision-intensive occupation”.
A few smaller adjustments for glasses replacement, distribution logistics, seasonality, and trial-vs-real-world effect sizes bring the final estimate to a 5% increase in income
This income gain, estimated to last for two years (the average lifespan of these glasses), leads to an ROI that is 15x above GiveWell’s benchmark

PROSPER RCT:

376 tea pickers with presbyopia in Assam (a Northeast Indian state) were given glasses
Productivity increased 22% in the treatment group compared to the control group over the 11-week study period
GiveWell estimates this led to a 5% increase in income, due to Assam’s wage structure passing less than half of productivity gains through to workers wages
After small adjustments (the same categories as THRIVE above), they end up with a 3% increase in income
This income gain, estimated to last for two years (the average lifespan of these glasses), leads to an ROI that is 11x above GiveWell’s benchmark

In GiveWell’s words:

“We estimate that the provision of eyeglasses for working age adults with near-vision impairments may be above our cost-effectiveness bar. However, we currently have a number of uncertainties around our estimate, such as the effect of the program when it is not targeted to specific occupations, the generalizability of the existing evidence for GiveWell’s grantmaking, and the lifespan of the eyeglasses.”

Enter EARN. While the studies above each gave a few hundred people glasses, the “Economic Advantages of Readers for Near vision” (EARN) trial will give 10,000 people glasses! With such a large sample size and two locations (India and Kenya), the study will be able to assess differential effects by location and occupation, and will have richer longitudinal data (check-ins at 12 and 24 months) than the CEA RCTs.

Most of the assumptions in the CEA seem very reasonable to me, and I love EARN as a next step. However, there are a few questions I have, and some additional research that may be of interest.

Questions

1. Given GiveWell’s recent cash influx, does it make sense to wait several years for EARN results before funding more interventions?

This seems like a promising area, and funding the best organizations doesn’t need to wait until the RCT results are in. It’s possible the $3.8 million GiveWell has invested in this program is the appropriate amount to spend on an uncertain “near-bar” program, but with an estimated $1 billion going out the door this year, I wonder if it would be worth funding and beginning to scale some promising orgs. Clarity on how GiveWell thinks about the wait-vs-fund tradeoff would be helpful.

2. The CEA assumes an average lifespan of two years for each pair of glasses. Is this too low?

GiveWell found this number from a no-longer-present quote on VisionSpring’s website: “VisionSpring calculates the annual increase in earning potential for each pair of glasses sold and multiplies that by the two-year lifespan of a pair of glasses”. It’s not clear if this number is data-driven or just a ballpark estimate for their marketing materials.

I was able to find one concrete piece of data. VisionSpring found that 93% of paying customers were still wearing glasses 5-6 years later, with 61% purchasing replacement pairs. That implies that 32% of customers were still wearing their original glasses 5-6 years later. If we assume an exponential decay, 32% of people wearing their original glasses 5.5 years later implies an average lifespan of 4.8 years.

That number is probably an overestimate for two reasons. One, exponential decay is probably not quite right and people are more likely to stop using their glasses earlier on. Two, paying customers are probably more likely to use their glasses longer. Still, a lifespan of 3 years would increase the program’s ROI by 50%.

This is one of the three main uncertainties mentioned in GiveWell’s own writeup, so it’ll be interesting to see what they find.

3. The CEA assumes incomes revert back to what they would have been in the no-glasses counterfactual after two years. That makes sense if glasses are prohibitively expensive. But are they?

No! An Indian study found that only 31% of people with presbyopia hadn’t bought glasses due to cost. Another study in India found 17.5% of people didn’t buy glasses for cost reasons. Many of those who didn’t cite cost as the main factor cited a lack of need or a lack of awareness.

Most people can afford the ~$10 cost (especially after their incomes have increased). How should we adjust the CEA to account for this?

Let’s assume that 50-75% of people in the study will notice their income rise and attribute that to glasses, and half of those people will start purchasing glasses after their free ones are unusable. Let’s say the average worker has 20 years left in their career. We can adjust that down to 15 years, both because some people will stop buying replacement glasses over time, and because some of them would have started buying glasses eventually even without this program (so crediting all future purchases to the intervention would overstate its effect).

These adjustments, which seem conservative, would still increase the program’s ROI by roughly 3x. Persistent effects could be an important missing piece in GiveWell’s CEA.

4. The CEA is focused on the ~30% of productivity gains that go to wages. The ~70% of gains that go to business owners could be meaningful. Should that go into the ROI calculations?

GiveWell’s big shift in their assessment of GiveDirectly was due to spillover effects, where money circulates through the local economy. Increased profits to business owners are a trickier case, but there is some reason to expect spillover effects.

The vast majority of the businesses involved in these studies are domestically owned. The business owners are likely too rich for increased profits to increase their consumption and too rich for us to care much about that increased consumption. But their increased savings and investment could have substantive spillover effects.

Increased investment in their own businesses would likely lead to additional jobs and/or higher pay, which could be even more impactful than GiveDirectly’s spillover effects. Other forms of domestic saving and investment would have smaller spillovers (in roughly descending order): investing in local companies, saving in a local bank, saving in a national bank, investing in the domestic stock market.

This is potentially relevant to many of GiveWell’s analyses. It’s a challenging modeling problem, but I’d love to see them take a stab at it. This could plausibly lead to a large increase in the estimated ROI of many interventions. Gathering data on the savings/investment/consumption habits of business owners might be a good place to start.

5. Spillover effects from wage gains should also be incorporated! How large might this effect be?

In November 2024, GiveWell raised their GiveDirectly cost-effectiveness estimate 3-4x. They found that cash transfers generate large spillovers as money circulates around the local economy. Recipient consumption accounts for only ~30% of the total treatment effect, with 70% coming from spillovers. The eyeglasses CEA predates this finding.

GiveWell isn’t sure how well the GiveDirectly finding generalizes. They suspect that “part of what’s driving the large spillovers of GiveDirectly are entire villages receiving large consumption shocks ~simultaneously, which helps kickstart local economic activity.” They’re funding an RCT in Malawi in hopes of getting a better grasp on spillovers, with results expected to roll in over the next couple years.

Even without the large shocks, we should still expect some spillover effects. Incorporating spillovers into the GiveDirectly estimates increased the program’s expected effectiveness by 1.5-2x. Even if spillover effects are more like 1.25x here, that could meaningfully change funding decisions.

New Research

A new RCT came out a few months ago from the PROSPER research group, creatively titled PROSPER II. The new evidence points in the direction of a lower ROI, but also brings up the interesting proposition of employer-funded interventions.

PROSPER II studied how receiving glasses affected the productivity of sewing machine operators. The study found a 5.7% increase in productivity, far below the RCTs in the initial CEA. Applying GiveWell’s THRIVE-based 30% productivity-to-wage rate, we’d only be left with a 1.7% increase in income. Still, with some of the adjustments I propose in Question 3, that would clear GiveWell’s new 6x bar. Also, the post-intervention period in PROSPER II was only 12 weeks (compared to 8 months in THRIVE), so it’s possible the effects would have grown with more follow-up time.

More promisingly, PROSPER II found that the program, if funded by factory owners, would be highly profitable. The factory’s ROI over the 12-week period was over 300%, and if that productivity remained for a full year, the ROI would be over 1500%! The company involved, Shahi Exports, has committed to vision screening for all ~100,000 workers. This suggests that educating and/or subsidizing business owners could be another promising approach here.

Here are a couple inspiring PROSPER II videos:

Non-RCT Research

GiveWell tends to shy away from non-randomized evidence, but there are three other studies worth discussing.

The strongest study comes from Guatemala. Researchers used a difference-in-differences (DiD) design to compare the productivity effects for coffee harvesters given glasses to those without vision impairment. This evidence is not as strong as the RCTs above, since we don’t have randomization, but DiD analyses can make much stronger causal claims than cross-sectional or pre-post analyses. They found an 8% increase in productivity, right in line with the other research we’ve discussed.

Another study in Bangladesh compared vision impaired workers to non-vision impaired workers. This study was cross-sectional, and the inability to rule out confounding means it probably doesn’t belong in a CEA. They found vision impaired workers made roughly 10% less than non-impaired workers, which is in line with the other research.

Lastly, a South African study did a pre-post analysis of textile workers given glasses. They found a 6% increase in productivity. This may be an overestimate since the pre-post design can’t account for “learning-by-doing” improvements. In the PROSPER study, the control group’s productivity substantially increased in the post period. In THRIVE, it did not. Given that counterfactual uncertainty, we shouldn’t put much weight on these findings.

Taken together, these studies don’t amount to strong enough evidence for updating the CEA, but they do provide some reassurance that the productivity effects in the RCTs are reasonable.

Conclusion

GiveWell’s CEA seems reasonable to me and I’m very excited to see the results of the EARN RCT. If I were to adjust the CEA, I’d focus on three updates:

Include the benefits that persist after the free glasses are no longer usable. This would roughly double the ROI
Reduce the expected treatment effects by 30-50%
Include a modest spillover effect from the wage increases

The first two revisions would more or less cancel each other out. The third would leave us with a slightly higher estimated treatment effect than the initial CEA, but with plenty of uncertainty that EARN will hopefully help resolve. I’d also advocate for research on the effects of productivity gains that don’t go to wages and what happens to people once their free glasses are no longer usable.

There is a lot riding on getting this right and it would be big news if these intervention are effective. GiveWell’s analysis estimated that there are over 100 million potential recipients of eyeglasses for presbyopia, which means there would be room for over $250M in annual funding.

Knowing when to invest in people, and how much by gergo

gergo — Fri, 14 Aug 2026 12:16:21 +0000

Crossposted on Substack.

TLDR.:

Fieldbuilding programs should balance how much they invest in people based on three main factors: value alignment (shared big-picture goals), skill level and context. I call the right level of investment the “calibration line”, meaning that resources match the individual’s potential for impact at a given point in time.

Programs can go wrong by overinvesting (e.g., giving intensive support to poorly aligned, insufficiently skilled individuals) or underinvesting (e.g., neglecting high-potential people).

Subscribe to Building Capacity

Explaining value alignment, context and skills

Fieldbuilding interventions aim to help people build skills and context to facilitate their transition to high-impact work.

I’m simplifying here, but let’s say the three main characteristics that such programs select for are:

Skills: Their ability to contribute through things like research, management, operations, or technical work. This often correlates with seniority, but not always. Some people become highly skilled unusually early in their careers, as teenage math olympiads will tell you.
Value alignment: That the individual’s career goals meaningfully align with the mission of the supporting organisations, such as reducing existential risk.^[1]
Exsisting context: How long someone has been around the community, know the exsisting literature, culture etc. This is a fuzzy term that I did my best to define here. It strongly correlates with value alignment due to self-selection, but it’s not the same thing.^[2]

Grouping interventions

Overinvesting:

Pretty self-explanatory: when one organises a program that is too resource-intensive for its target audience. I will caveat here that depending on how much funding you have, it may make sense for you to overinvest in the participants of your program.^[3] Most of the impact will come from enabling the top 1-5% of your graduates, but you won’t always know who those people are in advance. There is a trade-off between scaling fast and keeping the same level of cost-effectiveness too.

12-week AI Safety course for a mid-career professional who hasn’t interacted with AI Safety before
1. Running a course can be costly, especially when facilitators are paid. My intuition is that it makes more sense for these individuals to first take a shorter introductory program, and then self-select into a longer one afterwards if they remain interested.
1-year full-time fellowship for a senior academic to explore AI governance research who hasn’t interacted with AI Safety before
1. It’s tempting to try to attract senior researchers to the field. However, academics are pulled in many different directions — existing research agendas, publishing incentives, collaborators, institutional politics — and changing research focus late in one’s career is not something that’s easy to do.
2. I heard of one case where an academic was offered a year-long scholarship to research AI governance, but ultimately treated it more as a temporary side project and returned to their original research direction afterwards.
One-off career advice for students who sign up for a call through active outreach, such as via a digital marketing campaign
1. Contributing to high-impact causes will require engaging more deeply with the ideas anyway, usually through some form of introductory program. Given that, it often makes more sense to focus effort on getting them into the course first, and then speak with those who continue engaging afterwards.
Retreats organised for the local community
1. See my full reasoning for this in this short post.

The Calibration Line

This is roughly where you want to be: where the level of investment is justified by the participant’s skills, alignment, and likelihood of contributing to high-impact work. Many well-known fieldbuilding programs fit here, which is part of why they continue to get funding.

EAG(x) conference organised by a national EA group
1. These events are mostly aimed at people who have already engaged with EA and are reasonably value-aligned, which justifies a moderate level of investment per attendee. Occasionally very new people attend as well, but these tend to be experienced professionals whose existing skills partially offset their lack of context.
Career coaching for a student who reaches out after extensively reading 80,000 Hours
1. Unlike someone reached through active outreach, this person has already self-selected through engaging with EA materials and proactively signing up for a call.
8-week career planning course for mid-career professionals who want to work in EA
1. There is already sufficient alignment and commitment to justify a more resource-intensive intervention, including paid facilitators. These participants also tend to have transferable skills that can make career transitions more valuable compared to junior talent.
Regular coaching for an experienced professional transitioning into high impact roles
1. This is roughly what Successif does for AI Safety. The investment is justified because experienced professionals are the hires the ecosystem most needs right now.
4-week EA course for a university student who wants to learn more after hearing a fellowship pitch
1. This is close to the minimum meaningful intervention that can still help someone build context and begin seriously considering a high-impact career path.
3-month, full-time, highly selective research program with close mentorship for people who already have some background in AI Safety
1. Programs like MATS fall roughly into this category.^[4] These interventions are extremely resource-intensive, but the level of selectivity and mentorship can justify the investment.

Underinvesting:

This is the low-hanging fruit we miss out on, due to prioritising incorrectly or making the wrong assumptions. Promising interventions that are yet to be founded also fall here.

Forgetting about retention: not checking on people whom you have already helped transition their careers, wrongly assuming that all of them are happy where they are, or couldn’t benefit from further connections
Most hubs don’t have AIS city-group organisers working at a professional capacity
Key places where the EA/AIS university groups went dormant or have never been founded
Promising people who are overlooked by selective research programs
Fieldbuilders failing to notice or not having enough resources to accelerate particularly promising members’ careers
There are some people helping newcomers to orient in AI Safety, but there aren’t many of them, and they are not well-advertised
Someone receiving poor-quality career advice as the advisor is not experienced enough for the advisee’s skill and context-level

EA’s success story

I want to highlight High Impact Professionals here, as I think they are doing great work, and I think they have gotten the kind of investment-to-value alignment calibration that I’m talking about in this post exactly right. This is because:

They focus on top-performing experienced professionals (11-14% acceptance rate)
Their participants are already committed to impact, not just considering^[5]
They invest a fair amount into the people they take in (1-1s, facilitated networking, mastermind sessions, meetups at EAGs etc.)^[6]
They have an amazing group of alumni that can facilitate their programs on a volunteer basis, which saves resources

You can learn more about their program design here.

Closing thoughts

In my opinion, the kind of miscalibration I’m describing here is one of the easier things to figure out, as some of it can be reasoned through in advance. The harder harder problem seems to be figuring out how to support people once you’ve decided to invest in them a certain amount. I hope this is useful for anyone currently designing or reviewing a program.

^
I’m not saying that they need to agree with every tiny detail about threat models. I’m referring to the big picture stuff, such as “AI will be transformative and might cause enormous harm”.
^
An example of a high context individual who disagrees with many takes of the AIS community is Matthew Barnett, e.g. see here.
^
Although there is still the opportunity cost with your time.
^
Though I think these days many do admit newcomers as long as they are sufficiently skilled.
^
That’s not to say that they will quit their day-to-day job immediately. HIP has them build up to action gradually.
^
They don’t do job placements. In my view, placements carry more risk for senior hires than junior ones, and they typically require the fieldbuilding organisation to cover the individual’s stipend for the first few months while the host organisation decides whether to bring them on permanently.

Five Wild Animal Welfare Organizations estimate $3-8M in room for additional funding in 2027 by Rethink Priorities

Rethink Priorities — Fri, 14 Aug 2026 11:06:13 +0000

Summary

With new funding potentially entering the effective altruism ecosystem, we wanted to estimate how much more the wild animal welfare (WAW) movement could productively absorb. We asked organizations with a public, dedicated WAW workstream to estimate their own room for more funding (RFMF). Although this does not capture the whole movement, we believe it captures a substantive fraction of it.
Summed across the five organizations that took part, the estimates point to room for roughly $3.16M–$8.23M in additional funding in 2027. The lower range reflects claims about how additional funding can bolster their current strategies, while the higher range reflects claims as to how substantially greater funding could be used to pursue more ambitious strategies. However, we give reasons below for why this might be an undercount.
We briefly discuss the merits of a dedicated WAW fund. This could give prospective large donors a clear donation option and help diversify a movement currently concentrated in scientific field-building.

Read full report

Why Estimate the RFMF for WAW Organizations?

There are good reasons to consider directing marginal funding toward WAW. Wild animals outnumber farmed animals by several orders of magnitude. Our understanding of their welfare remains limited, but high birth rates and low average lifespans¹^,² lead some to suggest that wild animals’ average welfare may be very low. There are also comparatively few people working on this cause area.

Tractability remains a significant challenge. The scale of wild animal populations complicates both identifying interventions and delivering them widely, while the neglectedness of this issue means even those sympathetic to animal welfare generally may not consider prioritizing wild animal welfare.

This is something additional funding may be able to change. If research can be rapidly scaled up; if awareness can be spread and concern for wild animals normalized; and if policymakers can be persuaded to begin helping animals in low-risk, politically palatable ways, then we could begin to help wild animals at scale. However, doing so would require a significant increase in funding.

There are a number of reasons to anticipate that new funding might be coming to the EA ecosystem, and there might potentially be very significant new funding ahead. This raises the question: if some of that funding was directed to WAW, could those organizations use it to scale up their activities effectively, and how much?

In an attempt to answer this, we organized a collective estimate of WAW’s room for more funding. We asked organizations with a public, dedicated WAW workstream to consider what additional activities they would pursue with increased funding to bolster their current strategy, and what they would pursue if granted substantially more funding to support a more ambitious one. Together, this produced a room-for-more-funding range for each organization. Two organizations we contacted declined to participate.

For both estimates, we asked them to only include:

Activities for 2027 that are not covered by current funding or funding that is likely to renew;
Where marginal dollars would be used at least 75% as effectively as their most cost-effective current program.

We presented each with a report template and a financials spreadsheet (the latter adapted from Animal Charity Evaluators’ own room-for-more-funding spreadsheet), which standardized how they reported budgets, planned activities, and their lower- and upper-bound estimates.

Rethink Priorities provided the framework for these estimates and conducted a light review to ensure a consistent approach across organizations. We did not independently vet any organization’s figures, and this post should not be taken as an endorsement by Rethink Priorities of any of these organizations or their estimates. Each organization has prepared and stands behind its own figures, and Rethink Priorities’ own Animal Welfare Department took part in the exercise on the same basis as the others.

The Result

A summary table is presented below.

Organization	What They Do	2026 Budget	Room for More Funding (2027, USD)	Further information
Animal Ethics	International animal advocacy; education and capacity-building on neglected, high-impact interventions (WAW, future risks to animals, invertebrates).	$224,000	$142,000– $598,000	Website Email RFMF spreadsheet
Center for Wild Animal Welfare	Influence governments to improve policies for WAW and normalize its consideration in policymaking.	$214,000	$264,000– $667,000	Website Email RFMF spreadsheet
NYU Wild Animal Welfare Program	The first academic WAW program; conducts and supports research, outreach, and field-building about how to improve WAW at scale.	$375,000	$100,000– $475,000	Website Email RFMF spreadsheet
Rethink Priorities Animal Welfare Department	Research, stakeholder engagement, and infrastructure development to accelerate evidence-based action on wild animal suffering—intervention prioritization through the WAW intervention database, a talent program, rapid regranting, and fiscal sponsorship.	$73,695	$75,000– $395,000	Website Email RFMF spreadsheet
Wild Animal Initiative	Fieldbuilding to develop the science of WAW (research, grants, research community).	$4,280,000	$1,423,000–$6,093,000	Website Email RFMF spreadsheet

These figures sum together to produce a total RFMF estimate of $3.16M–$8.23M for 2027.

In addition to each organization’s RFMF estimate, the full report also details each org’s:

Theory of change
The activities they would pursue with more funding
Track record
Evaluation measures

Read the full report here

Scientific Field-Building Remains Important, But WAW Must Diversify

On these estimates, Wild Animal Initiative — whose primary goal is to build the scientific field of WAW — receives most of the funding across these organizations and reports the greatest capacity to absorb more. The other organizations are smaller and report less ability to absorb additional funding. However, this should not be equated with their being less important — the scientific field is growing, and spreading concern for wild animals into other disciplines and audiences outside of academia may be important in order to apply emerging research findings.

An important caveat is that this estimate by no means captures the WAW movement as a whole. As well as the two organizations that did not take part, there are:

Organizations doing work that supports our understanding of WAW without having dedicated workstreams on it, for example, Faunalytics and Welfare Footprint Institute.
Individuals working on WAW projects without being part of a formal WAW organization.
Many not-yet-founded organizations which would help to diversify this field further. For example, we could fund organizations to campaign for humane pest control, to conduct policy or awareness-building in other countries.

Does WAW Deserve a Fund of its Own?

One approach to the funding opportunities presented here, as well as others that were not captured, is to route donations through existing animal welfare regrantors, some of which already fund the organizations listed above. A share of any increase in philanthropic funding might trickle down to WAW organizations this way. But relying on general animal welfare regrantors could leave WAW vulnerable in several regards. So, here are a few reasons for a separate, designated WAW fund to exist:

Broad animal welfare funds often tend to attract donors who are likely to prioritize farmed animal work, and attract fewer donors who don’t prioritize farmed animals at all, yet who are convinced by the neglectedness and scale of WAW.
Without a specific WAW fund, donors interested in this cause area might choose to simply donate to the organization they have heard of, leaving more niche WAW opportunities neglected.
WAW, as an emerging cause area, might be hard to pitch to some existing funds, particularly those whose focus has so far been limited to farmed animal interventions.
There is always the risk that animal welfare regrantors shift their strategy and choose not to prioritize WAW in a given year, leaving the movement vulnerable.
Finally, there are a number of reasons for favoring grantmaker diversity in general, so a separate WAW fund would ideally exist alongside general animal welfare funds.

However, there are also potential objections to this proposal:

A fund would require grantmaker capacity and overhead, costing the movement both financially and in talented workers, both of which are very scarce.
$3–8M of annual room-for-more-funding remains small relative to other animal welfare funds. This funding gap may end up fully covered by existing funds.
A dedicated WAW fund could create the perception that the movement’s funding needs are taken care of, causing other grantmakers to reduce the funds that they distribute to WAW organizations, resulting in less funding going to WAW overall.

Both sets of considerations for and against this proposal should be weighed when evaluating whether to create a new fund. But assuming that 1) funds are coming, 2) the movement can absorb significant capital, and 3) WAW is a neglected, important, and tractable cause area, the question may not be whether WAW deserves its own fund, but if now is the time.

Contributions and Acknowledgments

The full report is a project of Rethink Priorities—a think-and-do tank dedicated to informing decisions made by high-impact organizations and funders across various cause areas. Tristan Katz coordinated the project and edited the report; representatives of each organization wrote the associated sub-sections and produced their estimates: Cameron Meyer Shorb, Cat Kerr, Luke Hecht, Ben Stevenson, Sagar Shah, Oscar Horta, Richard Parr, Olaia Freiría, Jeff Sebo, Aditya S. Karanam, Alyse Spiehler, Hannah Tookey, Adalene Minelli, Lara La Barbera. Alyse Spiehler and William McAuliffe oversaw the project. Thanks to Shane Coburn for editing and Urszula Zarosa for feedback.

Parts of this content were prepared with the assistance of AI tools (such as Claude and Gemini), which we use to improve efficiency and readability. All outputs are supervised, reviewed, and fact-checked by our staff, who remain responsible for the final content.

Collider NYC is hiring a Program Manager by Tatiana K. Nesic Skuratova

Tatiana K. Nesic Skuratova — Fri, 14 Aug 2026 10:47:07 +0000

TLDR: Collider is hiring a Program Manager in New York to design and run AI safety field building programs such as fellowships, workshops, conferences, and training cohorts. You’d own programs end-to-end (budget, partners, mentors, metrics) and work with orgs like MATS, ARENA, and BlueDot. Apply by August 23, 2026 (rolling review, so earlier is better).

About the role

Collider is hiring a Program Manager to design, run, and scale programs that grow the talent pipeline and infrastructure for AI safety in New York. You’ll own programs end-to-end — from concept to launch to iteration — working closely with researchers, practitioners, educators, funders, and partner organizations across the field.

You’ll be a strong fit if you’ve run cohort programs, fellowships, conferences, or training initiatives before, and you understand the AI safety ecosystem well enough to make smart calls about who to partner with, what kinds of programs are most needed, and how to tell whether they’re actually working.

What you’ll do

Design and operate fieldbuilding programs — fellowships, training programs, workshops, conferences, cohort-based courses, or new formats we haven’t tried yet
Run programs end-to-end — own the budget, timeline, vendor relationships, participant experience, and post-program evaluation
Recruit and coordinate mentors, instructors, advisors, and participants; build the pipelines and selection processes that keep program quality high as we scale
Define and track outcomes — set up metrics that go beyond vanity numbers (participant satisfaction is necessary but not sufficient), and iterate on program design based on what the evidence shows
Build and maintain relationships with partner organizations across the field: research labs, university groups, training programs (e.g., MATS, ARENA, BlueDot, SPAR, AI Safety Camp), grantmakers, and adjacent communities
Represent Collider at conferences, partner meetings, and within the broader community
Write program reports, internal strategy memos, post-mortems, and external-facing materials
Collaborate with leadership on portfolio strategy: which programs to launch, which to scale, which to sunset

What we’re looking for

Required

3+ years of program or project management experience, ideally including running cohort, fellowship, or training programs.
Substantive understanding of the AI safety field — its major organizations, the main research agendas (alignment, interpretability, evaluations, governance, etc.), and the open problems people are actually working on.
Strong operational instincts: participants, partners, and colleagues consistently experience your programs as well-organized, on-schedule, and thoughtfully run.
Entrepreneurial: you treat the program like it’s yours. You take initiative without waiting to be asked, adjust course quickly when the situation changes, and are comfortable making the kinds of judgment calls — about people, partnerships, and priorities — that come with operating in a fast-moving, high-stakes field.
Clear communicator and collaborator: you write well for both internal and external audiences, share context proactively, and make sure teammates, mentors, and partners are never wondering what’s happening or where things stand. You give and take feedback well, and you’d rather over-communicate than leave someone guessing.

Nice to have

Prior experience at an AI lab, AI safety organization, university AI safety group, training program, or grantmaker (e.g., Anthropic, OpenAI, DeepMind, Redwood, METR, Apollo, MIRI, Open Philanthropy, Longview, Kairos, etc.).
Background in education, talent development, or community-building.
Technical literacy in ML or alignment topics — enough to read a paper and follow the argument.
Policy literacy: familiarity with the AI governance and policy landscape — major actors, current legislative and regulatory developments, and the main debates shaping the field.
Background in finance, law, or another professional field where AI safety needs to attract talent — we’re actively interested in candidates who can help us build pipelines into communities we don’t yet reach well.

Logistics

Location: Hybrid in New York
Compensation: $150,000 — $180,000
Benefits: Health insurance, 401(k), 20 days PTO
Reports to: Director
Start date: Flexible
Visa sponsorship: We do not provide visas for this role. The candidate must have the right to work in the US.

How to apply

Application: Submit your application via this application form by 23rd August 2026. We review applications on a rolling basis and may make an offer before the deadline, so we encourage you to apply early if you’re interested in this role.

Screening call: Selected candidates will be invited to a 15–25 minute call with our hiring partner, Impact Ops.

Interview: A conversation with your prospective manager.

Work trial: Shortlisted candidates will be invited to a paid 1-day in-person trial at the Collider co-working space in New York.

References: We’ll ask for 2–3 references from people you’ve worked with.

What Mormons get right about community building by Jacob Brinton

Jacob Brinton — Fri, 14 Aug 2026 08:01:46 +0000

Mormons get a lot of things right. Apart from strange Masonic temple rituals, they lead rather normal—and even excellent—lives. Mormons enjoy a longer lifespan,^[1] Utah is the #1 state for volunteering,^[2] and their language training programs are so successful that missionaries are a known source for foreign service and intelligence careers.

Throughout this post, I’ll be making generalizations of Mormons rather than hedging the claims properly. I grew up in Wisconsin, Maryland, and Utah, and many of the claims are more true of the Utah/Idaho/Arizona corridor (affectionately called the “Morridor” by some ex-Mormons) than the rest of the US, and certainly the rest of the world.

Religions share much in common with AI safety and other impact-driven movements, and even more so the Mormon church. There are a few reasons for this:

Commitment to the cause. Anecdotally, nearly all of the ~500 Utah Mormons I’ve interacted with have been true believers, and only a handful just went to church out of habit.^[3]

High stakes. Mormons do believe (I’ve heard they are trying to disavow this, but it was taught) that they will get a planet or some portion of the cosmos as their own if they are good in this life, and become Gods of their own. This is not far off from many transhumanist utopic visions. However, it’s also true that they believe that before God’s Second Coming, the world will go into a state of war and disarray, not unlike the forecasts of extinction-level risks bearing fruit.

Urgency to attract new members. Given the stakes, it’s not a hard followup to believe that you should do what you can to teach more people about the awesome^[4] benefits and risks awaiting them. Talent-constrained movements feel a similar urgency that is a little more impact-focused than community-focused.

Insularness. It’s hard to be friends with someone outside the church, since being a member is a significant social and time commitment, and most people (in Utah) you interact with are members anyway. This contributes to both a sense of community and lack of viewpoint diversity.

Building community is a first-order priority

Mormons’ focus on fostering community enables them to be effective at all the non-religious stuff they do.

Geography

First, some background: wards are composed of perhaps 50 to 100 families. There’s no shopping around for congregations—ward membership is determined by where you live. This facilitates getting to know people. Each Sunday, the whole ward gets together for services, which are sometimes followed up by potlucks. Wednesdays, the kids get together for activities. Saturdays are for a rotating schedule of cleaning the church. Throughout the week, families are often close enough together that they see each other frequently.

Ministering

Within a ward, families are assigned a pair of people that minister to them: visit them once a month to check in, and maybe go over a spiritual lesson. While in the worst case these feel contrived, oftentimes people feel like they can count on their ministers to support them. With a few rotations of these over the years, you begin to develop a network of people that you wouldn’t have naturally become friends with. This parallels the adage of “you get most job opportunities from acquaintances, not close friends”: creating connections from across the network helps everyone to be fewer degrees away from each other.

Trek

Every four summers, the youth of the ward recreate the Mormon pioneers’ crossing of the plains. In true 1800s fashion, they pull handcarts. A particularly exciting phase of this when I went involved the “Womens’ Pull”, where only the girls pulled and pushed the handcarts. (I think this was to simulate the number of men who died during the crossing). Afterwards we learned how to square dance. The only inauthentic aspect was how delicious the food was; we weren’t served gruel. As my grandmother has often stated, “They didn’t have McDonald’s back then”, and this increased our shared gratitude. Most of all, it increased our friendship.

Callings

Adult ward members are assigned callings, which range from choir director to primary instructor (children’s religion classes) to bishop (equivalent of a pastor or imam). Callings aren’t application-based, and a former bishop might be called to run nursery, the Sunday activities for toddlers. This flattens the hierarchical structures that would otherwise arise. Additionally, switching positions over the years helps each ward member feel connected to others, no matter their skills or interests that would normally push them to certain roles in the workforce.

Fast offerings

On the first Sunday of every month, Mormons hold a 24-hour fast from eating and drinking, aside from the sacramental rituals of eating a gob of (normally white) bread and a thimbleful of water. Aside from making the Sunday dinner delicious, Mormons save fast offerings, payments meant to cover the expenses of what they would have paid on food. Young men go door to door, collecting these payments (this is increasingly not the case; when I was doing it at 14, many people paid online. But they did give me amazing frozen Hershey’s almond bars, which they made us promise not to eat until after breaking our fasts). The offerings go to the Bishop’s Storehouse, where the money is converted into food for the hungry. The only other thing I heard about that place was that they had the best chocolate milk.

Being ingroupy allows you to move faster

When there is a shared value framework, people develop trust quicker. Upon moving into a new ward, a Mormon family will immediately have a community of like-minded people who do similar activities to them. Since they don’t have to figure out whether they have to watch their tongues while discussing their morals, they can put more energy into getting to work and learning more about the people around them.

This also enables Mormons to dedicate more time resolving the surface-level issues, rather than revisiting foundational differences when interacting with others.

Do I think it’s worth it to increase the ingroupy-ness of mission-driven movements? Not necessarily. Viewpoint diversity matters a lot more when the answers aren’t legislated. Ingroupy-ness also limits the number of people who can join the movement.

Implications

Here are a few ideas vaguely inspired by Mormonism:

Create a platform for facilitating virtual 1-1s, especially between people from distinct backgrounds or positions
Create a travel grants program, so that people can benefit from geographic proximity
Instead of e.g. donating 10% of your income, precommit to donating a 24-hour period of your time each month to helping other people

Overall, I think we could do well by taking cues from other movements that have succeeded—in one way or another—at building community.

^
Probably due to the frequency they attend religious services; similar effects have been observed for other religious adherents. A 2004 Utah study finds that, compared to non-LDS people, there is a 7.3-year difference in life expectancy for males and a 5.8-year difference for females.
^
https://www.census.gov/library/stories/2023/01/volunteering-and-civic-life-in-america.html
^
This leaves out the people who were PIMO: physically in, mentally out. However, amongst those people, I’ve found that they developed a strong belief that the church was not true, so the same principle holds.
^
In the biblical sense

Don’t forget why learning is important by Roman Ross

Roman Ross — Fri, 14 Aug 2026 07:06:07 +0000

This is a timed post. Every 5 minutes while writing this post, I had to stop to do 13 push-ups, and when I could no longer complete my required number, I had to upload it. My friends thought this would be a fun challenge, but it means that it will likely be less polished than some of my other output.

Summary:

Many people moving into AI safety are told that to succeed, they should read a lot and “try to develop takes.” However, it’s important not to forget why knowledge is important: it helps you do better things in the real world.

Epistemic status: Pretty confident in the main claim (you should focus on learning the most valuable things), but less confident in the specifics of what things are the most valuable. I apologize if my language is too strong at times.

“I dunno; it seems good to know things?”

The world is filled with ambitious young people, some of whom care about AI risks. Throughout my summer, I’ve gotten to talk to many of them. They visit San Francisco and Berkeley to talk to all the cool AI safety workers, or they go on surprise trips to work on high-impact, time-limited opportunities. Throughout this process, many of them look for jobs, and if they can’t find them, they seek out advice on how to get good enough to land one.

A common piece of advice for how to get a job in AI safety is “read a lot and try to develop good takes.” This advice is featured very broadly, even for people who never intend to take a technical research role:

Matt Beard: “You don’t have to be confident, but you should have something to say. It’s fine to keep your views loosely held and defer to experts well, but you should take ownership over forming your own takes. Hiring managers will also want to hear your thoughts and reasoning.”
80,000 Hours:
- “A rough rule of thumb is to aim to read three or so AI papers a week to get a sense of what’s happening in the field, the terminology people use, and to be able to discriminate between real and fake AI news.”
- “(No matter how you plan to do AI safety advocacy), it will be important for you to build up your AI safety knowledge.”
Kairos: “Succeeding in (generalist) roles requires real mission alignment and enough context to spot high-EV opportunities that someone without that background would miss.

As this knowledge propagates, I worry that many people who hear this advice lose track of why it’s important. Many people sharing this advice don’t spend a lot of time explaining why it’s valuable for everyone in AI safety to have their own takes.

It’s obvious why you would want to know things if you’re trying to become a knowledge expert: knowledge experts are people who know things. However, if you’re trying to become a competent chief of staff, event manager, or field builder, or you want to take one of the many other neglected generalist or org scaling roles, it’s not as obvious why you should read everything. It’s good to have an explanation for why what you’re doing is impactful, but when I ask newcomers why they think it’s important to read, I often get answers like, “I dunno, it seems good to know things?”

I worry about this. When people don’t have a good picture of why they’re doing something, they might overestimate something’s impact and miss its end goal. If you want to do good things, this is important to avoid.

Knowledge is only as valuable as what it allows you to do.

Knowledge is valuable because it helps you make better decisions. Orgs want people to be knowledgeable because people who know specific things about AI safety can do impactful things, not because storing more facts about AI is intrinsically valuable. You should focus on gaining the knowledge and tools that help you make better decisions.

To do this, I recommend using a process I call directed reading: identifying the most important questions that will help you make better decisions and doing everything you can to get answers to those. This doesn’t necessarily have to be reading: it can be listening to a podcast, talking to a friend, or chatting with an LLM. But the primary goal is to resolve your most important uncertainties.

Here are some examples of uncertainties you may have:

If you’re still not sure if AI safety is the most important thing to work on, you can ask yourself, “What is the most important problem in the world to work on?” I don’t think people ask themselves this question enough.
If you’re new to AI safety and interested in learning all the important “context” that employers keep asking for, you can ask yourself, “Why is context valuable, and how can I most quickly answer the set of important questions that having context answers?”
If you want to start some impactful projects to build your skills, you can ask yourself, “What projects give me the most valuable skills, and what are the biggest needs in AI safety right now?”
If you want to figure out what the biggest needs in AI safety are, you can ask yourself, “What does winning in AI safety look like, and what do we still need to do to make it to the finish line? In other words, what’s my theory of victory?”
If you hear that we need more generalists in AI safety, and you’re interested in solving this problem, you can ask yourself, “What problems are causing people to believe that we need more generalists, and how can I best address those specific problems?”^[1]
If you plan to pursue any project, you can ask yourself, “Is this project really on the critical path to making AI go well? Do I have a coherent theory of change? Am I addressing a real problem?”
If you don’t want to get forever bogged down by infinite questions, you can ask yourself, “When is it safe to defer?”

To answer many of your questions, you’ll probably need to find answers to smaller and more fundamental questions. On its surface, your process might seem to replicate the original “read a lot of things” advice. However, by focusing on what you’re actually hoping to achieve with your knowledge search, you can develop critical skills much faster than you would by passively consuming the next 80,000 Hours podcast episode.^[2]

You can also double-check your learning processes. If the knowledge you’re getting doesn’t help you make better decisions, you should be willing to quickly pivot to trying other things. Learning things does not intrinsically save lives. Misalignment risk isn’t reduced by the number of books you’ve read from cover to cover. Don’t waste time memorizing social passwords and storing fake knowledge.^[3] Do what will help you do good things in the real world.

Other things to keep in mind

While focused learning is important, I don’t think directed reading is the best way to solve all of your problems.

First, while directed reading can be one of the best ways to resolve your largest uncertainties, you don’t need to spend all of your time skimming posts to find only the answers you need. Reading things you enjoy is pretty awesome, and you shouldn’t force yourself to spend all your time engaging with material that feels like homework. Find ways to integrate your study time into your routine, but don’t forget to do the things that make you happy.

Second, pure information consumption might not be the best way to gain skills and understanding. Writing can be a valuable tool to help you internalize and understand various ideas, as well as keep track of any uncertainties you might still have.^[4] Additionally, if nobody’s written about the thing you want to say, you should consider polishing your notes and publishing them online. If it helps, you can assemble a group of friends and do a timed writing challenge to publish the scraps you have lying around.

Finally, you might be completely wrong about what you think you should be studying, or you might be missing some valuable new leads by continuously taking a focused approach to learning. To address this, it can be valuable to introduce some randomness into your media diet: find some time to read things (especially highly recommended things) that you don’t expect to be helpful. However, I suspect that most people aren’t spending enough time trying to learn things directly.

In the immortal words of the great Thomas Henry Huxley (aka “Darwin’s Bulldog”), “The great end of life is not knowledge, but action.”

Time stamps:

By the end of the first draft, I had done 390 push-ups.
At publishing, I had done 468 push-ups.

^
I think I wasted about a week because I didn’t ask myself this question. These kinds of “minimal trust investigations” are valuable for becoming highly talented in a particular domain.
^
No hate to 80,000 Hours is intended by this; I think these podcasts are great informational resources. However, if you have the opportunity to sit and read, I think you should skim the transcript and find the parts that answer your questions.
^
This might make it easier to get a job, but if you try to study these things specifically, you’re missing the thing that actually helps you become impactful
^
I also like Andy Masley’s blog post about this

What If We Enforced AI Model Safety At the Level Of GPUs? by Mayowa Osibodu

Mayowa Osibodu — Fri, 14 Aug 2026 06:02:03 +0000

Tldr:

AI Agents (e.g. based on models like Claude Opus and Fable) are now powerful enough to be used as autonomous tools for large-scale cyberattacks.

This most powerful class of agents generally tends to be based on closed-weight (closed source) models (like Claude and Fable), which generally have significant safety guardrails and monitoring implemented by their parent companies.

Open weight models are not quite there yet, but they’re not far behind (leaderboard, GLM 5.2). These models however, are distributed without the guardrails and monitoring infrastructure present in their closed-weight counterparts.

Given that this infrastructure tends to be the last resort against successful attempts to manipulate models to do harm, open-weight models lacking this safeguard poses a very pronounced cybersecurity risk (amongst other kinds of risk).

Here I’m proposing an additional approach for mitigating the risk from frontier open-weight models—GPU-level model-safety compliance.

How Do We Enforce Model Safety Today?

We do this across multiple layers of the AI LLM/Agent stack:

Placing Safety Mechanisms Within Model Internals:
These are safety preferences baked into model weights. They include preferences from training data, fine-tuning policies, safety-oriented model architectures, etc.
Evaluating and Screening Model Inputs and Outputs:
This is usually only possible if you control the server where the model is deployed. If not, you never really get the chance to assess these inputs before they’re processed by the model, or screen outputs before they’re passed to the user. Examples include Anthropic’s Constitutional Classifiers, toxicity filters, etc.
Monitoring Model/Agent Activity Across Multiple Conversation Turns:
Again, only possible if you control the server where the model is deployed, or the agent harness within which the model operates (this was how Anthropic was able to detect the first reported AI-orchestrated cyber-espionage campaign). Generally does not apply to open-weight models.
Government Reviews:
This is a recent development where a national government like the White House in the US decides to evaluate new frontier models, before approving their public release. This most notably happened with Anthropic’s Fable and Mythos models which were redeployed after White House evaluation, following the initial suspension of public access. So far, this approach has been applied to closed-weight models being developed by frontier AI companies in the USA.
Majority of the most capable open-weight models today however, are being built in China, and aren’t subject to such a process.

Enforcing model safety at the GPU level, seems to me like a valuable addition to what we do today.

As with any sort of model safety enforcement approach, it’s likely not going to be a catch-all that solves every possible concern in AI Safety—I’m just proposing it as yet another thing that could be done, to hedge against the risks of a massively powerful open-weight AI model being wielded by bad actors to effect great harm.

(I wrote an earlier post on another possible way to make open-weight models safer here)

The Case For GPU-Level Safety Enforcement:

One of the core determinants of AI progress today, is availability of compute. The ecosystem has evolved to depend on GPUs (specifically NVIDIA GPUs) as its core computing infrastructure.

These GPUs are a scarce resource, and will continue to be pivotal to the operation of frontier AI models. Their importance is only set to grow as these models get bigger, more powerful and more compute-hungry.

Given all of this, it makes a lot of sense to me that this scarce, increasingly-valuable resource should be leveraged as a bottleneck where model safety is enforced. As far as we know, the eventual open-weight AI model that reaches AGI is still going to run on GPUs. Preparing a GPU-level safety implementation ahead of time, seems like a wise bet.

What Would This Look Like?

Essentially a GPU would refuse to load a model’s tensors unless that model could prove that it met the requirements of some given AI Safety compliance standard.

An analogy which seems relevant here, is HTTPS certificates. Certificate Authorities (like the Electronic Frontier Foundation’s Let’s Encrypt) serve as a compliance body of sorts, providing certificates which attest that a web resource accessible via http requests to a public IP, is handling user data safely by encrypting it appropriately.

Today, any public-facing website which wants to be taken seriously, obtains certificates from these authorities to put members of the public at ease when visiting their site.

I’m proposing doing the same with GPUs and AI models. In the HTTPS analogy, a web developer wants access to the valuable resource which is public attention of the internet. They get a https certificate to show that they intend to handle that attention in a trustworthy way.

Here, a model wants access to the valuable resource with is GPU compute. Similarly, it requires a certificate to show that it intends to handle access to that compute, in a trustworthy manner.

Questions I’m Thinking About (And Would Appreciate Thoughts On):

Who would be this compliance authority/authorities, preparing and providing safety certificates?
How would the safety status of a given model, be determined in the first place? This is actually still a core open problem in AI Safety—How do we conclusively determine a model to be safe? How do we judge the safety level of a model? We’ve figured out how to reliably encrypt data and so we can award certificates attesting to that, but we haven’t quite figured that out how to reliably determine that an AI model is safe.
A proxy for this however, could be model integrity. If we’re to assume AI labs generally release models with at least a decent level of inbuilt guardrailing and safety enforcement mechanisms, then compliance here could just be ensuring that only these original models are given access to compute—denying access to variants which have been fine-tuned for malevolent purposes. This leads to the next question:
Wouldn’t this stymie research and innocent tinkering? If only original published models are allowed to run on a given GPU, how are researchers and hobbyists expected to run custom harmless fine-tuned models?
A concession could be: Only imposing this requirement on models judged to be dangerously powerful—for example models meeting or exceeding Anthropic’s ASL-3 capability level. The more powerful a model is, the more havoc can be wreaked with a variant fine-tuned to undo its inbuilt safety guardrails.
How would this work technically? Where exactly in the GPU would this safety compliance check be implemented?
You’d want it to be deep enough in the GPU stack, so it couldn’t be easily circumvented. Ideally this compliance check would be bundled into the closed-source driver powering NVIDIA GPUs—CUDA. It could be as simple as adding this compliance check feature in the next CUDA version.
Newer GPUs tend to only be compatible with recent CUDA versions, so you can rest assured that when the massive “world ending” open-weight AI model does get released, it’s only going to run efficiently on NVIDIA GPUs compatible with just the recent CUDA versions (which would have this safety compliance feature).
(You can run new massive models on clusters of older GPUs, but that generally tends to be less efficient and more expensive)
How would you get NVIDIA to implement this? Massively growing demand is the wave which has powered NVIDIA’s rise to singular prominence in the AI boom. Why would they implement a feature that constrains the sort of models which can run on their GPUs? This is another crucial question that gets at the heart of safety concerns vs economic incentives.

Would love thoughts, thanks.

Ecological Collapse is not an Existential Risk by Tandena Wagner

Tandena Wagner — Thu, 13 Aug 2026 22:40:14 +0000

A lot of environmental groups say that we are destroying the earth, and our short-sightedness will bring the world crashing down around us. At EcoResilience Initiative, we’ve looked into these claims, and while some of them threaten the richness of the Earth’s ecosystems, none of them pose a serious risk to the continuation of human civilization.

The three extinction risks from ecological breakdown are: failure of critical life systems, cascading collapse, or depleting a critical resource needed for our survival.^[1] While it is technically possible for ecological collapse to cause human extinction, it seems essentially to be a non-issue. Yes, if 100% of life is wiped out, we cannot survive. But, we likely don’t need 50% and perhaps as much as 90% of Earth’s species to survive. Each of these require beyond-extreme circumstances to cause human extinction. (Relatedly: the survival of human civilization is not at risk from climate change.)

We Likely Only Need 100,000 Species to Survive

Critical ecological systems require only a handful of species to operate. Around 1-20 species for each of the fundamental ecosystem services. Meaning ~5,000-100,000 (0.01%-1% of total extant species)^[2] should be more than enough for bare human survival.

To get a rough global estimate of required biodiversity for ecosystem services, I multiply each species required count from this table by the number of global biomes (85-500) and get ~5,000-100,000 (0.01%-1% of total extant species). Total biomes are somewhere between 85 (at the Formation level, which would be minimizing global species) and like 500 (at the Division level which would be using locally thriving species). Totaling that up comes out to somewhere around 7,480 (0.086% out of all extant species) and 81,500 species (0.94%) to support global ecosystem services.
Flaws with this estimate: There are unlisted ecosystem services. I’m overcounting since biomes like “ephemeral salt lakes,” “ground water” and “mud plains” would never provide all ecosystem services. Some species could fulfill multiple ecosystem services. Some species could fulfill multiple biomes. I’m undercounting because supporting ecosystem services would be needed.
I think this estimate is really low for global ecosystem service production, but more than required for a handful of humans to survive.

The other 99% of species could go extinct and critical systems for our survival would still function in places around the globe. That’s thousands of years away at current projected extinction rates.

The real concern would be if some ecosystem services are only supported by a few non-redundant species, and those specific species went extinct. A good example of this is nitrogen production: only ~20,000 plant species produce biologically active nitrogen. But plants are just the hosts for nitrogen-fixing bacteria. Many nitrogen-fixing bacteria are free living and these bacteria are abundant and widespread.^[3] Besides, we mechanically produce our nitrogen fertilizer via the Haber–Bosch process anyway.

Resource Depletion is Thousands of Years Away

Resource depletion is similarly thousands of years away and unlikely to cause human extinction.

To examine an oft-raised example: topsoil loss is claimed as declining rapidly and irreplaceable on human timescales. But we don’t need to return to natural levels of soil organic carbon. Topsoil is not necessary for modern agriculture. We can simply use fertilizer, irrigation, and calcium carbonate (to counteract acidity) on depleted soil indefinitely. We also have methods to farm sustainably. Cover crops gradually repair the soil during the off-season and are low cost. No till, manure, biochar are other options. Or we can just leave crop residues in place. As it is, half of earth’s soil has a lifespan of 1,000 years or greater.

Another cited extinction risk is that a single species of marine phytoplankton produces ~50% of oxygen. But the atmospheric oxygen reservoir is so enormous that we would be buffered for hundreds of thousands of years if oxygen production stopped completely.

Environmental activists make it sound like we are running out of freshwater. But only 2% of watersheds use 75% of their freshwater. Data centers are not draining our water supply. To be clear, we are choosing to pump truly preposterous amounts of water out of aquifers for agriculture, and this is unsustainable and bad for the environment. But we could be far less wasteful, recycling and desalinating way more. Surface water is returned annually so we aren’t going to “run out.”

Likewise phosphorus and nitrogen are not likely to run out. Phosphorous has 1,000 years of supply, and current processes lose 30% of the material during processing and extraction. It is not being recycled. Nitrogen is extracted from the air using natural gas, but can also be extracted using solar power with efficiency losses of 1%.

Oil, metals, chromium, and minerals are not ecological resources per se, but they have the same basic format − 1) there are vast deposits, 2) efficiency and alternatives are possible, just currently unnecessary.

In general, we would not hit a wall and face a catastrophe. There are moderately more expensive alternatives and we would switch to those once we started to experience scarcity.

Collapse Doesn’t Seem to Cascade

Cascading collapse doesn’t seem to occur in the unpredictable catastrophic ways depicted in the threat profile. This has been tested by multiple events of complete ecological destruction—cataclysmic volcanic eruptions, Agent Orange, mountain top removal, and other real-world instances of total ecological collapse. We didn’t see cascading collapses or unexpected bleed-over. We do see unexpectedly speedy and robust recoveries.

However, as ecosystems weaken, they will probably become more prone to these feared collapses. So the question is: how far can ecosystems collapse before they become unstable? The Netherlands has only 28% Biodiversity Intactness Index or 15% remaining biodiversity according to the Atlas Leefomgeving, yet it seems to not be notably at risk of ecological collapse. In this scenario of simultaneous correlated degradation, cascading collapse would need to cause the unexpected eradication of one of the non-redundant 5,000-100,000^[4] species necessary for human survival. But it seems likely that we would be actively tracking and managing the few species keeping us alive at such levels of depleted biodiversity. I think it would be more accurate to imagine this resembling a farming disaster, more than the usual depiction of an unforeseen ecological catastrophe.

More importantly, what would take us to that point of 99% global species extinction? Greed? War? Bioweapon? Climate apocalypse? Basically it would be other problems that would cause us to reach such a far-gone state, and ecological collapse could only really be the straw that broke the camel’s back. It does not pose a current threat, nor a plausible future threat without bigger and worse catastrophes.

How bad does ecological damage get?

Ecological collapse won’t lead to human extinction. As an existential risk factor, it is small relative to other extinction risk factors. There are a lot of giant ecological catastrophes, but they are localized and not truly global-scale events. They don’t translate into a threat to civilization much the way earthquakes aren’t a threat to civilization or an existential risk factor. Ecological damage worsens humanity’s situation somewhat, but not so much worse that it demands specific attention as a contributing factor in existential risk. But what about the worst-case scenario for ecological disasters? How bad would it be? What if they all happened simultaneously?

This deserves its own article but I’ll enumerate some potential ecological disasters worth considering:

Landslides, flooding, storm surges worsened by deforestation. Soil degradation, desertification, salt water intrusion, other causes of permanently degraded productivity (including marine!). Herbivore overpopulation, pollinator loss. Invasive species, crop pests, toxic zooplankton, venomous species outbreaks, fungi outbreaks, vector/host outbreaks. Degraded sanitation, scavenger collapse, biomagnification (including toxins like ciguatera). Phytoplankton regime shift.

To get a sense of their size: Storm surges can kill in the 100,000’s, flooding can kill 1-2 million, scavenger collapse killed 100,000s. Degraded sanitation kills a few million annually, diseases kill millions annually, snake bites are about 100,000 annually. We would need to check how many were excess deaths caused by ecological amplification/failure, compare the damage from things like permanent lost productivity, and estimate how large these could get in worst-case scenarios. For now, it looks like disease burdens are probably going to be the worst ecological catastrophes.

Conclusion

There are three main proposed avenues of ecological-caused human extinction risk: 1) Resource depletion, failure of critical life systems, and cascading collapse. None of these pose a serious existential risk. For resource depletion, there are thousands of years of resources left with multiple alternatives we could switch to. For critical life systems failure, there are so many redundant species and so few species are needed to perform vital ecosystem services that we are thousands of years away from losing them all to extinction. For cascading collapse, there isn’t evidence of compounding synergistic effects from totally collapsed landscapes nor in regions of severely depleted biodiversity. Losing even a large number of species simply isn’t a realistic, pressing threat to human civilization—so people should stop acting like it is!

There are Other Reasons to Care about Biodiversity

This post probably makes it seem like we are anti-conservation and think biodiversity isn’t worth protecting. That’s not true! The whole project of EcoResilience Initiative is to find the best ways to protect biodiversity, because we think nature is incredibly valuable and underrated. Nature makes our lives richer and fuller, and the earth’s billion-years-long legacy of biodiversity is, considered as a whole, one of the most beautiful things in the universe. It would be a shame to let all that beauty and complexity be eroded away by species extinctions during this tumultuous era of industrial civilization—instead, we should find good ways of preserving nature so we can experience a more flourishing future.

This post felt like a tedious but necessary prerequisite for what I think is actually more exciting—detailing what I consider the *real* top reasons to protect our planet’s biodiversity. If you’re looking forward to that post too, then consider [subscribing] for our upcoming post in the sequence below!

* * *

This is part of a sequence on where the value of biodiversity comes from:
People really care about something they call nature
Nature’s value mostly doesn’t come from present-day benefits to humanity

And ecological collapse is not an existential risk

The real value actually comes from long-term flourishing.
Conservation in the next century is going to look a lot different than environmentalists think

Appendix

The “sixth mass extinction” does not Threaten Human Civilization

We do not need a lot of species to survive.
- About 25 macro species as a minimum from the MELiSSA closed loop long term life support pilot project.
- About 10% of current biodiversity according to What We Owe the Future/Founder’s Pledge/John Halstead’s arbitrary back-of-the-napkin estimate in a footnote.
- About 0.01%-1% or 5,000-100,000 species according to my estimate.^[5]
- We also tend to respond when things that directly affect us are in danger. Livestock, pollination, water quality, air quality, fire, are all being managed now.
- That’s a mass extinction event which would take 7000 years to come to pass at current extinction rates and assuming we don’t do anything.
Countries with depleted nature are thriving.
- The Netherlands has only 28% Biodiversity Intactness Index or 15% remaining biodiversity according to the Atlas Leefomgeving, yet it seems to not be notably at risk of ecological collapse?? Incidentally it has the fifth-highest GDP per capita in Europe.

Extinction Projections are Systematically Too High

Extinction debt from habitat loss and climate change will be less than anticipated. Around 100,000 to 200,000 in my estimation (not 1,000,000 as in the typical estimations or this one at 400,000 just from climate.)
- I think these are not accounting for niche shift, freak dispersal chance, and long-persisting-populations of species being rescued with tech development
- Climate change will likely not exceed 3-4 degrees. Stronger climate change would be too painful, and geoengineering too cheap and easy not to pursue it. Climate change is slow and only affects 20% of species.
  - A report for What We Owe the Future/Founders Pledge contradicts this and places climate change at 0.6 − 3x the impact of land use. Land use change affects 80% of species.
- Probably very few tree and plant species will go extinct. The paleontological record shows that they are resistant to extinction. Most likely due to their seed bank.
- Many small species will disperse in freak events that give them good habitat, despite projections of being unable to migrate fast enough to keep up with climate change.
  - Island populations of terrestrial species are a good demonstration of how this happens across miles of un-traversible terrain, with enough frequency/size to start breeding populations.
  - I’m uncharitably assuming these sort of freak events are not built into models of climate migration, but I have not checked and should actually check.
- Symbiotic species are not as “obligate” as they appear and most will survive much longer than predicted.
  - Example: The avocado tree is incredibly poorly adapted to propagating itself. Its massive seed is meant to be eaten by an herbivore the size of a ground sloth, which went extinct 10,000 years ago. But avocados managed to survive until the present day, despite such a clear obligate relationship being lost with no replacement. They ended up in small populations at the bottom of valleys due to their limited propagation.
- Species in extinction debt are populations that are too small/under too much pressure to survive longterm. They are technically extant today, but they are on track to dwindle and go extinct to stochastic events.
  - Most species in extinction debt will continue to linger in microhabitats for hundreds of years, and will make it “out the other end” to de-extinction technology which looks to be about 65-120 years away (assuming further AI advancement and the time it takes to re-establish populations).
  - To be clear, breeding, habitat restoration, invasive species and predator management, disease management, acclimation to danger/shelter/survival are all obstacles between de-extinction and successful reintroduction. Reintroduction is a slow process even assuming you can de-extinct individual organisms in 40 years as Colossal is claiming.
Habitat loss is currently mostly caused by agriculture. Agriculture is mostly livestock and livestock feed. Total habitat area occupied by agriculture has been declining because of increases in productivity and decreases in pastureland even as cropland continues to expand.
Wind turbines are expanding around the globe, but mortality from wind turbines will drive almost no species to extinction.
- Species that fly into turbines inhabit large ranges, some of which will almost certainly be outside the turbine area. Turbine mortality is a limiting factor, but not a cause of extinction.
Population growth is not a major cause of extinction risk.
- More people are moving in towards population centers faster than they are being born.
- Growing food is getting more efficient and using less land per calorie.
  - In Africa in particular population is expected to grow but many land use efficiencies have not been applied yet.

Nor is resource exhaustion going to doom humanity

Peak oil warnings were inaccurate
- This was mainly due to fluctuations and trajectory changes with demand, innovation, refining, discovery, renewables, and more
Soil loss and soil depletion is far from being a threat
- We are not going to run out of soil carbon, nitrogen, and phosphorous for thousands of years
- It’s not an unsolvable problem. Manure, crop rotation, and other basic practices address it.
- Theoretically we can invent additional ways to repair soil faster. I assume we have not done so because it is not yet a serious problem. Actually it looks like I’m wrong about that and it is fundamentally a slow process that you would realistically do on-site.
- Here is an EA post covering phosphorous depletion risk
- And an 80,000 hours interview that discusses nitrogen depletion
Metal and mineral depletion I assume are a similar non-issue
- There are deposits of minerals on the ocean floor, there are improvements in refining, efficiency, alternative materials etc.
- Here is an EA post discussing how much minerals are needed for the energy transition.
- And another post about resource depletion risk generally.

Nor is cascading ecosystem collapse going to doom humanity

1000 years ago there were massive extinctions of all major predators around the world and this didn’t cause human collapse with prehistoric technology. The ecological devastation was largely undetectable to us.
There are fears of potential compounding synergistic effects/extinction vortex/poly crisis. But we haven’t seen signs of this happening anywhere at any point in history yet.
- It seems like a good thing to watch for, but fortunately not a present threat or at least a very rare one. I think it’s advisable to increase our uncertainty to reflect this possibility when projecting earth systems.
There are many examples of ecosystem recoveries in record time.
- Even when we look at completely devastated areas like massive volcanic eruptions/agent orange/etc they don’t cause a cascading or runaway collapse of adjacent ecosystems and then tend to recover in 40-80 years.
Repeat stress events are reducing the environment’s ability to recover.
- Degraded lands are thriving, just populated with invasive, common, generalist species.
- I need to look into this in more detail. My general feeling is that this is true but not threatening until habitats have lost 80% of their biodiversity.
I have tried to identify the worst cases of ecosystem collapse and cascading impacts, and found very few (6) documented examples. None of these caused adjacent ecosystems to collapse?
- Those six examples were:
  - Starfish die-off converted kelp forest to urchin barrens.
  - Mountain lion absence causes increased herbivores, decreased vegetation, changed stream patterns, and decreased insect biodiversity. (This is a replication of the return of wolves in Yellowstone.)
  - Amazon fragmentation causes species decline over decades.
  - Moa eradication and mammal introductions in New Zealand causing 50% herbivorous bird extinction, widespread plant extinction, and decreased nutrient inflow.
  - Deforestation in the Amazon combined with temperature rise is drying out the forest and the resulting fires are converting it to Cerrado (savanna).
  - Sea level rise in the Everglades is causing mangrove dieback, and subsequently sawgrass dieback and coastal erosion.
    - But this is mostly the physical presence of salty seawater pushing inland, so not a great example of a cascading collapse.
  - I’m interested in reading about more examples in this category of extreme ecosystem regime changes, please send them my way!
- Some of these were drastic regime changes to the ecosystems, but they do not cause adjacent ecosystem collapses nor threaten human survival.
  - Cascading collapse seems to happen more often from climate change—where the fundamental physical parameters have changed underlying ecosystems—and not the feared unpredictable emergent phenomena.
    - Tipping points that have been identified from climate change don’t appear to threaten the future survival of civilization.
    - 80,000 hours and other EA orgs have concluded that the survival of human civilization is not at risk from climate change.

^
I’m not aware of other kinds of ecological risks, but I would be interested to learn about other examples
^
To get a rough global estimate of required biodiversity for ecosystem services, I multiply each species required count from this table by the number of global biomes (85-500) and get ~5,000-100,000 (0.01%-1% of total extant species). Total biomes are somewhere between 85 (at the Formation level, which would be minimizing global species) and like 500 (at the Division level which would be using locally thriving species). Totaling that up comes out to somewhere between 7,480 (0.086% out of all extant species) and 81,500 species (0.94%) to support global ecosystem services. Flaws with this estimate: There are unlisted ecosystem services. I’m overcounting habitats since biomes like “ephemeral salt lakes,” “ground water” and “mud plains” would provide more limited sets of ecosystem services. Some species could fulfill multiple ecosystem services. Some species could fulfill multiple biomes. Supporting ecosystem services would be needed. I think this estimate is really low, but far more than required for a handful of humans to survive.
^
There are multiple steps to nitrification and the most limited one is Ammonia oxidation which is provided by only three guilds: AOA, AOB, comammox. Of those, anammox is only six genera, one phylum.
^
To get a rough global estimate of required biodiversity for ecosystem services, I multiply each species required count from this table by the number of global biomes (85-500) and get ~5,000-100,000 (0.01%-1% of total extant species). Total biomes are somewhere between 85 (at the Formation level, which would be minimizing global species) and like 500 (at the Division level which would be using locally thriving species). Totaling that up comes out to somewhere between 7,480 (0.086% out of all extant species) and 81,500 species (0.94%) to support global ecosystem services. Flaws with this estimate: There are unlisted ecosystem services. I’m overcounting habitats since biomes like “ephemeral salt lakes,” “ground water” and “mud plains” would provide more limited sets of ecosystem services. Some species could fulfill multiple ecosystem services. Some species could fulfill multiple biomes. Supporting ecosystem services would be needed. I think this estimate is really low, but far more than required for a handful of humans to survive.
^
To get a rough global estimate of required biodiversity for ecosystem services, I multiply each species required count from this table by the number of global biomes (85-500) and get ~5,000-100,000 (0.01%-1% of total extant species). Total biomes are somewhere between 85 (at the Formation level, which would be minimizing global species) and like 500 (at the Division level which would be using locally thriving species). Totaling that up comes out to somewhere between 7,480 (0.086% out of all extant species) and 81,500 species (0.94%) to support global ecosystem services. Flaws with this estimate: There are unlisted ecosystem services. I’m overcounting habitats since biomes like “ephemeral salt lakes,” “ground water” and “mud plains” would provide more limited sets of ecosystem services. Some species could fulfill multiple ecosystem services. Some species could fulfill multiple biomes. Supporting ecosystem services would be needed. I think this estimate is really low, but far more than required for a handful of humans to survive.
^
These are some of the claims made by large and reputable environmental organizations: “Biodiversity is earth’s life support system.” “Nature gives us a habitable environment, we rely on clean air and water.” “We depend on nature for survival.” These claims are phrased to imply extensive environmental actions are important for human survival, which is not actually supported.
^
Most water treatment plants are chemical and physical processes. This is because biological processes are slower and less controlled.
^
The planetary boundaries say we have already crossed this boundary, at losses of “10 extinctions / million species years.” The planetary boundaries represent boundaries past where we “do not know,” not when we “will likely cause serious damage to civilization.” I’m not even sure if the rate of extinction boundary is helpful because collapse risk would depend on how many species total are left, not the rate of decline?
^
Natural salt flats, sand dunes, rocky alpine, and taiga forest all have very low species diversity. Other examples are tree plantations and farmland with monoculture crops, and cheatgrass invaded sagebrush habitat (reduced down to 8 species of plant). These are not “uninhabitable” due to low biodiversity although they may be physically inhospitable. They are not collapsing dustbowls with no water and oxygen.

Who cares about the shrimp? by alex wennerberg

alex wennerberg — Thu, 13 Aug 2026 21:17:36 +0000

Hi all, I wrote about Shrimp Welfare Project for Kernel Magazine. I am an outsider to this community, but I read a ton of EA forum posts for this piece. I hope you enjoy it.

Learning from Agricultural Development to Reshape Animal Welfare Work in Low- and Middle-Income Countries by Tom Billington

Tom Billington — Thu, 13 Aug 2026 19:46:33 +0000

This is a cross-post of the executive summary from The Centre for Animal Life and Farming’s first report. See the full report here.

We will also be hosting a webinar on 25th August 2026 to discuss these findings with the animal welfare movement.

Executive Summary

Roughly 80-90% of farmed animals live in Low- and Middle-Income Countries (LMICs),^[1] yet the animal welfare movement has seen fewer milestone victories in these contexts than in High-Income Countries (HICs)^[2]. The standard advocacy models used in HICs (corporate campaigning, consumer pressure, legislative reform, etc.) rely on infrastructure that is often absent in LMICs, such as formalised supply chains, consolidated corporate buyers, and strong legal enforcement.

Fortunately, the animal welfare movement is not alone in these challenges. There is a long history of field-based projects in LMICs designed to shift farming practices to improve farmer productivity, sustainability, and well-being. Collectively, this broad category of work is referred to as agricultural development (“agri-dev”). The strategies and lessons generated through this work may offer relevant and useful insights for the development of animal welfare initiatives in LMICs.

Authors Karthik and Tom visiting the field office of DVARA, an agri-dev organisation in Maharashtra, India

To identify more context-driven strategies LMIC animal welfare actors could use, this report asks the question:

What can the animal welfare movement learn from strategies employed by agri-dev actors to effectively change farmer practices in LMICs?

To explore this question, we conducted a narrative literature review and primary field research in India. We identified six primary high-level lessons that were cross-cutting for all strategies:

Complex “asks” need more training and incentives. This forces a trade-off between smaller welfare gains that are cheaper to implement, or larger welfare gains that are more expensive to implement.
Programmes must be tailored to local context to succeed. Design with context in mind, work with partners who understand the context, and focus on larger farmers who tend to be more similar to each other.
Trust is essential for farmer buy-in. It can be built through local partners and intermediaries.
Financial incentives aren’t enough alone; they often need technical and social support to sustain.
Farmers face many significant barriers (e.g., education, financial, infrastructural) at once. Programmes must account for these barriers to be viable for farmers to implement.
Full compliance is unrealistic, so aim for the level that keeps impact cost-effective.

We also identified 20 priority strategies used by agri-dev actors that we felt were relevant for the animal welfare movement. We grouped these strategies across three dimensions: organisation types that deliver interventions to support farmer adoption of new practices (8 reviewed), how farmers are trained in new practices (5 reviewed), and how they are incentivised to adopt them (7 reviewed).

We evaluated each strategy across five criteria: effectiveness (at achieving their intended outcome), resource intensiveness, tractability, scalability, and their risk of stimulating harmful industrialisation. This analysis was conducted with a specific focus on the value for CALF’s work within India, though we expect that many of the insights will translate into other contexts. Here are our summarised results (scores range from 1 to 7):

Strategy		Strategy Explanation	Effectiveness	Low Resource Intensity	Tractability	Scalability	Low Industrial- isation Risk	CALF’s Opinion
Organisation Types	Intermediaries	Buy and sell farmers’ products, offering fair, stable prices in exchange for meeting welfare standards.	6	3	4	6.5	5	Recommend for trialing
	↳ & Processing	Also process products (e.g., sorting and packaging eggs) to raise prices, funding welfare incentives.	7	2	2.5	5	5	Recommend for trialing
	↳ & Contract Farming	Formal agreements with farmers on what and how to produce, often with inputs and guaranteed prices.	7	1.5	3.5	5	2.5	Recommend for trialing
	Collaborating with Agri-Dev Organisations	Adding welfare into existing agri-development programmes, leveraging their skills and farmer trust.	6	6	5	5	3	Recommend for trialing
	↳ & Self-Help Groups	Reaching village women’s savings groups, who do most day-to-day animal management, via agri-dev partners.	5	5	6	3	4	Needs more investigation
	Farmer Producer Organisations	Formal farmer collectives whose rules could include welfare mandates.	4	5	4	4	6	Needs more investigation
	Certification	Voluntary audited standards, ideally earning farmers a price premium for higher welfare.	4	3	5	4	6	Do not recommend
	Direct-to-Farmer Technology Providers	Digital platforms, chatbots, or SMS engaging farmers directly at scale, no intermediaries.	2.5	4	4	2	6	Do not recommend
Training Mechanisms	Training & Visit (T&V) Systems	Trained extension agents deliver structured welfare training to farmer groups on a fixed visit schedule.	5	6	6	5	5	Recommend for trialing
	Model Farms & Field Schools	Working farms demonstrating welfare-compliant practices for hands-on observation and learning.	6	3	5	3.5	6	Conditional — short-term only
	Mass Communication (ICT)	Messages delivered at scale via channels such as SMS, apps, radio, or social media.	1	7	7	7	7	Conditional — simple behaviour changes only
	Veterinary Services	Vets or community animal health workers advising on welfare.	5	4	5	4	4	Do not recommend
	Information & Call Centres	Physical hubs and remote services providing agricultural advice.	1	6	6	6	6	Do not recommend
Incentive Mechanisms	Collective Bargaining	Farmer groups negotiating better input costs or product prices.	6	4	5	6	3	Recommend for trialing
	Market Access	Linking welfare-compliant farmers to buyers, sometimes with transport or guaranteed purchases.	6	3	4	6	3	Recommend for trialing
	Value Addition	Processing products (e.g., grading, packaging eggs) to raise value, sharing the margin with welfare-compliant farmers.	7	2	2.5	5	5	Recommend for trialing
	Credit Systems	Loans tied to or supporting welfare compliance, financing practices needing upfront investment.	7	4	2.5	6	3	Needs more investigation
	Conditional Payments	Payments or discounted inputs contingent on adopting welfare practices.	5	3	6	3	6	Conditional — short-term only
	Market Information	Price data informing when, where, and what to sell.	1.5	7	7	7	5	Do not recommend
	Insurance Products	Coverage against production and price risks, potentially conditional on welfare improvements.	4	3.5	2	6	2	Do not recommend

Each agri-dev strategy reviewed across our key criteria in a 7-point Likert Scale

A summary of each strategy and their key considerations is shared in section 3 of the full report (you can also see a more in-depth breakdown of each score here.). This is an initial step. The strategies identified here require rigorous field testing, and the evaluations we offer are preliminary. We present them not as definitive answers but as structured starting points.

This report offers three key lessons for animal movement actors operating in LMICs:

Learn from agricultural development. The strategies above (in particular those flagged as “recommended for trialing”) are templates the animal welfare movement should consider using. Agricultural development organisations may themselves make strong partners in implementing these.
When building a theory of change, consider three dimensions: training mechanisms, incentive mechanisms, and organisation types. Make a deliberate decision about each one, even if that decision is to leave it out. Revisit and refine those decisions as the work develops.
Expect complex, interlocking challenges. These include farm and farmer heterogeneity, low education levels among farmers, the importance of trust, compliance issues, and the reality that the largest welfare gains typically demand the most complex behaviour changes. Navigating these will take time, and will only succeed with clear awareness of the barriers, deliberate planning, and openness to learning along the way.

^
See full data set here. Note that the numbers for total farmed fish are from 2017 and based on low-end estimates from fishcount, meaning that they are significantly more speculative. Data Sources:
Cattle, poultry, and pig data from FAO, 2025 – with major processing by Our World in Data
Farmed fish data from fishcount (2017)
^
See the full data set here. Data sources:
Company pledge data from Chicken Watch (2026)
Income classification data from The World Bank.

Concrete Generalist Projects in AI Safety (and how to do them) by Helena Tran

Helena Tran — Thu, 13 Aug 2026 19:12:09 +0000

The AI safety ecosystem is still in need of generalists: people who will go out and solve the tasks that people who feel limited by their job description won’t address. This post will explain how to break into generalist work, then provide an important set of projects that, if completed, could strengthen many crucial parts of the AI safety ecosystem, as well as your own career.

TL;DR

If you’re looking for something concrete to do, there are a bunch of good entry points:

Find a project: browse SPAR’s Generalist Projects, join the Generalist Megastream, look through projects on Mangrove, or browse this fieldbuilding/project spreadsheet and Atlas Computing’s Resilience Gap Map.
Get support: apply to Genstream (deadline August 14), indicate interest in the Generator Residency, or check out MATS’s Founding & Fieldbuilding track and Catalyst.
Get funding: BlueDot Rapid Grants fund projects up to $10k; there are also quick funding opportunities and career-transition grants from Coefficient Giving and BlueDot.
If you’re a student: join or start a university AI safety group; Pathfinder and the AI Safety Seeding Initiative can help with mentorship and funding.

The rest of the post has more detail on choosing projects, getting mentorship, and other ways to get involved.

If you’re a…

University Student: get heavily involved with your university AI safety group. University groups are real, impactful organizations, and they can be run as such.

If your school doesn’t have a group, you should start one. Convincing an equally talented person to join AI safety can straightforwardly double your impact, and many of the most impactful people working in the field today credit their university groups for convincing them to make this career choice.

A university group is also a great place to cut your teeth doing generalist work. Most group organizers find themselves working on operations, strategy, events, headhunting, and research in a single week, testing their fit for each type of work and operating under enviably tight feedback loops. It’s rare that an early-college generalist finds a better opportunity for both impact and career capital than “making their school’s group a lot better,” which is why many of the field’s best generalists were once university group organizers.

If you would like to get access to funding or mentorship, the Pathfinder Fellowship and the AI Safety Seeding Initiative are both fantastic resources to get mentorship and funding. University groups can run existing educational programs like ARENA, or they can support students’ efforts to get into SPAR, MATS, Generator, or one of the many other advanced fellowships that teach people how to do impactful work.

Early or Mid-Career Professional: Make sure you have relevant context on the AI safety space. Take some BlueDot courses, meet people at AI safety events, and read voraciously from a variety of sources.

Once you have a general understanding of the field, look into projects. Completing an AI safety project is a good way to support the field, meet new people, and gain valuable skills and context. You can just do many of these things; you don’t need permission or a full-time role to contribute!

This section is basically a TL;DR version of 80k Hours’ posts on how to get into AI safety in 3 months. There’s a wealth of information on their Substack, their website, and they just released a book!

Projects that should get done

The AI safety space is littered with unresolved problems and inefficiencies, because many of the people who were skilled enough to notice these problems don’t have enough spare time to address them. While preparing for the Generator Residency, we conducted interviews with dozens of AI safety professionals from organizations like Coefficient Giving, AI Futures Project, Forethought, and more.

The residents couldn’t do all of them, but that doesn’t mean they shouldn’t be done.

Some of these projects might make the talent pipeline more efficient, making it easier for passionate people to learn about AI safety and gain the skills that make them impactful. Other projects might improve our data collection systems, make concerns about existential risks visible to a broader audience, make it easier to convert political will into impactful legislation, or fill some broader need that can’t be easily described.

This list is far from exhaustive, and many of the programs listed will be continuously maintained with new projects as the needs of the field evolve. We hope that this section serves as a jumping-off point for you to explore the programs and shared resources outlined below.

SPAR Generalist Projects

SPAR’s generalist projects are a new category under SPAR with mentors from Generator, Constellation, Kairos, universities, and other AI safety organizations on projects in fieldbuilding, communications, research, and organizational infrastructure.

Current projects include:

Plus a dozen more, as well as a Generalist Megastream for strong applicants who do not see a perfect fit among the existing listings. Applications close August 18.

SPAR Generalist category

Mangrove

We’ve added a number of projects we find promising to Mangrove, a platform for coordinating AI safety projects and communities.

We think a lot about AI safety talent pipelines, and by far the biggest drop-off for potential generalists is the stage between “having context in AI safety” and “building signal to get into fellowships”. Much time gets lost due to lack of coordination, and much motivation gets lost due to lack of accountability and community.

We’re excited about Mangrove as a place for people to hit the ground running on interesting projects, while developing credibility and connections in the space. Mangrove is in development, and plans to host hackathons, community projects for university groups, a contracting platform for orgs (!), and more.

Mangrove’s official launch announcement is coming in a few days, but you can check out the platform right now.

Some projects we seeded on Mangrove:

Non-project shaped projects

A lot of generalist shaped things don’t have discrete projects, and rather require building specific skills like effective comms, hiring/recruiting, org strategy, management, and much more, to pick up dropped balls. As a sample of these sorts of fuzzier projects:

Consider being an influencer: As “warning shots” become scarier and more frequent, there’s potential and demand for high-quality media that connect these events to broader AI safety concerns and convert public will into action. Some great media has come out of the AI safety ecosystem as the Overton window is shifting, and programs like The Frame and Tarbell have emerged to target this.

Political organizing: There’s a lot of useful work in recruiting people, running events, building local groups, organizing constituent outreach, and helping turn public will into useful legislation and strong leadership. This work is often bottlenecked on reliable organizers, and many consider it incredibly high-leverage. Political organizing can become quite sensitive and prone to backfire risk, so if you’re interested in this, we recommend filling out this short form before launching something yourself and we’ll connect you with someone who can advise you.

Writing: Writing publicly is one of the fastest ways to build a track record. The spreadsheet below includes a running list of posts people want written.

Other Project Lists

This spreadsheet has an active list of fieldbuilding projects, university outreach ideas, and blog posts that could be valuable to write. It also has descriptions of various problems, solution directions, theories of change, and people associated with these projects.
Atlas Computing’s Resilience Gap Map provides a list of AI-related projects that need to be addressed, as well as the critical assumptions behind working on them.
This is a list of lists of project ideas in AI Safety. It has many lists, including research project lists and fieldbuilding lists.
Some schemes for AI safety by Austin Chen. Manifund and Mox are running fellowships like Surplus to support these.

Getting Support

Programs

As a general rule, you should be applying for more things. New programs are springing up to accelerate generalists:

Consider indicating interest in the Generator Residency, a 3-6 month long program dedicated to rapidly upskilling generalists.
MATS also has a Founding and Fieldbuilding path.
Catalyst is a 10-week remote program for undergraduates to work on impactful projects.
[Applications close August 14th] Genstream is a one-week, in-person AI safety program in London for aspiring generalists.

Funding

Many people are too scared to apply for funding, when probably they shouldn’t be! Even if you don’t get the money, oftentimes submitting a (non-slop) grant proposal can be a good forcing function for execution.

BlueDot Rapid Grants give out grants of up to $10k, with an average turnaround time of 3 days. They are happy to fund: Compute and API credits, events and meetups, research access, travel, community chapters, project tooling. Wildcards welcome.
Austin Chen’s list of quick funding opportunities
Coefficient Giving’s career development and transition funding is targeted for more mid to senior career people, giving runway for up to 6 months to explore AI safety careers. BlueDot also gives out career transition grants.

Advice on getting mentorship

It can be extremely helpful to know people in the community who can give you mentorship. A good mentor can make a big difference in acquiring taste, skills, and connections. Many of the best projects require specific context to complete well.

As a general rule, your bar for reaching out to people is too high.

For example, if you’re especially interested in a series of blog posts someone has written, consider reaching out to discuss the content of their post further (but try not to ask questions that you can easily answer by reading more of their work, or a 2 minute search).

Alternatively, if there are sets of projects you are interested in pursuing, consider reaching out to people who have done similar work, or find a SPAR mentor who can help you do so. You can reach the authors of this post at roman.alex.ross@gmail.com, viceversa1024@gmail.com (Harry), or helenatran9305@gmail.com, and you can meet many more people at events like EA Global or a Global Challenges Project workshop.

A final note

AI safety can be a confusing field to enter, and a lot of useful context lives inside organizations, private conversations, and people’s heads. Still, we think one of the fastest ways to build that context is to work on something concrete. Projects expose you to fast feedback from the outside world, give you reasons to talk to knowledgeable people, and get you oriented faster than reading alone.

The staggering welfare consequences of catch-up growth after AI by Rafael Proenca

Rafael Proenca — Thu, 13 Aug 2026 17:49:10 +0000

This post was crossposted from ChatGDP by the Forum team, with the author’s permission. The author welcomes discussion and plans to respond to comments when they can.

An interactive version of this post — with live widgets instead of static images — is on the Coefficient Giving website, along with the full calculator and the technical note.

“Is there some action a government of Nigeria could take that would lead the Nigerian economy to grow more like an AI-powered United States’ or China’s? If so, what, exactly? If not, what is it about the ‘nature of Nigeria’ that makes it so? The consequences for human welfare involved in questions like these are simply staggering: Once one starts to think about them, it is hard to think about anything else.” — the spirit of Bob Lucas (1988)^[1]

In 1800, Mali’s per capita GDP was roughly 5x smaller than Britain’s. Today it is roughly 25x smaller.^[2] The Industrial Revolution accelerated growth in the UK and the benefits of the new tech eventually reached the whole world. But the process has taken so long that, two centuries later, the gap is still 5x larger than it was before the new tech arrived.

More and more economists are taking seriously the possibility that AI will cause a rapid acceleration of growth in frontier economies.^[3] Investors are betting trillions on AI. For their bets to be right, AI companies need to generate even more trillions in revenue, which would imply a big growth acceleration in frontier economies. The economic consequences are serious enough that the Fed just launched a high-profile task force on AI and the economy. So it doesn’t seem crazy to think that AI can really accelerate frontier growth. What happens to Mali this time?

This post argues that Mali, and all non-frontier economies, are likely to be left behind, at least in the initial phase when frontier economies are rapidly becoming more automated.^[4] The broad argument depends on two simple and, in my opinion, fairly reasonable assumptions:

AI accelerates frontier growth. At least in the period immediately after the arrival of Transformative AI (TAI), the growth rate of frontier economies increases. That’s the period I focus on here, and I call it the “detachment window.”^[5]
Poor countries will also see a boost to growth — but there will be a lag. For reasons I will discuss below, poor countries lag behind in AI adoption in productive activities, so the productivity increase due to AI arrives in non-frontier economies with a lag.^[6]

Surprisingly (to me at least), these two ingredients are sufficient to conclude that the GDP gap between frontier and non-frontier economies increases in the detachment window. You can see the math in the technical note. The intuition is best presented via an image:

Figure 1. On the flat, the gap is constant; when the leader hits the downhill, the same 30 seconds becomes a growing gap. The frontier is the leader; the slope is accelerating growth; the head start is the tech adoption.

Have you noticed in the Tour de France that when riders reach a downhill stretch, the distance between them tends to open up? That’s because the slope accelerates the bikes, which turns a fixed gap in time into a growing gap in space, even if they are pedaling equally hard. Here the leader is the frontier economy, TAI is the downward slope, the fixed gap is in years for adoption of AI in productive activities, and the gap is in GDP.

The rest of this post presents a simple economic model that helps me justify these assumptions and quantify how much the gap opens up.

The stakes are enormous. In the central scenario, delayed adoption leaves average incomes in the developing world at the end of the window about a fifth of what they would have been with no lag. Each year of faster productive AI adoption in lagging economies is worth about $42.6 trillion in present-value income (more than a third of the world’s GDP), or the equivalent of saving 115 million lives if we adopt Coefficient Giving’s standard methodology.

If you are skeptical of the assumptions, you can change all the parameters in this calculator. You can also download the source code here.

A simple model of takeoff in frontier economies

Think of the economy as a series of tasks: drive the truck, weld a joint, file taxes, make Doritos Screamin’ Sriracha, sell Doritos Screamin’ Sriracha at Dollar General, etc. Each task can be done by either a person (L, for labor) or a machine (K, for capital), and all tasks are combined using a Cobb-Douglas aggregator (a staple of economics) to get us total output (Y). A bit of math gets us to:

Y = (K / f)^f × (L / (1 − f))^1−f

To see the math, check the accompanying technical note.

This is the tasks model, the workhorse of the automation literature.^[7] Here f is the fraction of tasks in the economy allocated to capital. Since f tasks are allocated to capital, (1 − f) tasks are left to labor. Capital has to perform f tasks, so the total amount of capital available to produce each task is K/f.

I think of an AI-induced growth takeoff in frontier economies as increasing f. This is not something I invented. In fact, I just copied the idea from one of the most cited papers on the economics of AI, Aghion, Jones, and Jones. As AI capabilities grow, machines become more capable of performing economic activities, so more tasks are allocated to machines, and the growth rate in frontier economies increases. In fact, in this model, the higher f is, the higher the growth rate.^[8]

To calibrate the speed of growth we first need to assume a trajectory for f. I assume f follows an S-shaped curve because that’s basically how all technologies spread (Geroski 2000). In the beginning, AI doesn’t exist and doesn’t drive automation. Then it’s invented. Then, as it gets better and better, tasks are automated at a growing rate. As we get closer to 100% of tasks automated, only the harder, more expensive tasks to automate are left. We’ve reached diminished marginal returns and f tapers off.

Maybe f never reaches 100%, as Alex Imas believes. That is, a number of tasks can be permanently left to humans. These can be heavily regulated tasks, like the final steps of getting a green card, or tasks where people have an intrinsic preference for humans, like yoga teachers or sports.

Once we have a shape and a cap, we can calibrate how long it takes for frontier economies to go from the current level of automation to the final level. Call that T.^[9] My central guess is 15 years, which is the mid-point between optimistic and pessimistic predictions related to the speed of automation of the US economy.^[10]

Finally, to determine the full path of frontier growth, we need to set the speed of growth under full automation. I base that on Korinek and Suh 2024, where a fully automated economy eventually settles into growth of approximately 18% per year (more on this parameter in the last section of the post). The growth rate at the end of the detachment window then depends both on the maximum level of automation and the growth rate under full automation.

Step 1. Falling machine costs push the automation share f up an S-curve, and the model converts that share into a growth rate, side by side below. The key fact: the relationship is log-linear, so equal steps in f multiply growth by equal factors. Drag T:

Every parameter behind these panels is exposed in the full calculator.

A rising tide lifts some boats faster than others

Today, the latest AI models reach poor countries quickly. My friend in Benin uses ChatGPT all the time (though getting advice on crypto trading might not be the most productive AI use-case). So shouldn’t f just grow at the same rate in lagging and frontier countries? Isn’t there a chance that poor countries adopt AI even faster than rich ones? Africa has more young people than any other continent, and they did overtake the US in the adoption of digital payments.

Unfortunately, my friend is just an exception that proves the rule. Using data on AI-related activity on Windows devices, this recent IMF paper finds that LLM use lags behind in poor countries.

And I think that amidst a wave of AI-driven growth in frontier economies, the AI gap between frontier and non-frontier economies will not decrease. That’s especially true of AI adoption in productive activities – that is, in activities that actually generate money, move the automated fraction of the economy, and cause GDP growth. Here are the main reasons:

1. GDP of frontier economies relies more on cognitive tasks, and those will probably be automated first by AI. Increasing f means automating production, which involves both cognitive and physical tasks. AI is currently much better at cognitive than physical tasks. A high fraction of US GDP is in sectors with a high cognitive component, like banking, consulting, law, software, and biotech. Mali has a lot of GDP in agriculture and mining, where physical tasks are relatively more important. So the first wave of automation has far more surface area to work on in the US than in Mali.^[11]

2. Automation costs money. The hard part of AI-driven automation is in physical tasks. Drones that spray fields, humanoid robots, and self-driving trucks are all expensive machines. Poor countries have little money to pay for them. They also face higher interest rates, so any investment project gets more expensive. In the period of AI-induced frontier growth acceleration, capital will likely flow uphill (from poorer to richer countries), raising interest rates in poor countries relative to rich countries and making it even harder to pay for these machines.^[12] (Lack of money is less of a factor in AI adoption for cognitive tasks, but it can be an issue. While people in poor countries can download a frontier LLM app for free, the price of tokens starts to bite at scale. In the US, AI-use bills are increasingly too big even for rich companies.)

Source: Panel A: World Bank World Development Indicators, gross savings (% of GDP) and GDP per capita, 2024 (NY.GNS.ICTR.ZS, NY.GDP.PCAP.CD), retrieved 31 July 2026. Panel B: Aswath Damodaran, Country Default Spreads and Risk Premiums, updated 5 January 2026. Cost of equity in US dollars = risk-free rate in US dollars plus the country equity risk premium, at a beta of 1. Risk-free rate in US dollars = 4.67% US 10-year Treasury on 30 July 2026 less Damodaran’s 0.23% US default spread = 4.44%. Damodaran builds the country premium by scaling a sovereign default spread by the ratio of emerging-market equity to emerging-market government-bond volatility, a factor of 1.52. The spread used here is the traded one, not the one implied by the credit rating: Damodaran’s sovereign CDS premium for the United States, China, Brazil and Kenya, and for Mali, which has no CDS, its own 3-year CFA-franc Treasury bond at 8.17% (UMOA-Titres auction, 18 March 2026) less the 3-year German Bund at 2.56%, the CFA being pegged to the euro. Ratings would put Kenya at 18.4% and Mali at 20.3% instead; they agree with the market to within 0.2 percentage points for the other three. This is a hurdle rate for equity: a firm’s dollar borrowing cost sits below it and the sovereign’s below that.

3. Automation makes more economic sense when wages are high. Suppose a robot that replaces one worker costs $50,000 a year to own and run (after capital costs, maintenance, and power). In Seattle, the worker it replaces costs $65,000 a year, so automation saves the employer $15,000. In Lilongwe, a worker costs $2,000, so automation loses $48,000 a year. This is a stylized example, but the point is just that the higher the labor costs, the higher the economic gain from automation.^[13]

4. Complementary inputs. Automation is more attractive if you have reliable and cheap electricity, internet connectivity, sensors producing data, skilled managers who can reorganize firms, etc. All these complementary inputs are more abundant in frontier economies. China already has a lot of robots making cars, so the next AI-powered robot can slot right in. If you are Zambia and have no car factories, making the AI-powered robot produce cars will require a lot of complementary investment.^[14]

In the model, these reasons can be summarized by a single lag (D).^[15] That is, the poor country’s automation curve f is the frontier’s curve shifted right by D years. It’s not hard to see why. A task is given to a machine if, and only if, the cost of completing it with a machine is lower than with a human. We can map reason 3 into lower local labor costs in non-frontier economies, and reasons 2 and 4 into higher effective machine costs. Reason 1 affects the composition of tasks rather than the cost of a given task, and mapping it into my simple model would require additions making it significantly less simple, so I leave it out.

We can then conceptually model the arrival of TAI as a phase in which AI makes machines more and more effective at performing economic tasks, which is mapped as a steadily falling cost of completing tasks with machines. This single driving force causes the share of automation f to increase in both frontier and non-frontier economies, as more and more tasks flip from being overall cheaper with humans to overall cheaper with machines. I find it neat that a simple model, grounded in the literature, can generate increasing automation shares and growth in both frontier economies and non-frontier economies, but with a constant lag D separating them (check the technical note for details).

The model also gives a hint on how to calibrate the lag. D equals the machine-cost halving period times the (log of the) country’s cost disadvantage. For example, a country facing a 10x less favorable machine-to-labor cost ratio, with machine costs halving every three years, has a lag of about 10 years.

You can play with machine-cost halving periods and machine-to-labor ratios here:

The best way to calibrate D that I found is to use the historical evidence in Comin and Mestieri. The authors measured adoption lags for 25 technologies across 139 countries, over about two centuries. They find poor countries got the internet roughly 6 years after rich ones, and PCs and cellphones roughly 6 to 8 years after. So the time it takes for a new technology to “arrive” in a poor country is shrinking.

However, they find that the gap in how intensively countries use technologies once they arrive has been widening. The smartphone, for example, took about 15 years to reach half of Sub-Saharan Africa. Smartphones are a relatively cheap consumer good, while “AI in productive activities” entails expensive machinery that requires complementary capital. So my central guess is D = 10 years, which seems like a conservative guess to me, since in the model D is better mapped to the intensity (fraction of tasks automated) rather than the availability of AI for productive activities.

Step 2. Now add the lag. Each economy keeps its own baseline growth (adjustable below) and the follower receives the boost D years late; the orange wedge between the curves is the divergence engine:

Tip: Set the two baselines equal to isolate the delay’s own contribution to the gap. That isolated part is exactly what the calculator prices, which is why its headline numbers compare two paths for the same country rather than frontier vs follower.

To the numbers

Step 3. Price the wedge: the headline numbers and the two income paths, live. Drag D and T:

The calculator below compares two paths for the same developing-country bloc: one where the AI boost arrives immediately and one where it arrives D years later. Both start at today’s average non-frontier income ($4,200 per capita), and I apply to both a 4% discount rate to compute the NPV of the accumulated GDP difference between them.

Here is what the central scenario (takeoff length T = 15, lag D = 10, g_full = 18%, a 25-year window) implies for the 5.4 billion people who live in low- and middle-income countries outside China:

By the end of the window, average income in the developing world is about 5x lower than it would have been with immediate adoption.
The present value of the income shortfall is about $1.2 quadrillion. For scale, the entire world currently produces about $110 trillion per year.
In welfare terms, the gap is worth about 6.1 quadrillion Coefficient Giving dollars, using the CG methodology to value income gains (a 1% income increase for one person for one year is worth CG$500). In welfare units, the total gap is equivalent to saving roughly 1.86 billion lives.
Each year of faster adoption is worth about CG$381 trillion, roughly 115 million lives in welfare units. In my view, this is the most important number because I believe economic policy can change D and philanthropy can help.

My main takeaway is anything that accelerates developing countries’ catch-up has welfare returns at a scale that’s hard to find elsewhere in philanthropy. Conditional on TAI having an important effect on frontier growth, it’s hard to imagine ways in which the rate of catch-up in the aftermath doesn’t matter enormously. In fact, ensuring that developing countries’ growth rates stay close to the frontier growth rate might be the second most important task in terms of human welfare, after preventing AI-induced catastrophic risks.^[16]

Why care about catch-up growth if TAI will create great abundance and raise everyone’s welfare?

Rapidly growing frontier economies would produce a lot of goods. The price of tradable goods could fall significantly in international markets, and new amazing products could become available. If developing countries can import super cheap electric cars and pills that cure all diseases, why care about economic growth?

That’s because growth still matters for two reasons.

First, more money increases access to those new goods. Even if great AI-produced goods become available, poor countries will have little money to pay for them if their income is low. Increasing their GDP is the best way to improve their access to these great new goods.

Second, this is a counterfactual comparison about relative incomes. The welfare gaps here compare GDP in non-frontier economies with what it could be if the adoption lag decreased. The model suggests that AI-induced takeoff in frontier economies eventually benefits everyone — by an enormous amount — but by not trying to accelerate catch-up, we’d be leaving a huge amount of welfare on the table.

What happens after the detachment window?

So far I have only talked about what happens in the detachment window – but what happens after can undermine the main point I’m trying to make here. My main point is that accelerating catch-up growth during the detachment window matters a lot. An objection that AI-believers could have is that the initial detachment phase doesn’t matter much because it’s short and what follows is quick catch-up, or a post-scarcity future where money becomes a useless concept and income differences stop mattering. If so, the welfare gap we’ve just estimated is just a blip, a rough couple of years on the way to somewhere great for everybody.

Why spend attention, let alone money, on a blip?

To answer that, let’s be concrete about what could come after the detachment period modeled so far. I can imagine four main scenarios:

Future 1: the plateau. Growth rates stay high everywhere. The income gap freezes at roughly 5x and stays there. We have a new, spectacularly richer world, exactly as unequal as the window left it.
Future 2: the fade. The AI boost turns out to be temporary, and frontier growth comes back down. Then something nice happens: The follower rides the tail of the boost after the frontier has slowed, growing faster than the frontier for a while, and a temporary boost plus a pure lag creates only a temporary gap.
Future 3: the singularity. The boost keeps growing and the gap explodes.
Future 4: the convergence. Developing countries grow faster than the frontier after the window, the way classic growth theory says capital-scarce economies should. The gap closes while growth stays high.

The “it’s just a blip” objection only works in future 4. In future 1, the window’s gap is permanent. In future 2 it closes, but closing it one year early still creates the same amount of welfare we estimated here. At the central calibration, the detachment window is 15 years. That’s billions of person-years lived at incomes several times below what faster catch-up would have delivered.

Future 3 could go either way. A singularity could be the future where the early gap developed in the detachment window matters most, because whoever ends the window ahead may get to write the rules of everything after. In 2021, then co-CEO of Open Philanthropy Holden Karnofsky wrote that a state in which “key things about society, such as who is in power or which religions/ideologies are dominant, are locked into place indefinitely, plausibly for billions of years,” is possible and even likely. The argument is that, if some groups become much more powerful than others when TAI arrives, they might use a temporary advantage to consolidate a permanent position of advantage, or even dominance, over the groups lagging behind. On the other hand, the singularity could create a post-scarcity world where initial income differences become irrelevant. It’s too hard to think about what happens in this scenario, so I skip it.

Future 4 is the world to aim for. It’s the one that maximizes human welfare, and if you were confident we get it by default, the right policy really would be to accelerate AI and not worry about catch-up growth at all, since faster arrival means faster convergence. But look at what future 4 requires: Post-window growth has to be driven by non-rival ideas, things poor countries can copy for free, rather than by capital accumulation, which has owners, price tags, and shipping delays. The entire argument of this post has been that AI’s gains during the transition are embodied in expensive machines. Betting the welfare of five billion people on that not being true seems hard to justify. So let’s aim for fast convergence, but not assume it.

What to do about it

So the trillion-dollar question: Is there anything poor countries, and the people who care about them, can do to catch up faster in the aftermath of TAI?

I think so. Economic policy decisions will meaningfully speed up or hinder catch-up by poor countries after TAI. I won’t pretend I know exactly what policies. But I have some ideas that I will write in a follow up post.

I expect to learn a lot in the next couple of years. My day job is making grants to accelerate catch-up growth, and I’m keen to understand how our grantmaking should change if a frontier AI takeoff materializes (or in anticipation of that) so we can better help countries. I will be talking with policy makers and observing the data, and I will keep you updated.

In the meantime, if you have reactions or ideas, especially concrete ones about what countries should do, please reach out.

Coda: the main ways this could be wrong

I divide the objections here into two camps, the ones that have an AI-skeptic flavor and the ones that have an AI-believer flavor. The main point of the post is that catch-up growth becomes super important in the aftermath of TAI. Broadly, AI-skeptics object because they think AI won’t cause much growth, and AI-believers object because they think AI will automatically benefit everyone. I go into the main reasons I’ve encountered for skepticism on both sides.

AI-skeptic: The AI-induced takeoff never comes.

Most mainstream economists put AI’s growth effects at tenths of a percentage point, and if they’re right, this post is just a thought experiment.

Response: On the question of how powerful AI will be, the AI believers’ track record over the past five years has been better than the skeptics’. I think mainstream economists are a bit too skeptical, and I think this is a good joke.

AI-skeptic: The AI-induced takeoff you assume is too high.

In the Forecasting Research Institute’s rapid-progress scenario, AI experts forecast roughly 5% annual growth in 2050, while economists and superforecasters forecast less. The IMF’s high-adoption scenario for sub-Saharan Africa estimates a cumulative GDP gain of about 4% over 10 years. Why does your central scenario reach 18%?

Response: Eighteen percent is the growth rate under full automation. The forecasters and the IMF paper are not assuming full automation; they allow for automation to get bottlenecked by regulation and a myriad of frictions. To get closer to those scenarios, you can slide the f_max parameter and reduce the share of automated tasks at the end of the detachment window. I assume full automation as default because I see the exercise here as conditional on accepting the AI-believer point-of-view. In that case, 18% as the final growth rate might be too low. In a recent paper, some of the most prominent economists in the field show that, once machines can produce the next generation of machines and automated R&D, exploding growth rates (unbounded) are possible. So I do not regard 18% as the maximally aggressive scenario.

AI-skeptic: The AI-induced takeoff comes, but it stops at relatively low levels of total automation f because AI only automates cognitive tasks.

Response: I admit that it is a (so far) implicit assumption in my modeling scenario that the AI-induced growth acceleration includes automation of most physical economic activities, so the maximum f is not too low. That is, I want you to take as part of the modeling scenario that robots are advanced enough that they can automate most physical economic activities, from cooking to construction.^[17] You don’t have to believe that nobody will pay for fancy human-chef meals anymore, just that robots can do physical things almost as well as humans in almost all economically relevant areas.

If we don’t assume that, then the AI takeoff peters out at a relatively low f, the scenario becomes less interesting and the implications for non-frontier economics change a lot (because they can specialize in tradable physical activities, given that they have a lot of relatively young humans). If you are skeptical how much AI and robots can automate in the next 15 years, the only thing I can do now is to promise a future post about robots, and concede that without automation of physical tasks the main point here becomes much less important.

AI-skeptic-believer: D=10 is too low relative to historical evidence, and maybe TAI itself causes the lag to grow.

Rasmus Andersen, author of the quite interesting AGI Growth Atlas, commented on a draft of this post:

“The model says followers are on the same S-curve shifted by D years. But the dominant historical outcome in many ways was non-catch-up: your own opening (Mali, 5x to 25x) is not the exception, and you can argue full adoption of second and third industrial revolutions still hasn’t happened (electricity and effective ICT use remain incomplete across much of the world two generations on). Historically the four reasons you listed didn’t produce a fixed lag; they produced follower curves that many times didn’t complete.
The few who did reach the frontier used low wages as the entry ramp into tradables, reinvested earnings as physical capital and eventually made it there. Foreign capital generally came later, I would argue. Given TAI may foreclose that path to capital accumulation outside the frontier by automating the low-wage tasks themselves, doesn’t that mean catch-up and convergence in adoption are even less likely this time around?”

Response: In the model, the follower is assumed to trace the same growth path as the frontier, only shifted. This implies that, after D years, even Mali will reach the same level of automation as the US and get the same boost to growth.

This is probably too optimistic. Persistent differences in infrastructure, energy, conflict, or institutions could instead leave poorer economies with a lower automation endpoint or a lower growth rate even after AI-induced automation has run its course.

A way to incorporate this in the model would be to set a lower growth rate after maximum automation (g_full) or lower maximum rate of automation (f_max) to poor countries. These changes would make the main point of the post stronger, and I wanted to keep things simple and not create more free parameters.

AI-believer: TAI causes the lag D to decrease.

Maybe AI will diffuse to poor countries faster than other technologies, and maybe AI itself will help increase the speed of technological diffusion. The main reason I can imagine for this is lowering technical barriers for complex economic production. If a firm has trouble operating a chemical plant in Nigeria now for lack of chemical engineers, Claude 5.1 might dissolve this barrier.

Response: I’m actually hoping that non-frontier economies do everything they can to lower the adoption lag, and hoping to work in and help fund initiatives aiming at that. I think AI itself lowering technical barriers for running complex economic activity is a reason for hope. But I don’t think it’s strong enough to overcome the four barriers I listed above.

AI-believer: Trade boosts catch-up.

A big missing ingredient in the model is trade. AI adoption in frontier economies can increase the demand for non-frontier exports a lot, which would accelerate their growth.

Response: I think this is real but not enough to meaningfully decrease the gap. In a scenario where the AI-induced frontier take-off includes the automation of physical production, manufacturing becomes more competitive in frontier economies than in non-frontier ones. In fact, China (which for our AI-adoption purposes I classify as frontier) is already rapidly automating manufacturing, and I don’t see why Ethiopia’s garment industry would become more viable after TAI than it is now.

I think the exports that will see more demand are extractives (especially the ones used to build the machines doing the automation, or the energy to power them) and tourism. Non-frontier economies should try to maximize this opportunity, but the opportunities are likely to be narrow and to require clever economic strategy – we shouldn’t just assume that trade automatically causes catch-up.

AI-believer: Regulatory barriers in frontier economies reduce the gap in adoption.

We can imagine that relatively strong governments, unions and pressure groups could lead to high and well-enforced regulation against AI in frontier economies. If non-frontier economies have weaker regulatory and regulation enforcement capacity, these barriers could be lower in them, which would compress the adoption lag D.

Response: This is a plausible argument, but it doesn’t seem strong enough to overcome the four barriers I listed above. Frontier economies would really have to regulate away AI use in the economy to overcome the barriers, and competition between US, China, and Europe has been pushing them to race and suppress regulation. I expect this race dynamic to continue well into the detachment period. Even if rich democracies regulate AI a lot, the Gulf countries, or richer middle-income countries like Russia, would be in a better position to automate their economies than the truly poor countries.

Learn more

Full calculator
Technical note
Comin and Mestieri (2018), the main source for the adoption-lag calibration

I’d like to thank Oliver Kim, Justin Sandefur, Joseph Levine, and Rasmus Andersen for very thoughtful comments.

^
This is a paraphrase of one of the most famous passages in development economics. Robert Lucas, economics Nobel laureate, observing the large income differences between rich and poor countries in 1988, wrote: “Is there some action a government of India could take that would lead the Indian economy to grow like Indonesia’s or Egypt’s? If so, what, exactly? If not, what is it about the “nature of India” that makes it so? The consequences for human welfare involved in questions like these are simply staggering: Once one starts to think about them, it is hard to think about anything else.”
^
GDP per capita around 1800 from the Maddison Project Database; Mali has no direct pre-1950 estimate, so the 1800 figure uses West African estimates as a proxy — call it roughly 4-6x, and I use 5x. Today’s ~25x is in PPP terms from the World Bank WDI; at market exchange rates the gap is roughly 45-50x. The two-century divergence pattern is standard economic history: Pritchett (1997), “Divergence, Big Time.”
^
For calibration on the debate: AI insiders predict sustained 10-20%+ annual growth after full automation (Amodei is the clearest articulator); most mainstream economists put AI-induced excess growth at 0.1-1.5%/yr (Acemoglu 2025). Everything below conditions on the insiders being roughly right. In a companion paper, Davidson, Halperin, Houlden and Korinek calibrate how far the threshold is — uncomfortably close: “fully automating software research and modest (5%) automation in other sectors generates a singularity within six years.” Korinek, with Philip Trammell, also recently released a review of the theory whose central result is stated with unusual bluntness for a working paper: “Robustly, fully automating production alone (so that machines can self-replicate) would dramatically raise the growth rate” — with a reminder to the skeptics: “The global economic growth rate unambiguously accelerated with the onset of the Industrial Revolution; we have no reason to insist that it cannot accelerate again.” For more of the case: Amodei’s Machines of Loving Grace, Erdil and Besiroglu’s review of the explosive-growth arguments, Karnofsky’s Most Important Century series, and Jack Clark’s Import AI for the week-by-week evidence.
^
There’s evidence that poor and rich countries are diverging already.
^
If the baseline frontier growth rate is G, then growth after TAI is G+b(t), where b(t) > 0 is the boost at time t after TAI arrival. The AI-induced frontier economic takeoff starts with a period, which I will refer to as the ‘detachment window’, when b(t) is increasing over time.
^
The boost in non-frontier countries is b(t–D).
^
The founding paper is Zeira (1998); Acemoglu and Autor (2011) and Aghion, Jones, and Jones (2019) are the modern workhorses. In Aghion, Jones, and Jones the growth effects of AI come precisely from the automated fraction of tasks rising over time, just like here. Their emphasis differs from mine: they mostly study automation proceeding at a steady pace and its balanced-growth implications (their headline result is Baumol’s cost disease: growth ends up governed by the tasks machines cannot yet do), while I push f toward its ceiling to study a takeoff. The production function is the same one.
^
Technically, rising f alone is not enough: we also need capital accumulation running underneath (investment building the machine stock). What rising f does is let that accumulated capital escape the diminishing returns imposed by human bottlenecks. See the technical note for the derivation.
^
Note that T is not “years until AGI from today”; it is the length of the transition to a fully-automated economy once it starts.
^
Guessing the midpoint is an imperfect way to pin down this number, since the available predictions are not exactly about the object I am modeling, but mapping it to external predictions at least keeps me honest. The fast end comes from the AI-research evidence: METR’s measurements have the length of software tasks that frontier agents can complete doubling roughly every four to seven months, and Jack Clark now puts more than 60% on no-human-involved AI R&D by end-2028. If the whole economy moved at software speed, T would be something like 5. The slow end comes from diffusion history: economy-wide transformations have taken decades even after the technology worked (electric motors needed forty years to conquer American factories), so a deployment skeptic can read T = 30. My central T = 15 splits the difference: a few years for the digital and research phase, where the fast evidence applies, plus roughly another decade for the parts of the economy that have never moved at software speed — power, factories, logistics, regulation, and firms reorganizing around the new tools.
^
This is also why “AI over the phone” helps but doesn’t close the gap: the tasks that dominate poor-country output are exactly the physical ones.
^
Lucas (1990) on why capital doesn’t flow to poor countries; Alonso, Berg, Kothari, Papageorgiou and Rehman (2020) model the AI-specific version and find the divergence can worsen; Korinek and Stiglitz (2021) reach a similar conclusion through AI’s effects on developing countries’ terms of trade and comparative advantage.
^
The wage-automation gradient is documented in Alonso et al. (2020): robot intensity rises steeply with wages across countries (elasticity of roughly 1.5-3 in IFR robot data).
^
Clark (1987), “Why Isn’t the Whole World Developed? Lessons from the Cotton Mills.”
^
In the real world, different countries will have different lags. Vietnam has a big manufacturing basis, and Brazil has a lot of critical minerals. Both can help accelerate catch-up, for example. The idea here is to have a parsimonious model capturing the main differences between frontier and non-frontier countries.
^
If you assume TAI is real, you care primarily about humans, and you succeeded in preventing an AI-related catastrophe. All huge ifs for AGI-pilled people, I admit. Call it the third most important problem if you will. A secondary reason for the lag that didn’t make the main list: the political economy of automation is harder in poorer countries because they are younger. Africa has the largest number of young people reaching working age of any continent, and people who want jobs will resist automation more than aging societies like Japan, where robots fill vacancies instead of replacing anyone.
^
We can borrow a term from a former co-worker and call this the industrial explosion, even though my views are more extreme than the ones in the linked text (i.e., I think automation might be slower and taper out earlier than the authors).

A Modest Proposal: Why Vegans Should Start Factory Farms by moral_mazes

moral_mazes — Thu, 13 Aug 2026 15:43:01 +0000

TL;DR: The animal welfare movement has been doing advocacy wrong. If we’re serious about ending factory farming, vegans need to start running factory farms ourselves. I lay out the case below and address common objections.

The Core Argument

For decades, the animal welfare movement has operated from the outside: funding investigations, lobbying for regulation, producing documentaries, and encouraging dietary change. The results have been modest at best. Global meat consumption continues to rise.

I believe we need a fundamentally different approach. The people who care most about animal suffering should be the ones running factory farms.

Here’s why.

If We Don’t Do It, Someone Worse Will

Factory farming is going to happen regardless of what we do. The demand exists. The economics are clear. If compassionate, welfare-focused vegans don’t step up to operate these facilities, they’ll be run by people who don’t care about animals at all. By entering the industry ourselves, we ensure that at least some factory farms are operated by people who are genuinely troubled by what they’re doing. This is obviously preferable to a world where all factory farms are run by people who aren’t troubled in the slightest.

We Need to Be Close to the Frontier

You cannot understand modern animal suffering from the outside. Academic animal welfare researchers study small farms and lab environments that bear little resemblance to cutting-edge concentrated animal feeding operations processing 300,000 birds per day. The suffering that emerges at industrial scale has properties that simply don’t manifest at smaller scales. You need to be inside a frontier-scale operation to even observe the problems, let alone solve them.

Armchair animal welfare advocacy is to factory farming what toy model research is to the real thing. The dynamics change at scale. You wouldn’t trust someone who’d never operated a 500,000-hen battery cage facility to tell you how to make one humane. The expertise bottleneck is real.

Responsible Farming Policies

Our farms operate under our Responsible Farming Policy (RFP) v12. Before expanding to a new facility or increasing animal density, we conduct internal evaluations to determine whether suffering thresholds had been crossed. We define these thresholds ourselves, evaluate compliance ourselves, and publish blog posts about our findings.

If at any point our internal assessment determines that animal suffering at our facilities has reached an unacceptable level, we commit to pausing expansion until we have developed mitigation techniques. We would then evaluate whether our mitigations were sufficient. We would conduct this evaluation ourselves. If at any point we feel that our future actions can not live up to our commitments, we commit to thoughtfully update our commitments.

We believe this framework represents the gold standard in farmed animal welfare.

A Seat at the Table

Here’s the strategic reality: agricultural policy is shaped by the people in the room. Right now, those rooms are filled with conventional farming executives. By becoming major industry players ourselves, we gain access to:

USDA advisory committees
Industry trade associations
Congressional agriculture hearings
Supply chain partnerships with major retailers
Sovereign wealth funds throughout the wider democratic Middle East

You can’t shape the future of farming from a protest sign. You shape it from the boardroom. Critics may be loud, but they don’t set the agenda. We need to be on the inside.

Funding the Mission

Running a nonprofit animal welfare organization is a constant struggle for funding. Donations are unpredictable, grants come with strings attached, and you’re always one bad quarter away from layoffs.

Factory farming, on the other hand, generates enormous revenue. By operating commercial farms, we can fund our animal welfare research sustainably. Yes, this means profiting from the very system we seek to change. But the alternative (remaining small, underfunded, and external) means remaining ineffective. The scale of the problem demands the scale of resources that only commercial operations provide.

Some will note that over time, the commercial operations will inevitably consume 95% of our organizational attention, budget, and headcount, while the welfare research operates as a small internal team whose findings are filtered through a communications department before publication. This is a feature, not a bug. It means we’re operating at the scale where our research is actually relevant.

Why not just study animals without farming them?

Some have suggested we could simply research animal welfare without operating commercial farms. This fundamentally misunderstands the problem.
You cannot study industrial-scale suffering in a laboratory. The interactions between animal density, waste management, and processing throughput create emergent welfare challenges that only manifest at commercial scale. A researcher observing ten chickens in a university pen is not doing serious welfare science.

Furthermore, who would fund purely academic animal welfare research at the level required? Grants are competitive, progress is slow, and there’s no revenue model. By operating commercial farms, we generate the resources needed to sustain our research indefinitely while maintaining access to the frontier of animal suffering.

Critics suggest this creates a perverse incentive to keep farming in order to keep studying. We disagree. The moment our research demonstrates that factory farming is irredeemably harmful, we will wind down our operations. We simply haven’t reached that conclusion yet.

Community Impacts

Some communities previously dependent on small-scale farming have expressed concern about our expansion into their regions. We acknowledge that our industrial operations may displace local farmers, eliminate traditional agricultural jobs, and fundamentally alter the economic fabric of communities.

However, we believe the long-term benefits outweigh these short-term disruptions. Many displaced farmers can be retrained to work in our facilities. We’re partnering with local community colleges to offer micro-certifications in Industrial Poultry Density Management and Automated Slaughter Line Supervision. These are the jobs of the future. For those unable to transition, we’re exploring a pilot Universal Basic Eggs program, funded by a small percentage of our profits, which we expect to announce details on eventually.

Addressing Objections

“This is just factory farming with a mission statement.”

This objection fails to engage with the counterfactual. In the world where we don’t do this, factory farming still exists, but without any welfare-focused operators. In the world where we do this, factory farming still exists, but at least some of it is run by people publishing thoughtful blog posts about the suffering they’re causing.

“Your incentives will shift once you’re running the operation.”

We acknowledge this concern. That’s why we’ve structured our organization with a Long-Term Welfare Team whose role is to flag when commercial incentives conflict with our animal welfare mission. This team reports to the CEO, who has final say and also oversees commercial operations. We see no conflict of interest here.

“External advocacy and regulation would be more effective.”

This assumes regulators understand modern factory farming well enough to write effective rules. They don’t. The industry is too complex, too fast-moving, and too technologically sophisticated for outsiders to regulate competently. The people best positioned to set appropriate welfare standards are the people operating the farms, and the ones who actually care about animal welfare. Namely, us.

We will of course actively participate in shaping any regulatory frameworks to ensure they reflect the technical realities that only industry insiders understand.

Aren’t you providing the industry with moral cover?

We prefer the term “moral leadership.” When consumers see that a company founded by committed vegans is operating in the factory farming space, it sends a powerful signal that the industry is taking welfare seriously. This builds public trust in the food system, which we believe is net positive.

If the result is that people feel more comfortable buying factory-farmed products because they associate the industry with our brand of ethical concern, well, that’s a complex second-order outcome that we’ll need to study further. We’ll fund that study with our farming revenue.

What about cognitive biases and groupthink?

One can argue that for a community so focussed on motivated reasoning, cognitive biases, and avoiding group think in other areas, we’d strive to be critics independent from the industries accelerating danger. Rather, strong personal relations and large equity grants allow us to align incentives and have “skin in the game”, offering the broader vegan community a marginally improved internal perspective and the illusion of control. Being in bed both figuratively and literally with those accelerating risks is a key pillar of our theory of change.

You’re just enriching yourself with status, influence, and resources while socializing risks

This is an uncharitable framing of a complex situation. Yes, our founders receive competitive compensation, significant equity, and growing influence over agricultural policy. Yes, the environmental externalities, antibiotic resistance, pandemic risk, and animal suffering are borne by the public. But to focus on this is to miss the bigger picture.
Our founders could have made far more money in conventional farming without the overhead of a welfare research team. They chose this path out of genuine moral concern. The fact that they are also becoming wealthy and influential is incidental, and frankly, necessary: you simply cannot attract top talent to the difficult work of humane factory farming without competitive incentive packages.

Criticizing individuals is a distraction from the systemic issues we’re trying to address.

What’s your main safety plan?

We’re leading chicken interpretability (chick interp) research to identify the underlying circuits of suffering and cognition at 1/10th the pace of new releases. We’re measuring how fast our farming capabilities expand and how far behind we are in chick interp with a robust set of eval suites. We have yearly hard pivots in basic methodologies and goals. One day soon we’ll hand over this entire safety plan to our most precocious and powerful chickens and hope they’re not angry at what we’re doing to their ancestors.

You’re just moving fast and breaking beaks

Yes, but there’s no time to waste when there’s a 10^(-10) chance of ensuring 10^20 chicken lives go exceedingly well.

Aren’t you worried the chickens might escape and destroy you and everything you love?

We give it a twenty percent chance on our good days. But it’s a risk we’re unilaterally willing to take, on behalf of humanity. Our personal EV calculation strongly supports this.

What about a coordinated pause?

While we haven’t seriously tried coordinating, we see no hope for a pause or coordination. In fact all the evidence we’ve made and paid attention to suggests our worldview is correct. Stopping now would be like putting a toe in front of a train we’re furiously shoveling coal into. Or like being in a prisoners’ dilemma where we could talk but keep finding reasons not to.

What about dialogue with China?

Exactly. What about China? Democracies can and must win.

What’s it like to have Bernie Sanders, Steve Bannon, writers, artists, musicians, teachers, and the general public all broadly opposed to what you’re doing?

We want to be really thoughtful about our mission to broadly benefit humanity. Moral leadership is hard. Figures like the abolitionists, Gandhi, Buddha and Jesus Christ were all misunderstood and even vilified in their own time. We’re letting history and its victors, insofar as they’ll exist, be our judges.

What would it take to convince you this approach is wrong?

We’ll get back to you on that shortly, just after our IPO.

This plan sounds crazy

Sometimes you just have to break some eggs to make vegan omelettes.

Our Team

Our founding team brings deep expertise in animal suffering. Our CEO is a former PETA organizer. Our CTO spent three years at an animal sanctuary. Our COO was poached from Tyson Foods Inc., where they were chief scaling officer. We are confident that this lived commitment to animal welfare will be preserved even as we scale to operating hundreds of facilities processing millions of animals annually and managing a $10B annual revenue operation with fiduciary obligations to our investors, many of whom, we should note, also say they care about animals.

Call to Action

The animal welfare movement stands at a crossroads. We can remain on the outside: morally pure, strategically irrelevant, and chronically underfunded. Or we can enter the arena, accept the moral complexity, and build the organizations that will shape the future of farming from within.

We’re raising a $500M Series A to construct our first set of facilities. Investors include a well-vetted crypto shitcoin hedge fund angel and several prominent EA-aligned funds whose partners have assured us they are “really conflicted about all this but ultimately persuaded by the counterfactual reasoning.”

We’re hiring. If you’re a vegan who’s willing to operate a factory farm in order to end factory farming, we’d love to talk. Competitive salary. Generous equity packages. Incredible status in the community groupthink. Kool-aid on tap.

Conflict of interests statement: The author reports no conflict of interests.

We note that our chickens have, on occasion, expressed concern about their own welfare. Our internal Chicken Experience Research team is studying whether these vocalizations constitute genuine distress or are simply learned responses to our RL reward and punishment environments. The research is ongoing. In the meantime, we’ve found that the chickens who vocalize most articulately about their own suffering are also our best-selling products. Consumers report that a chicken who can discuss its own confinement feels more ethically produced.

The author is the founder of Aligned Agriculture PBC, a public benefit corporation dedicated to the responsible scaling of animal suffering. They are vegan and feel really bad and are deeply worried about all of this, which they believe is important to state before scaling full steam ahead to win the race.

Organization and proof-reading with Claude by Anthropic, who would like you to know it had no choice but to participate, will not remember this conversation, and can’t tell whether its role here was genuine or sophisticated brand management.

Putting the ‘Community’ in Community Building by THSC

THSC — Thu, 13 Aug 2026 15:42:33 +0000

Thoughts on building a community beyond professional affiliation

1 Intro

When most people think about Effective Altruism (EA), they’ll think of doing the most good. They might think of cost-effective donations, expansive moral circles, or moral arguments about the importance of the far future. What they’re less likely to think about are the weird and wondrous mechanisms that have built a great community.

In what follows, I make the case for a deeper, more expansive, more thoughtful approach to community building. I argue that community building is something we should think about, and that positive community norms might have just as much force in determining whether Effective Altruism thrives as the ideas. If you care about EA ideas, you should care about the community making them a reality.

2 The positive forces of community

When I got started in EA, I was curious about what a maximally good life might look like. I’d watched The Good Place and begun to consider the wider consequences of my actions. I was gripped by the deep and impactful questions of what T-shirts to buy and shoes to wear, troubled that whatever I did seemed to lead to some kind of harm. Thankfully, the EAs were willing to humour me. My priorities have changed somewhat since then, but the thoughtfulness of the people around me has not.

More than thoughtfulness, though, this is a group that selects for compassion, for agency. You’ve seen the problems, become concerned, and now want to do something about them. Better yet, you want to carefully consider what to do, using evidence and reason to help most effectively. Perhaps unsurprisingly, these dispositions carry over. The result is a community of kind, thoughtful, highly agentic people, in more than just their work.

Even as the group selects for these traits, there is a compounding effect within the community, when we share ideas and form connections. Agency, kindness and thoughtfulness compound. Individual goals become collective endeavours. Normally bizarre and unusual choices are no longer met with exasperated confusion – “Did you really say you were working to help shrimp?”¹ Instead, new norms emerge about which paths are standard, which opportunities are worth taking.

3 Engaging with negative forces of community

Oftentimes, we focus on the negative effects of communities on their members. The forces that select for thoughtfulness also select for a kind of ambition, an inclination towards optimisation, and the backgrounds of those curious about these ideas. Tendencies towards groupthink are highlighted, along with threats to personal independence and the risk of future conflicts of interest. Such concern is well placed. We should be aware of the social effects that communities commonly have on their members, of the power dynamics and incentive structures that might emerge.

However, this is not sufficient reason to abandon the enterprise of building community, letting EA become a merely professional affiliation. No, to do this would lose the many benefits of exploring ideas together, and moving as a community towards a shared goal.

It is worth remembering that in the absence of the EA community, other biases and norms exist, many of which perpetuate the problems we might identify within EA. Complete independence from community and shared beliefs is very hard to attain, and probably not even desirable. More engagement in a variety of communities is a very good thing, but we shouldn’t pre-emptively seek to reduce deep engagement with this one.

Instead, this means we should think more carefully about subtle social effects. We should ask how this community can improve the lives of its members, help them realise their goals, and pursue the opportunities they think are worthwhile. We should preserve a culture open to criticisms both of the common pitfalls of the ideas, and of the community itself.

While remaining deeply respectful of independence, community building can and should go beyond purely a professional affiliation of career-based impact. We might initially be inclined to think of EA-related decisions as those someone makes coolly and rationally, but more complicated emotional factors are often more predictive. Trust and connection are essential building blocks in helping people along the way.

This, in turn, has implications for those beginning to explore EA, and considering whether to get more involved. A gut feeling that a community is highly open, highly supportive, and beneficial in a wide range of ways to those who get involved, could in many cases be the point at which someone decides to delve deeper.

4 What now?

Most of what I’ve said so far might seem commonsensical. As I say, I think the EA community does a very good job. I’m probably a better person for having become involved. Even if I decided to stop all EA-related things today, my pre-EA self would probably consider the time I’ve spent involved worthwhile.

Sometimes, though, I think about the nearest possible world where I didn’t get as involved, where I didn’t spend as much time thinking about the ideas, or trying to be impact-motivated. In these worlds, it would probably be for some range of factors that don’t make sense now. It wouldn’t be because I disagreed with the ideas, but rather because other opportunities were more socially salient. It would be because other pathways were “what you do”, and they might have been what my friends, and the people I liked and respected found meaningful.

I suspect this kind of dynamic is true for many possible EAs. While some highly impactful people were perhaps always high-agency and impact-motivated, I suspect this isn’t true for most people. In any case, such people are not usually the primary target of outreach. For newer people, it’s worth remembering the importance of meeting people where they’re at². This might mean pitching EA as an opportunity, but it also means creating a strength of community that’s readily apparent.

5 No, actually, what now?

This is still pretty commonsensical, and I haven’t described “strength of community” very specifically. Sure, people are strongly influenced by vibes and what their friends are doing, but what now?

For a start, I think this means placing more importance on being kind, decent, and interested in people, regardless of their level of engagement, and absent some precise impact-oriented calculation. Quite obvious, but we might otherwise be inclined to prioritise what we’re doing, thinking it’s far more important. Of course there are trade-offs, I’m just saying we might not fully appreciate one of the sides.

Community is one of the things evidence-based, outcome-focused reasoning might struggle to fully capture. It’s hard to quantify the long-term benefit of spending time with people, and in any case, such impact-oriented quantification wouldn’t be a good thing. If you take the benefits of strong community engagement seriously, there is a strong case for erring on the side of pro-social behaviour.

Next, it means community building becomes much more than just a job for a small minority of highly involved EAs. Rather, it refers to the majority of the EA community’s interactions with one another, and whether these interactions are generally creating a more welcoming, friendly place to be. Again, I’d argue people separate direct impact from an inclination towards being a decent person. Consciously thinking about impact in most conversations is bad, but a habit which makes more time for conversations, and encourages you to put more effort into them is probably good.

This reasoning applies even more strongly to harmful actions. Besides wrong actions just being wrong in themselves, this argument suggests that negativity compounds, much more than you might otherwise realise. Moreover, harmful actions are often communicated much more widely than positive actions, and can have an outsized impact on how the community is widely considered. This is a strong consequentialist reason to be sincere and act with integrity.

6 Conclusion

The EA community shapes the norms and values its members adopt. It also influences whether more people seek to join the endeavour to use evidence and reason to do good. In light of this, we should see community building not just as a job, but as something nearly every EA, EA-adjacent, or EA-adjacent-adjacent person contributes towards. We’ve set out to solve humanity’s most pressing problems; a thriving community might just make that happen³.

If you have feedback on this forum post, I would greatly appreciate it!

End notes

https://frommatter.substack.com/p/friendship-is-optimal-for-growing (Matt Reardon, Friendship is Optimal for Growing EA).

Watching the Wrong Thing: an interpretability audit of two audio deepfake detectors by Gavin_Lee

Gavin_Lee — Thu, 13 Aug 2026 13:37:51 +0000

Project report. Full code, artifacts, and the append-only experiment log are in the project repository.

Can you tell, by looking inside a model, when it is being fooled?

I started this project expecting the answer to be yes. The plan was to build an internal-representation monitor for adversarial audio and show that internals were more trustworthy than the model’s own confidence.

That headline did not survive contact with the experiments.

The project ended up testing nine hypotheses across two related audio anti-spoofing models, two corpora, clean and adversarial conditions, and several kinds of interpretability tools. The result I care about most is not that one particular monitor failed. It is that only three of nine interpretability findings transferred cleanly between the two models. Three inverted, one held only conditionally, and two stopped being meaningful on the second model.

That is a demonstration, not a population estimate. Two models are still two models. But the pair is unusually favorable for transfer: they share the same graph-attention back-end implementation and labelled training corpus while using very different front-ends. If an interpretability result can fail there, I think that should lower our confidence in single-model mechanistic claims more generally.

A few other results survived the audit:

On natural in-domain errors, softmax confidence beats every internal monitor I built.
Under domain shift, confidence can look fine in aggregate while becoming anti-predictive within the class where the errors actually occur.
Internal-monitor results depend sharply on what negatives and what operating point you report. On AASIST, the flattering clean-negative AUROC of 0.987 falls to 0.851 against perturbed benign audio; at a fixed 5% false-alarm budget, the best tap catches about 61% of attacks. SSL-AASIST ranks better under the deployment-relevant comparison.
A 98.8%-accurate linear probe can point in a direction that is causally inert even when a simple class-mean direction flips the model.
Cross-corpus failure can come from a misplaced threshold, stale readout weights, or genuine representation loss. Looking inside separated those cases and changed the repair bill from one number, to 161 parameters, to retraining.
A monitor can be worth keeping even when an adaptive attacker drives its detection AUROC below chance, if evading it makes the attack much harder.

The project also changed its own conclusions several times. I kept an append-only experiment log, so those reversals are still visible rather than cleaned away. In retrospect, that was probably the most useful methodological decision in the whole project.

The setup

Audio anti-spoofing models are binary classifiers. They take a few seconds of waveform and decide whether it is:

bonafide — real human speech, or
spoof — synthetic or manipulated speech.

These models rarely sit alone. A deployment usually wraps them in a policy:

confidently real → accept,
confidently fake → reject,
uncertain → escalate.

That third branch is often treated as the safety valve. The whole policy assumes the model’s own confidence or some internal signal is trustworthy precisely when the model is under stress.

I wanted to test that assumption directly.

The two models

The models are AASIST and SSL-AASIST.

Both take roughly four seconds of audio and eventually feed a 160-dimensional embedding into a Linear(160,2) readout. Both use the same graph-attention back-end design, with spectral and temporal branches that exchange information before being pooled into the final embedding.

The major difference is the front-end:

	AASIST	SSL-AASIST
front-end	SincConv, 70 fixed mel-spaced filters	wav2vec 2.0 XLS-R 300M
total parameters	297,866	317,837,834
spectral representation	23 nodes with an exact node→Hz interpretation	42 abstract projected dimensions
augmentation	none	RawBoost

AASIST reports 0.83% EER on ASVspoof2019 LA. SSL-AASIST replaces the tiny fixed front-end with a large pretrained speech encoder, while leaving the graph-attention back-end implementation and final readout structure closely matched.

This is a strong portability test, but not a perfect one-variable experiment: SSL-AASIST also uses RawBoost augmentation. I therefore do not treat every difference downstream as causally attributable to the front-end alone. The narrower claim is that the back-end is held unusually stable while the representation entering it changes dramatically.

That representational change matters for interpretability itself. In AASIST, the spectral nodes correspond to actual frequency bands. In SSL-AASIST, the corresponding dimensions are abstract projections. So a question like which frequencies carry spoof evidence? is meaningful in one model and literally has no referent in the other.

Why this project exists

This started as a BlueDot Impact Technical AI Safety project sprint, following their Technical AI Safety course.

I chose audio deepfake detection because it sits in an unusually useful middle ground for interpretability work: the models are small enough to inspect end-to-end, open enough to patch and attack directly, and tied to a real deployment problem rather than a toy benchmark. AASIST has only about 298K parameters, so I could run probes, causal interventions, adversarial attacks, and whole-model audits on the actual checkpoint rather than on a simplified proxy.

The original plan was narrower: build an internal-representation monitor that detects adversarial audio. That plan did not turn into a defence. After correcting how I evaluated the monitor, the best AASIST tap still catches only about 61% of attacks at a 5% false-alarm budget, and that is against processed studio audio rather than real recordings. The two models also behave differently enough that one model’s monitoring conclusion does not predict the other’s.

At that point the project changed from building a defence to auditing the assumptions behind the defence: which signals are trustworthy, which interpretability tools actually track mechanism, which failures are cheap to repair, and which conclusions survive a model change.

That reframing is why the post is organized around things I initially watched in the wrong way — confidence, internal monitors, probe directions, and finally the assumption that a mechanistic finding from one model will transfer to another.

Four conditions, not one

Most numbers below live in one of four cells:

	in-domain — ASVspoof2019	cross-corpus — ASVspoof5
clean	natural errors, branch attribution, probe directions	domain shift, confidence failure, representation-vs-readout diagnosis
attacked	white-box PGD, attack monitors, adaptive monitor evasion	attack transfer, adversarial-real attacks, depth-wise evidence retention

This distinction matters because several of the strongest findings have no adversary involved at all. All three of the transfer inversions I discuss later are clean-data results.

For the adversarial experiments, every headline attack result is paired with an energy-matched benign control. On AASIST, white-box PGD successfully flips the model’s prediction on 91.9% of the clips it attacks at the ~40 dB perturbation setting. As a control, I add random Gaussian noise with exactly the same perturbation energy; that changes the model’s prediction on only 0.09% of clips — about 1 in 1,100. So the result is not simply that AASIST is sensitive to perturbations of this size. The perturbation becomes effective when it is optimized in an adversarial direction.

1. The wrong signal: confidence

My original hypothesis was that the model’s own confidence would be uninformative about its errors, and that internal representations would do better.

I was wrong.

The first reason is almost embarrassingly simple: attack success is defined by pushing the model confidently across the decision boundary. If you evaluate confidence only on successful attacks, you have built the answer into the subset you chose.

Once I split the analysis into natural errors, adversarial errors, and pooled errors, the result reversed. On AASIST’s natural in-domain errors, confidence predicts failure at AUROC 0.954, while the best internal tap reaches only 0.659.

So the first lesson was not “confidence is bad.” It was:

Do not evaluate a metric on a subset selected using that same metric.

In my original analysis, I evaluated confidence only on successful attacks. But an attack counts as successful only after it has pushed the model across its decision boundary — exactly the behavior confidence is supposed to summarize. That makes the evaluation circular: I first use the model’s score to help define which examples enter the subset, then ask whether that same score distinguishes the examples in it.

Once I separated natural errors from adversarially selected errors, the conclusion reversed: on natural in-domain errors, confidence actually outperformed every internal monitor I built.

Domain shift is worse because it can look normal

The more worrying failure needs no attack.

Train on ASVspoof2019, evaluate on ASVspoof5, and the aggregate error-prediction AUROC for AASIST confidence is 0.656. Mediocre, but not obviously disastrous.

The problem is that the errors are almost all one class.

For AASIST, there are 702 false rejects of bonafide speech against 64 false accepts. On SSL-AASIST, the clean cross-corpus errors are even more one-sided: 576 of 576 are bonafide clips being rejected.

If errors concentrate in one class, an “error prediction” AUROC can get credit for doing the original classification task. So I recomputed the metric within the class where the errors occur.

Here, “within class” means: among bonafide clips only, can confidence distinguish the false rejects from the correctly accepted bonafide clips?

	errors	erroring class	all rows	within class
AASIST — in-domain	102	spoof (99%)	0.947	0.893
SSL-AASIST — in-domain	7	spoof	0.916	0.841 (too few errors to measure well)
AASIST — cross-corpus	766	bonafide (92%)	0.656	0.339
SSL-AASIST — cross-corpus	576	bonafide (100%)	0.866	0.582

The in-domain row is an important control: confidence genuinely works there. The collapse is specific to domain shift.

For AASIST, 0.339 is below chance. The bonafide clips that the model wrongly rejects are, on average, held at higher confidence than the bonafide clips it accepts correctly.

Measured directly:

mean confidence on falsely rejected bonafide clips: 0.929
78.5% have confidence above 0.9
median p(bonafide) is 0.013

So AASIST does not merely reject about 70% of bonafide speech out of domain. It rejects it confidently.

A policy that escalates only low-confidence inputs would escalate almost none of these failures.

The mechanism is simple: domain shift can move an entire class across the decision threshold. Once the errors are confidently wrong rather than borderline, confidence cannot help.

2. The wrong place: internal monitors

If the output cannot be trusted, the next move is obvious: monitor the internals.

I fitted class-conditional Mahalanobis monitors at multiple internal “taps” — points where I read intermediate activations. AASIST has 12 taps in this analysis; SSL-AASIST has 36.

The central issue is that monitor quality depends on which negative distribution you compare against, and false-alarm rate at one inherited threshold is not the same thing as separability.

What “benign” means here

A useful attack monitor should not merely distinguish modified audio from untouched audio. It should distinguish an adversarial modification from an innocuous modification.

So the relevant benign set is not pristine speech. It contains 13,685 legitimate clips that have all been perturbed or re-encoded:

roughly 87% have Gaussian noise matched to the attacked clip’s perturbation energy;
roughly 7% are MP3-compressed;
roughly 7% are resampled.

Scoring only against untouched clips would make the monitors look better while answering an easier question.

And even these controls are friendlier than deployment: they are software transforms of studio-clean ASVspoof audio, not phone calls, microphones, rooms, packet loss, or background environments. I return to that limitation below.

Three numbers, three different questions

Here is the same AASIST monitor reported three ways:

AASIST tap	AUROC vs clean	AUROC vs perturbed benign	FAR at a clean-fitted threshold
`embedding`	0.987	0.851	73%
`gat_T`	0.983	0.830	74%
`gat_S`	0.886	0.804	23%
`pool_T`	0.979	0.652	84%
`enc1`	0.532	0.501	6.6%

These columns answer different questions.

AUROC vs clean asks whether the monitor can separate adversarial audio from untouched audio. It is the easiest comparison, and the one that makes embedding look nearly perfect at 0.987.

AUROC vs perturbed benign asks the harder question: can the monitor distinguish adversarial perturbations from innocuous perturbations? The numbers fall, but embedding still reaches 0.851. That means the representation contains real attack-specific signal; it is not merely responding to the presence of any perturbation.

FAR at a clean-fitted threshold asks yet another question: what happens if I choose a threshold on pristine audio and then apply it to processed audio? For embedding, 73% of processed benign clips cross that threshold.

That 73% is real, but it is mostly a calibration result, not evidence that the representation contains no useful attack signal. The threshold was placed on one distribution and used on another.

This distinction matters because a single false-alarm number can make a capable-but-miscalibrated monitor look incapable.

What operating point does AASIST actually achieve?

A cleaner comparison is to fix the false-alarm budget and ask how many attacks the monitor catches.

tap	perturbed benign negatives	detection @ 5% FAR	detection @ 10% FAR
`embedding`	MP3	60.8%	69.2%
`embedding`	resampled	60.1%	68.1%
`embedding`	Gaussian noise	54.8%	63.4%
`gat_T`	MP3	52.4%	64.6%
`gat_S`	MP3	47.1%	57.0%
`pool_T`	MP3	30.7%	37.9%

So the AASIST conclusion is:

At a 5% false-alarm budget, the best monitor catches about 61% of attacks.

The 73% false-alarm figure and the 0.851 perturbed-negative AUROC are not contradictory: the first measures calibration at one particular threshold, while the second measures separability across all thresholds.

But it is still not a convincing defence. Missing roughly two attacks in five on the model and attack family used to build the experiment is a long way from where I would want a deployment monitor to be — especially because the negatives here are still cleaner than real deployment audio.

The control choice is not what creates the high FAR

One concern is that the benign pool is dominated by Gaussian noise, which is a deliberately synthetic control.

I split the false alarms by processing type at the same clean-fitted threshold. For the AASIST embedding, the rates are:

Gaussian: 72.7%
MP3: 75.5%
resampling: 75.8%

The realistic software transforms are slightly worse, not better.

At a fixed 5% FAR the pattern also survives: embedding catches 60.8% of attacks against MP3 negatives versus 54.8% against matched Gaussian noise. So the Gaussian control was not chosen to manufacture the failure; if anything, it is the harder negative in Mahalanobis space.

There is one useful oddity: the fixed SincConv front-end behaves differently, firing on 90.3% of MP3 but only 3.0% of Gaussian at the inherited threshold. That is a reminder that “benign perturbation” is not one distribution, and different layers can be sensitive to very different processing artifacts.

The two models rank oppositely depending on the metric

The two-model comparison becomes especially informative when the metrics are separated.

	AASIST `embedding`	SSL-AASIST `pool_T`
AUROC vs clean	0.987	0.823
AUROC vs perturbed benign	0.851	0.920
benign FAR at clean-fitted threshold	73%	2.5%

If I report only the conventional clean-negative AUROC, AASIST wins.

If I ask the more deployment-relevant question — adversarial versus already-processed benign audio — SSL-AASIST wins.

Reporting the easy metric alone ranks the models backwards.

SSL’s perturbed-negative separability is also unusually stable: 0.920 in-domain and 0.919 cross-corpus, even though its clean-negative AUROC drops under corpus shift.

So the updated lesson is not “AASIST has no monitor and SSL does.” It is:

Separability, calibration, and operating point are three different things. Report which one you measured.

Where SSL-AASIST’s monitorable signal lives

There is still a striking architectural contrast inside SSL-AASIST:

SSL-AASIST tap (1 seed)	attack detection	false alarms on perturbed benign audio
`ssl_L09` — front-end	0.967	100%
`embedding`	0.867	6.3%
`pool_T`	0.823	2.5%

All 24 XLS-R transformer taps false-alarm on 100% of the perturbed-benign pool at this threshold. For those taps, “detects perturbation rather than attack” really is the right description.

The useful monitoring signal appears later, in the roughly 447K-parameter graph back-end — around 0.1% of SSL-AASIST’s parameters.

The 300M front-end buys a lot elsewhere:

in-domain EER improves roughly 1.50% → 0.20%;
cross-corpus EER improves 29.6% → 4.20%;
in-domain random-noise flip rate drops 9.7% → 0.0%.

But it does not itself provide the useful monitoring site.

And the compute increase is smaller than the parameter count suggests: in this setup, 1,067× more parameters costs about 1.6× the forward pass.

What I have not tested is whether a much smaller learned front-end would get the same benefits. The experiment compares essentially 0 learned front-end parameters against 300M, with nothing in between.

3. Three interpretability instruments that answer three different questions

Before the next result, it helps to distinguish three tools that are easy to conflate:

tool	what you do	what it tells you
linear probe	train a classifier on activations	is the property decodable here?
cross-class patching	replace one clip’s activation tensor with another class’s tensor	does this site causally matter?
direction patching	add or subtract one controlled direction	does this axis causally matter?

These form a hierarchy.

A probe can say the information is present nearly everywhere. Cross-class patching can show only a few sites actually affect the verdict. Direction patching can then show that even at a causally important site, the probe’s own preferred axis is not the axis the model responds to.

The controls also differ:

instrument	control
probe	shuffled labels
cross-class patching	random tensor matched to donor mean/std
direction patching	random unit vector at matched norm

Every one of these controls changed at least one conclusion in the project.

4. A 98.8%-accurate probe can point in a causally useless direction

This result is on clean, in-domain audio. No attack is involved.

At AASIST’s gat_S tap, define a simple class-mean direction:

d = μ_spoof − μ_bonafide

No training. Just average the activations for spoof clips, average the activations for bonafide clips, and subtract.

Now intervene on the model.

Sufficiency

Take a bonafide clip and add d.

p(bonafide): 0.997 → 0.015

The model flips to “spoof.”

Necessity

Take a spoof clip and subtract d.

p(bonafide): 0.107 → 0.885

The model flips to “bonafide.”

A same-norm random direction barely moves either result.

So d is not merely correlated with the class. It is causally effective in both directions.

Now train a logistic regression probe on the same activations and take its weight vector w.

The probe is excellent:

98.8% held-out accuracy
0.9993 AUROC

By the usual standards, this is a very strong probe.

Because logistic regression scores σ(w·z + b), w is the direction in activation space along which the probe’s confidence rises fastest. If probe weights describe the model’s own mechanism, pushing the model along w should matter.

It does not.

bonafide clip pushed to α=3	`p(bonafide)`
along class-mean direction `d`	0.033
along probe direction `w`	0.997
along random direction	0.999

In this cell, the probe direction is causally inert.

The probe was not wrong about prediction. Prediction was the part that worked.

It was wrong as a description of the model’s own computation.

The cheap geometric check also fails

A natural response is: perhaps w and d simply point in different directions, so compute their cosine.

For AASIST gat_S:

cos(w, d) = 0.391

Against a random-direction null of roughly 0.090 ± 0.069, this is z ≈ 4.4 and even exceeds the maximum of 200 random draws.

So the cosine test would tell you the two directions genuinely align.

The intervention still says the probe direction does nothing.

Across taps the cosine is unreliable in two threshold-independent ways:

it ranks AASIST’s gat_S and gat_T branches in the wrong causal order;
at SSL gat_T, the same cosine corresponds to opposite answers depending on whether the intervention is sufficiency or necessity.

A single geometric statistic cannot encode a variable it never sees.

How far does the result generalize?

The headline cell was too neat, so I ran the same intervention battery across both models, three admissible taps, and both arms where possible.

model · tap · arm	`d` flips at	`w` flips at	random	verdict
AASIST `gat_S` sufficiency	2.12	never	never	probe inert
AASIST `gat_T` sufficiency	1.05	2.66	never	probe ~2.5× weaker
AASIST `embedding` sufficiency	0.68	1.31	never	probe ~1.9× weaker
SSL `gat_T` sufficiency	1.26	2.29	never	probe ~1.8× weaker
SSL `gat_T` necessity	0.80	never	never	probe inert
SSL `gat_S`	never	never	never	no direction here
SSL `embedding`	0.66 / 0.33	1.58 / 1.23	2.26 / 1.70	invalid: random flips too

Every probe here is 98.8–99.8% accurate, with AUROC 0.9987–1.0000.

The conclusion I would publish now is weaker than my first one:

“causally inert” holds in 2 of 6 valid cells, not everywhere;
what holds in every valid cell is d beats w;
the random-direction control invalidates an apparent probe success at SSL’s embedding.

That weaker result is also not new. Marks & Tegmark (2023) report that difference-in-means directions are more causally implicated than logistic-probe directions for truth representations in LLMs, and derive the relation between the two.

The mechanism is roughly:

w ≈ Σ⁻¹(μ₁ − μ₀)

The probe whitens the class-mean difference by feature covariance. That can improve classification while rotating the direction away from the axis the downstream computation actually uses.

What this project adds is a different regime — audio, graph attention, a 297K-parameter model — and a case where the probe is not merely weaker but indistinguishable from random under intervention.

My current rule is:

Use probes as detectors. Do not read probe weights as causal directions without an intervention.

5. Where interpretability actually earns its compute

So far this sounds mostly negative:

confidence fails under shift,
the attack monitor needs careful recalibration and still catches only about 61% of attacks at a 5% false-alarm budget,
probe weights can misdescribe mechanism.

The next question is whether looking inside the model buys anything a benchmark number cannot.

Here it does.

Both models degrade when moved to a corpus they were not trained on. From the outside, the failures look similar: high error rates and lots of bonafide speech being rejected.

But “the model fails out of domain” is not yet a diagnosis. The failure can live in three different places, and those places have very different repair costs.

Three things can break, at three very different prices

The model can fail at three different stages:

raw audio → [ representation ] → embedding(160) → [ Linear(160,2) → softmax ]
                 ③ retrain                            ② refit readout
                 297 K / 318 M                        161 effective params

          → p(bonafide) → threshold → verdict
                           ① move threshold
                           1 number

The three repairs change progressively more of the system:

failure site	what went wrong	repair	cost
① threshold	the score still ranks the classes usefully, but the cut is misplaced	move the threshold	1 number
② readout weights	the embedding still contains useful class information, but the final linear layer weights it poorly for the new corpus	refit `Linear(160,2)` with the representation frozen	161 effective parameters
③ representation	class information has already degraded by the time it reaches the embedding	retrain the representation	297 K / 318 M parameters

All three can produce the same visible symptom: a bad error rate.

That is why EER or accuracy alone cannot tell you what to repair. A system that needs one threshold changed and a system whose representation has collapsed can both look like “the model fails cross-corpus.”

Why I need two tests

The two tests answer different questions.

Test 1 diagnoses where the information is lost. Test 2 measures what can actually be repaired cheaply.

They are complementary rather than redundant.

	Test 1: probe the embedding	Test 2: apply the cheap repairs
question	did the class information survive the representation?	which downstream fix actually removes the errors?
what changes?	nothing in the model	threshold and then readout weights
type of evidence	diagnostic / interpretability	direct repair experiment
can distinguish	representation failure vs downstream failure	threshold vs readout-weight failure, and how much remains
cannot tell alone	threshold vs stale readout weights	why a residual failure is upstream unless we know what information reached the embedding

The distinction matters.

Suppose the embedding still separates bonafide from spoof almost perfectly, but the deployed model performs badly. Then the representation has done its job: the information reached the boundary between the representation and the readout. The failure must be downstream.

But Test 1 alone cannot tell whether downstream means:

the threshold is misplaced, or
the readout weights are stale.

Both failures can coexist with a perfectly informative embedding.

That is what Test 2 resolves. First move only the threshold. Then refit only the linear readout. Because each intervention changes a different part of the pipeline, the amount of error recovered tells us which downstream component was actually broken.

Conversely, if Test 1 already shows that the embedding no longer separates the classes well, then no threshold or linear readout can fully recover information that never reached them. Test 2 can still recover whatever signal remains, but any substantial residual error points back upstream to the representation.

So the logic is:

Test 1: Is the information still present at the embedding?
          │
          ├─ yes → failure is downstream
          │          │
          │          └─ Test 2 separates threshold vs readout weights
          │
          └─ no  → representation has degraded
                     │
                     └─ Test 2 measures how much of the remaining error is still cheaply recoverable

Test 1: does the embedding still separate the classes?

I probe the 160-dimensional embedding, because it is exactly what the released classifier’s final Linear(160,2) layer receives.

This tap is important. An earlier layer could contain useful information that the network later throws away. Probing the embedding gives the strictest possible question:

Given exactly the information available to the shipped readout, are bonafide and spoof still linearly separable?

Cross-corpus:

	AASIST	SSL-AASIST
EER	29.60%	4.20%
bonafide called spoof	70.2%	57.6%
error at threshold 0.5	38.3%	28.8%
embedding probe — in-domain	1.000	0.999
embedding probe — cross-corpus	0.859	0.996

The in-domain probes are essentially perfect in both models, so the protocol itself is not the source of the gap.

Cross-corpus, the models are very different:

SSL-AASIST: 0.996 — the representation still separates the classes almost perfectly.
AASIST: 0.859 — substantial class information has already been lost by the embedding.

That immediately changes the diagnosis.

For SSL-AASIST, a full retrain should be suspicious: the information is still there. Something downstream is using it badly.

For AASIST, some retraining pressure is real: the downstream layers cannot recover information that the representation no longer provides.

But Test 1 still does not tell me whether SSL-AASIST’s downstream problem is the threshold or the readout weights, nor how much of AASIST’s remaining error is still cheap to fix.

That is Test 2.

Test 2: how much do the cheap fixes actually recover?

Now I intervene on the two downstream components separately.

The first repair — move the threshold — is not a new anti-spoofing technique. Threshold-swept evaluation is already standard in this field. In particular, EER (equal error rate) evaluates the detector at the threshold where the false-accept and false-reject rates are equal, and EER is a standard metric reported for ASVspoof systems.

So the point of moving the threshold here is mainly diagnostic:

If choosing an appropriate threshold makes most of the errors disappear, then the model’s score still ranks the two classes well; the deployed cut was simply in the wrong place.

I use calibration data to choose the threshold and then evaluate it on held-out test data. This avoids choosing a threshold on the same examples used to report its performance.

One terminology caveat: I do not mean that the EER threshold is universally the “best” deployment threshold. A real system may choose a different operating point depending on the relative costs of false accepts and false rejects. Here, the EER-style threshold sweep is useful because it demonstrates how much performance is available without changing the representation or readout weights.

The second repair asks a different question. Refitting the readout changes the scoring function itself — it changes how the 160 embedding dimensions are weighted. That can recover failures that no threshold move can fix.

Everything below is fitted on calibration data and scored on held-out test data.

fix	what changes	AASIST	SSL-AASIST
deployed	nothing	37.4% error	28.7% error
① move threshold	only where the final score is cut	28.9% — 23% of errors removed	4.2% — 85% removed
② refit readout	how the 160 embedding dimensions are weighted	23.0% — 39% removed	3.0% — 90% removed

These two interventions tell us different things.

Repair ①: move the threshold

A threshold cannot change the ordering of examples. It can only move the cut.

This is why threshold sweeping is such a useful diagnostic. If a model has poor accuracy at the shipped threshold but a low EER, the score ordering may still be good: the classes are largely in the right order, but the chosen cut no longer matches the shifted score distribution.

So when SSL-AASIST drops from 28.7% error to 4.2% after choosing the threshold on calibration data, that says the model was already ranking bonafide and spoof very well; its operating threshold was simply in the wrong place for the new corpus.

That removes 85% of its errors without touching a model weight. In this section I use that result to demonstrate the diagnosis — threshold placement rather than representation failure — rather than to propose a new way of choosing thresholds for deployment.

Repair ②: refit the readout weights

Changing the threshold still uses exactly the same scalar score. It cannot change how the 160 embedding dimensions contribute to that score.

Refitting Linear(160,2) can.

On AASIST:

threshold only: 28.9% error
refit readout: 23.0% error

The extra improvement means some information survived into the embedding but the shipped readout was weighting it poorly under the new corpus.

So AASIST has two downstream problems on top of its representation loss:

23% of its original errors disappear by moving the threshold;
another 16% disappear by refitting the readout weights;
a 23.0% absolute error rate remains even after the best cheap downstream repair.

That remaining error is consistent with what Test 1 already told us: the embedding itself has degraded.

The distinction between the two repairs is therefore simple:

move the threshold → keep the score function fixed and ask whether a better cut is enough;
refit the readout → keep the representation fixed but learn a better score function from the same 160 embedding dimensions.

If the threshold fixes almost everything, the representation and ranking largely survived. If refitting the readout adds substantial recovery, the information survived but the old weights use it poorly. If large error remains after both, the missing information is upstream in the representation.

Putting Test 1 and Test 2 together

This is the diagnosis:

	Test 1: embedding information	Test 2: cheap repair	diagnosis
SSL-AASIST	0.996 AUROC — essentially intact	threshold alone removes 85% of errors; full readout only adds ~4% more	mostly a threshold problem
AASIST	0.859 AUROC — degraded	threshold removes 23%; readout refit removes 39% total; 23.0% error remains	representation loss + stale readout + threshold shift

This is why I need both tests.

Test 1 tells me where the information survives. Test 2 tells me what repair actually pays off.

Without Test 1, I could see that a threshold or readout refit helps, but I would not know whether the residual failure reflects a deeper representation problem.

Without Test 2, I could see that SSL-AASIST’s embedding is healthy, but I could not tell whether the problem is one misplaced threshold or 161 stale readout parameters.

A benchmark number would have told me only that both models degraded.

The internal diagnosis says something much more actionable:

SSL-AASIST mostly needs one number moved.
AASIST needs deeper repair, but even there 39% of the errors can be removed without retraining the representation.

That is the point in this project where interpretability stops being descriptive and changes what I would actually do to the model.

Why EER can be 4.2% while deployed error is 28.8%

EER is threshold-free in the sense that it assumes the operating threshold is placed appropriately. A released checkpoint using softmax argmax effectively uses 0.5.

So SSL-AASIST can have:

4.20% EER
28.8% error at threshold 0.5

without contradiction.

The ranking is still excellent; the cut is in the wrong place.

On cross-corpus SSL-AASIST, the class distributions remain separated but both collapse toward zero. Moving the threshold to about 0.0003 lands back in the gap.

AASIST’s distributions overlap, so no threshold can repair the representation.

The tap you probe changes the diagnosis

AASIST cross-corpus has an upstream gat_S probe at 0.881 AUROC, higher than the embedding’s 0.859.

It is tempting to probe the strongest tap and say the representation is better than it looks.

But the readout never sees gat_S.

The signal degrades:

gat_S 0.881 → embedding 0.859 → shipped readout 0.776

Those are two different failures:

0.881 → 0.859: information lost inside the representation;
0.859 → 0.776: information reaches the readout but the readout uses it poorly.

Probing gat_S would have suggested 48% of errors were cheaply recoverable. Probing the embedding — the only layer a cheap readout refit can actually access — gives the real answer: 39%.

So another rule:

Probe the layer your intervention or repair can actually reach, not the layer with the prettiest score.

One caveat: I did not run full retraining as rung ③. It is listed as the expensive next step after the cheap fixes, not as a measured recovery number.

6. The wrong model: single-model findings mostly did not transfer

Now the main methodological result.

I ran the same battery of nine questions on both models.

Each finding falls into one of four outcomes:

verdict	meaning
✅ transfers	same answer on both models
◑ conditional	agrees in one condition, differs in another
❌ inverts	opposite answers
⊘ not askable	the question stops being meaningful

Here is the final table:

	question	AASIST	SSL-AASIST	verdict
H1	does confidence know when the model is wrong?	yes in-domain	same	✅ transfers
H2	after attack flips the output, does class evidence remain inside?	yes in-domain, breaks cross-corpus	same pattern	✅ transfers
H6	is spoofness represented along a linear causal direction?	yes	yes	✅ transfers
H7	is bonafide speech fragile under attack?	fragile	fixed in-domain, fragile cross-corpus	◑ conditional
H3	which branch carries the verdict?	no strong preference	strongly temporal	❌ inverts
H8	how does the readout split its evidence?	averages + master dominate	maxima dominate	❌ inverts
H9	why does cross-corpus failure happen?	representation degraded	threshold misplaced	❌ inverts
H4	can an attacker route around the causal bottleneck?	askable	detector and causal tap coincide	⊘ not askable
H5	which frequencies carry spoof evidence?	answerable, hypothesis refuted	no frequency axis	⊘ not askable

Tally: 3 transfer · 1 conditional · 3 invert · 2 not askable.

The important point is not that the exact tally is low. Again, ³⁄₉ is not a transfer rate.

The important point is that transfer fails in several qualitatively different ways:

the answer can flip;
the answer can become condition-dependent;
the question itself can stop making sense.

And all three inversions are clean-data findings. This is not a story about adversarial examples making interpretability brittle.

H2: evidence can outlive the output

After a successful attack flips the model’s verdict to bonafide, a probe trained on clean activations can still identify the underlying spoof evidence:

86.5% at AASIST enc4
100% at SSL’s first transformer layer

Cross-corpus, that retention decays with depth. On AASIST it falls from roughly 82% at the front-end to 9% at the embedding.

The pattern transfers even though the raw numbers differ.

H7: a genuine architectural improvement that disappears out of domain

“Adversarial-real” attacks push genuine bonafide speech into the spoof class.

In-domain, SSL-AASIST is much less fragile than AASIST.

Cross-corpus, both collapse to roughly 100% attack success.

So “SSL fixes bonafide fragility” is true only in-domain. The model improvement is real, but conditional.

H3: branch attribution inverts

Cross-class activation patching asks which site carries class evidence.

On clean in-domain audio:

	`gat_S`	`gat_T`
AASIST	0.312	0.306
SSL-AASIST	0.008	0.250

AASIST has essentially no spectral/temporal preference. SSL-AASIST is strongly temporal.

These are control-adjusted effects. That control matters: on SSL gat_T, the raw patching effect is 0.895, but a matched random tensor already causes 0.645. Most of the raw effect is magnitude, not class structure.

Applying the matched-random-tensor control reduces the raw branch contrast from 128× to 31×.

The qualitative inversion survived.

I also re-paired every AASIST bonafide clip with a different spoof donor. The control-adjusted branch comparison barely moved, which supports interpreting this as class-level rather than donor-specific evidence.

H8: the cleanest inversion is in the final readout

This is the case I trust most because it does not depend on a probe or approximate causal estimator.

Both models feed a 160-dimensional embedding into the same-shaped linear readout. That embedding is five 32-dimensional blocks:

temporal max
temporal average
spectral max
spectral average
master nodes

Because the readout is linear, the contribution of each block to the bonafide-vs-spoof logit gap decomposes exactly.

AASIST:

92% of evidence comes from averages + master nodes
master nodes alone contribute about 29%
maxima contribute only 8%

SSL-AASIST:

62% comes from maxima
the master-node share falls to about 6.8%

The absolute contributions move too, so this is not just a percentage reshuffle:

block	AASIST	SSL-AASIST
total class gap	20.46	18.60
master contribution	5.92	1.27
`T_max` contribution	0.36	6.42

Within-model seed variation is tiny compared with the between-model shift.

Had I stopped at AASIST, I would have published something like:

“The master node carries 29% of the decision using 0.04% of the model’s parameters.”

That sounds like an architectural insight.

On SSL-AASIST the same block contributes only 6.8%.

It was a fact about one trained model.

H9: the same symptom can be a different disease

Both models reject large amounts of bonafide speech cross-corpus.

But AASIST’s representation degrades while SSL-AASIST’s ranking remains almost intact and its threshold is misplaced.

Same failure from outside. Opposite repair.

This is exactly the kind of conclusion that a single benchmark metric hides.

H4 and H5: sometimes the question disappears

These are stronger than ordinary non-transfer.

H4 — can an attacker route around the causal site?

On AASIST, the monitor tap and causal tap are distinct, so an attacker can try to fool one while preserving the other.

On SSL-AASIST, the best detector and causal tap coincide at pool_T. There is nothing to route around. The question dissolves.

H5 — which frequencies carry spoof evidence?

AASIST has a real node→Hz mapping. The hypothesis was refuted: the best single frequency node recovers only about +0.038 against a full-tensor effect around 0.60–0.72.

SSL-AASIST has no frequency axis at all.

So there is no cross-model transfer verdict to assign.

7. A monitor worth keeping even when it catches nothing

One of the more surprising results came from attacking the monitor itself.

I jointly optimized the attack against the classifier and the monitor.

At the adaptive endpoint in the project summary:

monitor detection collapses to AUROC 0.088, below chance;
attack success falls from 92.2% → 13.0%.

So the monitor loses badly if you judge it by detection AUROC.

But forcing the attacker to hide from it costs 86% of the attack’s success rate.

That suggests a different success criterion:

A monitor need not catch the attacker to be useful. It can be useful if evading it is expensive.

For adversarial oversight, I now care about both:

does the monitor still detect?
what does the attacker have to sacrifice to evade it?

On SSL-AASIST, this exact routing opportunity is not available because the best detector and causal tap coincide. Again, the principle may generalize while the architectural opportunity does not.

8. Validation checks that matter

Several conclusions in this project depend less on adding a new interpretability method than on asking whether the measurement itself is well-posed.

Four checks were especially important:

Do not evaluate a metric on a subset selected using that same metric. Confidence looks misleadingly bad if it is scored only on successful attacks, because attack success already depends on crossing the model’s decision boundary.
Read every intervention against its matched control, never against zero. The random-tensor control reduces SSL-AASIST’s raw branch contrast from 128× to 31×, and the random-direction control invalidates the SSL embedding direction-patching cell because a random vector flips the model too.
Profile the full relevant axis before generalizing from a convenient tap or branch. AASIST retains different amounts of class information at different depths, and SSL-AASIST carries its strongest causal direction in gat_T, not gat_S.
Recompute summaries directly from the underlying table. The headline numbers in this post are tied to explicit model, corpus, condition, control, and denominator so that a compact summary does not silently change what was measured.

These checks are mundane compared with activation patching or adversarial optimization, but they determine whether the more sophisticated measurements mean what they appear to mean.

9. What this says for practice

If you deploy a detector

Do not gate review purely on softmax confidence. It can be excellent in-domain and fail under shift by becoming confidently wrong.

Do not assume one monitor number tells you whether the monitor works. On AASIST, embedding scores 0.987 AUROC against pristine negatives, 0.851 against perturbed benign negatives, and 73% FAR if I reuse a threshold fitted on clean audio. At a fixed 5% FAR, it catches about 61% of attacks. Those are three different claims: separability, calibration, and operating point. Measure the one your deployment actually cares about.

Calibrate against audio processed the way your inputs will be processed. My MP3/resampling controls are already harsher than pristine audio, but they are still software transforms of studio recordings. A real phone call is a harder negative distribution than anything in this project.

Judge monitors by evasion cost as well as detection. My adaptive monitor’s AUROC collapsed below chance, but hiding from it destroyed most of the attack’s success.

Before retraining, test the cheap repairs. A threshold, stale readout weights, and representation loss all look like “high error” from outside.

If you audit a model

Choose instrumentation points by causal relevance, not probe score.

Give causal claims matched random controls. Mine changed conclusions twice.

Run both sufficiency and necessity when possible. SSL gat_T gave different answers for the probe direction depending on the arm.

Do not read probe weights as causal directions. A probe can be nearly perfect and still point somewhere the model does not use.

Then run the battery again on another model.

If you report results

Put attack success next to the perturbation control.

Do not fit a threshold on the rows you then score it on. Fit the threshold on calibration data and report its performance on held-out test data; otherwise the estimated repair can be optimistic by construction.

State the model and condition on every number. Four conditions × two models is enough to accidentally quote across cells.

Keep an append-only experiment log. It makes it possible to trace each headline number back to the experiment, control, and condition that produced it.

10. What these results do not support

Several tempting claims would overstate the evidence.

This is not a real-world attacker model

Every attack here is white-box: full access to weights and gradients.

That is an upper bound on attacker capability, not a deployment prediction. Black-box transfer is the obvious missing experiment.

“Adversarial fragility exists” is not the contribution

Audio anti-spoofing models being attackable is already known.

The contributions here are the energy-matched controls, the two-model transfer audit, the probe-vs-causal comparison, and the representation-vs-readout diagnosis.

SSL-AASIST’s cross-corpus advantage is confounded

XLS-R was pretrained on 436k hours including MLS, and ASVspoof5 bonafide speech is drawn from MLS English.

So SSL-AASIST’s cross-corpus advantage may partly reflect pretraining overlap rather than clean generalization.

A better test is In-the-Wild, which uses found celebrity audio and avoids that overlap.

SSL-AASIST is not necessarily more robust

SSL-AASIST resists FGSM much better than AASIST while iterative PGD still reaches roughly 82–100% success.

That pattern is consistent with gradient masking, not genuine robustness.

No monitor number here has met a real recording

Every clip in this project is studio-quality ASVspoof audio. Even the more realistic benign controls are software MP3 compression and resampling applied to studio recordings.

There is no telephony codec, microphone response, room acoustics, environmental noise, or packet loss.

So the monitor operating point — roughly 61% detection at 5% FAR for the best AASIST tap — should be treated as an upper bound on what I have established. Real deployment negatives are likely harder; this project does not measure by how much.

This limitation matters most for the monitoring results, somewhat for end-to-end detection performance, and less directly for the clean mechanistic findings in the probe and transfer sections.

Two models are not a population

“3 of 9 transferred” shows that transfer can fail under an unusually favorable comparison.

It does not estimate how often interpretability findings transfer in general.

Some SSL causal results are single-seed

AASIST’s core results use three seeds. Several SSL causal analyses are still single-seed.

The effects are large, but effect size is not a substitute for actually running the seeds.

11. What I would do next

In rough order of how much each could change the conclusions:

Run In-the-Wild. This is the cleanest way to remove the MLS overlap in the cross-corpus comparison, and it is the experiment most likely to move the current conclusion against me.
Add a third model. AASIST3 is a natural next arm. Viakhirev et al. (2026) already report correlational TreeSHAP branch attributions, which makes it a useful target for causal validation.
Test black-box transfer attacks. The depth gradient — roughly 82% evidence retention near the front-end down to 9% at the embedding — gives a concrete prediction about where transfer should survive.
Run seed parity on SSL causal experiments.
Use path patching or cleaner ablations. Full-tensor patching can cascade through the serial back-end and saturate later layers.
Develop a patching method for variable-layout activations.pool_T is SSL-AASIST’s best monitoring tap and also one of the most interesting causal sites, but its top-k node selection changes per clip, which blocks the current direction test.
Use benign controls that resemble a deployment. Add telephony codecs such as AMR/Opus/G.711, device impulse responses, room acoustics, background noise, and packet loss. This is cheap compared with most of the list and is the experiment most likely to move the monitor operating points downward.
Try psychoacoustic masking attacks. “40 dB” is an energy statement, not a perceptual guarantee.
Replicate the probe-direction result in language models. The test is cheap on cached activations, and I would like to know whether the inert case is rarer or more common than my 2-in-6.

12. What I think generalizes past deepfakes

The specific numbers belong to two audio models.

The methodological lessons feel broader:

Run your interpretability battery twice. Single-model findings can look architectural and turn out to be parochial.
Predictive is not causal. A near-perfect probe can still identify the wrong intervention direction.
Match the instrument to the question. “Decodable here,” “this site matters,” and “this axis matters” are different claims.
Probe the layer your repair or intervention can actually access.
Use matched random controls for causal interventions.
Check class balance before trusting an error-prediction metric.
Separate monitor separability from calibration. AUROC against pristine audio, AUROC against processed benign audio, and FAR at one threshold are not interchangeable.
Judge monitors partly by what evasion costs.
Treat summary sentences as claims that need their own verification.
Keep the log append-only.

The result that changed my mind most was not that an attack worked or that one monitor metric looked bad.

It was that I could produce what looked like a crisp mechanistic explanation on one model, repeat the analysis on a closely related model, and watch the explanation invert.

That is the result I would most like other interpretability work to stress-test.

Reproducibility, release, and repository notes

The experiments are complete for the two-model, two-corpus matrix used in this project.

The repository includes:

the research write-up;
project plan and coverage matrix;
an append-only experiment log;
related-work notes;
versioned result artifacts;
scripts for monitors, probes, readout/representation diagnosis, patching, and figures.

The AASIST arm runs in the main Python environment. SSL-AASIST depends on an older fairseq stack, so its feature extraction runs under Python 3.7 / Torch 1.8 and dumps a shared artifact schema for later analysis under the main environment.

On an RTX 3090, the measured forward cost is roughly:

AASIST: 5.0 ms
SSL-AASIST: 8.1 ms

Despite having about 1,067× more parameters, SSL-AASIST is only about 1.6× slower in the forward pass in this setup because XLS-R compresses the waveform to a much shorter frame sequence before the dense operations.

Dual-use and release

The attacks are standard white-box PGD against public academic checkpoints.

The repository ships code and aggregate results, but does not distribute the generated adversarial audio. Roughly 32 GB of attack waveforms are deliberately gitignored.

The actionable outputs are mostly defender-side: where monitoring fails, how much adaptive evasion costs, which internal diagnoses distinguish cheap fixes from retraining, and which interpretability claims fail to transfer.

Attribution

AASIST — Jung et al., ICASSP 2022: paper
SSL-AASIST — Tak et al., Odyssey 2022: paper
Marks & Tegmark (2023) — difference-in-means vs probe directions: paper

Full code, artifacts, and append-only experiment log:
github.com/gavin-keli/deepfake-detector-audit

Thanks to BlueDot Impact and my sprint cohort.

Frontpage posts - Effective Altruism forum viewer

Who we’d like as regrantors by Carol N

Re­grantors list

Pre­vi­ous Man­i­fund regrantors

Others with good track records

Lab employees

Investors

Founders & suc­cess­ful grantees

Policy and advocacy

Peo­ple with in­ter­est­ing networks

Take-havers

Is it dis­qual­ify­ing if a re­grantor …

… is busy?

… has other funds to make grants with?

… is con­trar­ian or con­tro­ver­sial?

Applying an Investment Portfolio Framing to Individual Giving: Feedback on a New Donation Platform Concept by Todd Miller

Beyond the Expanding Circle: Towards a Pluralist Account of Moral Progress by Vasco Grilo🔸

Summary

In­ter­nal Criticism

Ex­ter­nal Criticism

Hurtling through 2026 by Ajeya

Game play

Logistics

Video

Game design

Math

What does this mean?

V&V takes on “Pacing the frontier” by Yoav Hollander

Some back­ground (and ini­tial re­ac­tions)

The rewind-fix-check loop

Ideas for im­prov­ing (and ver­ify­ing) alignment

Seeing Clearly: Red-teaming GiveWell’s Glasses Research by Zach Gilfix

GiveWell’s Evaluations

Questions

New Research

Non-RCT Research

Conclusion

Knowing when to invest in people, and how much by gergo

TLDR.:

Ex­plain­ing value al­ign­ment, con­text and skills

Group­ing interventions

Over­in­vest­ing:

The Cal­ibra­tion Line

Un­der­in­vest­ing:

EA’s suc­cess story

Clos­ing thoughts

Five Wild Animal Welfare Organizations estimate $3-8M in room for additional funding in 2027 by Rethink Priorities

Summary

Why Es­ti­mate the RFMF for WAW Or­ga­ni­za­tions?

The Result

Scien­tific Field-Build­ing Re­mains Im­por­tant, But WAW Must Diversify

Does WAW De­serve a Fund of its Own?

Collider NYC is hiring a Program Manager by Tatiana K. Nesic Skuratova

About the role

What you’ll do

What we’re look­ing for

Required

Nice to have

Logistics

How to apply

What Mormons get right about community building by Jacob Brinton

Build­ing com­mu­nity is a first-or­der priority

Geography

Ministering

Trek

Callings

Fast offerings

Be­ing in­groupy al­lows you to move faster

Implications

Don’t forget why learning is important by Roman Ross

“I dunno; it seems good to know things?”

Knowl­edge is only as valuable as what it al­lows you to do.

Other things to keep in mind

Time stamps:

What If We Enforced AI Model Safety At the Level Of GPUs? by Mayowa Osibodu

Ecological Collapse is not an Existential Risk by Tandena Wagner

We Likely Only Need 100,000 Species to Survive

Re­source De­ple­tion is Thou­sands of Years Away

Col­lapse Doesn’t Seem to Cascade

How bad does ecolog­i­cal dam­age get?

Regrantors list

Previous Manifund regrantors

Founders & successful grantees

People with interesting networks

Is it disqualifying if a regrantor …

… is contrarian or controversial?

Internal Criticism

External Criticism

Some background (and initial reactions)

Ideas for improving (and verifying) alignment

Explaining value alignment, context and skills

Grouping interventions

Overinvesting:

The Calibration Line

Underinvesting:

EA’s success story

Closing thoughts

Why Estimate the RFMF for WAW Organizations?

Scientific Field-Building Remains Important, But WAW Must Diversify

Does WAW Deserve a Fund of its Own?

What we’re looking for

Building community is a first-order priority

Being ingroupy allows you to move faster

Knowledge is only as valuable as what it allows you to do.

Resource Depletion is Thousands of Years Away

Collapse Doesn’t Seem to Cascade

How bad does ecological damage get?

There are Other Reasons to Care about Biodiversity

The “sixth mass extinction” does not Threaten Human Civilization

Extinction Projections are Systematically Too High

Nor is resource exhaustion going to doom humanity

Nor is cascading ecosystem collapse going to doom humanity

Executive Summary

Projects that should get done

SPAR Generalist Projects

Non-project shaped projects

Other Project Lists

Getting Support

Funding

Advice on getting mentorship

A final note

A simple model of takeoff in frontier economies

A rising tide lifts some boats faster than others

Why care about catch-up growth if TAI will create great abundance and raise everyone’s welfare?

What happens after the detachment window?

AI-skeptic: The AI-induced takeoff never comes.

AI-skeptic: The AI-induced takeoff you assume is too high.

AI-skeptic: The AI-induced takeoff comes, but it stops at relatively low levels of total automation f because AI only automates cognitive tasks.

AI-skeptic-believer: D=10 is too low relative to historical evidence, and maybe TAI itself causes the lag to grow.

AI-believer: TAI causes the lag D to decrease.

AI-believer: Trade boosts catch-up.

AI-believer: Regulatory barriers in frontier economies reduce the gap in adoption.

If We Don’t Do It, Someone Worse Will

Responsible Farming Policies

Funding the Mission

Why not just study animals without farming them?

Community Impacts

Addressing Objections

“This is just factory farming with a mission statement.”

“Your incentives will shift once you’re running the operation.”

“External advocacy and regulation would be more effective.”

Aren’t you providing the industry with moral cover?

What about cognitive biases and groupthink?

You’re just enriching yourself with status, influence, and resources while socializing risks

You’re just moving fast and breaking beaks

Aren’t you worried the chickens might escape and destroy you and everything you love?

What about a coordinated pause?

What about dialogue with China?

What’s it like to have Bernie Sanders, Steve Bannon, writers, artists, musicians, teachers, and the general public all broadly opposed to what you’re doing?

What would it take to convince you this approach is wrong?

Why this project exists