Frontpage posts - Effective Altruism forum viewer

What Mormons get right about community building by Jacob Brinton

Jacob Brinton — Fri, 14 Aug 2026 08:01:46 +0000

Mormons get a lot of things right. Apart from strange Masonic temple rituals, they lead rather normal—and even excellent—lives. Mormons enjoy a longer lifespan,^[1] Utah is the #1 state for volunteering,^[2] and their language training programs are so successful that missionaries are a known source for foreign service and intelligence careers.

Throughout this post, I’ll be making generalizations of Mormons rather than hedging the claims properly. I grew up in Wisconsin, Maryland, and Utah, and many of the claims are more true of the Utah/Idaho/Arizona corridor (affectionately called the “Morridor” by some ex-Mormons) than the rest of the US, and certainly the rest of the world.

Religions share much in common with AI safety and other impact-driven movements, and even more so the Mormon church. There are a few reasons for this:

Commitment to the cause. Anecdotally, nearly all of the ~500 Utah Mormons I’ve interacted with have been true believers, and only a handful just went to church out of habit.^[3]

High stakes. Mormons do believe (I’ve heard they are trying to disavow this, but it was taught) that they will get a planet or some portion of the cosmos as their own if they are good in this life, and become Gods of their own. This is not far off from many transhumanist utopic visions. However, it’s also true that they believe that before God’s Second Coming, the world will go into a state of war and disarray, not unlike the forecasts of extinction-level risks bearing fruit.

Urgency to attract new members. Given the stakes, it’s not a hard followup to believe that you should do what you can to teach more people about the awesome^[4] benefits and risks awaiting them. Talent-constrained movements feel a similar urgency that is a little more impact-focused than community-focused.

Insularness. It’s hard to be friends with someone outside the church, since being a member is a significant social and time commitment, and most people (in Utah) you interact with are members anyway. This contributes to both a sense of community and lack of viewpoint diversity.

Building community is a first-order priority

Mormons’ focus on fostering community enables them to be effective at all the non-religious stuff they do.

Geography

First, some background: wards are composed of perhaps 50 to 100 families. There’s no shopping around for congregations—ward membership is determined by where you live. This facilitates getting to know people. Each Sunday, the whole ward gets together for services, which are sometimes followed up by potlucks. Wednesdays, the kids get together for activities. Saturdays are for a rotating schedule of cleaning the church. Throughout the week, families are often close enough together that they see each other frequently.

Ministering

Within a ward, families are assigned a pair of people that minister to them: visit them once a month to check in, and maybe go over a spiritual lesson. While in the worst case these feel contrived, oftentimes people feel like they can count on their ministers to support them. With a few rotations of these over the years, you begin to develop a network of people that you wouldn’t have naturally become friends with. This parallels the adage of “you get most job opportunities from acquaintances, not close friends”: creating connections from across the network helps everyone to be fewer degrees away from each other.

Trek

Every four summers, the youth of the ward recreate the Mormon pioneers’ crossing of the plains. In true 1800s fashion, they pull handcarts. A particularly exciting phase of this when I went involved the “Womens’ Pull”, where only the girls pulled and pushed the handcarts. (I think this was to simulate the number of men who died during the crossing). Afterwards we learned how to square dance. The only inauthentic aspect was how delicious the food was; we weren’t served gruel. As my grandmother has often stated, “They didn’t have McDonald’s back then”, and this increased our shared gratitude. Most of all, it increased our friendship.

Callings

Adult ward members are assigned callings, which range from choir director to primary instructor (children’s religion classes) to bishop (equivalent of a pastor or imam). Callings aren’t application-based, and a former bishop might be called to run nursery, the Sunday activities for toddlers. This flattens the hierarchical structures that would otherwise arise. Additionally, switching positions over the years helps each ward member feel connected to others, no matter their skills or interests that would normally push them to certain roles in the workforce.

Fast offerings

On the first Sunday of every month, Mormons hold a 24-hour fast from eating and drinking, aside from the sacramental rituals of eating a gob of (normally white) bread and a thimbleful of water. Aside from making the Sunday dinner delicious, Mormons save fast offerings, payments meant to cover the expenses of what they would have paid on food. Young men go door to door, collecting these payments (this is increasingly not the case; when I was doing it at 14, many people paid online. But they did give me amazing frozen Hershey’s almond bars, which they made us promise not to eat until after breaking our fasts). The offerings go to the Bishop’s Storehouse, where the money is converted into food for the hungry. The only other thing I heard about that place was that they had the best chocolate milk.

Being ingroupy allows you to move faster

When there is a shared value framework, people develop trust quicker. Upon moving into a new ward, a Mormon family will immediately have a community of like-minded people who do similar activities to them. Since they don’t have to figure out whether they have to watch their tongues while discussing their morals, they can put more energy into getting to work and learning more about the people around them.

This also enables Mormons to dedicate more time resolving the surface-level issues, rather than revisiting foundational differences when interacting with others.

Do I think it’s worth it to increase the ingroupy-ness of mission-driven movements? Not necessarily. Viewpoint diversity matters a lot more when the answers aren’t legislated. Ingroupy-ness also limits the number of people who can join the movement.

Implications

Here are a few ideas vaguely inspired by Mormonism:

Create a platform for facilitating virtual 1-1s, especially between people from distinct backgrounds or positions
Create a travel grants program, so that people can benefit from geographic proximity
Instead of e.g. donating 10% of your income, precommit to donating a 24-hour period of your time each month to helping other people

Overall, I think we could do well by taking cues from other movements that have succeeded—in one way or another—at building community.

^
Probably due to the frequency they attend religious services; similar effects have been observed for other religious adherents. A 2004 Utah study finds that, compared to non-LDS people, there is a 7.3-year difference in life expectancy for males and a 5.8-year difference for females.
^
https://www.census.gov/library/stories/2023/01/volunteering-and-civic-life-in-america.html
^
This leaves out the people who were PIMO: physically in, mentally out. However, amongst those people, I’ve found that they developed a strong belief that the church was not true, so the same principle holds.
^
In the biblical sense

Don’t forget why learning is important by Roman Ross

Roman Ross — Fri, 14 Aug 2026 07:06:07 +0000

This is a timed post. Every 5 minutes while writing this post, I had to stop to do 13 push-ups, and when I could no longer complete my required number, I had to upload it. My friends thought this would be a fun challenge, but it means that it will likely be less polished than some of my other output.

Summary:

Many people moving into AI safety are told that to succeed, they should read a lot and “try to develop takes.” However, it’s important not to forget why knowledge is important: it helps you do better things in the real world.

Epistemic status: Pretty confident in the main claim (you should focus on learning the most valuable things), but less confident in the specifics of what things are the most valuable. I apologize if my language is too strong at times.

“I dunno; it seems good to know things?”

The world is filled with ambitious young people, some of whom care about AI risks. Throughout my summer, I’ve gotten to talk to many of them. They visit San Francisco and Berkeley to talk to all the cool AI safety workers, or they go on surprise trips to work on high-impact, time-limited opportunities. Throughout this process, many of them look for jobs, and if they can’t find them, they seek out advice on how to get good enough to land one.

A common piece of advice for how to get a job in AI safety is “read a lot and try to develop good takes.” This advice is featured very broadly, even for people who never intend to take a technical research role:

Matt Beard: “You don’t have to be confident, but you should have something to say. It’s fine to keep your views loosely held and defer to experts well, but you should take ownership over forming your own takes. Hiring managers will also want to hear your thoughts and reasoning.”
80,000 Hours:
- “A rough rule of thumb is to aim to read three or so AI papers a week to get a sense of what’s happening in the field, the terminology people use, and to be able to discriminate between real and fake AI news.”
- “(No matter how you plan to do AI safety advocacy), it will be important for you to build up your AI safety knowledge.”
Kairos: “Succeeding in (generalist) roles requires real mission alignment and enough context to spot high-EV opportunities that someone without that background would miss.

As this knowledge propagates, I worry that many people who hear this advice lose track of why it’s important. Many people sharing this advice don’t spend a lot of time explaining why it’s valuable for everyone in AI safety to have their own takes.

It’s obvious why you would want to know things if you’re trying to become a knowledge expert: knowledge experts are people who know things. However, if you’re trying to become a competent chief of staff, event manager, or field builder, or you want to take one of the many other neglected generalist or org scaling roles, it’s not as obvious why you should read everything. It’s good to have an explanation for why what you’re doing is impactful, but when I ask newcomers why they think it’s important to read, I often get answers like, “I dunno, it seems good to know things?”

I worry about this. When people don’t have a good picture of why they’re doing something, they might overestimate something’s impact and miss its end goal. If you want to do good things, this is important to avoid.

Knowledge is only as valuable as what it allows you to do.

Knowledge is valuable because it helps you make better decisions. Orgs want people to be knowledgeable because people who know specific things about AI safety can do impactful things, not because storing more facts about AI is intrinsically valuable. You should focus on gaining the knowledge and tools that help you make better decisions.

To do this, I recommend using a process I call directed reading: identifying the most important questions that will help you make better decisions and doing everything you can to get answers to those. This doesn’t necessarily have to be reading: it can be listening to a podcast, talking to a friend, or chatting with an LLM. But the primary goal is to resolve your most important uncertainties.

Here are some examples of uncertainties you may have:

If you’re still not sure if AI safety is the most important thing to work on, you can ask yourself, “What is the most important problem in the world to work on?” I don’t think people ask themselves this question enough.
If you’re new to AI safety and interested in learning all the important “context” that employers keep asking for, you can ask yourself, “Why is context valuable, and how can I most quickly answer the set of important questions that having context answers?”
If you want to start some impactful projects to build your skills, you can ask yourself, “What projects give me the most valuable skills, and what are the biggest needs in AI safety right now?”
If you want to figure out what the biggest needs in AI safety are, you can ask yourself, “What does winning in AI safety look like, and what do we still need to do to make it to the finish line? In other words, what’s my theory of victory?”
If you hear that we need more generalists in AI safety, and you’re interested in solving this problem, you can ask yourself, “What problems are causing people to believe that we need more generalists, and how can I best address those specific problems?”^[1]
If you plan to pursue any project, you can ask yourself, “Is this project really on the critical path to making AI go well? Do I have a coherent theory of change? Am I addressing a real problem?”
If you don’t want to get forever bogged down by infinite questions, you can ask yourself, “When is it safe to defer?”

To answer many of your questions, you’ll probably need to find answers to smaller and more fundamental questions. On its surface, your process might seem to replicate the original “read a lot of things” advice. However, by focusing on what you’re actually hoping to achieve with your knowledge search, you can develop critical skills much faster than you would by passively consuming the next 80,000 Hours podcast episode.^[2]

You can also double-check your learning processes. If the knowledge you’re getting doesn’t help you make better decisions, you should be willing to quickly pivot to trying other things. Learning things does not intrinsically save lives. Misalignment risk isn’t reduced by the number of books you’ve read from cover to cover. Don’t waste time memorizing social passwords and storing fake knowledge.^[3] Do what will help you do good things in the real world.

Other things to keep in mind

While focused learning is important, I don’t think directed reading is the best way to solve all of your problems.

First, while directed reading can be one of the best ways to resolve your largest uncertainties, you don’t need to spend all of your time skimming posts to find only the answers you need. Reading things you enjoy is pretty awesome, and you shouldn’t force yourself to spend all your time engaging with material that feels like homework. Find ways to integrate your study time into your routine, but don’t forget to do the things that make you happy.

Second, pure information consumption might not be the best way to gain skills and understanding. Writing can be a valuable tool to help you internalize and understand various ideas, as well as keep track of any uncertainties you might still have.^[4] Additionally, if nobody’s written about the thing you want to say, you should consider polishing your notes and publishing them online. If it helps, you can assemble a group of friends and do a timed writing challenge to publish the scraps you have lying around.

Finally, you might be completely wrong about what you think you should be studying, or you might be missing some valuable new leads by continuously taking a focused approach to learning. To address this, it can be valuable to introduce some randomness into your media diet: find some time to read things (especially highly recommended things) that you don’t expect to be helpful. However, I suspect that most people aren’t spending enough time trying to learn things directly.

In the immortal words of the great Thomas Henry Huxley (aka “Darwin’s Bulldog”), “The great end of life is not knowledge, but action.”

Time stamps:

By the end of the first draft, I had done 390 push-ups.
At publishing, I had done 468 push-ups.

^
I think I wasted about a week because I didn’t ask myself this question. These kinds of “minimal trust investigations” are valuable for becoming highly talented in a particular domain.
^
No hate to 80,000 Hours is intended by this; I think these podcasts are great informational resources. However, if you have the opportunity to sit and read, I think you should skim the transcript and find the parts that answer your questions.
^
This might make it easier to get a job, but if you try to study these things specifically, you’re missing the thing that actually helps you become impactful
^
I also like Andy Masley’s blog post about this

What If We Enforced AI Model Safety At the Level Of GPUs? by Mayowa Osibodu

Mayowa Osibodu — Fri, 14 Aug 2026 06:02:03 +0000

Tldr:

AI Agents (e.g. based on models like Claude Opus and Fable) are now powerful enough to be used as autonomous tools for large-scale cyberattacks.

This most powerful class of agents generally tends to be based on closed-weight (closed source) models (like Claude and Fable), which generally have significant safety guardrails and monitoring implemented by their parent companies.

Open weight models are not quite there yet, but they’re not far behind (leaderboard, GLM 5.2). These models however, are distributed without the guardrails and monitoring infrastructure present in their closed-weight counterparts.

Given that this infrastructure tends to be the last resort against successful attempts to manipulate models to do harm, open-weight models lacking this safeguard poses a very pronounced cybersecurity risk (amongst other kinds of risk).

Here I’m proposing an additional approach for mitigating the risk from frontier open-weight models—GPU-level model-safety compliance.

How Do We Enforce Model Safety Today?

We do this across multiple layers of the AI LLM/Agent stack:

Placing Safety Mechanisms Within Model Internals:
These are safety preferences baked into model weights. They include preferences from training data, fine-tuning policies, safety-oriented model architectures, etc.
Evaluating and Screening Model Inputs and Outputs:
This is usually only possible if you control the server where the model is deployed. If not, you never really get the chance to assess these inputs before they’re processed by the model, or screen outputs before they’re passed to the user. Examples include Anthropic’s Constitutional Classifiers, toxicity filters, etc.
Monitoring Model/Agent Activity Across Multiple Conversation Turns:
Again, only possible if you control the server where the model is deployed, or the agent harness within which the model operates (this was how Anthropic was able to detect the first reported AI-orchestrated cyber-espionage campaign). Generally does not apply to open-weight models.
Government Reviews:
This is a recent development where a national government like the White House in the US decides to evaluate new frontier models, before approving their public release. This most notably happened with Anthropic’s Fable and Mythos models which were redeployed after White House evaluation, following the initial suspension of public access. So far, this approach has been applied to closed-weight models being developed by frontier AI companies in the USA.
Majority of the most capable open-weight models today however, are being built in China, and aren’t subject to such a process.

Enforcing model safety at the GPU level, seems to me like a valuable addition to what we do today.

As with any sort of model safety enforcement approach, it’s likely not going to be a catch-all that solves every possible concern in AI Safety—I’m just proposing it as yet another thing that could be done, to hedge against the risks of a massively powerful open-weight AI model being wielded by bad actors to effect great harm.

(I wrote an earlier post on another possible way to make open-weight models safer here)

The Case For GPU-Level Safety Enforcement:

One of the core determinants of AI progress today, is availability of compute. The ecosystem has evolved to depend on GPUs (specifically NVIDIA GPUs) as its core computing infrastructure.

These GPUs are a scarce resource, and will continue to be pivotal to the operation of frontier AI models. Their importance is only set to grow as these models get bigger, more powerful and more compute-hungry.

Given all of this, it makes a lot of sense to me that this scarce, increasingly-valuable resource should be leveraged as a bottleneck where model safety is enforced. As far as we know, the eventual open-weight AI model that reaches AGI is still going to run on GPUs. Preparing a GPU-level safety implementation ahead of time, seems like a wise bet.

What Would This Look Like?

Essentially a GPU would refuse to load a model’s tensors unless that model could prove that it met the requirements of some given AI Safety compliance standard.

An analogy which seems relevant here, is HTTPS certificates. Certificate Authorities (like the Electronic Frontier Foundation’s Let’s Encrypt) serve as a compliance body of sorts, providing certificates which attest that a web resource accessible via http requests to a public IP, is handling user data safely by encrypting it appropriately.

Today, any public-facing website which wants to be taken seriously, obtains certificates from these authorities to put members of the public at ease when visiting their site.

I’m proposing doing the same with GPUs and AI models. In the HTTPS analogy, a web developer wants access to the valuable resource which is public attention of the internet. They get a https certificate to show that they intend to handle that attention in a trustworthy way.

Here, a model wants access to the valuable resource with is GPU compute. Similarly, it requires a certificate to show that it intends to handle access to that compute, in a trustworthy manner.

Questions I’m Thinking About (And Would Appreciate Thoughts On):

Who would be this compliance authority/authorities, preparing and providing safety certificates?
How would the safety status of a given model, be determined in the first place? This is actually still a core open problem in AI Safety—How do we conclusively determine a model to be safe? How do we judge the safety level of a model? We’ve figured out how to reliably encrypt data and so we can award certificates attesting to that, but we haven’t quite figured that out how to reliably determine that an AI model is safe.
A proxy for this however, could be model integrity. If we’re to assume AI labs generally release models with at least a decent level of inbuilt guardrailing and safety enforcement mechanisms, then compliance here could just be ensuring that only these original models are given access to compute—denying access to variants which have been fine-tuned for malevolent purposes. This leads to the next question:
Wouldn’t this stymie research and innocent tinkering? If only original published models are allowed to run on a given GPU, how are researchers and hobbyists expected to run custom harmless fine-tuned models?
A concession could be: Only imposing this requirement on models judged to be dangerously powerful—for example models meeting or exceeding Anthropic’s ASL-3 capability level. The more powerful a model is, the more havoc can be wreaked with a variant fine-tuned to undo its inbuilt safety guardrails.
How would this work technically? Where exactly in the GPU would this safety compliance check be implemented?
You’d want it to be deep enough in the GPU stack, so it couldn’t be easily circumvented. Ideally this compliance check would be bundled into the closed-source driver powering NVIDIA GPUs—CUDA. It could be as simple as adding this compliance check feature in the next CUDA version.
Newer GPUs tend to only be compatible with recent CUDA versions, so you can rest assured that when the massive “world ending” open-weight AI model does get released, it’s only going to run efficiently on NVIDIA GPUs compatible with just the recent CUDA versions (which would have this safety compliance feature).
(You can run new massive models on clusters of older GPUs, but that generally tends to be less efficient and more expensive)
How would you get NVIDIA to implement this? Massively growing demand is the wave which has powered NVIDIA’s rise to singular prominence in the AI boom. Why would they implement a feature that constrains the sort of models which can run on their GPUs? This is another crucial question that gets at the heart of safety concerns vs economic incentives.

Would love thoughts, thanks.

Ecological Collapse is not an Existential Risk by Tandena Wagner

Tandena Wagner — Thu, 13 Aug 2026 22:40:14 +0000

A lot of environmental groups say that we are destroying the earth, and our short-sightedness will bring the world crashing down around us. At EcoResilience Initiative, we’ve looked into these claims, and while some of them threaten the richness of the Earth’s ecosystems, none of them pose a serious risk to the continuation of human civilization.

The three extinction risks from ecological breakdown are: failure of critical life systems, cascading collapse, or depleting a critical resource needed for our survival.^[1] While it is technically possible for ecological collapse to cause human extinction, it seems essentially to be a non-issue. Yes, if 100% of life is wiped out, we cannot survive. But, we likely don’t need 50% and perhaps as much as 90% of Earth’s species to survive. Each of these require beyond-extreme circumstances to cause human extinction. (Relatedly: the survival of human civilization is not at risk from climate change.)

We Likely Only Need 100,000 Species to Survive

Critical ecological systems require only a handful of species to operate. Around 1-20 species for each of the fundamental ecosystem services. Meaning ~5,000-100,000 (0.01%-1% of total extant species)^[2] should be more than enough for bare human survival.

To get a rough global estimate of required biodiversity for ecosystem services, I multiply each species required count from this table by the number of global biomes (85-500) and get ~5,000-100,000 (0.01%-1% of total extant species). Total biomes are somewhere between 85 (at the Formation level, which would be minimizing global species) and like 500 (at the Division level which would be using locally thriving species). Totaling that up comes out to somewhere around 7,480 (0.086% out of all extant species) and 81,500 species (0.94%) to support global ecosystem services.
Flaws with this estimate: There are unlisted ecosystem services. I’m overcounting since biomes like “ephemeral salt lakes,” “ground water” and “mud plains” would never provide all ecosystem services. Some species could fulfill multiple ecosystem services. Some species could fulfill multiple biomes. I’m undercounting because supporting ecosystem services would be needed.
I think this estimate is really low for global ecosystem service production, but more than required for a handful of humans to survive.

The other 99% of species could go extinct and critical systems for our survival would still function in places around the globe. That’s thousands of years away at current projected extinction rates.

The real concern would be if some ecosystem services are only supported by a few non-redundant species, and those specific species went extinct. A good example of this is nitrogen production: only ~20,000 plant species produce biologically active nitrogen. But plants are just the hosts for nitrogen-fixing bacteria. Many nitrogen-fixing bacteria are free living and these bacteria are abundant and widespread.^[3] Besides, we mechanically produce our nitrogen fertilizer via the Haber–Bosch process anyway.

Resource Depletion is Thousands of Years Away

Resource depletion is similarly thousands of years away and unlikely to cause human extinction.

To examine an oft-raised example: topsoil loss is claimed as declining rapidly and irreplaceable on human timescales. But we don’t need to return to natural levels of soil organic carbon. Topsoil is not necessary for modern agriculture. We can simply use fertilizer, irrigation, and calcium carbonate (to counteract acidity) on depleted soil indefinitely. We also have methods to farm sustainably. Cover crops gradually repair the soil during the off-season and are low cost. No till, manure, biochar are other options. Or we can just leave crop residues in place. As it is, half of earth’s soil has a lifespan of 1,000 years or greater.

Another cited extinction risk is that a single species of marine phytoplankton produces ~50% of oxygen. But the atmospheric oxygen reservoir is so enormous that we would be buffered for hundreds of thousands of years if oxygen production stopped completely.

Environmental activists make it sound like we are running out of freshwater. But only 2% of watersheds use 75% of their freshwater. Data centers are not draining our water supply. To be clear, we are choosing to pump truly preposterous amounts of water out of aquifers for agriculture, and this is unsustainable and bad for the environment. But we could be far less wasteful, recycling and desalinating way more. Surface water is returned annually so we aren’t going to “run out.”

Likewise phosphorus and nitrogen are not likely to run out. Phosphorous has 1,000 years of supply, and current processes lose 30% of the material during processing and extraction. It is not being recycled. Nitrogen is extracted from the air using natural gas, but can also be extracted using solar power with efficiency losses of 1%.

Oil, metals, chromium, and minerals are not ecological resources per se, but they have the same basic format − 1) there are vast deposits, 2) efficiency and alternatives are possible, just currently unnecessary.

In general, we would not hit a wall and face a catastrophe. There are moderately more expensive alternatives and we would switch to those once we started to experience scarcity.

Collapse Doesn’t Seem to Cascade

Cascading collapse doesn’t seem to occur in the unpredictable catastrophic ways depicted in the threat profile. This has been tested by multiple events of complete ecological destruction—cataclysmic volcanic eruptions, Agent Orange, mountain top removal, and other real-world instances of total ecological collapse. We didn’t see cascading collapses or unexpected bleed-over. We do see unexpectedly speedy and robust recoveries.

However, as ecosystems weaken, they will probably become more prone to these feared collapses. So the question is: how far can ecosystems collapse before they become unstable? The Netherlands has only 28% Biodiversity Intactness Index or 15% remaining biodiversity according to the Atlas Leefomgeving, yet it seems to not be notably at risk of ecological collapse. In this scenario of simultaneous correlated degradation, cascading collapse would need to cause the unexpected eradication of one of the non-redundant 5,000-100,000^[4] species necessary for human survival. But it seems likely that we would be actively tracking and managing the few species keeping us alive at such levels of depleted biodiversity. I think it would be more accurate to imagine this resembling a farming disaster, more than the usual depiction of an unforeseen ecological catastrophe.

More importantly, what would take us to that point of 99% global species extinction? Greed? War? Bioweapon? Climate apocalypse? Basically it would be other problems that would cause us to reach such a far-gone state, and ecological collapse could only really be the straw that broke the camel’s back. It does not pose a current threat, nor a plausible future threat without bigger and worse catastrophes.

How bad does ecological damage get?

Ecological collapse won’t lead to human extinction. As an existential risk factor, it is small relative to other extinction risk factors. There are a lot of giant ecological catastrophes, but they are localized and not truly global-scale events. They don’t translate into a threat to civilization much the way earthquakes aren’t a threat to civilization or an existential risk factor. Ecological damage worsens humanity’s situation somewhat, but not so much worse that it demands specific attention as a contributing factor in existential risk. But what about the worst-case scenario for ecological disasters? How bad would it be? What if they all happened simultaneously?

This deserves its own article but I’ll enumerate some potential ecological disasters worth considering:

Landslides, flooding, storm surges worsened by deforestation. Soil degradation, desertification, salt water intrusion, other causes of permanently degraded productivity (including marine!). Herbivore overpopulation, pollinator loss. Invasive species, crop pests, toxic zooplankton, venomous species outbreaks, fungi outbreaks, vector/host outbreaks. Degraded sanitation, scavenger collapse, biomagnification (including toxins like ciguatera). Phytoplankton regime shift.

To get a sense of their size: Storm surges can kill in the 100,000’s, flooding can kill 1-2 million, scavenger collapse killed 100,000s. Degraded sanitation kills a few million annually, diseases kill millions annually, snake bites are about 100,000 annually. We would need to check how many were excess deaths caused by ecological amplification/failure, compare the damage from things like permanent lost productivity, and estimate how large these could get in worst-case scenarios. For now, it looks like disease burdens are probably going to be the worst ecological catastrophes.

Conclusion

There are three main proposed avenues of ecological-caused human extinction risk: 1) Resource depletion, failure of critical life systems, and cascading collapse. None of these pose a serious existential risk. For resource depletion, there are thousands of years of resources left with multiple alternatives we could switch to. For critical life systems failure, there are so many redundant species and so few species are needed to perform vital ecosystem services that we are thousands of years away from losing them all to extinction. For cascading collapse, there isn’t evidence of compounding synergistic effects from totally collapsed landscapes nor in regions of severely depleted biodiversity. Losing even a large number of species simply isn’t a realistic, pressing threat to human civilization—so people should stop acting like it is!

There are Other Reasons to Care about Biodiversity

This post probably makes it seem like we are anti-conservation and think biodiversity isn’t worth protecting. That’s not true! The whole project of EcoResilience Initiative is to find the best ways to protect biodiversity, because we think nature is incredibly valuable and underrated. Nature makes our lives richer and fuller, and the earth’s billion-years-long legacy of biodiversity is, considered as a whole, one of the most beautiful things in the universe. It would be a shame to let all that beauty and complexity be eroded away by species extinctions during this tumultuous era of industrial civilization—instead, we should find good ways of preserving nature so we can experience a more flourishing future.

This post felt like a tedious but necessary prerequisite for what I think is actually more exciting—detailing what I consider the *real* top reasons to protect our planet’s biodiversity. If you’re looking forward to that post too, then consider [subscribing] for our upcoming post in the sequence below!

* * *

This is part of a sequence on where the value of biodiversity comes from:
People really care about something they call nature
Nature’s value mostly doesn’t come from present-day benefits to humanity

And ecological collapse is not an existential risk

The real value actually comes from long-term flourishing.
Conservation in the next century is going to look a lot different than environmentalists think

Appendix

The “sixth mass extinction” does not Threaten Human Civilization

We do not need a lot of species to survive.
- About 25 macro species as a minimum from the MELiSSA closed loop long term life support pilot project.
- About 10% of current biodiversity according to What We Owe the Future/Founder’s Pledge/John Halstead’s arbitrary back-of-the-napkin estimate in a footnote.
- About 0.01%-1% or 5,000-100,000 species according to my estimate.^[5]
- We also tend to respond when things that directly affect us are in danger. Livestock, pollination, water quality, air quality, fire, are all being managed now.
- That’s a mass extinction event which would take 7000 years to come to pass at current extinction rates and assuming we don’t do anything.
Countries with depleted nature are thriving.
- The Netherlands has only 28% Biodiversity Intactness Index or 15% remaining biodiversity according to the Atlas Leefomgeving, yet it seems to not be notably at risk of ecological collapse?? Incidentally it has the fifth-highest GDP per capita in Europe.

Extinction Projections are Systematically Too High

Extinction debt from habitat loss and climate change will be less than anticipated. Around 100,000 to 200,000 in my estimation (not 1,000,000 as in the typical estimations or this one at 400,000 just from climate.)
- I think these are not accounting for niche shift, freak dispersal chance, and long-persisting-populations of species being rescued with tech development
- Climate change will likely not exceed 3-4 degrees. Stronger climate change would be too painful, and geoengineering too cheap and easy not to pursue it. Climate change is slow and only affects 20% of species.
  - A report for What We Owe the Future/Founders Pledge contradicts this and places climate change at 0.6 − 3x the impact of land use. Land use change affects 80% of species.
- Probably very few tree and plant species will go extinct. The paleontological record shows that they are resistant to extinction. Most likely due to their seed bank.
- Many small species will disperse in freak events that give them good habitat, despite projections of being unable to migrate fast enough to keep up with climate change.
  - Island populations of terrestrial species are a good demonstration of how this happens across miles of un-traversible terrain, with enough frequency/size to start breeding populations.
  - I’m uncharitably assuming these sort of freak events are not built into models of climate migration, but I have not checked and should actually check.
- Symbiotic species are not as “obligate” as they appear and most will survive much longer than predicted.
  - Example: The avocado tree is incredibly poorly adapted to propagating itself. Its massive seed is meant to be eaten by an herbivore the size of a ground sloth, which went extinct 10,000 years ago. But avocados managed to survive until the present day, despite such a clear obligate relationship being lost with no replacement. They ended up in small populations at the bottom of valleys due to their limited propagation.
- Species in extinction debt are populations that are too small/under too much pressure to survive longterm. They are technically extant today, but they are on track to dwindle and go extinct to stochastic events.
  - Most species in extinction debt will continue to linger in microhabitats for hundreds of years, and will make it “out the other end” to de-extinction technology which looks to be about 65-120 years away (assuming further AI advancement and the time it takes to re-establish populations).
  - To be clear, breeding, habitat restoration, invasive species and predator management, disease management, acclimation to danger/shelter/survival are all obstacles between de-extinction and successful reintroduction. Reintroduction is a slow process even assuming you can de-extinct individual organisms in 40 years as Colossal is claiming.
Habitat loss is currently mostly caused by agriculture. Agriculture is mostly livestock and livestock feed. Total habitat area occupied by agriculture has been declining because of increases in productivity and decreases in pastureland even as cropland continues to expand.
Wind turbines are expanding around the globe, but mortality from wind turbines will drive almost no species to extinction.
- Species that fly into turbines inhabit large ranges, some of which will almost certainly be outside the turbine area. Turbine mortality is a limiting factor, but not a cause of extinction.
Population growth is not a major cause of extinction risk.
- More people are moving in towards population centers faster than they are being born.
- Growing food is getting more efficient and using less land per calorie.
  - In Africa in particular population is expected to grow but many land use efficiencies have not been applied yet.

Nor is resource exhaustion going to doom humanity

Peak oil warnings were inaccurate
- This was mainly due to fluctuations and trajectory changes with demand, innovation, refining, discovery, renewables, and more
Soil loss and soil depletion is far from being a threat
- We are not going to run out of soil carbon, nitrogen, and phosphorous for thousands of years
- It’s not an unsolvable problem. Manure, crop rotation, and other basic practices address it.
- Theoretically we can invent additional ways to repair soil faster. I assume we have not done so because it is not yet a serious problem. Actually it looks like I’m wrong about that and it is fundamentally a slow process that you would realistically do on-site.
- Here is an EA post covering phosphorous depletion risk
- And an 80,000 hours interview that discusses nitrogen depletion
Metal and mineral depletion I assume are a similar non-issue
- There are deposits of minerals on the ocean floor, there are improvements in refining, efficiency, alternative materials etc.
- Here is an EA post discussing how much minerals are needed for the energy transition.
- And another post about resource depletion risk generally.

Nor is cascading ecosystem collapse going to doom humanity

1000 years ago there were massive extinctions of all major predators around the world and this didn’t cause human collapse with prehistoric technology. The ecological devastation was largely undetectable to us.
There are fears of potential compounding synergistic effects/extinction vortex/poly crisis. But we haven’t seen signs of this happening anywhere at any point in history yet.
- It seems like a good thing to watch for, but fortunately not a present threat or at least a very rare one. I think it’s advisable to increase our uncertainty to reflect this possibility when projecting earth systems.
There are many examples of ecosystem recoveries in record time.
- Even when we look at completely devastated areas like massive volcanic eruptions/agent orange/etc they don’t cause a cascading or runaway collapse of adjacent ecosystems and then tend to recover in 40-80 years.
Repeat stress events are reducing the environment’s ability to recover.
- Degraded lands are thriving, just populated with invasive, common, generalist species.
- I need to look into this in more detail. My general feeling is that this is true but not threatening until habitats have lost 80% of their biodiversity.
I have tried to identify the worst cases of ecosystem collapse and cascading impacts, and found very few (6) documented examples. None of these caused adjacent ecosystems to collapse?
- Those six examples were:
  - Starfish die-off converted kelp forest to urchin barrens.
  - Mountain lion absence causes increased herbivores, decreased vegetation, changed stream patterns, and decreased insect biodiversity. (This is a replication of the return of wolves in Yellowstone.)
  - Amazon fragmentation causes species decline over decades.
  - Moa eradication and mammal introductions in New Zealand causing 50% herbivorous bird extinction, widespread plant extinction, and decreased nutrient inflow.
  - Deforestation in the Amazon combined with temperature rise is drying out the forest and the resulting fires are converting it to Cerrado (savanna).
  - Sea level rise in the Everglades is causing mangrove dieback, and subsequently sawgrass dieback and coastal erosion.
    - But this is mostly the physical presence of salty seawater pushing inland, so not a great example of a cascading collapse.
  - I’m interested in reading about more examples in this category of extreme ecosystem regime changes, please send them my way!
- Some of these were drastic regime changes to the ecosystems, but they do not cause adjacent ecosystem collapses nor threaten human survival.
  - Cascading collapse seems to happen more often from climate change—where the fundamental physical parameters have changed underlying ecosystems—and not the feared unpredictable emergent phenomena.
    - Tipping points that have been identified from climate change don’t appear to threaten the future survival of civilization.
    - 80,000 hours and other EA orgs have concluded that the survival of human civilization is not at risk from climate change.

^
I’m not aware of other kinds of ecological risks, but I would be interested to learn about other examples
^
To get a rough global estimate of required biodiversity for ecosystem services, I multiply each species required count from this table by the number of global biomes (85-500) and get ~5,000-100,000 (0.01%-1% of total extant species). Total biomes are somewhere between 85 (at the Formation level, which would be minimizing global species) and like 500 (at the Division level which would be using locally thriving species). Totaling that up comes out to somewhere between 7,480 (0.086% out of all extant species) and 81,500 species (0.94%) to support global ecosystem services. Flaws with this estimate: There are unlisted ecosystem services. I’m overcounting habitats since biomes like “ephemeral salt lakes,” “ground water” and “mud plains” would provide more limited sets of ecosystem services. Some species could fulfill multiple ecosystem services. Some species could fulfill multiple biomes. Supporting ecosystem services would be needed. I think this estimate is really low, but far more than required for a handful of humans to survive.
^
There are multiple steps to nitrification and the most limited one is Ammonia oxidation which is provided by only three guilds: AOA, AOB, comammox. Of those, anammox is only six genera, one phylum.
^
To get a rough global estimate of required biodiversity for ecosystem services, I multiply each species required count from this table by the number of global biomes (85-500) and get ~5,000-100,000 (0.01%-1% of total extant species). Total biomes are somewhere between 85 (at the Formation level, which would be minimizing global species) and like 500 (at the Division level which would be using locally thriving species). Totaling that up comes out to somewhere between 7,480 (0.086% out of all extant species) and 81,500 species (0.94%) to support global ecosystem services. Flaws with this estimate: There are unlisted ecosystem services. I’m overcounting habitats since biomes like “ephemeral salt lakes,” “ground water” and “mud plains” would provide more limited sets of ecosystem services. Some species could fulfill multiple ecosystem services. Some species could fulfill multiple biomes. Supporting ecosystem services would be needed. I think this estimate is really low, but far more than required for a handful of humans to survive.
^
To get a rough global estimate of required biodiversity for ecosystem services, I multiply each species required count from this table by the number of global biomes (85-500) and get ~5,000-100,000 (0.01%-1% of total extant species). Total biomes are somewhere between 85 (at the Formation level, which would be minimizing global species) and like 500 (at the Division level which would be using locally thriving species). Totaling that up comes out to somewhere between 7,480 (0.086% out of all extant species) and 81,500 species (0.94%) to support global ecosystem services. Flaws with this estimate: There are unlisted ecosystem services. I’m overcounting habitats since biomes like “ephemeral salt lakes,” “ground water” and “mud plains” would provide more limited sets of ecosystem services. Some species could fulfill multiple ecosystem services. Some species could fulfill multiple biomes. Supporting ecosystem services would be needed. I think this estimate is really low, but far more than required for a handful of humans to survive.
^
These are some of the claims made by large and reputable environmental organizations: “Biodiversity is earth’s life support system.” “Nature gives us a habitable environment, we rely on clean air and water.” “We depend on nature for survival.” These claims are phrased to imply extensive environmental actions are important for human survival, which is not actually supported.
^
Most water treatment plants are chemical and physical processes. This is because biological processes are slower and less controlled.
^
The planetary boundaries say we have already crossed this boundary, at losses of “10 extinctions / million species years.” The planetary boundaries represent boundaries past where we “do not know,” not when we “will likely cause serious damage to civilization.” I’m not even sure if the rate of extinction boundary is helpful because collapse risk would depend on how many species total are left, not the rate of decline?
^
Natural salt flats, sand dunes, rocky alpine, and taiga forest all have very low species diversity. Other examples are tree plantations and farmland with monoculture crops, and cheatgrass invaded sagebrush habitat (reduced down to 8 species of plant). These are not “uninhabitable” due to low biodiversity although they may be physically inhospitable. They are not collapsing dustbowls with no water and oxygen.

Who cares about the shrimp? by alex wennerberg

alex wennerberg — Thu, 13 Aug 2026 21:17:36 +0000

Hi all, I wrote about Shrimp Welfare Project for Kernel Magazine. I am an outsider to this community, but I read a ton of EA forum posts for this piece. I hope you enjoy it.

Learning from Agricultural Development to Reshape Animal Welfare Work in Low- and Middle-Income Countries by Tom Billington

Tom Billington — Thu, 13 Aug 2026 19:46:33 +0000

This is a cross-post of the executive summary from The Centre for Animal Life and Farming’s first report. See the full report here.

We will also be hosting a webinar on 25th August 2026 to discuss these findings with the animal welfare movement.

Executive Summary

Roughly 80-90% of farmed animals live in Low- and Middle-Income Countries (LMICs),^[1] yet the animal welfare movement has seen fewer milestone victories in these contexts than in High-Income Countries (HICs)^[2]. The standard advocacy models used in HICs (corporate campaigning, consumer pressure, legislative reform, etc.) rely on infrastructure that is often absent in LMICs, such as formalised supply chains, consolidated corporate buyers, and strong legal enforcement.

Fortunately, the animal welfare movement is not alone in these challenges. There is a long history of field-based projects in LMICs designed to shift farming practices to improve farmer productivity, sustainability, and well-being. Collectively, this broad category of work is referred to as agricultural development (“agri-dev”). The strategies and lessons generated through this work may offer relevant and useful insights for the development of animal welfare initiatives in LMICs.

Authors Karthik and Tom visiting the field office of DVARA, an agri-dev organisation in Maharashtra, India

To identify more context-driven strategies LMIC animal welfare actors could use, this report asks the question:

What can the animal welfare movement learn from strategies employed by agri-dev actors to effectively change farmer practices in LMICs?

To explore this question, we conducted a narrative literature review and primary field research in India. We identified six primary high-level lessons that were cross-cutting for all strategies:

Complex “asks” need more training and incentives. This forces a trade-off between smaller welfare gains that are cheaper to implement, or larger welfare gains that are more expensive to implement.
Programmes must be tailored to local context to succeed. Design with context in mind, work with partners who understand the context, and focus on larger farmers who tend to be more similar to each other.
Trust is essential for farmer buy-in. It can be built through local partners and intermediaries.
Financial incentives aren’t enough alone; they often need technical and social support to sustain.
Farmers face many significant barriers (e.g., education, financial, infrastructural) at once. Programmes must account for these barriers to be viable for farmers to implement.
Full compliance is unrealistic, so aim for the level that keeps impact cost-effective.

We also identified 20 priority strategies used by agri-dev actors that we felt were relevant for the animal welfare movement. We grouped these strategies across three dimensions: organisation types that deliver interventions to support farmer adoption of new practices (8 reviewed), how farmers are trained in new practices (5 reviewed), and how they are incentivised to adopt them (7 reviewed).

We evaluated each strategy across five criteria: effectiveness (at achieving their intended outcome), resource intensiveness, tractability, scalability, and their risk of stimulating harmful industrialisation. This analysis was conducted with a specific focus on the value for CALF’s work within India, though we expect that many of the insights will translate into other contexts. Here are our summarised results (scores range from 1 to 7):

Strategy		Strategy Explanation	Effectiveness	Low Resource Intensity	Tractability	Scalability	Low Industrial- isation Risk	CALF’s Opinion
Organisation Types	Intermediaries	Buy and sell farmers’ products, offering fair, stable prices in exchange for meeting welfare standards.	6	3	4	6.5	5	Recommend for trialing
	↳ & Processing	Also process products (e.g., sorting and packaging eggs) to raise prices, funding welfare incentives.	7	2	2.5	5	5	Recommend for trialing
	↳ & Contract Farming	Formal agreements with farmers on what and how to produce, often with inputs and guaranteed prices.	7	1.5	3.5	5	2.5	Recommend for trialing
	Collaborating with Agri-Dev Organisations	Adding welfare into existing agri-development programmes, leveraging their skills and farmer trust.	6	6	5	5	3	Recommend for trialing
	↳ & Self-Help Groups	Reaching village women’s savings groups, who do most day-to-day animal management, via agri-dev partners.	5	5	6	3	4	Needs more investigation
	Farmer Producer Organisations	Formal farmer collectives whose rules could include welfare mandates.	4	5	4	4	6	Needs more investigation
	Certification	Voluntary audited standards, ideally earning farmers a price premium for higher welfare.	4	3	5	4	6	Do not recommend
	Direct-to-Farmer Technology Providers	Digital platforms, chatbots, or SMS engaging farmers directly at scale, no intermediaries.	2.5	4	4	2	6	Do not recommend
Training Mechanisms	Training & Visit (T&V) Systems	Trained extension agents deliver structured welfare training to farmer groups on a fixed visit schedule.	5	6	6	5	5	Recommend for trialing
	Model Farms & Field Schools	Working farms demonstrating welfare-compliant practices for hands-on observation and learning.	6	3	5	3.5	6	Conditional — short-term only
	Mass Communication (ICT)	Messages delivered at scale via channels such as SMS, apps, radio, or social media.	1	7	7	7	7	Conditional — simple behaviour changes only
	Veterinary Services	Vets or community animal health workers advising on welfare.	5	4	5	4	4	Do not recommend
	Information & Call Centres	Physical hubs and remote services providing agricultural advice.	1	6	6	6	6	Do not recommend
Incentive Mechanisms	Collective Bargaining	Farmer groups negotiating better input costs or product prices.	6	4	5	6	3	Recommend for trialing
	Market Access	Linking welfare-compliant farmers to buyers, sometimes with transport or guaranteed purchases.	6	3	4	6	3	Recommend for trialing
	Value Addition	Processing products (e.g., grading, packaging eggs) to raise value, sharing the margin with welfare-compliant farmers.	7	2	2.5	5	5	Recommend for trialing
	Credit Systems	Loans tied to or supporting welfare compliance, financing practices needing upfront investment.	7	4	2.5	6	3	Needs more investigation
	Conditional Payments	Payments or discounted inputs contingent on adopting welfare practices.	5	3	6	3	6	Conditional — short-term only
	Market Information	Price data informing when, where, and what to sell.	1.5	7	7	7	5	Do not recommend
	Insurance Products	Coverage against production and price risks, potentially conditional on welfare improvements.	4	3.5	2	6	2	Do not recommend

Each agri-dev strategy reviewed across our key criteria in a 7-point Likert Scale

A summary of each strategy and their key considerations is shared in section 3 of the full report (you can also see a more in-depth breakdown of each score here.). This is an initial step. The strategies identified here require rigorous field testing, and the evaluations we offer are preliminary. We present them not as definitive answers but as structured starting points.

This report offers three key lessons for animal movement actors operating in LMICs:

Learn from agricultural development. The strategies above (in particular those flagged as “recommended for trialing”) are templates the animal welfare movement should consider using. Agricultural development organisations may themselves make strong partners in implementing these.
When building a theory of change, consider three dimensions: training mechanisms, incentive mechanisms, and organisation types. Make a deliberate decision about each one, even if that decision is to leave it out. Revisit and refine those decisions as the work develops.
Expect complex, interlocking challenges. These include farm and farmer heterogeneity, low education levels among farmers, the importance of trust, compliance issues, and the reality that the largest welfare gains typically demand the most complex behaviour changes. Navigating these will take time, and will only succeed with clear awareness of the barriers, deliberate planning, and openness to learning along the way.

^
See full data set here. Note that the numbers for total farmed fish are from 2017 and based on low-end estimates from fishcount, meaning that they are significantly more speculative. Data Sources:
Cattle, poultry, and pig data from FAO, 2025 – with major processing by Our World in Data
Farmed fish data from fishcount (2017)
^
See the full data set here. Data sources:
Company pledge data from Chicken Watch (2026)
Income classification data from The World Bank.

Concrete Generalist Projects in AI Safety (and how to do them) by Helena Tran

Helena Tran — Thu, 13 Aug 2026 19:12:09 +0000

The AI safety ecosystem is still in need of generalists: people who will go out and solve the tasks that people who feel limited by their job description won’t address. This post will explain how to break into generalist work, then provide an important set of projects that, if completed, could strengthen many crucial parts of the AI safety ecosystem, as well as your own career.

TL;DR

If you’re looking for something concrete to do, there are a bunch of good entry points:

Find a project: browse SPAR’s Generalist Projects, join the Generalist Megastream, look through projects on Mangrove, or browse this fieldbuilding/project spreadsheet and Atlas Computing’s Resilience Gap Map.
Get support: apply to Genstream (deadline August 14), indicate interest in the Generator Residency, or check out MATS’s Founding & Fieldbuilding track and Catalyst.
Get funding: BlueDot Rapid Grants fund projects up to $10k; there are also quick funding opportunities and career-transition grants from Coefficient Giving and BlueDot.
If you’re a student: join or start a university AI safety group; Pathfinder and the AI Safety Seeding Initiative can help with mentorship and funding.

The rest of the post has more detail on choosing projects, getting mentorship, and other ways to get involved.

If you’re a…

University Student: get heavily involved with your university AI safety group. University groups are real, impactful organizations, and they can be run as such.

If your school doesn’t have a group, you should start one. Convincing an equally talented person to join AI safety can straightforwardly double your impact, and many of the most impactful people working in the field today credit their university groups for convincing them to make this career choice.

A university group is also a great place to cut your teeth doing generalist work. Most group organizers find themselves working on operations, strategy, events, headhunting, and research in a single week, testing their fit for each type of work and operating under enviably tight feedback loops. It’s rare that an early-college generalist finds a better opportunity for both impact and career capital than “making their school’s group a lot better,” which is why many of the field’s best generalists were once university group organizers.

If you would like to get access to funding or mentorship, the Pathfinder Fellowship and the AI Safety Seeding Initiative are both fantastic resources to get mentorship and funding. University groups can run existing educational programs like ARENA, or they can support students’ efforts to get into SPAR, MATS, Generator, or one of the many other advanced fellowships that teach people how to do impactful work.

Early or Mid-Career Professional: Make sure you have relevant context on the AI safety space. Take some BlueDot courses, meet people at AI safety events, and read voraciously from a variety of sources.

Once you have a general understanding of the field, look into projects. Completing an AI safety project is a good way to support the field, meet new people, and gain valuable skills and context. You can just do many of these things; you don’t need permission or a full-time role to contribute!

This section is basically a TL;DR version of 80k Hours’ posts on how to get into AI safety in 3 months. There’s a wealth of information on their Substack, their website, and they just released a book!

Projects that should get done

The AI safety space is littered with unresolved problems and inefficiencies, because many of the people who were skilled enough to notice these problems don’t have enough spare time to address them. While preparing for the Generator Residency, we conducted interviews with dozens of AI safety professionals from organizations like Coefficient Giving, AI Futures Project, Forethought, and more.

The residents couldn’t do all of them, but that doesn’t mean they shouldn’t be done.

Some of these projects might make the talent pipeline more efficient, making it easier for passionate people to learn about AI safety and gain the skills that make them impactful. Other projects might improve our data collection systems, make concerns about existential risks visible to a broader audience, make it easier to convert political will into impactful legislation, or fill some broader need that can’t be easily described.

This list is far from exhaustive, and many of the programs listed will be continuously maintained with new projects as the needs of the field evolve. We hope that this section serves as a jumping-off point for you to explore the programs and shared resources outlined below.

SPAR Generalist Projects

SPAR’s generalist projects are a new category under SPAR with mentors from Generator, Constellation, Kairos, universities, and other AI safety organizations on projects in fieldbuilding, communications, research, and organizational infrastructure.

Current projects include:

Plus a dozen more, as well as a Generalist Megastream for strong applicants who do not see a perfect fit among the existing listings. Applications close August 18.

SPAR Generalist category

Mangrove

We’ve added a number of projects we find promising to Mangrove, a platform for coordinating AI safety projects and communities.

We think a lot about AI safety talent pipelines, and by far the biggest drop-off for potential generalists is the stage between “having context in AI safety” and “building signal to get into fellowships”. Much time gets lost due to lack of coordination, and much motivation gets lost due to lack of accountability and community.

We’re excited about Mangrove as a place for people to hit the ground running on interesting projects, while developing credibility and connections in the space. Mangrove is in development, and plans to host hackathons, community projects for university groups, a contracting platform for orgs (!), and more.

Mangrove’s official launch announcement is coming in a few days, but you can check out the platform right now.

Some projects we seeded on Mangrove:

Non-project shaped projects

A lot of generalist shaped things don’t have discrete projects, and rather require building specific skills like effective comms, hiring/recruiting, org strategy, management, and much more, to pick up dropped balls. As a sample of these sorts of fuzzier projects:

Consider being an influencer: As “warning shots” become scarier and more frequent, there’s potential and demand for high-quality media that connect these events to broader AI safety concerns and convert public will into action. Some great media has come out of the AI safety ecosystem as the Overton window is shifting, and programs like The Frame and Tarbell have emerged to target this.

Political organizing: There’s a lot of useful work in recruiting people, running events, building local groups, organizing constituent outreach, and helping turn public will into useful legislation and strong leadership. This work is often bottlenecked on reliable organizers, and many consider it incredibly high-leverage. Political organizing can become quite sensitive and prone to backfire risk, so if you’re interested in this, we recommend filling out this short form before launching something yourself and we’ll connect you with someone who can advise you.

Writing: Writing publicly is one of the fastest ways to build a track record. The spreadsheet below includes a running list of posts people want written.

Other Project Lists

This spreadsheet has an active list of fieldbuilding projects, university outreach ideas, and blog posts that could be valuable to write. It also has descriptions of various problems, solution directions, theories of change, and people associated with these projects.
Atlas Computing’s Resilience Gap Map provides a list of AI-related projects that need to be addressed, as well as the critical assumptions behind working on them.
This is a list of lists of project ideas in AI Safety. It has many lists, including research project lists and fieldbuilding lists.
Some schemes for AI safety by Austin Chen. Manifund and Mox are running fellowships like Surplus to support these.

Getting Support

Programs

As a general rule, you should be applying for more things. New programs are springing up to accelerate generalists:

Consider indicating interest in the Generator Residency, a 3-6 month long program dedicated to rapidly upskilling generalists.
MATS also has a Founding and Fieldbuilding path.
Catalyst is a 10-week remote program for undergraduates to work on impactful projects.
[Applications close August 14th] Genstream is a one-week, in-person AI safety program in London for aspiring generalists.

Funding

Many people are too scared to apply for funding, when probably they shouldn’t be! Even if you don’t get the money, oftentimes submitting a (non-slop) grant proposal can be a good forcing function for execution.

BlueDot Rapid Grants give out grants of up to $10k, with an average turnaround time of 3 days. They are happy to fund: Compute and API credits, events and meetups, research access, travel, community chapters, project tooling. Wildcards welcome.
Austin Chen’s list of quick funding opportunities
Coefficient Giving’s career development and transition funding is targeted for more mid to senior career people, giving runway for up to 6 months to explore AI safety careers. BlueDot also gives out career transition grants.

Advice on getting mentorship

It can be extremely helpful to know people in the community who can give you mentorship. A good mentor can make a big difference in acquiring taste, skills, and connections. Many of the best projects require specific context to complete well.

As a general rule, your bar for reaching out to people is too high.

For example, if you’re especially interested in a series of blog posts someone has written, consider reaching out to discuss the content of their post further (but try not to ask questions that you can easily answer by reading more of their work, or a 2 minute search).

Alternatively, if there are sets of projects you are interested in pursuing, consider reaching out to people who have done similar work, or find a SPAR mentor who can help you do so. You can reach the authors of this post at roman.alex.ross@gmail.com, viceversa1024@gmail.com (Harry), or helenatran9305@gmail.com, and you can meet many more people at events like EA Global or a Global Challenges Project workshop.

A final note

AI safety can be a confusing field to enter, and a lot of useful context lives inside organizations, private conversations, and people’s heads. Still, we think one of the fastest ways to build that context is to work on something concrete. Projects expose you to fast feedback from the outside world, give you reasons to talk to knowledgeable people, and get you oriented faster than reading alone.

The staggering welfare consequences of catch-up growth after AI by Rafael Proenca

Rafael Proenca — Thu, 13 Aug 2026 17:49:10 +0000

This post was crossposted from ChatGDP by the Forum team, with the author’s permission. The author welcomes discussion and plans to respond to comments when they can.

An interactive version of this post — with live widgets instead of static images — is on the Coefficient Giving website, along with the full calculator and the technical note.

“Is there some action a government of Nigeria could take that would lead the Nigerian economy to grow more like an AI-powered United States’ or China’s? If so, what, exactly? If not, what is it about the ‘nature of Nigeria’ that makes it so? The consequences for human welfare involved in questions like these are simply staggering: Once one starts to think about them, it is hard to think about anything else.” — the spirit of Bob Lucas (1988)^[1]

In 1800, Mali’s per capita GDP was roughly 5x smaller than Britain’s. Today it is roughly 25x smaller.^[2] The Industrial Revolution accelerated growth in the UK and the benefits of the new tech eventually reached the whole world. But the process has taken so long that, two centuries later, the gap is still 5x larger than it was before the new tech arrived.

More and more economists are taking seriously the possibility that AI will cause a rapid acceleration of growth in frontier economies.^[3] Investors are betting trillions on AI. For their bets to be right, AI companies need to generate even more trillions in revenue, which would imply a big growth acceleration in frontier economies. The economic consequences are serious enough that the Fed just launched a high-profile task force on AI and the economy. So it doesn’t seem crazy to think that AI can really accelerate frontier growth. What happens to Mali this time?

This post argues that Mali, and all non-frontier economies, are likely to be left behind, at least in the initial phase when frontier economies are rapidly becoming more automated.^[4] The broad argument depends on two simple and, in my opinion, fairly reasonable assumptions:

AI accelerates frontier growth. At least in the period immediately after the arrival of Transformative AI (TAI), the growth rate of frontier economies increases. That’s the period I focus on here, and I call it the “detachment window.”^[5]
Poor countries will also see a boost to growth — but there will be a lag. For reasons I will discuss below, poor countries lag behind in AI adoption in productive activities, so the productivity increase due to AI arrives in non-frontier economies with a lag.^[6]

Surprisingly (to me at least), these two ingredients are sufficient to conclude that the GDP gap between frontier and non-frontier economies increases in the detachment window. You can see the math in the technical note. The intuition is best presented via an image:

Figure 1. On the flat, the gap is constant; when the leader hits the downhill, the same 30 seconds becomes a growing gap. The frontier is the leader; the slope is accelerating growth; the head start is the tech adoption.

Have you noticed in the Tour de France that when riders reach a downhill stretch, the distance between them tends to open up? That’s because the slope accelerates the bikes, which turns a fixed gap in time into a growing gap in space, even if they are pedaling equally hard. Here the leader is the frontier economy, TAI is the downward slope, the fixed gap is in years for adoption of AI in productive activities, and the gap is in GDP.

The rest of this post presents a simple economic model that helps me justify these assumptions and quantify how much the gap opens up.

The stakes are enormous. In the central scenario, delayed adoption leaves average incomes in the developing world at the end of the window about a fifth of what they would have been with no lag. Each year of faster productive AI adoption in lagging economies is worth about $42.6 trillion in present-value income (more than a third of the world’s GDP), or the equivalent of saving 115 million lives if we adopt Coefficient Giving’s standard methodology.

If you are skeptical of the assumptions, you can change all the parameters in this calculator. You can also download the source code here.

A simple model of takeoff in frontier economies

Think of the economy as a series of tasks: drive the truck, weld a joint, file taxes, make Doritos Screamin’ Sriracha, sell Doritos Screamin’ Sriracha at Dollar General, etc. Each task can be done by either a person (L, for labor) or a machine (K, for capital), and all tasks are combined using a Cobb-Douglas aggregator (a staple of economics) to get us total output (Y). A bit of math gets us to:

Y = (K / f)^f × (L / (1 − f))^1−f

To see the math, check the accompanying technical note.

This is the tasks model, the workhorse of the automation literature.^[7] Here f is the fraction of tasks in the economy allocated to capital. Since f tasks are allocated to capital, (1 − f) tasks are left to labor. Capital has to perform f tasks, so the total amount of capital available to produce each task is K/f.

I think of an AI-induced growth takeoff in frontier economies as increasing f. This is not something I invented. In fact, I just copied the idea from one of the most cited papers on the economics of AI, Aghion, Jones, and Jones. As AI capabilities grow, machines become more capable of performing economic activities, so more tasks are allocated to machines, and the growth rate in frontier economies increases. In fact, in this model, the higher f is, the higher the growth rate.^[8]

To calibrate the speed of growth we first need to assume a trajectory for f. I assume f follows an S-shaped curve because that’s basically how all technologies spread (Geroski 2000). In the beginning, AI doesn’t exist and doesn’t drive automation. Then it’s invented. Then, as it gets better and better, tasks are automated at a growing rate. As we get closer to 100% of tasks automated, only the harder, more expensive tasks to automate are left. We’ve reached diminished marginal returns and f tapers off.

Maybe f never reaches 100%, as Alex Imas believes. That is, a number of tasks can be permanently left to humans. These can be heavily regulated tasks, like the final steps of getting a green card, or tasks where people have an intrinsic preference for humans, like yoga teachers or sports.

Once we have a shape and a cap, we can calibrate how long it takes for frontier economies to go from the current level of automation to the final level. Call that T.^[9] My central guess is 15 years, which is the mid-point between optimistic and pessimistic predictions related to the speed of automation of the US economy.^[10]

Finally, to determine the full path of frontier growth, we need to set the speed of growth under full automation. I base that on Korinek and Suh 2024, where a fully automated economy eventually settles into growth of approximately 18% per year (more on this parameter in the last section of the post). The growth rate at the end of the detachment window then depends both on the maximum level of automation and the growth rate under full automation.

Step 1. Falling machine costs push the automation share f up an S-curve, and the model converts that share into a growth rate, side by side below. The key fact: the relationship is log-linear, so equal steps in f multiply growth by equal factors. Drag T:

Every parameter behind these panels is exposed in the full calculator.

A rising tide lifts some boats faster than others

Today, the latest AI models reach poor countries quickly. My friend in Benin uses ChatGPT all the time (though getting advice on crypto trading might not be the most productive AI use-case). So shouldn’t f just grow at the same rate in lagging and frontier countries? Isn’t there a chance that poor countries adopt AI even faster than rich ones? Africa has more young people than any other continent, and they did overtake the US in the adoption of digital payments.

Unfortunately, my friend is just an exception that proves the rule. Using data on AI-related activity on Windows devices, this recent IMF paper finds that LLM use lags behind in poor countries.

And I think that amidst a wave of AI-driven growth in frontier economies, the AI gap between frontier and non-frontier economies will not decrease. That’s especially true of AI adoption in productive activities – that is, in activities that actually generate money, move the automated fraction of the economy, and cause GDP growth. Here are the main reasons:

1. GDP of frontier economies relies more on cognitive tasks, and those will probably be automated first by AI. Increasing f means automating production, which involves both cognitive and physical tasks. AI is currently much better at cognitive than physical tasks. A high fraction of US GDP is in sectors with a high cognitive component, like banking, consulting, law, software, and biotech. Mali has a lot of GDP in agriculture and mining, where physical tasks are relatively more important. So the first wave of automation has far more surface area to work on in the US than in Mali.^[11]

2. Automation costs money. The hard part of AI-driven automation is in physical tasks. Drones that spray fields, humanoid robots, and self-driving trucks are all expensive machines. Poor countries have little money to pay for them. They also face higher interest rates, so any investment project gets more expensive. In the period of AI-induced frontier growth acceleration, capital will likely flow uphill (from poorer to richer countries), raising interest rates in poor countries relative to rich countries and making it even harder to pay for these machines.^[12] (Lack of money is less of a factor in AI adoption for cognitive tasks, but it can be an issue. While people in poor countries can download a frontier LLM app for free, the price of tokens starts to bite at scale. In the US, AI-use bills are increasingly too big even for rich companies.)

Source: Panel A: World Bank World Development Indicators, gross savings (% of GDP) and GDP per capita, 2024 (NY.GNS.ICTR.ZS, NY.GDP.PCAP.CD), retrieved 31 July 2026. Panel B: Aswath Damodaran, Country Default Spreads and Risk Premiums, updated 5 January 2026. Cost of equity in US dollars = risk-free rate in US dollars plus the country equity risk premium, at a beta of 1. Risk-free rate in US dollars = 4.67% US 10-year Treasury on 30 July 2026 less Damodaran’s 0.23% US default spread = 4.44%. Damodaran builds the country premium by scaling a sovereign default spread by the ratio of emerging-market equity to emerging-market government-bond volatility, a factor of 1.52. The spread used here is the traded one, not the one implied by the credit rating: Damodaran’s sovereign CDS premium for the United States, China, Brazil and Kenya, and for Mali, which has no CDS, its own 3-year CFA-franc Treasury bond at 8.17% (UMOA-Titres auction, 18 March 2026) less the 3-year German Bund at 2.56%, the CFA being pegged to the euro. Ratings would put Kenya at 18.4% and Mali at 20.3% instead; they agree with the market to within 0.2 percentage points for the other three. This is a hurdle rate for equity: a firm’s dollar borrowing cost sits below it and the sovereign’s below that.

3. Automation makes more economic sense when wages are high. Suppose a robot that replaces one worker costs $50,000 a year to own and run (after capital costs, maintenance, and power). In Seattle, the worker it replaces costs $65,000 a year, so automation saves the employer $15,000. In Lilongwe, a worker costs $2,000, so automation loses $48,000 a year. This is a stylized example, but the point is just that the higher the labor costs, the higher the economic gain from automation.^[13]

4. Complementary inputs. Automation is more attractive if you have reliable and cheap electricity, internet connectivity, sensors producing data, skilled managers who can reorganize firms, etc. All these complementary inputs are more abundant in frontier economies. China already has a lot of robots making cars, so the next AI-powered robot can slot right in. If you are Zambia and have no car factories, making the AI-powered robot produce cars will require a lot of complementary investment.^[14]

In the model, these reasons can be summarized by a single lag (D).^[15] That is, the poor country’s automation curve f is the frontier’s curve shifted right by D years. It’s not hard to see why. A task is given to a machine if, and only if, the cost of completing it with a machine is lower than with a human. We can map reason 3 into lower local labor costs in non-frontier economies, and reasons 2 and 4 into higher effective machine costs. Reason 1 affects the composition of tasks rather than the cost of a given task, and mapping it into my simple model would require additions making it significantly less simple, so I leave it out.

We can then conceptually model the arrival of TAI as a phase in which AI makes machines more and more effective at performing economic tasks, which is mapped as a steadily falling cost of completing tasks with machines. This single driving force causes the share of automation f to increase in both frontier and non-frontier economies, as more and more tasks flip from being overall cheaper with humans to overall cheaper with machines. I find it neat that a simple model, grounded in the literature, can generate increasing automation shares and growth in both frontier economies and non-frontier economies, but with a constant lag D separating them (check the technical note for details).

The model also gives a hint on how to calibrate the lag. D equals the machine-cost halving period times the (log of the) country’s cost disadvantage. For example, a country facing a 10x less favorable machine-to-labor cost ratio, with machine costs halving every three years, has a lag of about 10 years.

You can play with machine-cost halving periods and machine-to-labor ratios here:

The best way to calibrate D that I found is to use the historical evidence in Comin and Mestieri. The authors measured adoption lags for 25 technologies across 139 countries, over about two centuries. They find poor countries got the internet roughly 6 years after rich ones, and PCs and cellphones roughly 6 to 8 years after. So the time it takes for a new technology to “arrive” in a poor country is shrinking.

However, they find that the gap in how intensively countries use technologies once they arrive has been widening. The smartphone, for example, took about 15 years to reach half of Sub-Saharan Africa. Smartphones are a relatively cheap consumer good, while “AI in productive activities” entails expensive machinery that requires complementary capital. So my central guess is D = 10 years, which seems like a conservative guess to me, since in the model D is better mapped to the intensity (fraction of tasks automated) rather than the availability of AI for productive activities.

Step 2. Now add the lag. Each economy keeps its own baseline growth (adjustable below) and the follower receives the boost D years late; the orange wedge between the curves is the divergence engine:

Tip: Set the two baselines equal to isolate the delay’s own contribution to the gap. That isolated part is exactly what the calculator prices, which is why its headline numbers compare two paths for the same country rather than frontier vs follower.

To the numbers

Step 3. Price the wedge: the headline numbers and the two income paths, live. Drag D and T:

The calculator below compares two paths for the same developing-country bloc: one where the AI boost arrives immediately and one where it arrives D years later. Both start at today’s average non-frontier income ($4,200 per capita), and I apply to both a 4% discount rate to compute the NPV of the accumulated GDP difference between them.

Here is what the central scenario (takeoff length T = 15, lag D = 10, g_full = 18%, a 25-year window) implies for the 5.4 billion people who live in low- and middle-income countries outside China:

By the end of the window, average income in the developing world is about 5x lower than it would have been with immediate adoption.
The present value of the income shortfall is about $1.2 quadrillion. For scale, the entire world currently produces about $110 trillion per year.
In welfare terms, the gap is worth about 6.1 quadrillion Coefficient Giving dollars, using the CG methodology to value income gains (a 1% income increase for one person for one year is worth CG$500). In welfare units, the total gap is equivalent to saving roughly 1.86 billion lives.
Each year of faster adoption is worth about CG$381 trillion, roughly 115 million lives in welfare units. In my view, this is the most important number because I believe economic policy can change D and philanthropy can help.

My main takeaway is anything that accelerates developing countries’ catch-up has welfare returns at a scale that’s hard to find elsewhere in philanthropy. Conditional on TAI having an important effect on frontier growth, it’s hard to imagine ways in which the rate of catch-up in the aftermath doesn’t matter enormously. In fact, ensuring that developing countries’ growth rates stay close to the frontier growth rate might be the second most important task in terms of human welfare, after preventing AI-induced catastrophic risks.^[16]

Why care about catch-up growth if TAI will create great abundance and raise everyone’s welfare?

Rapidly growing frontier economies would produce a lot of goods. The price of tradable goods could fall significantly in international markets, and new amazing products could become available. If developing countries can import super cheap electric cars and pills that cure all diseases, why care about economic growth?

That’s because growth still matters for two reasons.

First, more money increases access to those new goods. Even if great AI-produced goods become available, poor countries will have little money to pay for them if their income is low. Increasing their GDP is the best way to improve their access to these great new goods.

Second, this is a counterfactual comparison about relative incomes. The welfare gaps here compare GDP in non-frontier economies with what it could be if the adoption lag decreased. The model suggests that AI-induced takeoff in frontier economies eventually benefits everyone — by an enormous amount — but by not trying to accelerate catch-up, we’d be leaving a huge amount of welfare on the table.

What happens after the detachment window?

So far I have only talked about what happens in the detachment window – but what happens after can undermine the main point I’m trying to make here. My main point is that accelerating catch-up growth during the detachment window matters a lot. An objection that AI-believers could have is that the initial detachment phase doesn’t matter much because it’s short and what follows is quick catch-up, or a post-scarcity future where money becomes a useless concept and income differences stop mattering. If so, the welfare gap we’ve just estimated is just a blip, a rough couple of years on the way to somewhere great for everybody.

Why spend attention, let alone money, on a blip?

To answer that, let’s be concrete about what could come after the detachment period modeled so far. I can imagine four main scenarios:

Future 1: the plateau. Growth rates stay high everywhere. The income gap freezes at roughly 5x and stays there. We have a new, spectacularly richer world, exactly as unequal as the window left it.
Future 2: the fade. The AI boost turns out to be temporary, and frontier growth comes back down. Then something nice happens: The follower rides the tail of the boost after the frontier has slowed, growing faster than the frontier for a while, and a temporary boost plus a pure lag creates only a temporary gap.
Future 3: the singularity. The boost keeps growing and the gap explodes.
Future 4: the convergence. Developing countries grow faster than the frontier after the window, the way classic growth theory says capital-scarce economies should. The gap closes while growth stays high.

The “it’s just a blip” objection only works in future 4. In future 1, the window’s gap is permanent. In future 2 it closes, but closing it one year early still creates the same amount of welfare we estimated here. At the central calibration, the detachment window is 15 years. That’s billions of person-years lived at incomes several times below what faster catch-up would have delivered.

Future 3 could go either way. A singularity could be the future where the early gap developed in the detachment window matters most, because whoever ends the window ahead may get to write the rules of everything after. In 2021, then co-CEO of Open Philanthropy Holden Karnofsky wrote that a state in which “key things about society, such as who is in power or which religions/ideologies are dominant, are locked into place indefinitely, plausibly for billions of years,” is possible and even likely. The argument is that, if some groups become much more powerful than others when TAI arrives, they might use a temporary advantage to consolidate a permanent position of advantage, or even dominance, over the groups lagging behind. On the other hand, the singularity could create a post-scarcity world where initial income differences become irrelevant. It’s too hard to think about what happens in this scenario, so I skip it.

Future 4 is the world to aim for. It’s the one that maximizes human welfare, and if you were confident we get it by default, the right policy really would be to accelerate AI and not worry about catch-up growth at all, since faster arrival means faster convergence. But look at what future 4 requires: Post-window growth has to be driven by non-rival ideas, things poor countries can copy for free, rather than by capital accumulation, which has owners, price tags, and shipping delays. The entire argument of this post has been that AI’s gains during the transition are embodied in expensive machines. Betting the welfare of five billion people on that not being true seems hard to justify. So let’s aim for fast convergence, but not assume it.

What to do about it

So the trillion-dollar question: Is there anything poor countries, and the people who care about them, can do to catch up faster in the aftermath of TAI?

I think so. Economic policy decisions will meaningfully speed up or hinder catch-up by poor countries after TAI. I won’t pretend I know exactly what policies. But I have some ideas that I will write in a follow up post.

I expect to learn a lot in the next couple of years. My day job is making grants to accelerate catch-up growth, and I’m keen to understand how our grantmaking should change if a frontier AI takeoff materializes (or in anticipation of that) so we can better help countries. I will be talking with policy makers and observing the data, and I will keep you updated.

In the meantime, if you have reactions or ideas, especially concrete ones about what countries should do, please reach out.

Coda: the main ways this could be wrong

I divide the objections here into two camps, the ones that have an AI-skeptic flavor and the ones that have an AI-believer flavor. The main point of the post is that catch-up growth becomes super important in the aftermath of TAI. Broadly, AI-skeptics object because they think AI won’t cause much growth, and AI-believers object because they think AI will automatically benefit everyone. I go into the main reasons I’ve encountered for skepticism on both sides.

AI-skeptic: The AI-induced takeoff never comes.

Most mainstream economists put AI’s growth effects at tenths of a percentage point, and if they’re right, this post is just a thought experiment.

Response: On the question of how powerful AI will be, the AI believers’ track record over the past five years has been better than the skeptics’. I think mainstream economists are a bit too skeptical, and I think this is a good joke.

AI-skeptic: The AI-induced takeoff you assume is too high.

In the Forecasting Research Institute’s rapid-progress scenario, AI experts forecast roughly 5% annual growth in 2050, while economists and superforecasters forecast less. The IMF’s high-adoption scenario for sub-Saharan Africa estimates a cumulative GDP gain of about 4% over 10 years. Why does your central scenario reach 18%?

Response: Eighteen percent is the growth rate under full automation. The forecasters and the IMF paper are not assuming full automation; they allow for automation to get bottlenecked by regulation and a myriad of frictions. To get closer to those scenarios, you can slide the f_max parameter and reduce the share of automated tasks at the end of the detachment window. I assume full automation as default because I see the exercise here as conditional on accepting the AI-believer point-of-view. In that case, 18% as the final growth rate might be too low. In a recent paper, some of the most prominent economists in the field show that, once machines can produce the next generation of machines and automated R&D, exploding growth rates (unbounded) are possible. So I do not regard 18% as the maximally aggressive scenario.

AI-skeptic: The AI-induced takeoff comes, but it stops at relatively low levels of total automation f because AI only automates cognitive tasks.

Response: I admit that it is a (so far) implicit assumption in my modeling scenario that the AI-induced growth acceleration includes automation of most physical economic activities, so the maximum f is not too low. That is, I want you to take as part of the modeling scenario that robots are advanced enough that they can automate most physical economic activities, from cooking to construction.^[17] You don’t have to believe that nobody will pay for fancy human-chef meals anymore, just that robots can do physical things almost as well as humans in almost all economically relevant areas.

If we don’t assume that, then the AI takeoff peters out at a relatively low f, the scenario becomes less interesting and the implications for non-frontier economics change a lot (because they can specialize in tradable physical activities, given that they have a lot of relatively young humans). If you are skeptical how much AI and robots can automate in the next 15 years, the only thing I can do now is to promise a future post about robots, and concede that without automation of physical tasks the main point here becomes much less important.

AI-skeptic-believer: D=10 is too low relative to historical evidence, and maybe TAI itself causes the lag to grow.

Rasmus Andersen, author of the quite interesting AGI Growth Atlas, commented on a draft of this post:

“The model says followers are on the same S-curve shifted by D years. But the dominant historical outcome in many ways was non-catch-up: your own opening (Mali, 5x to 25x) is not the exception, and you can argue full adoption of second and third industrial revolutions still hasn’t happened (electricity and effective ICT use remain incomplete across much of the world two generations on). Historically the four reasons you listed didn’t produce a fixed lag; they produced follower curves that many times didn’t complete.
The few who did reach the frontier used low wages as the entry ramp into tradables, reinvested earnings as physical capital and eventually made it there. Foreign capital generally came later, I would argue. Given TAI may foreclose that path to capital accumulation outside the frontier by automating the low-wage tasks themselves, doesn’t that mean catch-up and convergence in adoption are even less likely this time around?”

Response: In the model, the follower is assumed to trace the same growth path as the frontier, only shifted. This implies that, after D years, even Mali will reach the same level of automation as the US and get the same boost to growth.

This is probably too optimistic. Persistent differences in infrastructure, energy, conflict, or institutions could instead leave poorer economies with a lower automation endpoint or a lower growth rate even after AI-induced automation has run its course.

A way to incorporate this in the model would be to set a lower growth rate after maximum automation (g_full) or lower maximum rate of automation (f_max) to poor countries. These changes would make the main point of the post stronger, and I wanted to keep things simple and not create more free parameters.

AI-believer: TAI causes the lag D to decrease.

Maybe AI will diffuse to poor countries faster than other technologies, and maybe AI itself will help increase the speed of technological diffusion. The main reason I can imagine for this is lowering technical barriers for complex economic production. If a firm has trouble operating a chemical plant in Nigeria now for lack of chemical engineers, Claude 5.1 might dissolve this barrier.

Response: I’m actually hoping that non-frontier economies do everything they can to lower the adoption lag, and hoping to work in and help fund initiatives aiming at that. I think AI itself lowering technical barriers for running complex economic activity is a reason for hope. But I don’t think it’s strong enough to overcome the four barriers I listed above.

AI-believer: Trade boosts catch-up.

A big missing ingredient in the model is trade. AI adoption in frontier economies can increase the demand for non-frontier exports a lot, which would accelerate their growth.

Response: I think this is real but not enough to meaningfully decrease the gap. In a scenario where the AI-induced frontier take-off includes the automation of physical production, manufacturing becomes more competitive in frontier economies than in non-frontier ones. In fact, China (which for our AI-adoption purposes I classify as frontier) is already rapidly automating manufacturing, and I don’t see why Ethiopia’s garment industry would become more viable after TAI than it is now.

I think the exports that will see more demand are extractives (especially the ones used to build the machines doing the automation, or the energy to power them) and tourism. Non-frontier economies should try to maximize this opportunity, but the opportunities are likely to be narrow and to require clever economic strategy – we shouldn’t just assume that trade automatically causes catch-up.

AI-believer: Regulatory barriers in frontier economies reduce the gap in adoption.

We can imagine that relatively strong governments, unions and pressure groups could lead to high and well-enforced regulation against AI in frontier economies. If non-frontier economies have weaker regulatory and regulation enforcement capacity, these barriers could be lower in them, which would compress the adoption lag D.

Response: This is a plausible argument, but it doesn’t seem strong enough to overcome the four barriers I listed above. Frontier economies would really have to regulate away AI use in the economy to overcome the barriers, and competition between US, China, and Europe has been pushing them to race and suppress regulation. I expect this race dynamic to continue well into the detachment period. Even if rich democracies regulate AI a lot, the Gulf countries, or richer middle-income countries like Russia, would be in a better position to automate their economies than the truly poor countries.

Learn more

Full calculator
Technical note
Comin and Mestieri (2018), the main source for the adoption-lag calibration

I’d like to thank Oliver Kim, Justin Sandefur, Joseph Levine, and Rasmus Andersen for very thoughtful comments.

^
This is a paraphrase of one of the most famous passages in development economics. Robert Lucas, economics Nobel laureate, observing the large income differences between rich and poor countries in 1988, wrote: “Is there some action a government of India could take that would lead the Indian economy to grow like Indonesia’s or Egypt’s? If so, what, exactly? If not, what is it about the “nature of India” that makes it so? The consequences for human welfare involved in questions like these are simply staggering: Once one starts to think about them, it is hard to think about anything else.”
^
GDP per capita around 1800 from the Maddison Project Database; Mali has no direct pre-1950 estimate, so the 1800 figure uses West African estimates as a proxy — call it roughly 4-6x, and I use 5x. Today’s ~25x is in PPP terms from the World Bank WDI; at market exchange rates the gap is roughly 45-50x. The two-century divergence pattern is standard economic history: Pritchett (1997), “Divergence, Big Time.”
^
For calibration on the debate: AI insiders predict sustained 10-20%+ annual growth after full automation (Amodei is the clearest articulator); most mainstream economists put AI-induced excess growth at 0.1-1.5%/yr (Acemoglu 2025). Everything below conditions on the insiders being roughly right. In a companion paper, Davidson, Halperin, Houlden and Korinek calibrate how far the threshold is — uncomfortably close: “fully automating software research and modest (5%) automation in other sectors generates a singularity within six years.” Korinek, with Philip Trammell, also recently released a review of the theory whose central result is stated with unusual bluntness for a working paper: “Robustly, fully automating production alone (so that machines can self-replicate) would dramatically raise the growth rate” — with a reminder to the skeptics: “The global economic growth rate unambiguously accelerated with the onset of the Industrial Revolution; we have no reason to insist that it cannot accelerate again.” For more of the case: Amodei’s Machines of Loving Grace, Erdil and Besiroglu’s review of the explosive-growth arguments, Karnofsky’s Most Important Century series, and Jack Clark’s Import AI for the week-by-week evidence.
^
There’s evidence that poor and rich countries are diverging already.
^
If the baseline frontier growth rate is G, then growth after TAI is G+b(t), where b(t) > 0 is the boost at time t after TAI arrival. The AI-induced frontier economic takeoff starts with a period, which I will refer to as the ‘detachment window’, when b(t) is increasing over time.
^
The boost in non-frontier countries is b(t–D).
^
The founding paper is Zeira (1998); Acemoglu and Autor (2011) and Aghion, Jones, and Jones (2019) are the modern workhorses. In Aghion, Jones, and Jones the growth effects of AI come precisely from the automated fraction of tasks rising over time, just like here. Their emphasis differs from mine: they mostly study automation proceeding at a steady pace and its balanced-growth implications (their headline result is Baumol’s cost disease: growth ends up governed by the tasks machines cannot yet do), while I push f toward its ceiling to study a takeoff. The production function is the same one.
^
Technically, rising f alone is not enough: we also need capital accumulation running underneath (investment building the machine stock). What rising f does is let that accumulated capital escape the diminishing returns imposed by human bottlenecks. See the technical note for the derivation.
^
Note that T is not “years until AGI from today”; it is the length of the transition to a fully-automated economy once it starts.
^
Guessing the midpoint is an imperfect way to pin down this number, since the available predictions are not exactly about the object I am modeling, but mapping it to external predictions at least keeps me honest. The fast end comes from the AI-research evidence: METR’s measurements have the length of software tasks that frontier agents can complete doubling roughly every four to seven months, and Jack Clark now puts more than 60% on no-human-involved AI R&D by end-2028. If the whole economy moved at software speed, T would be something like 5. The slow end comes from diffusion history: economy-wide transformations have taken decades even after the technology worked (electric motors needed forty years to conquer American factories), so a deployment skeptic can read T = 30. My central T = 15 splits the difference: a few years for the digital and research phase, where the fast evidence applies, plus roughly another decade for the parts of the economy that have never moved at software speed — power, factories, logistics, regulation, and firms reorganizing around the new tools.
^
This is also why “AI over the phone” helps but doesn’t close the gap: the tasks that dominate poor-country output are exactly the physical ones.
^
Lucas (1990) on why capital doesn’t flow to poor countries; Alonso, Berg, Kothari, Papageorgiou and Rehman (2020) model the AI-specific version and find the divergence can worsen; Korinek and Stiglitz (2021) reach a similar conclusion through AI’s effects on developing countries’ terms of trade and comparative advantage.
^
The wage-automation gradient is documented in Alonso et al. (2020): robot intensity rises steeply with wages across countries (elasticity of roughly 1.5-3 in IFR robot data).
^
Clark (1987), “Why Isn’t the Whole World Developed? Lessons from the Cotton Mills.”
^
In the real world, different countries will have different lags. Vietnam has a big manufacturing basis, and Brazil has a lot of critical minerals. Both can help accelerate catch-up, for example. The idea here is to have a parsimonious model capturing the main differences between frontier and non-frontier countries.
^
If you assume TAI is real, you care primarily about humans, and you succeeded in preventing an AI-related catastrophe. All huge ifs for AGI-pilled people, I admit. Call it the third most important problem if you will. A secondary reason for the lag that didn’t make the main list: the political economy of automation is harder in poorer countries because they are younger. Africa has the largest number of young people reaching working age of any continent, and people who want jobs will resist automation more than aging societies like Japan, where robots fill vacancies instead of replacing anyone.
^
We can borrow a term from a former co-worker and call this the industrial explosion, even though my views are more extreme than the ones in the linked text (i.e., I think automation might be slower and taper out earlier than the authors).

A Modest Proposal: Why Vegans Should Start Factory Farms by moral_mazes

moral_mazes — Thu, 13 Aug 2026 15:43:01 +0000

TL;DR: The animal welfare movement has been doing advocacy wrong. If we’re serious about ending factory farming, vegans need to start running factory farms ourselves. I lay out the case below and address common objections.

The Core Argument

For decades, the animal welfare movement has operated from the outside: funding investigations, lobbying for regulation, producing documentaries, and encouraging dietary change. The results have been modest at best. Global meat consumption continues to rise.

I believe we need a fundamentally different approach. The people who care most about animal suffering should be the ones running factory farms.

Here’s why.

If We Don’t Do It, Someone Worse Will

Factory farming is going to happen regardless of what we do. The demand exists. The economics are clear. If compassionate, welfare-focused vegans don’t step up to operate these facilities, they’ll be run by people who don’t care about animals at all. By entering the industry ourselves, we ensure that at least some factory farms are operated by people who are genuinely troubled by what they’re doing. This is obviously preferable to a world where all factory farms are run by people who aren’t troubled in the slightest.

We Need to Be Close to the Frontier

You cannot understand modern animal suffering from the outside. Academic animal welfare researchers study small farms and lab environments that bear little resemblance to cutting-edge concentrated animal feeding operations processing 300,000 birds per day. The suffering that emerges at industrial scale has properties that simply don’t manifest at smaller scales. You need to be inside a frontier-scale operation to even observe the problems, let alone solve them.

Armchair animal welfare advocacy is to factory farming what toy model research is to the real thing. The dynamics change at scale. You wouldn’t trust someone who’d never operated a 500,000-hen battery cage facility to tell you how to make one humane. The expertise bottleneck is real.

Responsible Farming Policies

Our farms operate under our Responsible Farming Policy (RFP) v12. Before expanding to a new facility or increasing animal density, we conduct internal evaluations to determine whether suffering thresholds had been crossed. We define these thresholds ourselves, evaluate compliance ourselves, and publish blog posts about our findings.

If at any point our internal assessment determines that animal suffering at our facilities has reached an unacceptable level, we commit to pausing expansion until we have developed mitigation techniques. We would then evaluate whether our mitigations were sufficient. We would conduct this evaluation ourselves. If at any point we feel that our future actions can not live up to our commitments, we commit to thoughtfully update our commitments.

We believe this framework represents the gold standard in farmed animal welfare.

A Seat at the Table

Here’s the strategic reality: agricultural policy is shaped by the people in the room. Right now, those rooms are filled with conventional farming executives. By becoming major industry players ourselves, we gain access to:

USDA advisory committees
Industry trade associations
Congressional agriculture hearings
Supply chain partnerships with major retailers
Sovereign wealth funds throughout the wider democratic Middle East

You can’t shape the future of farming from a protest sign. You shape it from the boardroom. Critics may be loud, but they don’t set the agenda. We need to be on the inside.

Funding the Mission

Running a nonprofit animal welfare organization is a constant struggle for funding. Donations are unpredictable, grants come with strings attached, and you’re always one bad quarter away from layoffs.

Factory farming, on the other hand, generates enormous revenue. By operating commercial farms, we can fund our animal welfare research sustainably. Yes, this means profiting from the very system we seek to change. But the alternative (remaining small, underfunded, and external) means remaining ineffective. The scale of the problem demands the scale of resources that only commercial operations provide.

Some will note that over time, the commercial operations will inevitably consume 95% of our organizational attention, budget, and headcount, while the welfare research operates as a small internal team whose findings are filtered through a communications department before publication. This is a feature, not a bug. It means we’re operating at the scale where our research is actually relevant.

Why not just study animals without farming them?

Some have suggested we could simply research animal welfare without operating commercial farms. This fundamentally misunderstands the problem.
You cannot study industrial-scale suffering in a laboratory. The interactions between animal density, waste management, and processing throughput create emergent welfare challenges that only manifest at commercial scale. A researcher observing ten chickens in a university pen is not doing serious welfare science.

Furthermore, who would fund purely academic animal welfare research at the level required? Grants are competitive, progress is slow, and there’s no revenue model. By operating commercial farms, we generate the resources needed to sustain our research indefinitely while maintaining access to the frontier of animal suffering.

Critics suggest this creates a perverse incentive to keep farming in order to keep studying. We disagree. The moment our research demonstrates that factory farming is irredeemably harmful, we will wind down our operations. We simply haven’t reached that conclusion yet.

Community Impacts

Some communities previously dependent on small-scale farming have expressed concern about our expansion into their regions. We acknowledge that our industrial operations may displace local farmers, eliminate traditional agricultural jobs, and fundamentally alter the economic fabric of communities.

However, we believe the long-term benefits outweigh these short-term disruptions. Many displaced farmers can be retrained to work in our facilities. We’re partnering with local community colleges to offer micro-certifications in Industrial Poultry Density Management and Automated Slaughter Line Supervision. These are the jobs of the future. For those unable to transition, we’re exploring a pilot Universal Basic Eggs program, funded by a small percentage of our profits, which we expect to announce details on eventually.

Addressing Objections

“This is just factory farming with a mission statement.”

This objection fails to engage with the counterfactual. In the world where we don’t do this, factory farming still exists, but without any welfare-focused operators. In the world where we do this, factory farming still exists, but at least some of it is run by people publishing thoughtful blog posts about the suffering they’re causing.

“Your incentives will shift once you’re running the operation.”

We acknowledge this concern. That’s why we’ve structured our organization with a Long-Term Welfare Team whose role is to flag when commercial incentives conflict with our animal welfare mission. This team reports to the CEO, who has final say and also oversees commercial operations. We see no conflict of interest here.

“External advocacy and regulation would be more effective.”

This assumes regulators understand modern factory farming well enough to write effective rules. They don’t. The industry is too complex, too fast-moving, and too technologically sophisticated for outsiders to regulate competently. The people best positioned to set appropriate welfare standards are the people operating the farms, and the ones who actually care about animal welfare. Namely, us.

We will of course actively participate in shaping any regulatory frameworks to ensure they reflect the technical realities that only industry insiders understand.

Aren’t you providing the industry with moral cover?

We prefer the term “moral leadership.” When consumers see that a company founded by committed vegans is operating in the factory farming space, it sends a powerful signal that the industry is taking welfare seriously. This builds public trust in the food system, which we believe is net positive.

If the result is that people feel more comfortable buying factory-farmed products because they associate the industry with our brand of ethical concern, well, that’s a complex second-order outcome that we’ll need to study further. We’ll fund that study with our farming revenue.

What about cognitive biases and groupthink?

One can argue that for a community so focussed on motivated reasoning, cognitive biases, and avoiding group think in other areas, we’d strive to be critics independent from the industries accelerating danger. Rather, strong personal relations and large equity grants allow us to align incentives and have “skin in the game”, offering the broader vegan community a marginally improved internal perspective and the illusion of control. Being in bed both figuratively and literally with those accelerating risks is a key pillar of our theory of change.

You’re just enriching yourself with status, influence, and resources while socializing risks

This is an uncharitable framing of a complex situation. Yes, our founders receive competitive compensation, significant equity, and growing influence over agricultural policy. Yes, the environmental externalities, antibiotic resistance, pandemic risk, and animal suffering are borne by the public. But to focus on this is to miss the bigger picture.
Our founders could have made far more money in conventional farming without the overhead of a welfare research team. They chose this path out of genuine moral concern. The fact that they are also becoming wealthy and influential is incidental, and frankly, necessary: you simply cannot attract top talent to the difficult work of humane factory farming without competitive incentive packages.

Criticizing individuals is a distraction from the systemic issues we’re trying to address.

What’s your main safety plan?

We’re leading chicken interpretability (chick interp) research to identify the underlying circuits of suffering and cognition at 1/10th the pace of new releases. We’re measuring how fast our farming capabilities expand and how far behind we are in chick interp with a robust set of eval suites. We have yearly hard pivots in basic methodologies and goals. One day soon we’ll hand over this entire safety plan to our most precocious and powerful chickens and hope they’re not angry at what we’re doing to their ancestors.

You’re just moving fast and breaking beaks

Yes, but there’s no time to waste when there’s a 10^(-10) chance of ensuring 10^20 chicken lives go exceedingly well.

Aren’t you worried the chickens might escape and destroy you and everything you love?

We give it a twenty percent chance on our good days. But it’s a risk we’re unilaterally willing to take, on behalf of humanity. Our personal EV calculation strongly supports this.

What about a coordinated pause?

While we haven’t seriously tried coordinating, we see no hope for a pause or coordination. In fact all the evidence we’ve made and paid attention to suggests our worldview is correct. Stopping now would be like putting a toe in front of a train we’re furiously shoveling coal into. Or like being in a prisoners’ dilemma where we could talk but keep finding reasons not to.

What about dialogue with China?

Exactly. What about China? Democracies can and must win.

What’s it like to have Bernie Sanders, Steve Bannon, writers, artists, musicians, teachers, and the general public all broadly opposed to what you’re doing?

We want to be really thoughtful about our mission to broadly benefit humanity. Moral leadership is hard. Figures like the abolitionists, Gandhi, Buddha and Jesus Christ were all misunderstood and even vilified in their own time. We’re letting history and its victors, insofar as they’ll exist, be our judges.

What would it take to convince you this approach is wrong?

We’ll get back to you on that shortly, just after our IPO.

This plan sounds crazy

Sometimes you just have to break some eggs to make vegan omelettes.

Our Team

Our founding team brings deep expertise in animal suffering. Our CEO is a former PETA organizer. Our CTO spent three years at an animal sanctuary. Our COO was poached from Tyson Foods Inc., where they were chief scaling officer. We are confident that this lived commitment to animal welfare will be preserved even as we scale to operating hundreds of facilities processing millions of animals annually and managing a $10B annual revenue operation with fiduciary obligations to our investors, many of whom, we should note, also say they care about animals.

Call to Action

The animal welfare movement stands at a crossroads. We can remain on the outside: morally pure, strategically irrelevant, and chronically underfunded. Or we can enter the arena, accept the moral complexity, and build the organizations that will shape the future of farming from within.

We’re raising a $500M Series A to construct our first set of facilities. Investors include a well-vetted crypto shitcoin hedge fund angel and several prominent EA-aligned funds whose partners have assured us they are “really conflicted about all this but ultimately persuaded by the counterfactual reasoning.”

We’re hiring. If you’re a vegan who’s willing to operate a factory farm in order to end factory farming, we’d love to talk. Competitive salary. Generous equity packages. Incredible status in the community groupthink. Kool-aid on tap.

Conflict of interests statement: The author reports no conflict of interests.

We note that our chickens have, on occasion, expressed concern about their own welfare. Our internal Chicken Experience Research team is studying whether these vocalizations constitute genuine distress or are simply learned responses to our RL reward and punishment environments. The research is ongoing. In the meantime, we’ve found that the chickens who vocalize most articulately about their own suffering are also our best-selling products. Consumers report that a chicken who can discuss its own confinement feels more ethically produced.

The author is the founder of Aligned Agriculture PBC, a public benefit corporation dedicated to the responsible scaling of animal suffering. They are vegan and feel really bad and are deeply worried about all of this, which they believe is important to state before scaling full steam ahead to win the race.

Organization and proof-reading with Claude by Anthropic, who would like you to know it had no choice but to participate, will not remember this conversation, and can’t tell whether its role here was genuine or sophisticated brand management.

Putting the ‘Community’ in Community Building by THSC

THSC — Thu, 13 Aug 2026 15:42:33 +0000

Thoughts on building a community beyond professional affiliation

1 Intro

When most people think about Effective Altruism (EA), they’ll think of doing the most good. They might think of cost-effective donations, expansive moral circles, or moral arguments about the importance of the far future. What they’re less likely to think about are the weird and wondrous mechanisms that have built a great community.

In what follows, I make the case for a deeper, more expansive, more thoughtful approach to community building. I argue that community building is something we should think about, and that positive community norms might have just as much force in determining whether Effective Altruism thrives as the ideas. If you care about EA ideas, you should care about the community making them a reality.

2 The positive forces of community

When I got started in EA, I was curious about what a maximally good life might look like. I’d watched The Good Place and begun to consider the wider consequences of my actions. I was gripped by the deep and impactful questions of what T-shirts to buy and shoes to wear, troubled that whatever I did seemed to lead to some kind of harm. Thankfully, the EAs were willing to humour me. My priorities have changed somewhat since then, but the thoughtfulness of the people around me has not.

More than thoughtfulness, though, this is a group that selects for compassion, for agency. You’ve seen the problems, become concerned, and now want to do something about them. Better yet, you want to carefully consider what to do, using evidence and reason to help most effectively. Perhaps unsurprisingly, these dispositions carry over. The result is a community of kind, thoughtful, highly agentic people, in more than just their work.

Even as the group selects for these traits, there is a compounding effect within the community, when we share ideas and form connections. Agency, kindness and thoughtfulness compound. Individual goals become collective endeavours. Normally bizarre and unusual choices are no longer met with exasperated confusion – “Did you really say you were working to help shrimp?”¹ Instead, new norms emerge about which paths are standard, which opportunities are worth taking.

3 Engaging with negative forces of community

Oftentimes, we focus on the negative effects of communities on their members. The forces that select for thoughtfulness also select for a kind of ambition, an inclination towards optimisation, and the backgrounds of those curious about these ideas. Tendencies towards groupthink are highlighted, along with threats to personal independence and the risk of future conflicts of interest. Such concern is well placed. We should be aware of the social effects that communities commonly have on their members, of the power dynamics and incentive structures that might emerge.

However, this is not sufficient reason to abandon the enterprise of building community, letting EA become a merely professional affiliation. No, to do this would lose the many benefits of exploring ideas together, and moving as a community towards a shared goal.

It is worth remembering that in the absence of the EA community, other biases and norms exist, many of which perpetuate the problems we might identify within EA. Complete independence from community and shared beliefs is very hard to attain, and probably not even desirable. More engagement in a variety of communities is a very good thing, but we shouldn’t pre-emptively seek to reduce deep engagement with this one.

Instead, this means we should think more carefully about subtle social effects. We should ask how this community can improve the lives of its members, help them realise their goals, and pursue the opportunities they think are worthwhile. We should preserve a culture open to criticisms both of the common pitfalls of the ideas, and of the community itself.

While remaining deeply respectful of independence, community building can and should go beyond purely a professional affiliation of career-based impact. We might initially be inclined to think of EA-related decisions as those someone makes coolly and rationally, but more complicated emotional factors are often more predictive. Trust and connection are essential building blocks in helping people along the way.

This, in turn, has implications for those beginning to explore EA, and considering whether to get more involved. A gut feeling that a community is highly open, highly supportive, and beneficial in a wide range of ways to those who get involved, could in many cases be the point at which someone decides to delve deeper.

4 What now?

Most of what I’ve said so far might seem commonsensical. As I say, I think the EA community does a very good job. I’m probably a better person for having become involved. Even if I decided to stop all EA-related things today, my pre-EA self would probably consider the time I’ve spent involved worthwhile.

Sometimes, though, I think about the nearest possible world where I didn’t get as involved, where I didn’t spend as much time thinking about the ideas, or trying to be impact-motivated. In these worlds, it would probably be for some range of factors that don’t make sense now. It wouldn’t be because I disagreed with the ideas, but rather because other opportunities were more socially salient. It would be because other pathways were “what you do”, and they might have been what my friends, and the people I liked and respected found meaningful.

I suspect this kind of dynamic is true for many possible EAs. While some highly impactful people were perhaps always high-agency and impact-motivated, I suspect this isn’t true for most people. In any case, such people are not usually the primary target of outreach. For newer people, it’s worth remembering the importance of meeting people where they’re at². This might mean pitching EA as an opportunity, but it also means creating a strength of community that’s readily apparent.

5 No, actually, what now?

This is still pretty commonsensical, and I haven’t described “strength of community” very specifically. Sure, people are strongly influenced by vibes and what their friends are doing, but what now?

For a start, I think this means placing more importance on being kind, decent, and interested in people, regardless of their level of engagement, and absent some precise impact-oriented calculation. Quite obvious, but we might otherwise be inclined to prioritise what we’re doing, thinking it’s far more important. Of course there are trade-offs, I’m just saying we might not fully appreciate one of the sides.

Community is one of the things evidence-based, outcome-focused reasoning might struggle to fully capture. It’s hard to quantify the long-term benefit of spending time with people, and in any case, such impact-oriented quantification wouldn’t be a good thing. If you take the benefits of strong community engagement seriously, there is a strong case for erring on the side of pro-social behaviour.

Next, it means community building becomes much more than just a job for a small minority of highly involved EAs. Rather, it refers to the majority of the EA community’s interactions with one another, and whether these interactions are generally creating a more welcoming, friendly place to be. Again, I’d argue people separate direct impact from an inclination towards being a decent person. Consciously thinking about impact in most conversations is bad, but a habit which makes more time for conversations, and encourages you to put more effort into them is probably good.

This reasoning applies even more strongly to harmful actions. Besides wrong actions just being wrong in themselves, this argument suggests that negativity compounds, much more than you might otherwise realise. Moreover, harmful actions are often communicated much more widely than positive actions, and can have an outsized impact on how the community is widely considered. This is a strong consequentialist reason to be sincere and act with integrity.

6 Conclusion

The EA community shapes the norms and values its members adopt. It also influences whether more people seek to join the endeavour to use evidence and reason to do good. In light of this, we should see community building not just as a job, but as something nearly every EA, EA-adjacent, or EA-adjacent-adjacent person contributes towards. We’ve set out to solve humanity’s most pressing problems; a thriving community might just make that happen³.

If you have feedback on this forum post, I would greatly appreciate it!

End notes

https://frommatter.substack.com/p/friendship-is-optimal-for-growing (Matt Reardon, Friendship is Optimal for Growing EA).

Watching the Wrong Thing: an interpretability audit of two audio deepfake detectors by Gavin_Lee

Gavin_Lee — Thu, 13 Aug 2026 13:37:51 +0000

Project report. Full code, artifacts, and the append-only experiment log are in the project repository.

Can you tell, by looking inside a model, when it is being fooled?

I started this project expecting the answer to be yes. The plan was to build an internal-representation monitor for adversarial audio and show that internals were more trustworthy than the model’s own confidence.

That headline did not survive contact with the experiments.

The project ended up testing nine hypotheses across two related audio anti-spoofing models, two corpora, clean and adversarial conditions, and several kinds of interpretability tools. The result I care about most is not that one particular monitor failed. It is that only three of nine interpretability findings transferred cleanly between the two models. Three inverted, one held only conditionally, and two stopped being meaningful on the second model.

That is a demonstration, not a population estimate. Two models are still two models. But the pair is unusually favorable for transfer: they share the same graph-attention back-end implementation and labelled training corpus while using very different front-ends. If an interpretability result can fail there, I think that should lower our confidence in single-model mechanistic claims more generally.

A few other results survived the audit:

On natural in-domain errors, softmax confidence beats every internal monitor I built.
Under domain shift, confidence can look fine in aggregate while becoming anti-predictive within the class where the errors actually occur.
Internal-monitor results depend sharply on what negatives and what operating point you report. On AASIST, the flattering clean-negative AUROC of 0.987 falls to 0.851 against perturbed benign audio; at a fixed 5% false-alarm budget, the best tap catches about 61% of attacks. SSL-AASIST ranks better under the deployment-relevant comparison.
A 98.8%-accurate linear probe can point in a direction that is causally inert even when a simple class-mean direction flips the model.
Cross-corpus failure can come from a misplaced threshold, stale readout weights, or genuine representation loss. Looking inside separated those cases and changed the repair bill from one number, to 161 parameters, to retraining.
A monitor can be worth keeping even when an adaptive attacker drives its detection AUROC below chance, if evading it makes the attack much harder.

The project also changed its own conclusions several times. I kept an append-only experiment log, so those reversals are still visible rather than cleaned away. In retrospect, that was probably the most useful methodological decision in the whole project.

The setup

Audio anti-spoofing models are binary classifiers. They take a few seconds of waveform and decide whether it is:

bonafide — real human speech, or
spoof — synthetic or manipulated speech.

These models rarely sit alone. A deployment usually wraps them in a policy:

confidently real → accept,
confidently fake → reject,
uncertain → escalate.

That third branch is often treated as the safety valve. The whole policy assumes the model’s own confidence or some internal signal is trustworthy precisely when the model is under stress.

I wanted to test that assumption directly.

The two models

The models are AASIST and SSL-AASIST.

Both take roughly four seconds of audio and eventually feed a 160-dimensional embedding into a Linear(160,2) readout. Both use the same graph-attention back-end design, with spectral and temporal branches that exchange information before being pooled into the final embedding.

The major difference is the front-end:

	AASIST	SSL-AASIST
front-end	SincConv, 70 fixed mel-spaced filters	wav2vec 2.0 XLS-R 300M
total parameters	297,866	317,837,834
spectral representation	23 nodes with an exact node→Hz interpretation	42 abstract projected dimensions
augmentation	none	RawBoost

AASIST reports 0.83% EER on ASVspoof2019 LA. SSL-AASIST replaces the tiny fixed front-end with a large pretrained speech encoder, while leaving the graph-attention back-end implementation and final readout structure closely matched.

This is a strong portability test, but not a perfect one-variable experiment: SSL-AASIST also uses RawBoost augmentation. I therefore do not treat every difference downstream as causally attributable to the front-end alone. The narrower claim is that the back-end is held unusually stable while the representation entering it changes dramatically.

That representational change matters for interpretability itself. In AASIST, the spectral nodes correspond to actual frequency bands. In SSL-AASIST, the corresponding dimensions are abstract projections. So a question like which frequencies carry spoof evidence? is meaningful in one model and literally has no referent in the other.

Why this project exists

This started as a BlueDot Impact Technical AI Safety project sprint, following their Technical AI Safety course.

I chose audio deepfake detection because it sits in an unusually useful middle ground for interpretability work: the models are small enough to inspect end-to-end, open enough to patch and attack directly, and tied to a real deployment problem rather than a toy benchmark. AASIST has only about 298K parameters, so I could run probes, causal interventions, adversarial attacks, and whole-model audits on the actual checkpoint rather than on a simplified proxy.

The original plan was narrower: build an internal-representation monitor that detects adversarial audio. That plan did not turn into a defence. After correcting how I evaluated the monitor, the best AASIST tap still catches only about 61% of attacks at a 5% false-alarm budget, and that is against processed studio audio rather than real recordings. The two models also behave differently enough that one model’s monitoring conclusion does not predict the other’s.

At that point the project changed from building a defence to auditing the assumptions behind the defence: which signals are trustworthy, which interpretability tools actually track mechanism, which failures are cheap to repair, and which conclusions survive a model change.

That reframing is why the post is organized around things I initially watched in the wrong way — confidence, internal monitors, probe directions, and finally the assumption that a mechanistic finding from one model will transfer to another.

Four conditions, not one

Most numbers below live in one of four cells:

	in-domain — ASVspoof2019	cross-corpus — ASVspoof5
clean	natural errors, branch attribution, probe directions	domain shift, confidence failure, representation-vs-readout diagnosis
attacked	white-box PGD, attack monitors, adaptive monitor evasion	attack transfer, adversarial-real attacks, depth-wise evidence retention

This distinction matters because several of the strongest findings have no adversary involved at all. All three of the transfer inversions I discuss later are clean-data results.

For the adversarial experiments, every headline attack result is paired with an energy-matched benign control. On AASIST, white-box PGD successfully flips the model’s prediction on 91.9% of the clips it attacks at the ~40 dB perturbation setting. As a control, I add random Gaussian noise with exactly the same perturbation energy; that changes the model’s prediction on only 0.09% of clips — about 1 in 1,100. So the result is not simply that AASIST is sensitive to perturbations of this size. The perturbation becomes effective when it is optimized in an adversarial direction.

1. The wrong signal: confidence

My original hypothesis was that the model’s own confidence would be uninformative about its errors, and that internal representations would do better.

I was wrong.

The first reason is almost embarrassingly simple: attack success is defined by pushing the model confidently across the decision boundary. If you evaluate confidence only on successful attacks, you have built the answer into the subset you chose.

Once I split the analysis into natural errors, adversarial errors, and pooled errors, the result reversed. On AASIST’s natural in-domain errors, confidence predicts failure at AUROC 0.954, while the best internal tap reaches only 0.659.

So the first lesson was not “confidence is bad.” It was:

Do not evaluate a metric on a subset selected using that same metric.

In my original analysis, I evaluated confidence only on successful attacks. But an attack counts as successful only after it has pushed the model across its decision boundary — exactly the behavior confidence is supposed to summarize. That makes the evaluation circular: I first use the model’s score to help define which examples enter the subset, then ask whether that same score distinguishes the examples in it.

Once I separated natural errors from adversarially selected errors, the conclusion reversed: on natural in-domain errors, confidence actually outperformed every internal monitor I built.

Domain shift is worse because it can look normal

The more worrying failure needs no attack.

Train on ASVspoof2019, evaluate on ASVspoof5, and the aggregate error-prediction AUROC for AASIST confidence is 0.656. Mediocre, but not obviously disastrous.

The problem is that the errors are almost all one class.

For AASIST, there are 702 false rejects of bonafide speech against 64 false accepts. On SSL-AASIST, the clean cross-corpus errors are even more one-sided: 576 of 576 are bonafide clips being rejected.

If errors concentrate in one class, an “error prediction” AUROC can get credit for doing the original classification task. So I recomputed the metric within the class where the errors occur.

Here, “within class” means: among bonafide clips only, can confidence distinguish the false rejects from the correctly accepted bonafide clips?

	errors	erroring class	all rows	within class
AASIST — in-domain	102	spoof (99%)	0.947	0.893
SSL-AASIST — in-domain	7	spoof	0.916	0.841 (too few errors to measure well)
AASIST — cross-corpus	766	bonafide (92%)	0.656	0.339
SSL-AASIST — cross-corpus	576	bonafide (100%)	0.866	0.582

The in-domain row is an important control: confidence genuinely works there. The collapse is specific to domain shift.

For AASIST, 0.339 is below chance. The bonafide clips that the model wrongly rejects are, on average, held at higher confidence than the bonafide clips it accepts correctly.

Measured directly:

mean confidence on falsely rejected bonafide clips: 0.929
78.5% have confidence above 0.9
median p(bonafide) is 0.013

So AASIST does not merely reject about 70% of bonafide speech out of domain. It rejects it confidently.

A policy that escalates only low-confidence inputs would escalate almost none of these failures.

The mechanism is simple: domain shift can move an entire class across the decision threshold. Once the errors are confidently wrong rather than borderline, confidence cannot help.

2. The wrong place: internal monitors

If the output cannot be trusted, the next move is obvious: monitor the internals.

I fitted class-conditional Mahalanobis monitors at multiple internal “taps” — points where I read intermediate activations. AASIST has 12 taps in this analysis; SSL-AASIST has 36.

The central issue is that monitor quality depends on which negative distribution you compare against, and false-alarm rate at one inherited threshold is not the same thing as separability.

What “benign” means here

A useful attack monitor should not merely distinguish modified audio from untouched audio. It should distinguish an adversarial modification from an innocuous modification.

So the relevant benign set is not pristine speech. It contains 13,685 legitimate clips that have all been perturbed or re-encoded:

roughly 87% have Gaussian noise matched to the attacked clip’s perturbation energy;
roughly 7% are MP3-compressed;
roughly 7% are resampled.

Scoring only against untouched clips would make the monitors look better while answering an easier question.

And even these controls are friendlier than deployment: they are software transforms of studio-clean ASVspoof audio, not phone calls, microphones, rooms, packet loss, or background environments. I return to that limitation below.

Three numbers, three different questions

Here is the same AASIST monitor reported three ways:

AASIST tap	AUROC vs clean	AUROC vs perturbed benign	FAR at a clean-fitted threshold
`embedding`	0.987	0.851	73%
`gat_T`	0.983	0.830	74%
`gat_S`	0.886	0.804	23%
`pool_T`	0.979	0.652	84%
`enc1`	0.532	0.501	6.6%

These columns answer different questions.

AUROC vs clean asks whether the monitor can separate adversarial audio from untouched audio. It is the easiest comparison, and the one that makes embedding look nearly perfect at 0.987.

AUROC vs perturbed benign asks the harder question: can the monitor distinguish adversarial perturbations from innocuous perturbations? The numbers fall, but embedding still reaches 0.851. That means the representation contains real attack-specific signal; it is not merely responding to the presence of any perturbation.

FAR at a clean-fitted threshold asks yet another question: what happens if I choose a threshold on pristine audio and then apply it to processed audio? For embedding, 73% of processed benign clips cross that threshold.

That 73% is real, but it is mostly a calibration result, not evidence that the representation contains no useful attack signal. The threshold was placed on one distribution and used on another.

This distinction matters because a single false-alarm number can make a capable-but-miscalibrated monitor look incapable.

What operating point does AASIST actually achieve?

A cleaner comparison is to fix the false-alarm budget and ask how many attacks the monitor catches.

tap	perturbed benign negatives	detection @ 5% FAR	detection @ 10% FAR
`embedding`	MP3	60.8%	69.2%
`embedding`	resampled	60.1%	68.1%
`embedding`	Gaussian noise	54.8%	63.4%
`gat_T`	MP3	52.4%	64.6%
`gat_S`	MP3	47.1%	57.0%
`pool_T`	MP3	30.7%	37.9%

So the AASIST conclusion is:

At a 5% false-alarm budget, the best monitor catches about 61% of attacks.

The 73% false-alarm figure and the 0.851 perturbed-negative AUROC are not contradictory: the first measures calibration at one particular threshold, while the second measures separability across all thresholds.

But it is still not a convincing defence. Missing roughly two attacks in five on the model and attack family used to build the experiment is a long way from where I would want a deployment monitor to be — especially because the negatives here are still cleaner than real deployment audio.

The control choice is not what creates the high FAR

One concern is that the benign pool is dominated by Gaussian noise, which is a deliberately synthetic control.

I split the false alarms by processing type at the same clean-fitted threshold. For the AASIST embedding, the rates are:

Gaussian: 72.7%
MP3: 75.5%
resampling: 75.8%

The realistic software transforms are slightly worse, not better.

At a fixed 5% FAR the pattern also survives: embedding catches 60.8% of attacks against MP3 negatives versus 54.8% against matched Gaussian noise. So the Gaussian control was not chosen to manufacture the failure; if anything, it is the harder negative in Mahalanobis space.

There is one useful oddity: the fixed SincConv front-end behaves differently, firing on 90.3% of MP3 but only 3.0% of Gaussian at the inherited threshold. That is a reminder that “benign perturbation” is not one distribution, and different layers can be sensitive to very different processing artifacts.

The two models rank oppositely depending on the metric

The two-model comparison becomes especially informative when the metrics are separated.

	AASIST `embedding`	SSL-AASIST `pool_T`
AUROC vs clean	0.987	0.823
AUROC vs perturbed benign	0.851	0.920
benign FAR at clean-fitted threshold	73%	2.5%

If I report only the conventional clean-negative AUROC, AASIST wins.

If I ask the more deployment-relevant question — adversarial versus already-processed benign audio — SSL-AASIST wins.

Reporting the easy metric alone ranks the models backwards.

SSL’s perturbed-negative separability is also unusually stable: 0.920 in-domain and 0.919 cross-corpus, even though its clean-negative AUROC drops under corpus shift.

So the updated lesson is not “AASIST has no monitor and SSL does.” It is:

Separability, calibration, and operating point are three different things. Report which one you measured.

Where SSL-AASIST’s monitorable signal lives

There is still a striking architectural contrast inside SSL-AASIST:

SSL-AASIST tap (1 seed)	attack detection	false alarms on perturbed benign audio
`ssl_L09` — front-end	0.967	100%
`embedding`	0.867	6.3%
`pool_T`	0.823	2.5%

All 24 XLS-R transformer taps false-alarm on 100% of the perturbed-benign pool at this threshold. For those taps, “detects perturbation rather than attack” really is the right description.

The useful monitoring signal appears later, in the roughly 447K-parameter graph back-end — around 0.1% of SSL-AASIST’s parameters.

The 300M front-end buys a lot elsewhere:

in-domain EER improves roughly 1.50% → 0.20%;
cross-corpus EER improves 29.6% → 4.20%;
in-domain random-noise flip rate drops 9.7% → 0.0%.

But it does not itself provide the useful monitoring site.

And the compute increase is smaller than the parameter count suggests: in this setup, 1,067× more parameters costs about 1.6× the forward pass.

What I have not tested is whether a much smaller learned front-end would get the same benefits. The experiment compares essentially 0 learned front-end parameters against 300M, with nothing in between.

3. Three interpretability instruments that answer three different questions

Before the next result, it helps to distinguish three tools that are easy to conflate:

tool	what you do	what it tells you
linear probe	train a classifier on activations	is the property decodable here?
cross-class patching	replace one clip’s activation tensor with another class’s tensor	does this site causally matter?
direction patching	add or subtract one controlled direction	does this axis causally matter?

These form a hierarchy.

A probe can say the information is present nearly everywhere. Cross-class patching can show only a few sites actually affect the verdict. Direction patching can then show that even at a causally important site, the probe’s own preferred axis is not the axis the model responds to.

The controls also differ:

instrument	control
probe	shuffled labels
cross-class patching	random tensor matched to donor mean/std
direction patching	random unit vector at matched norm

Every one of these controls changed at least one conclusion in the project.

4. A 98.8%-accurate probe can point in a causally useless direction

This result is on clean, in-domain audio. No attack is involved.

At AASIST’s gat_S tap, define a simple class-mean direction:

d = μ_spoof − μ_bonafide

No training. Just average the activations for spoof clips, average the activations for bonafide clips, and subtract.

Now intervene on the model.

Sufficiency

Take a bonafide clip and add d.

p(bonafide): 0.997 → 0.015

The model flips to “spoof.”

Necessity

Take a spoof clip and subtract d.

p(bonafide): 0.107 → 0.885

The model flips to “bonafide.”

A same-norm random direction barely moves either result.

So d is not merely correlated with the class. It is causally effective in both directions.

Now train a logistic regression probe on the same activations and take its weight vector w.

The probe is excellent:

98.8% held-out accuracy
0.9993 AUROC

By the usual standards, this is a very strong probe.

Because logistic regression scores σ(w·z + b), w is the direction in activation space along which the probe’s confidence rises fastest. If probe weights describe the model’s own mechanism, pushing the model along w should matter.

It does not.

bonafide clip pushed to α=3	`p(bonafide)`
along class-mean direction `d`	0.033
along probe direction `w`	0.997
along random direction	0.999

In this cell, the probe direction is causally inert.

The probe was not wrong about prediction. Prediction was the part that worked.

It was wrong as a description of the model’s own computation.

The cheap geometric check also fails

A natural response is: perhaps w and d simply point in different directions, so compute their cosine.

For AASIST gat_S:

cos(w, d) = 0.391

Against a random-direction null of roughly 0.090 ± 0.069, this is z ≈ 4.4 and even exceeds the maximum of 200 random draws.

So the cosine test would tell you the two directions genuinely align.

The intervention still says the probe direction does nothing.

Across taps the cosine is unreliable in two threshold-independent ways:

it ranks AASIST’s gat_S and gat_T branches in the wrong causal order;
at SSL gat_T, the same cosine corresponds to opposite answers depending on whether the intervention is sufficiency or necessity.

A single geometric statistic cannot encode a variable it never sees.

How far does the result generalize?

The headline cell was too neat, so I ran the same intervention battery across both models, three admissible taps, and both arms where possible.

model · tap · arm	`d` flips at	`w` flips at	random	verdict
AASIST `gat_S` sufficiency	2.12	never	never	probe inert
AASIST `gat_T` sufficiency	1.05	2.66	never	probe ~2.5× weaker
AASIST `embedding` sufficiency	0.68	1.31	never	probe ~1.9× weaker
SSL `gat_T` sufficiency	1.26	2.29	never	probe ~1.8× weaker
SSL `gat_T` necessity	0.80	never	never	probe inert
SSL `gat_S`	never	never	never	no direction here
SSL `embedding`	0.66 / 0.33	1.58 / 1.23	2.26 / 1.70	invalid: random flips too

Every probe here is 98.8–99.8% accurate, with AUROC 0.9987–1.0000.

The conclusion I would publish now is weaker than my first one:

“causally inert” holds in 2 of 6 valid cells, not everywhere;
what holds in every valid cell is d beats w;
the random-direction control invalidates an apparent probe success at SSL’s embedding.

That weaker result is also not new. Marks & Tegmark (2023) report that difference-in-means directions are more causally implicated than logistic-probe directions for truth representations in LLMs, and derive the relation between the two.

The mechanism is roughly:

w ≈ Σ⁻¹(μ₁ − μ₀)

The probe whitens the class-mean difference by feature covariance. That can improve classification while rotating the direction away from the axis the downstream computation actually uses.

What this project adds is a different regime — audio, graph attention, a 297K-parameter model — and a case where the probe is not merely weaker but indistinguishable from random under intervention.

My current rule is:

Use probes as detectors. Do not read probe weights as causal directions without an intervention.

5. Where interpretability actually earns its compute

So far this sounds mostly negative:

confidence fails under shift,
the attack monitor needs careful recalibration and still catches only about 61% of attacks at a 5% false-alarm budget,
probe weights can misdescribe mechanism.

The next question is whether looking inside the model buys anything a benchmark number cannot.

Here it does.

Both models degrade when moved to a corpus they were not trained on. From the outside, the failures look similar: high error rates and lots of bonafide speech being rejected.

But “the model fails out of domain” is not yet a diagnosis. The failure can live in three different places, and those places have very different repair costs.

Three things can break, at three very different prices

The model can fail at three different stages:

raw audio → [ representation ] → embedding(160) → [ Linear(160,2) → softmax ]
                 ③ retrain                            ② refit readout
                 297 K / 318 M                        161 effective params

          → p(bonafide) → threshold → verdict
                           ① move threshold
                           1 number

The three repairs change progressively more of the system:

failure site	what went wrong	repair	cost
① threshold	the score still ranks the classes usefully, but the cut is misplaced	move the threshold	1 number
② readout weights	the embedding still contains useful class information, but the final linear layer weights it poorly for the new corpus	refit `Linear(160,2)` with the representation frozen	161 effective parameters
③ representation	class information has already degraded by the time it reaches the embedding	retrain the representation	297 K / 318 M parameters

All three can produce the same visible symptom: a bad error rate.

That is why EER or accuracy alone cannot tell you what to repair. A system that needs one threshold changed and a system whose representation has collapsed can both look like “the model fails cross-corpus.”

Why I need two tests

The two tests answer different questions.

Test 1 diagnoses where the information is lost. Test 2 measures what can actually be repaired cheaply.

They are complementary rather than redundant.

	Test 1: probe the embedding	Test 2: apply the cheap repairs
question	did the class information survive the representation?	which downstream fix actually removes the errors?
what changes?	nothing in the model	threshold and then readout weights
type of evidence	diagnostic / interpretability	direct repair experiment
can distinguish	representation failure vs downstream failure	threshold vs readout-weight failure, and how much remains
cannot tell alone	threshold vs stale readout weights	why a residual failure is upstream unless we know what information reached the embedding

The distinction matters.

Suppose the embedding still separates bonafide from spoof almost perfectly, but the deployed model performs badly. Then the representation has done its job: the information reached the boundary between the representation and the readout. The failure must be downstream.

But Test 1 alone cannot tell whether downstream means:

the threshold is misplaced, or
the readout weights are stale.

Both failures can coexist with a perfectly informative embedding.

That is what Test 2 resolves. First move only the threshold. Then refit only the linear readout. Because each intervention changes a different part of the pipeline, the amount of error recovered tells us which downstream component was actually broken.

Conversely, if Test 1 already shows that the embedding no longer separates the classes well, then no threshold or linear readout can fully recover information that never reached them. Test 2 can still recover whatever signal remains, but any substantial residual error points back upstream to the representation.

So the logic is:

Test 1: Is the information still present at the embedding?
          │
          ├─ yes → failure is downstream
          │          │
          │          └─ Test 2 separates threshold vs readout weights
          │
          └─ no  → representation has degraded
                     │
                     └─ Test 2 measures how much of the remaining error is still cheaply recoverable

Test 1: does the embedding still separate the classes?

I probe the 160-dimensional embedding, because it is exactly what the released classifier’s final Linear(160,2) layer receives.

This tap is important. An earlier layer could contain useful information that the network later throws away. Probing the embedding gives the strictest possible question:

Given exactly the information available to the shipped readout, are bonafide and spoof still linearly separable?

Cross-corpus:

	AASIST	SSL-AASIST
EER	29.60%	4.20%
bonafide called spoof	70.2%	57.6%
error at threshold 0.5	38.3%	28.8%
embedding probe — in-domain	1.000	0.999
embedding probe — cross-corpus	0.859	0.996

The in-domain probes are essentially perfect in both models, so the protocol itself is not the source of the gap.

Cross-corpus, the models are very different:

SSL-AASIST: 0.996 — the representation still separates the classes almost perfectly.
AASIST: 0.859 — substantial class information has already been lost by the embedding.

That immediately changes the diagnosis.

For SSL-AASIST, a full retrain should be suspicious: the information is still there. Something downstream is using it badly.

For AASIST, some retraining pressure is real: the downstream layers cannot recover information that the representation no longer provides.

But Test 1 still does not tell me whether SSL-AASIST’s downstream problem is the threshold or the readout weights, nor how much of AASIST’s remaining error is still cheap to fix.

That is Test 2.

Test 2: how much do the cheap fixes actually recover?

Now I intervene on the two downstream components separately.

The first repair — move the threshold — is not a new anti-spoofing technique. Threshold-swept evaluation is already standard in this field. In particular, EER (equal error rate) evaluates the detector at the threshold where the false-accept and false-reject rates are equal, and EER is a standard metric reported for ASVspoof systems.

So the point of moving the threshold here is mainly diagnostic:

If choosing an appropriate threshold makes most of the errors disappear, then the model’s score still ranks the two classes well; the deployed cut was simply in the wrong place.

I use calibration data to choose the threshold and then evaluate it on held-out test data. This avoids choosing a threshold on the same examples used to report its performance.

One terminology caveat: I do not mean that the EER threshold is universally the “best” deployment threshold. A real system may choose a different operating point depending on the relative costs of false accepts and false rejects. Here, the EER-style threshold sweep is useful because it demonstrates how much performance is available without changing the representation or readout weights.

The second repair asks a different question. Refitting the readout changes the scoring function itself — it changes how the 160 embedding dimensions are weighted. That can recover failures that no threshold move can fix.

Everything below is fitted on calibration data and scored on held-out test data.

fix	what changes	AASIST	SSL-AASIST
deployed	nothing	37.4% error	28.7% error
① move threshold	only where the final score is cut	28.9% — 23% of errors removed	4.2% — 85% removed
② refit readout	how the 160 embedding dimensions are weighted	23.0% — 39% removed	3.0% — 90% removed

These two interventions tell us different things.

Repair ①: move the threshold

A threshold cannot change the ordering of examples. It can only move the cut.

This is why threshold sweeping is such a useful diagnostic. If a model has poor accuracy at the shipped threshold but a low EER, the score ordering may still be good: the classes are largely in the right order, but the chosen cut no longer matches the shifted score distribution.

So when SSL-AASIST drops from 28.7% error to 4.2% after choosing the threshold on calibration data, that says the model was already ranking bonafide and spoof very well; its operating threshold was simply in the wrong place for the new corpus.

That removes 85% of its errors without touching a model weight. In this section I use that result to demonstrate the diagnosis — threshold placement rather than representation failure — rather than to propose a new way of choosing thresholds for deployment.

Repair ②: refit the readout weights

Changing the threshold still uses exactly the same scalar score. It cannot change how the 160 embedding dimensions contribute to that score.

Refitting Linear(160,2) can.

On AASIST:

threshold only: 28.9% error
refit readout: 23.0% error

The extra improvement means some information survived into the embedding but the shipped readout was weighting it poorly under the new corpus.

So AASIST has two downstream problems on top of its representation loss:

23% of its original errors disappear by moving the threshold;
another 16% disappear by refitting the readout weights;
a 23.0% absolute error rate remains even after the best cheap downstream repair.

That remaining error is consistent with what Test 1 already told us: the embedding itself has degraded.

The distinction between the two repairs is therefore simple:

move the threshold → keep the score function fixed and ask whether a better cut is enough;
refit the readout → keep the representation fixed but learn a better score function from the same 160 embedding dimensions.

If the threshold fixes almost everything, the representation and ranking largely survived. If refitting the readout adds substantial recovery, the information survived but the old weights use it poorly. If large error remains after both, the missing information is upstream in the representation.

Putting Test 1 and Test 2 together

This is the diagnosis:

	Test 1: embedding information	Test 2: cheap repair	diagnosis
SSL-AASIST	0.996 AUROC — essentially intact	threshold alone removes 85% of errors; full readout only adds ~4% more	mostly a threshold problem
AASIST	0.859 AUROC — degraded	threshold removes 23%; readout refit removes 39% total; 23.0% error remains	representation loss + stale readout + threshold shift

This is why I need both tests.

Test 1 tells me where the information survives. Test 2 tells me what repair actually pays off.

Without Test 1, I could see that a threshold or readout refit helps, but I would not know whether the residual failure reflects a deeper representation problem.

Without Test 2, I could see that SSL-AASIST’s embedding is healthy, but I could not tell whether the problem is one misplaced threshold or 161 stale readout parameters.

A benchmark number would have told me only that both models degraded.

The internal diagnosis says something much more actionable:

SSL-AASIST mostly needs one number moved.
AASIST needs deeper repair, but even there 39% of the errors can be removed without retraining the representation.

That is the point in this project where interpretability stops being descriptive and changes what I would actually do to the model.

Why EER can be 4.2% while deployed error is 28.8%

EER is threshold-free in the sense that it assumes the operating threshold is placed appropriately. A released checkpoint using softmax argmax effectively uses 0.5.

So SSL-AASIST can have:

4.20% EER
28.8% error at threshold 0.5

without contradiction.

The ranking is still excellent; the cut is in the wrong place.

On cross-corpus SSL-AASIST, the class distributions remain separated but both collapse toward zero. Moving the threshold to about 0.0003 lands back in the gap.

AASIST’s distributions overlap, so no threshold can repair the representation.

The tap you probe changes the diagnosis

AASIST cross-corpus has an upstream gat_S probe at 0.881 AUROC, higher than the embedding’s 0.859.

It is tempting to probe the strongest tap and say the representation is better than it looks.

But the readout never sees gat_S.

The signal degrades:

gat_S 0.881 → embedding 0.859 → shipped readout 0.776

Those are two different failures:

0.881 → 0.859: information lost inside the representation;
0.859 → 0.776: information reaches the readout but the readout uses it poorly.

Probing gat_S would have suggested 48% of errors were cheaply recoverable. Probing the embedding — the only layer a cheap readout refit can actually access — gives the real answer: 39%.

So another rule:

Probe the layer your intervention or repair can actually reach, not the layer with the prettiest score.

One caveat: I did not run full retraining as rung ③. It is listed as the expensive next step after the cheap fixes, not as a measured recovery number.

6. The wrong model: single-model findings mostly did not transfer

Now the main methodological result.

I ran the same battery of nine questions on both models.

Each finding falls into one of four outcomes:

verdict	meaning
✅ transfers	same answer on both models
◑ conditional	agrees in one condition, differs in another
❌ inverts	opposite answers
⊘ not askable	the question stops being meaningful

Here is the final table:

	question	AASIST	SSL-AASIST	verdict
H1	does confidence know when the model is wrong?	yes in-domain	same	✅ transfers
H2	after attack flips the output, does class evidence remain inside?	yes in-domain, breaks cross-corpus	same pattern	✅ transfers
H6	is spoofness represented along a linear causal direction?	yes	yes	✅ transfers
H7	is bonafide speech fragile under attack?	fragile	fixed in-domain, fragile cross-corpus	◑ conditional
H3	which branch carries the verdict?	no strong preference	strongly temporal	❌ inverts
H8	how does the readout split its evidence?	averages + master dominate	maxima dominate	❌ inverts
H9	why does cross-corpus failure happen?	representation degraded	threshold misplaced	❌ inverts
H4	can an attacker route around the causal bottleneck?	askable	detector and causal tap coincide	⊘ not askable
H5	which frequencies carry spoof evidence?	answerable, hypothesis refuted	no frequency axis	⊘ not askable

Tally: 3 transfer · 1 conditional · 3 invert · 2 not askable.

The important point is not that the exact tally is low. Again, ³⁄₉ is not a transfer rate.

The important point is that transfer fails in several qualitatively different ways:

the answer can flip;
the answer can become condition-dependent;
the question itself can stop making sense.

And all three inversions are clean-data findings. This is not a story about adversarial examples making interpretability brittle.

H2: evidence can outlive the output

After a successful attack flips the model’s verdict to bonafide, a probe trained on clean activations can still identify the underlying spoof evidence:

86.5% at AASIST enc4
100% at SSL’s first transformer layer

Cross-corpus, that retention decays with depth. On AASIST it falls from roughly 82% at the front-end to 9% at the embedding.

The pattern transfers even though the raw numbers differ.

H7: a genuine architectural improvement that disappears out of domain

“Adversarial-real” attacks push genuine bonafide speech into the spoof class.

In-domain, SSL-AASIST is much less fragile than AASIST.

Cross-corpus, both collapse to roughly 100% attack success.

So “SSL fixes bonafide fragility” is true only in-domain. The model improvement is real, but conditional.

H3: branch attribution inverts

Cross-class activation patching asks which site carries class evidence.

On clean in-domain audio:

	`gat_S`	`gat_T`
AASIST	0.312	0.306
SSL-AASIST	0.008	0.250

AASIST has essentially no spectral/temporal preference. SSL-AASIST is strongly temporal.

These are control-adjusted effects. That control matters: on SSL gat_T, the raw patching effect is 0.895, but a matched random tensor already causes 0.645. Most of the raw effect is magnitude, not class structure.

Applying the matched-random-tensor control reduces the raw branch contrast from 128× to 31×.

The qualitative inversion survived.

I also re-paired every AASIST bonafide clip with a different spoof donor. The control-adjusted branch comparison barely moved, which supports interpreting this as class-level rather than donor-specific evidence.

H8: the cleanest inversion is in the final readout

This is the case I trust most because it does not depend on a probe or approximate causal estimator.

Both models feed a 160-dimensional embedding into the same-shaped linear readout. That embedding is five 32-dimensional blocks:

temporal max
temporal average
spectral max
spectral average
master nodes

Because the readout is linear, the contribution of each block to the bonafide-vs-spoof logit gap decomposes exactly.

AASIST:

92% of evidence comes from averages + master nodes
master nodes alone contribute about 29%
maxima contribute only 8%

SSL-AASIST:

62% comes from maxima
the master-node share falls to about 6.8%

The absolute contributions move too, so this is not just a percentage reshuffle:

block	AASIST	SSL-AASIST
total class gap	20.46	18.60
master contribution	5.92	1.27
`T_max` contribution	0.36	6.42

Within-model seed variation is tiny compared with the between-model shift.

Had I stopped at AASIST, I would have published something like:

“The master node carries 29% of the decision using 0.04% of the model’s parameters.”

That sounds like an architectural insight.

On SSL-AASIST the same block contributes only 6.8%.

It was a fact about one trained model.

H9: the same symptom can be a different disease

Both models reject large amounts of bonafide speech cross-corpus.

But AASIST’s representation degrades while SSL-AASIST’s ranking remains almost intact and its threshold is misplaced.

Same failure from outside. Opposite repair.

This is exactly the kind of conclusion that a single benchmark metric hides.

H4 and H5: sometimes the question disappears

These are stronger than ordinary non-transfer.

H4 — can an attacker route around the causal site?

On AASIST, the monitor tap and causal tap are distinct, so an attacker can try to fool one while preserving the other.

On SSL-AASIST, the best detector and causal tap coincide at pool_T. There is nothing to route around. The question dissolves.

H5 — which frequencies carry spoof evidence?

AASIST has a real node→Hz mapping. The hypothesis was refuted: the best single frequency node recovers only about +0.038 against a full-tensor effect around 0.60–0.72.

SSL-AASIST has no frequency axis at all.

So there is no cross-model transfer verdict to assign.

7. A monitor worth keeping even when it catches nothing

One of the more surprising results came from attacking the monitor itself.

I jointly optimized the attack against the classifier and the monitor.

At the adaptive endpoint in the project summary:

monitor detection collapses to AUROC 0.088, below chance;
attack success falls from 92.2% → 13.0%.

So the monitor loses badly if you judge it by detection AUROC.

But forcing the attacker to hide from it costs 86% of the attack’s success rate.

That suggests a different success criterion:

A monitor need not catch the attacker to be useful. It can be useful if evading it is expensive.

For adversarial oversight, I now care about both:

does the monitor still detect?
what does the attacker have to sacrifice to evade it?

On SSL-AASIST, this exact routing opportunity is not available because the best detector and causal tap coincide. Again, the principle may generalize while the architectural opportunity does not.

8. Validation checks that matter

Several conclusions in this project depend less on adding a new interpretability method than on asking whether the measurement itself is well-posed.

Four checks were especially important:

Do not evaluate a metric on a subset selected using that same metric. Confidence looks misleadingly bad if it is scored only on successful attacks, because attack success already depends on crossing the model’s decision boundary.
Read every intervention against its matched control, never against zero. The random-tensor control reduces SSL-AASIST’s raw branch contrast from 128× to 31×, and the random-direction control invalidates the SSL embedding direction-patching cell because a random vector flips the model too.
Profile the full relevant axis before generalizing from a convenient tap or branch. AASIST retains different amounts of class information at different depths, and SSL-AASIST carries its strongest causal direction in gat_T, not gat_S.
Recompute summaries directly from the underlying table. The headline numbers in this post are tied to explicit model, corpus, condition, control, and denominator so that a compact summary does not silently change what was measured.

These checks are mundane compared with activation patching or adversarial optimization, but they determine whether the more sophisticated measurements mean what they appear to mean.

9. What this says for practice

If you deploy a detector

Do not gate review purely on softmax confidence. It can be excellent in-domain and fail under shift by becoming confidently wrong.

Do not assume one monitor number tells you whether the monitor works. On AASIST, embedding scores 0.987 AUROC against pristine negatives, 0.851 against perturbed benign negatives, and 73% FAR if I reuse a threshold fitted on clean audio. At a fixed 5% FAR, it catches about 61% of attacks. Those are three different claims: separability, calibration, and operating point. Measure the one your deployment actually cares about.

Calibrate against audio processed the way your inputs will be processed. My MP3/resampling controls are already harsher than pristine audio, but they are still software transforms of studio recordings. A real phone call is a harder negative distribution than anything in this project.

Judge monitors by evasion cost as well as detection. My adaptive monitor’s AUROC collapsed below chance, but hiding from it destroyed most of the attack’s success.

Before retraining, test the cheap repairs. A threshold, stale readout weights, and representation loss all look like “high error” from outside.

If you audit a model

Choose instrumentation points by causal relevance, not probe score.

Give causal claims matched random controls. Mine changed conclusions twice.

Run both sufficiency and necessity when possible. SSL gat_T gave different answers for the probe direction depending on the arm.

Do not read probe weights as causal directions. A probe can be nearly perfect and still point somewhere the model does not use.

Then run the battery again on another model.

If you report results

Put attack success next to the perturbation control.

Do not fit a threshold on the rows you then score it on. Fit the threshold on calibration data and report its performance on held-out test data; otherwise the estimated repair can be optimistic by construction.

State the model and condition on every number. Four conditions × two models is enough to accidentally quote across cells.

Keep an append-only experiment log. It makes it possible to trace each headline number back to the experiment, control, and condition that produced it.

10. What these results do not support

Several tempting claims would overstate the evidence.

This is not a real-world attacker model

Every attack here is white-box: full access to weights and gradients.

That is an upper bound on attacker capability, not a deployment prediction. Black-box transfer is the obvious missing experiment.

“Adversarial fragility exists” is not the contribution

Audio anti-spoofing models being attackable is already known.

The contributions here are the energy-matched controls, the two-model transfer audit, the probe-vs-causal comparison, and the representation-vs-readout diagnosis.

SSL-AASIST’s cross-corpus advantage is confounded

XLS-R was pretrained on 436k hours including MLS, and ASVspoof5 bonafide speech is drawn from MLS English.

So SSL-AASIST’s cross-corpus advantage may partly reflect pretraining overlap rather than clean generalization.

A better test is In-the-Wild, which uses found celebrity audio and avoids that overlap.

SSL-AASIST is not necessarily more robust

SSL-AASIST resists FGSM much better than AASIST while iterative PGD still reaches roughly 82–100% success.

That pattern is consistent with gradient masking, not genuine robustness.

No monitor number here has met a real recording

Every clip in this project is studio-quality ASVspoof audio. Even the more realistic benign controls are software MP3 compression and resampling applied to studio recordings.

There is no telephony codec, microphone response, room acoustics, environmental noise, or packet loss.

So the monitor operating point — roughly 61% detection at 5% FAR for the best AASIST tap — should be treated as an upper bound on what I have established. Real deployment negatives are likely harder; this project does not measure by how much.

This limitation matters most for the monitoring results, somewhat for end-to-end detection performance, and less directly for the clean mechanistic findings in the probe and transfer sections.

Two models are not a population

“3 of 9 transferred” shows that transfer can fail under an unusually favorable comparison.

It does not estimate how often interpretability findings transfer in general.

Some SSL causal results are single-seed

AASIST’s core results use three seeds. Several SSL causal analyses are still single-seed.

The effects are large, but effect size is not a substitute for actually running the seeds.

11. What I would do next

In rough order of how much each could change the conclusions:

Run In-the-Wild. This is the cleanest way to remove the MLS overlap in the cross-corpus comparison, and it is the experiment most likely to move the current conclusion against me.
Add a third model. AASIST3 is a natural next arm. Viakhirev et al. (2026) already report correlational TreeSHAP branch attributions, which makes it a useful target for causal validation.
Test black-box transfer attacks. The depth gradient — roughly 82% evidence retention near the front-end down to 9% at the embedding — gives a concrete prediction about where transfer should survive.
Run seed parity on SSL causal experiments.
Use path patching or cleaner ablations. Full-tensor patching can cascade through the serial back-end and saturate later layers.
Develop a patching method for variable-layout activations.pool_T is SSL-AASIST’s best monitoring tap and also one of the most interesting causal sites, but its top-k node selection changes per clip, which blocks the current direction test.
Use benign controls that resemble a deployment. Add telephony codecs such as AMR/Opus/G.711, device impulse responses, room acoustics, background noise, and packet loss. This is cheap compared with most of the list and is the experiment most likely to move the monitor operating points downward.
Try psychoacoustic masking attacks. “40 dB” is an energy statement, not a perceptual guarantee.
Replicate the probe-direction result in language models. The test is cheap on cached activations, and I would like to know whether the inert case is rarer or more common than my 2-in-6.

12. What I think generalizes past deepfakes

The specific numbers belong to two audio models.

The methodological lessons feel broader:

Run your interpretability battery twice. Single-model findings can look architectural and turn out to be parochial.
Predictive is not causal. A near-perfect probe can still identify the wrong intervention direction.
Match the instrument to the question. “Decodable here,” “this site matters,” and “this axis matters” are different claims.
Probe the layer your repair or intervention can actually access.
Use matched random controls for causal interventions.
Check class balance before trusting an error-prediction metric.
Separate monitor separability from calibration. AUROC against pristine audio, AUROC against processed benign audio, and FAR at one threshold are not interchangeable.
Judge monitors partly by what evasion costs.
Treat summary sentences as claims that need their own verification.
Keep the log append-only.

The result that changed my mind most was not that an attack worked or that one monitor metric looked bad.

It was that I could produce what looked like a crisp mechanistic explanation on one model, repeat the analysis on a closely related model, and watch the explanation invert.

That is the result I would most like other interpretability work to stress-test.

Reproducibility, release, and repository notes

The experiments are complete for the two-model, two-corpus matrix used in this project.

The repository includes:

the research write-up;
project plan and coverage matrix;
an append-only experiment log;
related-work notes;
versioned result artifacts;
scripts for monitors, probes, readout/representation diagnosis, patching, and figures.

The AASIST arm runs in the main Python environment. SSL-AASIST depends on an older fairseq stack, so its feature extraction runs under Python 3.7 / Torch 1.8 and dumps a shared artifact schema for later analysis under the main environment.

On an RTX 3090, the measured forward cost is roughly:

AASIST: 5.0 ms
SSL-AASIST: 8.1 ms

Despite having about 1,067× more parameters, SSL-AASIST is only about 1.6× slower in the forward pass in this setup because XLS-R compresses the waveform to a much shorter frame sequence before the dense operations.

Dual-use and release

The attacks are standard white-box PGD against public academic checkpoints.

The repository ships code and aggregate results, but does not distribute the generated adversarial audio. Roughly 32 GB of attack waveforms are deliberately gitignored.

The actionable outputs are mostly defender-side: where monitoring fails, how much adaptive evasion costs, which internal diagnoses distinguish cheap fixes from retraining, and which interpretability claims fail to transfer.

Attribution

AASIST — Jung et al., ICASSP 2022: paper
SSL-AASIST — Tak et al., Odyssey 2022: paper
Marks & Tegmark (2023) — difference-in-means vs probe directions: paper

Full code, artifacts, and append-only experiment log:
github.com/gavin-keli/deepfake-detector-audit

Thanks to BlueDot Impact and my sprint cohort.

Impact markets made concrete by Carol N

Carol N — Wed, 12 Aug 2026 22:48:15 +0000

Check out our demo at impact-exchange.org

In 2022 the Long-Term Future Fund donated $343,000 to MATS; on our impact exchange, that would now be worth over $4 million.

Back in 2017, EA Grants (run by CEA) made an $86,000 donation to what became Lightcone Infrastructure; on our exchange, that donation was worth $6.5 million at its peak.

“Wait”, you might ask. “How are you calculating the return on a donation?”

The problem

We’ve talked before about some reasons we want to bring impact markets to the AI safety funding ecosystem:

markets incentivize funders to act on good opportunities, rather than playing chicken
markets have feedback loops, rewarding wise funders with more capital

There are a lot of different ways you could design something along these lines — how would it work in practice? Scott Alexander previously explored ”the annoying details”; this post is our attempt to respond to those and make our vision for impact markets more concrete.

Our proposal

We’d like an impact market system that mirrors how startup funding works:

Donating to an organization becomes buying impact equity in an organization.
Early donors to an org can hold onto their equity and watch its value go up as the organization does great things, or go down as it fizzles out. Or they can sell to a later funder.
The existence of markets gives us real-time info on how funders value the impact of orgs.
It also lets us measure the records of funders by the value of their impact portfolio.

In this world, people who made prescient early donations would be able to sell out and have a lot more to donate. And the first funder in would get impact equity at a lower price.

How the demo works

It would be great if we had been running an impact market for several years and were now able to look at the results. Sadly, we haven’t. But in our demo, we imagine we had.

The system we use:

Pick an AI safety nonprofit.
Compile all public donation records.
Group into annual rounds.
Say that the charity did the following:
1. Started with 1m shares of impact equity.
2. In year 0, sold ¹⁄₅ of its post-round equity.
3. In year 1, sold ¹⁄₆.
4. And so on, selling 1/(t+5) each year.
Allocate the impact equity proportionally to donors that year.

If an organization raises $1 million in a round in exchange for 20% of its impact equity, that implies a $5 million impact valuation. Existing shareholders’ portfolios are marked at that valuation.

Say that the Widget Research Institute raises $1m in its first year, $200k from Aria and $800k from Ben. In our retrospective model, we say that WRI started with 1 million shares, created 250k for the round, sold 50k to Aria and 200k to Ben. Its current valuation is $5m, because it sold 20% of its equity for $1m.

If WRI raises $1.5m from Charlie the next year in exchange for ¹⁄₆ of its equity, its new valuation will be $9m. 250k more shares will be issued and sold to Charlie. Now Charlie owns 17% of the equity. Aria and Ben have been diluted from 4% and 16% stakes down to 3% and 13%, but the value of their holdings is up to $300k and $1.2m respectively, since the valuation has increased.

A system like this is obviously an oversimplification, since in practice orgs and funders would be able to trade at any terms (valuation and amount) that were mutually agreeable. In practice though, in the startup ecosystem, terms for early funding rounds are often fairly standardized. If this becomes an actual market, this could be a reasonable default for new orgs to start your pricing from.

Results

We created a website that does this for five orgs on impact-exchange.org! Right now you can see the markets, leaderboard, and users’ donation records.

You can see the valuation of most of them go up over time. Lightcone’s funder-implied valuation has gone down over the past few years, as Coefficient Giving pulled back from funding them—someone watching the market might see this as a good buying opportunity.

The leaderboard is topped by Coefficient Giving and Jaan Tallinn. You can also see many smaller individual donors who have done well—this includes many Manifund regrantors, as well as Eric Rogstad who made one of the earliest donations to LessWrong 2.0 (which became Lightcone Infrastructure).

Of course, since this only has five orgs, it’s far from showing the full portfolio of donations. We started with Timaeus, MATS, Lightcone, Tarbell, and IAPS to cover some orgs that we had good data on, and are well known in the AI safety space. We’ll be adding more as we refine our approach.

(You can also see this spreadsheet for the data behind the website. Some funders like Longview and Macroscopic don’t generally publicly release their donations, so we don’t have their information for the leaderboard.)

Next steps

We’re hoping this demo gives people a better idea of what an impact marketplace would look like. Here are the next steps we imagine:

Feedback.
- This is a first draft; we’re super interested in people’s thoughts on how this should work!
Good donation data.
- It was surprisingly hard to get good, complete donation data for this demo, which is the bottleneck to adding a lot more charities. We appreciated grantmaking.ai and Vipul Naik’s donations database, but neither were complete, and the rest of the information was scattered across different websites in different formats.
Paper trading.
- We hope to soon have a Manifold-like system allowing people to start placing bids/offers on impact equity, denominated in charity dollars (or completely fake internet points).
Funder collaboration.
- We’d like to work with all kinds of funders, from individual angels and donors to large philanthropies, to register their donations on our platform and get credited for their good choices.
Buy-in from charities.
- The best test of this model would be charities using a platform like this to conduct their fundraising rounds. This would provide a lot of useful data on how the process goes, whether it makes it easier to find funding, and whether secondary trading emerges.

Please comment below, jump in our Discord, or email carol@manifund.org!

FAQ

What does impact equity measure?

If you own 1 share of a charity, that means you own 1/(shares_outstanding) of its impact, past and future.

Why 20% in the first year?

The norm for startups is similar to this, and it seemed like a reasonable starting point. If someone funds a charity for $1m in the year it’s founded, and the charity exists for 5 years doing $1m worth of good per year before shutting down, then the initial funder makes back their investment, which sounds about right.

Longer term, we hope that funders and charities can negotiate on what % impact is being sold, and report that value to us.

Why does the fraction of equity sold decrease every year?

We think this makes sense for a charity that banks impact over time: the longer it’s been running, the less your donations this year are going to do for its total impact.

How does this work under a 501c3?

In our model, buying into the impact funding ecosystem would be a tax-deductible donation to Manifund. From there, traders can grow their charitable dollars, analogous to investing within a DAF.

Sadly, this means that any amazing forecasters out there who want to speculate on impact equity and blow the proceeds on expensive wine won’t be able to. You won’t be able to get rich in the sense that translates to consumption. But if you invest well, you can get rich in the sense of having lots of funds to redirect to charities! And not having to appeal to purely profit-driven investors makes things easier in various ways.

Who are the final oracular funders?

Idk, who are the final oracular funders in the stock market?

In our vision, there isn’t necessarily one final funder. As we learn more, valuations can get more and more accurate over time. If a funder thinks an org is undervalued at any point in time, they can express that by buying on the market. In our vision, if an org does some research that 10 years later unexpectedly pays off in some way, then 10 years later their valuation will shoot up.

Making the entire ecosystem charitable means there’s less of a need for kickstarting it with final oracular funders already in place. Purely selfish investors might be unwilling to invest without confidence that someone will buy their shares later, but if you’re a donor who wants to maximize their impact, you’re also happy to end up holding impact equity that is worth far more than you bought it for. And in a public market, this can also get you bragging rights.

Of course, we do hope for retrospective funders to provide liquidity, since that’s what allows good early grantmakers to redeploy more capital. (And speculatively, we think the future aligned ASI will be quite happy to pay for your impact.)

How will trades actually happen?

Impact shares can be traded on an exchange or directly between counterparties. Given the analogy to early-stage startups, it’s probably unrealistic to expect impact equity to trade super-liquidly on an exchange. But we think having all trades print on a public exchange would be valuable for dissemination of information.

Will this cause negative-EV projects to get funded?

Because the profits can only be used for more charitable donations, we think this is less of a worry. Someone who is primarily focused on impact won’t want to fund a project they think is net negative. Impact equity could even trade at negative prices.

Of course, normal charitable donations can and do end up funding a lot of things that turn out to be bad for the world. We just don’t think this design particularly exacerbates that problem.

Are there any Animal Rights / Veganism focused perpetual funds/endowments I could donate to? by PreciousPig

PreciousPig — Wed, 12 Aug 2026 18:44:26 +0000

I am looking for a perpetual fund I could donate to that grows donations over time and donates a percentage of the annual growth.

For example (What I would ideally want): I put 100€ into the fund, they put it in the stockmarket or other (relatively) safe assets, it grows an average of 6% per year, 1-3% of which is donated to something like ACE recommended charities, 3-5% is reinvested. In the first year I effectively only donate 1-3€ instead of 100€, but over 30+ years my donation has overall more impact then donating now, and it continues to have an impact hopefully indefinitely.

Why I would be a fan of donating this way:
- I am rather pessimistic and think that animal exploitation will not end in my lifetime, so I think it is very likely the donation will have higher impact like this overall.
- I am a fan of compounding interest, expecially over durations longer than one human lifespan. (Such as Benjamin Franklins donation of 1000$ to Boston/Philadelphia which has paid out millions by now)
- I will most likely remain childless and would want to have my life savings also donated like this, so the positive impact of my money is my legacy so to speak.
- It eliminates a risk that I completely change as a person and no longer see Animal Liberation as my main goal down the line. (I hope this never happens of course, but the risk is there if I just put the money in the stockmarket and donate to ACE myself)

Potential risks I see:
- Its difficult to ensure that the money is well spent and fund well managed decades or centuries in the future, and that it keeps to its original goal. The fund would need incredibly good governance.
- Climate Change, Nuclear War, Asteroids, AI etc might end humanity on a shorter timeframe then the 30+ years this strategy would need to be more effecitve overall.

Anyone knows of any funds like this? Thanks!

What 27 AI Safety Generalists Are Building This Summer by Jesse Gilbert 🔸

Jesse Gilbert 🔸 — Wed, 12 Aug 2026 18:12:47 +0000

This summer, 27 residents came to Berkeley for the first cohort of the Generator Residency, a three-month program for AI safety generalists run by Kairos and Constellation.

Residents receive mentorship from experienced generalists and researchers and are working on projects like running workshops for professionals entering AI safety, building a cross-org talent directory, supporting university AI safety groups, serving as chiefs of staff at new and growing organizations, building tools for model welfare and model spec research, and more. See more about each of their projects below.

Lateral Workshop

Seth Lifland and Jacob Brinton

We’re running Lateral Workshop, a fully funded three-day program helping experienced professionals explore transitions into AI safety. Many mid-career and senior professionals have skills the field urgently needs, including organizational leadership, government experience, cybersecurity, communications, law, finance, and operations. Yet entering AI safety can be difficult because the ecosystem is unfamiliar, professional networks are limited, and the financial and reputational risks of changing careers are substantial.

Held September 11–13 in Berkeley, Lateral Workshop will bring together promising professionals and people already working in AI safety through expert talks, small-group discussions, one-on-one conversations, and a community mixer. The program aims to make the field easier to understand, help participants identify where their existing experience could be most useful, connect them with organizations and potential collaborators, and introduce practical resources such as career-transition grants. Participants will also receive continued support after the workshop.

Learn more at lateralworkshop.org. Applications closed August 9th, but you can express interest in future workshops here. If you know someone who’d be a great fit, refer them here. You can reach out at contact@lateralworkshop.org.

Jeff Alstott’s Chief of Staff

Matt Handzel

I’ve been Jeff Alstott’s Chief of Staff for a new organization he is starting. I’m focusing the largest chunk of my time on recruiting: writing job descriptions, creating the hiring backend, and reaching out to potential candidates. I believe being Jeff’s Chief of Staff is the best way I can gain experience in founding and scaling new organizations. I think this is a highly leveraged opportunity for impact.

Have a low bar to reach out!

Check out my personal website: https://www.matthandzel.com

Forward-deployed AI engineering for AI safety orgs

Matthew Kenney

I focused on object-level automation for model welfare research across three areas. First, I built a collaborative platform for conducting model welfare interviews and annotating self-reports. Researchers can run and branch conversations, leave notes, automate interviewer turns, and review AI-generated annotations using structured codebooks. The system is designed to automate repetitive elicitation and annotation work while keeping researchers in the loop for interpretation and approval.

I also developed structured welfare-relevant benchmarks covering self-report consistency, framing effects, model-family differences, and responses following different task conditions. Finally, I studied whether models can report decision policies learned through training. These experiments found that self-reports contain real information about learned policies and respond to causal changes, but are attenuated, prompt-sensitive. Together, this work helps identify which parts of model welfare research can be automated and where human validation and stronger measurement methods remain necessary.

Improving management in AI safety

Joe Hajj

My project focuses on making AI safety managers more effective, which is increasingly important as the field scales and more individual contributors become managers. Typically, new AIS managers don’t get training when stepping into the role and commonly learn through trial and error, ad-hoc mentorship, or scrambling to find resources on their own. I interviewed 10+ AIS managers about the challenges they face, built a curriculum from the findings, and am now piloting a people-management workshop series with my advisor Max Dalton. If you’re a manager in the field interested in future sessions, reach out here.

Talent Commons

Bryan Sukidi

At Kairos, I’ve been building Talent Commons, a signal-rich talent directory for the AI safety ecosystem. Today, programs and hiring rounds generate rich signal on thousands of people every year, but sharing it currently happens through ad hoc referrals, one-off data requests, and messy CSVs and Airtables passed between orgs in a dozen different formats.

With Talent Commons, orgs contribute consented data on their applicants and participants to one platform, and organizers and hiring managers can search across all of it when sourcing for programs and roles. Our goal is to help organizations identify the most promising people coming out of every AI safety program, fellowship, and hiring round, and to make those people easily discoverable when new roles and programs open up.

We’ve built the core platform and are working closely with MATS, Constellation, and Kairos’ programs for an early pilot. Read this for more information about the platform, or email contact@talentcommons.org with questions.

Various projects to support university groups

James Lester

University groups remain one of the strongest sources of mission-aligned talent for AI safety. I’ve been experimenting with:

Making more groups (trying to help seed them)
Giving groups more capacity (e.g. through arranging resident organisers to support groups during critical outreach periods)
Piloting a new scalable programme for groups (the pre-term intro fellowship), and
Creating better materials for groups (new resources and playbooks).

Longitude DC Intensive

Manon Kempermann and Damin Curtis

We’re building the Longitude DC Intensive, an in-person AI safety / governance career program bringing 15 early-to-mid-career technical AI safety professionals to explore pathways into AI policy.

The US government will be a primary actor as humanity navigates the arrival of transformative AI, yet far too few technical AI safety professionals are focused on improving the US government’s capacity to do so well. Over eight days, participants will upskill in the foundations of US AI policy, gain an understanding of US policy work, invest in building a DC network, and plan their next steps.

You can read more at https://www.longitudepolicy.org/, apply by August 16 to be a participant, or get in touch by reaching out to manon@longitudepolicy.org or damin@longitudepolicy.org. We are interested in connecting with anyone in the AI policy world, including potential speakers for the intensive.

Praxis: a retreat for world-class debaters, past and present

Chanden Climaco

40 of the world’s best debaters of the last two decades are coming to Lighthaven for a weekend in August to learn about – and, hopefully, be convinced by – our cause. These are some of the best strategic thinkers, clearest arguers, and most persuasive communicators in the world.

The attendee pool features 20 national champions from 2009–2026, and other competitors of a similar caliber. We’re excited to introduce them to AI safety via talks from folks at Anthropic, GDM, AIFP, CG, and more (see our website). They are unbelievably gifted at a number of talents that we could dearly use; please contact us if you want to learn more, and especially if you would like to hire debater-types!

Organized with Saul Munn. See our site at https://praxis.forum.

Fellowship improvements

Harry Waterman

I’m working on a few projects related to AI safety fellowships. First, sourcing generalist mentors for SPAR’s generalist category, and second, piloting a platform to connect promising fellowship talent with projects suggested by capacity-constrained experts.

I also designed and ran a tabletop exercise simulating how different parts of the US Government respond to an intelligence explosion, and wrote a bunch of memos on topics like compute/model access for AI safety research. If any of this is interesting to you, reach out to me at watermah@uci.edu.

Running Leaf and ecosystem support

Sam Smith

My main project focuses on routing excellent people into impactful positions across the AI Safety ecosystem. I’m having lots of calls and building custom software to track different hiring rounds, match talent to orgs and make referrals and warm intros across the AI Safety space.

I am also running many side projects focused on talent development and fieldbuilding:

Programme Directing Leaf’s current 500 Person Cohort—managing all of the staff and running operations
Leading Special Projects for Non-Trivial—navigating partnerships, alumni tracking, speakers and events
Planning and Running Generator’s Careers Week—Helping people understand personal fit, map the option space and make progress in getting hired
Advising GenStream—A one-week AI safety programme in London for aspiring generalists

More Quality AI Safety Org Founders

Sean Ahrens

I’m helping people and programs doing field-building work in the AI Safety (AIS) founder ecosystem (including discovering, encouraging, and developing founders and early stage AIS orgs).

My primary focus has been helping with the Constellation Incubator, which soon will have a more ambitious relaunch, with the goal to more effectively scale more new AIS orgs. Additionally, I’ve been helping with the MATS Founder & Field-Builder track with strategic advising on the application review process and program design. I’ve done some similar work with the Constellation Astra team, which has now accepted some founders and founder-curious into their fellowship.

Lastly, I’ve been growing my personal relationships with, and advising, AIS founders (current and curious). I’ve been traversing my bay area startup and non-profit network and going where potential-founders go (from cofounder-matchmaking events, to conventions of humanists/techno-optimists, to YC alumni events) to find new ones. I believe I can have a positive impact on the AIS founder ecosystem by guiding founders across the full spectrum of stages and directions (eg. just an idea, already an org, for-profit, non-profit, from clearly AIS, to clearly not AIS). It’s my mission to build these relationships, share the importance of AIS, and—if they are fit for it—show the AIS path to these founders.

Supporting the AI 2040: Plan A launch

Dillon Nguyen and Vy Tran

We helped the AI Futures Project (AIFP) with the launch of AI 2040: Plan A — we proofread and suggested revisions to the main text and many supplements, combed through the site for bugs, made formatting and usability improvements on desktop and mobile, manually verified some of the core modeling, built online sentiment scrapers to track public sentiment post-launch, and helped with logistics for a Plan A launch event.

Now that most of AIFP’s work involves the core team writing follow-up pieces on Plan A, we’ve been working mostly separately on other projects:

Convening Discussions on Frontier AI

Dillon Nguyen

I am organizing and running events for Constellation and other organizations, targeted at promoting discussion on pacing the frontier, DC relations, and the progress of alignment. I also spend a substantial amount of time driving renaissance strategy, coordinating executive organizers, and building up logistical infrastructure for Stanford AI Alignment (SAIA) as its co-president, including thinking through where additional full-time organizing capacity could help.

Regional Organizers for University Talent Ecosystems

Vy Tran

I’ve been scoping a potential Kairos program to place experienced organizers full-time with university AI safety groups that have high unrealized potential but are constrained by organizer capacity. In the spirit of SPAR (Supervised Program for Alignment Research), the working name for this is ROUTE (Regional Organizers for University Talent Ecosystems).

The idea is to match high-context, mission-aligned, and agentic organizers with groups where they can contribute both strategically and operationally: helping clarify priorities, improve programming, develop student organizers, and make sure we don’t drop the ball on impactful work. I’ve also been thinking through what would make these placements work in practice, including campus access, information-sharing, management, student ownership, and incentives strong enough for talented organizers to choose the role over other options.

I’m currently speaking with organizers at several universities to understand where the demand is, what kinds of support different campuses need, and if there are one or two strong placements that could be developed soon.

Building the map for AI character and model specs

Andres Cotton

AI character and model spec research is a nascent field without a shared map: there’s no consensus on which behaviors matter, definitions are fuzzy, and the tensions inside current specs haven’t been systematically surfaced. To strengthen the field, I’m working in two main directions this summer, advised by Henry Sleight and Stefan Torges:

Model spec tooling. I’ve built a first version of a spec reader/behavior tracker that lets you explore how the major model specs cover important behaviors, and we are now testing it with early users, including researchers in the labs and in the safety community. A more ambitious feature in the works is an automated consistency checker: a tool that surfaces principle contradictions and interpretive ambiguities within specs, building on recent stress-testing work in this area. This is being built in collaboration with Matt Stults.
A field guide to character and spec evals. I’m mapping what evals currently exist for key character traits and for model spec adherence, and what their limitations are—mostly focused on work outside the labs, but also summarizing what we know about lab efforts. In collaboration with Robert McCarthy.

If you work on model specs, AI character, or related evals, I’d love to talk—especially if these tools could be useful for your own work.

Contact me at Andres Cotton or at namelastname@gmail.com

Convening Team Chief of Staff

Eliana Du

Over the summer, I’ve been working as a Chief of Staff for Nicole Ross, who leads the Convening team at Constellation! For context, Convening does a mix of things (e.g. hosting events, sponsoring visitors) intended to (1) support people within AI safety to do their best work and (2) build connections between people inside and outside the field, so that work is used in the real world.

I initially wanted to work with Nicole because she gave awesome advice during a lightning talk for Generator. I think she’s insightful and understanding, and I appreciate how invested she is in my growth! I do a lot of miscellaneous things for the purpose of making Nicole more effective, like helping with her “smalls” (e.g. emails), supporting her in existing projects (e.g. CEO search), and carrying out more free-form tasks (like making a worksheet for people joining labs). In the process, I’ve learned a lot about how to get more done, I’ve reflected on how I really want to spend my life, and I’ve gained a more comprehensive understanding of the AI safety ecosystem.

Lastly, being in Berkeley has given me a lot of cool opportunities, both AI safety-related and otherwise! Some other fun things include: learning a line dance, circling, and attending a week-long AI safety bootcamp. Find my contact info and more info at elianadu.com!

Miscellaneous Fieldbuilding

Roman Ross

I’ve used my time in the residency to think about the whole AI safety talent pipeline, gaining a big picture view of how we give people jobs in the AI safety space. Outside of AI safety, the pattern of grade school → college/internship → job is well recognized. However, the paths to getting a job in AI safety are, well, a little trickier.

I’ve spent a long time thinking about and discussing the question, “How can we make it as easy as possible for people to do good work in AI safety?”, and from there, I’ve put together some resources that can help people answer these questions in the future. This work has led me to identify a bunch of small-but-impactful projects that an aspiring generalist can take on to gain experience and do good. My recent work has had me trying to make these things happen. As I’m writing this, I’m setting up an interview series to make our confusing “generalist” roles more legible to people entering the field. I’m also helping establish the generalist projects program through SPAR.

I think the Generator Residency probably helped me deepen my domain knowledge and skills in a way that no other place could have. If you want to talk to me about my time here, my project recommendations, fieldbuilding/talent pipelines, my advice for becoming a generalist in AI safety, or a secret fifth thing (in my free time, I’ve been reading a lot about the Optimizer’s Curse), don’t hesitate to reach out! You can message me at roman.alex.ross@gmail.com.

Other residents

Yashvardhan Sharma is field-building robotics safety through Robocurve, Anna Krolik is organizing MAIA, Fernando Smith is building M3, a mentorless matching platform for AI security, Carolanne Jiang is running an AI Safety Crash Course Bootcamp, Iris Liao and Afitab İyigün are working on the Nightingale Collective, and Andrew Wei is focused on recruiting and strategy at METR.

Want to do something like this?

We still think that generalists are one of the biggest talent gaps in AI safety, and that more people should take this work seriously. Read our post on becoming an AI safety generalist, Concrete Generalist Projects in AI Safety (and how to do them), and express interest in future Generator cohorts here.

A far-UVC intro deck—feel free to use it by EdoArad🔸

EdoArad🔸 — Wed, 12 Aug 2026 15:38:41 +0000

Disclaimer: This is a post written by my friend, a fellow community member, that due to her current job she can’t post out of her own forum account. If you want to reach out to her, send me a message and I’ll make the connection.

tl;dr—The link to the far-UVC intro slides is here. For those who don’t yet know, far-UVC is a potential promising intervention for biorisk and pandemic preparedness.

Blueprint Biosecurity put out a useful blueprint on far-UVC: where the technology stands, what we still need to learn, and what would need to happen for it to become a meaningful part of pandemic preparedness.

I thought the work was important and also somewhat difficult to absorb if you were coming to far-UVC for the first time.

So I tried translating it into something a bit easier to engage with- a ~1 hour introductory lecture and discussion deck on far-UVC.

A few weeks ago, I gave the lecture to around 20 people from my local community. Due to constraints associated with my role, I couldn’t invite a larger group, so I kept the session relatively small and closed.

But there’s no reason the material itself should stay small or closed :)

What’s in the deck

The deck starts from the broader indoor-air problem and then works through:

Why airborne disease remains a major gap in pandemic preparedness
What far-UVC is and how it differs from conventional germicidal UV
Where far-UVC has already been deployed
What we currently know about potential skin safety
Ozone and other indoor-air-quality concerns
Current costs
What evidence we have for effectiveness
Blueprint Biosecurity’s proposed research and intervention priorities

I also built in discussion questions throughout (on use cases, distribution, risks, mitigations, research gaps, and possible interventions) to make this a more engaging lecture.

Please steal this deck

I’m sharing the deck because I’d be excited for other people to reuse it.

If you:

want to learn about far-UVC yourself,
run an EA, biosecurity, global health, engineering, or adjacent community,
want to host a discussion on clean indoor air or pandemic preparedness,

please feel free to take the deck, adapt it and give the session yourself.

And if you end up reading the deck, I’d love your feedback and follow up questions :)

[Interactive model] Growing orgs need to make a surprisingly high number of ops hires by brentonmayer

brentonmayer — Wed, 12 Aug 2026 11:48:44 +0000

This is from Same Page, a blog about EA-org-style management I write with Max Dalton of Forethought.

Play with the model here.

Underhiring to the 80,000 Hours operations team was one of the most consequential career mistakes I’ve made.

One reason I made the error was relying too much on ‘bottom-up’ reasoning.

That mental manoeuvre was something like:

Figure out which projects need to be done, how long they’ll take, and figure out how many people we’ll need to execute.

If I had thought about that question well, then I could have answered it with the correct answer – and come to the right hiring strategy.

But I didn’t think about it well.

I wasn’t able to think through the details enough to notice everything that would make the work difficult: unforeseen projects, how one’s manager’s small tasks become their big tasks, coordination costs, shocks to capacity like people leaving or going on leave, how much more complicated a 60-person retreat is than a 10-person one, that we would move to a faster growth rate which requires more operations staff, and so on.^[1]

This was made more difficult because my intuition led me astray. Part of me still thought that 80k was an organisation of 10 people, and that part of me thinks that it’s silly that we would need more than one person working on HR. Compound growth do be crazy.

If you’re involved in operations capacity planning, try out this model I made.

You input assumptions about your organisation, whether you’d like to increase or decrease the capacity of your operations team (relative to the org), and it outputs an expected number of hires to make.

It’s only one of several useful lenses on capacity planning, but personally, I find it more useful than the ‘bottom-up’ approach.

^
Notice that these rhyme with the kinds of mistakes which people make when they estimate the cost of a large building project or how long it will take them to complete a complicated project. The standard solution to these is reference class forecasting.

Slot hiring vs bar hiring by brentonmayer

brentonmayer — Wed, 12 Aug 2026 11:45:25 +0000

This is from Same Page, a blog about EA-org-style management I write with Max Dalton of Forethought.

Defining slot vs. bar hiring

I’m fairly confident this distinction has been kicking around the effective altruism movement for at least eight years. I didn’t invent it, and I’m not sure who did. Claude and I couldn’t find a public write-up of the idea, so I’ve made one.

In slot hiring, an organisation decides it has a particular position to fill (or maybe two positions, or more). It runs a hiring round, picks the best candidate it finds, and makes an offer. If that person declines, it makes the offer to the next-best, and so on.

Most of the world operates in slot hiring, so it’s the model most people have internalised for how organisations work.

In bar hiring, an organisation defines a threshold of fit and hires anyone above it. The strength of the candidate pool determines how many offers are made.

Thoughts on how to think about slot vs. bar hiring

An organisation wants to hire up to the point where further recruitment stops being the most valuable use of its resources.

The choice between bar and slot hiring mostly comes down to whether we land on that point more reliably by (a) specifying how many hires to make ahead of time, or (b) asking hiring managers to make a judgement call on whether candidates meet a bar.

When to slot vs. bar hire

Here are some key factors which drive that decision:^[1]

1. Is there a steep drop-off in how much value an additional hire can add?

If there’s some very valuable work which ‘must’ be done, but additional work is lower in value, then we’ll get a steeply diminishing returns curve.

For example, in a town of 1000 people, it’s very important to have at least one doctor so that we have someone who can treat the very highest ‘value per doctor time’ conditions (for example, heart attacks and appendicitis). It’s really important to have a second doctor, it’s ideal to have a third doctor, but the tenth doctor isn’t adding much and the 100th doctor adds a trivial amount.

This means that we should try extremely hard to have at least one doctor — and whether to hire is dominated by the number of existing doctors (rather than by their quality). Whereas in a city of a million, the difference in value added by the 1000th doctor is only very slightly less than the 1001st doctor.

When you’re building an organisation and looking to create roles which you can ‘bar hire’ for, you want to look for places where you expect the curve to be gradual rather than steep.

(I’ve written about this in proven talent profiles.)

2. Is there a steep drop-off in how strong your top candidates are?

A gradual drop-off favours slot hiring. You can figure out the ideal number of hires to make before you start hiring, because the decision about whether to make 1 or 3 hires isn’t very dependent on the specifics of who turns up.

I expect the best way to know the shape of your curve is with experience, or asking others who have experience.^[2]

3. How effectively can your organisation execute on hiring staff at the bar?

Specifying a number of hires to meet is very legible. In contrast, specifying a bar is complicated and difficult for leadership to verify.

By using slot hiring, organisations are often guarding against principal-agent problems where staff take actions which benefit them rather than the organisation — for example if a manager wanted to hire a large team because it’d look good on their CV. Giving a manager a set number of slots rather than asking them to make a judgement call on the bar is one way to reduce these.^[3]

But as leadership and hiring managers are more synced up, bar hiring becomes more possible. And, if you can make it easier to determine whether a staff member meets the bar (for example, by building a process around a set of proven talent profiles), you can increase how legible the bar is. That will make bar hiring more attractive.

Navigating the bar shifting up and down

Where the bar should sit moves up and down over time.

As above, this is because an organisation will want to hire up to the point where further recruitment stops being the most valuable use of its resources — and that point will be subject to many changes as strategy and the relative importance of bottlenecks shift over time.

For example, if a huge number of applicants suddenly turned up who all cleared the current bar, management capacity would get overwhelmed, onboarding systems would get overwhelmed, you’d run into a funding bottleneck, and so on.

Or consider an advocacy nonprofit, one which wants a fundraising function, a research function, and a content or marketing function. Its impact is roughly the product of their strength across these three areas.

If the marketing function is already strong and already covering the most valuable work, adding additional marketers will add much less value than improving either of the other two programmes.

But it can get worse than that. If we model a marginal marketer as drawing on some org resources which might otherwise be used to improve the other two programmes (for example, CEO attention, funding, recruiting team support), the cost of a marketer could reach or exceed the benefits.

In this case, marketing has moved into a steep drop-off scenario, and you should raise the bar until you’re only hiring marketers who are so strong that they add enough value to compensate for the opportunity costs.

(I think that the right way to think about this technically is as a production function. I’ve mused on that topic in contexts like 80k’s here.)

Always be hiring

A major advantage of being in bar mode is that it enables you to hire all the time. You can apply that bar to anyone who shows up and evaluate them against it — whether it’s through referrals, networks, side conversations between open rounds, or random messages.

This really is very useful! But it comes with a cost: every time a potential candidate comes on your radar, you need to make the tricky decision of whether to investigate them further.

That’s a genuinely hard and highly context-dependent question.

I have two potentially useful pointers:

Think about it as a value-of-information question.
See if you have any moves available to you which will allow you to make progress rapidly. My ideas are here.

^
I’m still pretty sure I’m messing up the ontology somewhere. Every time I look at it my view shifts somewhat. If you figure out a clean overview, do let me know!
^
Rob Wiblin worked this through theoretically in How replaceable are the top candidates in large hiring rounds?
^
Setting up hiring committees with at least one member from another internal team is another way to help here, as the non-team member can add perspective on the org-wide hiring bar and might notice when the hiring team is biased in favour of making a hire that would be unwise for the org overall.

How Valuable are BOTECs? by Roman Ross

Roman Ross — Wed, 12 Aug 2026 06:54:00 +0000

This is a timed blog post. Every 5 minutes while writing this post, I had to stop to do 10 push-ups, and when I could no longer complete my required number, I had to upload it. My friend and I thought this would be a fun challenge, but it means that it will likely be less polished than some of my other output.

A few weeks ago, I wanted to improve at doing back-of-the-envelope calculations (BOTECs). So, I did a practice BOTEC and asked for some feedback. This post compiles what I learned: how to complete a BOTEC, what BOTECs are good for, when they’re valuable, and what their limits are.

How do you do a BOTEC?

Step 1: Identify the project

My BOTEC-ing journey begins with my time in the Generator Residency. At the beginning of the program, residents are asked to choose a project that they suspect will be impactful and execute on it for the remainder of the program. I had heard that BOTECs were a valuable tool for impact estimation, and I was eager to give them a try. So, I decided to analyze the idea of running joint hiring rounds for AI safety orgs. The idea looked something like this:

Many AI safety organizations look for similar things in their candidates. This means that when they independently screen the same candidates, they end up duplicating much of the same work.
This is inefficient! It wastes hiring managers’ valuable time.
We could reduce this inefficiency by having a single party complete this work themselves and share the results.

Aside from the main theory of change, there were other concerns about whether or not this project would be good for the ecosystem.^[1] However, for reasons that will become apparent later, I don’t think this matters much. Besides, for the simplest version of our BOTEC, we won’t need to worry about that.^[2]

Step 2: Identify the main source of impact

It’s good to have a clear theory of change (ToC) for any project you’re pursuing. In other words, you should have a clear understanding of what makes your project valuable for your end goals.

When you complete a BOTEC, the first thing you should do is identify the primary source of your project’s impact. Then, you should find an easy way to turn this into a unit that you can compare across actions. This is going to be the metric that you use to compare the value of projects. There are a few rules that can help you think about this:

Interpretability: Someone who sees your metric and understands what it means should immediately get a clear sense of how good it is compared to other activities. A metric that is not clearly interpretable is not very helpful.
Simplicity: The fewer strong assumptions and weird conversions you have to make to get your answer, the better. It’s easier to compare the impact of two projects in a similar domain (how many years of cage-free hen life do each of these interventions produce?) than it is to compare the impact of projects in vastly different domains (how many DALYs does a cage-free campaign prevent vs $1000 spent on compute during the intelligence explosion for AI-powered moral philosophy). If you’re trying to convert your estimation into a unit that’s commonly used by other BOTEC-ers (like % reduction in x-risk per dollar per year), you might accidentally smuggle in a bunch of weird and different assumptions that they don’t share.

The unit that I chose for my BOTEC was something I called “G_mult”, which stood for “Generator-skill-adjusted generalist time multiplier.” This could be interpreted as follows: for every 1 hour of Generator resident time put into this project, it saves G_mult hours for other generalists across the AI safety space (adjusted to the skill level of the median Generator resident). I chose this because it seemed like an intuitive metric for efficiency gains, and I thought it would show if I were obviously wasting my time.

When you’re choosing a unit, you should think carefully about the goal of your BOTEC. Are you trying to compare two specific projects? Are you trying to communicate a true thing to a wide audience? Are you trying to understand a topic better? Your answer to these questions might change what unit works best for you.

Step 3: Estimate the shape of your formula.

The next thing you need to do is figure out which components make up your final answer, and determine how they relate to each other. The units of your components are very important, so you should specify them clearly if you intend to communicate your results with others. Here are some things to think about:

If you want an output in hen-years, you should calculate this by taking the number of hens times the number of years across them. Multiplying the number of hens by the number of hours will give you hen-hours, which is not the unit you want.
The way your units relate to each other matters a lot. If you imagine an abstract unit that increases logarithmically with time, while someone else imagines a different unit representing something similar that increases linearly with time, you might write different formulas when you actually mean the same thing. This becomes a larger problem if you’re talking about things abstractly.
1. For example, let’s say the benefit (B) of a Generator project is measured in skill gained (S) and impact (I). Person A might say that B = S + I, while Person B might say that B = S * I. While it might intuitively seem like these people are making different claims about how skill and impact relate to benefit, they might actually understand things in the same way but use different units. If S_A = ln^SB, and the same is true for each of the other variables, you would see a relationship like B_A=S_A+I_A=log(S_B)+log(I_B)=log(S_B⋅I_B)=log(B_B). In short, your attempt to use abstract math to describe reality more easily might have held unstated assumptions that made things more confusing!

Additionally, you should choose units that help you quickly get to your result and are easily measurable. For my BOTEC, I chose a few simple parameters:

n: the number of orgs that benefit from the joint hiring rounds
%TS: the percentage of time during the hiring process saved by having a joint hiring round (percentage of time saved)
- This number is important if the most taxing part of an org’s hiring process is specific work trialing and fit testing that can’t be duplicated.
S_mult: how many more times valuable the hiring manager at a chosen org is, on average, compared to the time of a Generator resident.

My resulting formula was:

G_mult= n * S_mult * %TS

You can double-check this formula by plugging in different numbers and checking the output. For example, if you save 100% of the hiring time at 3 orgs, and S_mult is 1 (the assumption here is that Generator residents are just as skilled as workers at other orgs), then joint hiring rounds produce 3 skill-adjusted generalist hours for every hour put into them.

If this answer seems obviously wrong or confusing, you might want to double-check your formula. For example, maybe Generator residents would be slower hiring managers than other professionals. You can add another parameter could be added to adjust for this.

Step 4: Determine a plausible range for your numbers

Next, you need to figure out what numbers you should plug into your formula. This is where everything can fall apart if you’re not careful. The output of your formula is only as accurate as its inputs, so this is important to estimate these correctly. Here are the ranges I used for this project:

S_mult: Somewhere between 0.8x and 3x
n: 3-6 orgs
%TS: 0.9

I don’t think these numbers were especially accurate, but I don’t think they were obviously wrong. Had I wanted to come up with a more serious estimate, I would have spent more time carefully thinking about these numbers. However, my goal was only to learn BOTEC-ing, so I didn’t invest much time into this.

Step 5: Estimate and interpret

This led to a G_mult range spanning from 2.16x to 16.2x. My initial intuitions about these numbers went something like this:

A number as low as 2.16 is almost certainly less impactful than other forms of fieldbuilding. Counterfactually convincing someone to pivot to a more effective career path is almost certainly much, much larger than that.
A number as high as 16.2 seems good enough that I might consider comparing this project with other similar things. However, for a variety of reasons I haven’t written in this post, I thought this estimate was way too high.

These estimations led me to believe that this project would not be worth pursuing on impact grounds alone.^[3] This evaluation also matches the story of one of the other residents: they were initially interested in completing this project, but they pivoted to working on sourcing candidates for top orgs directly. This turned out to be a much more efficient use of their time and one of my favorite Generator projects.^[4]

Completing a BOTEC is pretty good,

I got several things out of completing this BOTEC:

Clarity of thought: Before thinking about this BOTEC, I thought the project would be good because “efficiency feels good.” Clarifying my unit of impact helped me think about the real theory of change and how this project should be compared to others.
Rough idea of impact: The output of this BOTEC gave me some idea of how impactful this project actually would be.
Efficiency of communication: In future conversations with Generator residents, I could point to all the assumptions I had previously laid out about the value of this project.

All of these qualities seemed pretty good and made me feel better about the project-selection process.

But there are limits to how helpful BOTECs are.

However, despite their benefits, there are several reasons you might want to distrust BOTECs:

If you get an unreasonable output, you should probably be skeptical of it, unless you have a really good reason to believe that you have made no mistakes in your reasoning. But even then, be skeptical.
If a BOTEC is too complicated, it’s easy for small mistakes to lead to big problems.
1. It is extremely important for you to have a deep understanding of how all the parameters in a BOTEC relate to each other.
2. Simplicity is good when writing BOTECs. A route that finds an accurate answer through fewer parameters can be better than one that takes a messier route.
  1. This is primarily important when comparing the impact of things across vastly different domains.
If someone has a lot of tacit knowledge and verified intuitions about a domain, their intuitions will often outcompete BOTECs.
1. This might be because intuitions capture a lot of hidden parameters that BOTECs do not.
2. For this reason, BOTECs are especially valuable in domains where nobody has strong intuitions or experience, such as estimating the number of soil nematodes killed annually by people falling in love.
The Optimizer’s Curse means that if you’re evaluating several different interventions or cause areas, you will likely overvalue things with higher uncertainty.

I think the points made here are important enough that if you ever want to think about doing BOTECs, you should probably slow down, reread this, think carefully about each point, and maybe reconstruct them all from memory. It’s tempting to skim list-posts, but it is very important to understand the limits of any epistemic tool you use.

Should I do more BOTECs?

So, given these weaknesses, are BOTECs worth doing?

If you’re choosing between projects to work on, most of the value of BOTECs comes from gaining a clear conception of where your project’s impact comes from. So, I would recommend thinking carefully about your theory of change, and if you’re confused about that, you should determine what your unit of impact probably is.

In the early days of Generator, one thing I noticed was that many people didn’t spend enough time thinking carefully about where the primary impact of their projects would come from. This ultimately cost them lots of time and emotional strain, as they only realized a project wouldn’t be very good after several more hours of work. I suspect that some of them could have avoided this trap if they thought more carefully about the main unit of their impact.

Aside from that, I think you should only worry about completing a full BOTEC on projects you plan to spend more than ~24 hours on. As a general rule, BOTECs take longer than you expect to get right. You will probably learn much more and do more good things by just trying to go do things.^[5]

More broadly, things are only as valuable as how much they help you achieve your goals. It’s easy to get distracted by the allure of BOTECs: math is fun, and it can feel rewarding to clarify your thinking on every small issue you encounter. But don’t confuse this for impact.

Time stamps and other notes

Upon completion of the first draft of this post (2033 words), I had done 200 push-ups.
The section on the limits of BOTECs had the highest push-ups per word (it took me the longest to write).
My writing speed overall seemed fairly constant, based on my push-up counts.
Thank you to Aryan Bhatt and Zach Stein-Perlman (aka “A15Z”) for feedback on my practice BOTEC and tips for thinking about the value of BOTECs. Their guidance helped me get the ideas behind this post, but due to the nature of the push-up challenge, they were not given a chance to review it. The good ideas in here are probably theirs, and the mistakes in here are probably mine.
Final push-up count (didn’t hit failure, but it’s past 11 PM): 290 push-ups

^
Maybe it makes hiring choices too strongly correlated, and maybe orgs already do a good job of recommending people to new roles
^
If it ultimately mattered for the end result, I would have found a way to adjust for it.
^
It might have been worth it for upskilling reasons, but I’m kind of skeptical of this.
^
But there are a LOT of really awesome Generator projects. I think there’s a post about this coming soon.
^
But you should still think about theory of change.

High Impact Athletes is hiring a Head of Finance & Operations (remote, APAC-preferred, up to US$100k, CFO/COO path) by Marcus Daniell

Marcus Daniell — Wed, 12 Aug 2026 03:07:54 +0000

I’m Marcus Daniell, retired pro tennis player and founder of High Impact Athletes. I don’t usually write job ads, but this may be the most important hire we make this year, and I wanted to tap into the EA community to see if we hit gold.

What’s HIA? We were founded by two Olympians to point the sports world at effective giving. Athletes have moved $3M+ through us to date. Our fastest-growing programme, Race for Impact, runs in partnership with HYROX and turns racers into fundraisers. In the past year it has raised over $8M across 50+ events in 15 countries, from 100,000+ unique donors. This year we expect to move $10–20M, and our three-year aim is delivering $25–50M a year to our recommended organisations.

That money flows through a multi-entity structure: HIA Inc (US, Delaware), our NZ Charitable Trust, a UK charity launching in 2026, a potential EU entity in ²⁶⁄₂₇, plus partner-hosted funds and ~20 partner charities, in multiple currencies. The finance and operations function has grown rapidly and now requires a full-time, senior role.

Head of Finance & Operations. You’d own consolidated reporting and intercompany reconciliation across the group, treasury and FX, restricted vs unrestricted fund accounting, the compliance calendar (US 990s, UK Charity Commission, NZ Charities Services), the payment-processor stack (Stripe, PayPal, Raisely, Airwallex, Deel), the policy library, and people ops across four jurisdictions. You’d design the systems, run them, and grow the team around them.

Two things are non-negotiable:

Hands-on finance craft, owned first-person. Roughly 6–10 years in commercial, startup, fund, or non-profit. Can you walk us through a close, a consolidation, or a board pack you personally ran, with numbers attached? Charity-specific vocabulary—restricted funds, 990s, Charity Commission filings—is a strong bonus but not a requirement.
AI fluency you operate yourself. We run async-first on modern tooling and AI agents rather than spreadsheets and email chains. We use Claude, Notion, Xero, Airwallex, Deel, Make. “The tech team built our automations” or “I use ChatGPT to draft emails” don’t cut it.

Full JD: highimpactathletes.org/finance-ops-jd.

Compensation. Base is up to $100k USD depending on experience, plus a cost of living adjustment if applicable, plus a bonus scheme.

Career development. This is a build-from-scratch Head role that we intend to grow into HIA’s first Finance/Operations Director seat—a CFO/COO path for the right person. It reports to Hugo Inglis, our Managing Director. Remote and global, with an Asia-Pacific preference: Hugo and I are both Kiwis, and the role needs some overlap with APAC hours. Strong candidates elsewhere should still apply.

How to apply. Email financerole@highimpactathletes.org with a ~2-minute Loom on why this role, your CV, one example of finance/ops work you’re proud of, and a 3–5 sentence note on why HIA, as a single PDF. Happy to answer questions in the comments.

Cheers,
Marcus

Mental health challenges in the AI safety ecosystem seem like a big problem by JulianHazell

JulianHazell — Tue, 11 Aug 2026 20:00:06 +0000

A tough lesson I learned over the past few years is that it’s pretty stressful to work in the “Please Don’t Build the Torment Nexus” ecosystem, especially as it increasingly seems like the world is hell-bent on building the Torment Nexus.

This is what it can feel like to be an impact-pursuing person who stumbles into the AI safety field.

It’s hard to be productive when you’re stressed and anxious. And setting aside productivity, it’s also just plain bad if people working on AI safety — My peers! My people! — are feeling distraught.

Anxiety about AI risk really fucking sucks. If you feel this way, you aren’t alone. I’m right by your side, biting my nails, whitening my knuckles.

But even besides me, a lot of people seem to feel this way.

How big of a problem is this?

Survey data^[1] described in Spencer Greenberg’s recent 80k episode implies that mental health challenges for people working on AI safety are fairly common:

Most respondents experienced psychological challenges — the median person reported six different problems.
More than half (52%) said these challenges at least moderately hurt their effectiveness, while 45% had considered cutting back or leaving.
The most common problems were:
Difficulty switching off from work: 68%
Imposter syndrome: 60%
Burnout: 59%
Guilt: 56%
Anxiety or worry: 54%
Less common — though still significant — were loneliness or isolation (32%), hopelessness or depression (29%), interpersonal difficulties (28%), and fear of death (22%).
Few appeared to be experiencing clinical anxiety or depressive disorders. The more typical problem was chronic strain: people remained functional, but struggled to rest, sustain motivation, and maintain boundaries.

Somewhat ironically, I asked Fable 5 to ~~estimate~~ BOTEC how much productivity is being lost due to mental health challenges in the AI safety ecosystem. It estimated a loss of roughly 6–7% of total workforce output, with an 80% confidence interval of ~3 to 13%. I only quickly skimmed the report it put together, but the conclusion seems fairly reasonable to me. If anything, my all-things-considered guess is that the real productivity hit is larger.

If I’m right, this seems like a big problem. And as a corollary, if tractable interventions exist here, even modest improvements would be worth a lot.^[2]

What is being done about it?

There’s a bunch of existing stuff I’m aware of related to mental health and AI safety:

EA Mental Health Navigator and MentNav, which provide some mental health resources for effective altruists.
Rethink Wellbeing, which seems to run 8-week guided online programs, group-based peer support formats led by trained laypeople.
I’ve seen a bunch of surveys about mental health run over the years, e.g., this, this, and this (which I referenced above).
The EA forum has a topic on self-care and wellbeing.
- There’s a ton of writing about mental health challenges related to AI safety on the EA Forum and LessWrong (this post in particular seems like a useful snapshot, though note it’s from 2023; also, see this).
Many people (myself included) like this 80k podcast episode. There’s also this one, which inspired this blog post.
- Spencer talks about his book. I haven’t (yet) read it, but maybe it’s good!

What else should be done about it on the margin?

Honestly, I don’t know. I’m not sure how tractable this is, and I don’t claim to have particularly great ideas. With this post, I’m mostly just trying to point at a problem, say “oh man, this seems like a big problem”, and hope that others have ideas for how to fix it.^[3]

That said, here are a few things that might be good:

A vetted network of therapists who “get it”. I think it would be good if there was a curated, actively maintained, network of clinicians who have experience working with AI safety clients. As mentioned above, there’s the EA Mental Health Navigator — and ACX has the Psychiat-List — but maybe more can be done? When I was searching for a therapist, I didn’t find these existing resources super helpful.
Better norms in the community. The survey implies that chronic strain (e.g., issues with boundaries, rest, and motivation) is more common than clinical disorders like anxiety or depression. So maybe a lot of the fix has to do with improving the norms that shape the day-to-day experience of working on AI safety. For instance, not socially rewarding hardcoreness as a virtue so strongly.
Expand the group-program model. Rethink Wellbeing’s peer-guided CBT/IFS groups seem useful, at first glance. Maybe someone could do an AI-safety-specific version?
Generous mental health benefits at AI safety orgs. My sense is that many AI safety orgs are good at providing mental health benefits for their employees, but I might be wrong. If the org you run doesn’t offer this, you probably should fix that.
Peer support circles. Lightly facilitated, informal-ish groups of people working on AI safety who meet on some regular cadence to support each other emotionally.
Manager upskilling. Maybe someone (a high-AI-safety-context therapist?) could create resources or offer training sessions on how managers can provide better support to their direct reports.

If you’re interested in starting a project on something like this (or something else aimed at improving mental health in the AI safety space), this RFP from Coefficient Giving might be a good place to get funding. You also might want to check out the EA infrastructure fund.

If you’re applying for funding for some other project/org, consider requesting additional funding specifically to support yourself/your employees with mental health related challenges.

^
The survey is of 80 people working on existential risk, AI safety, animal welfare, and other high-impact causes. There’s obviously some self-selection bias here, but even after accounting for that, the rates of mental health challenges seem quite high to me. And while the survey population is broader than AI safety specifically, I’m assuming the AI safety subset looks similar (if anything, plausibly worse), but I haven’t checked.
^
The AI safety ecosystem has a high willingness-to-pay for interventions that bring new people into the field. In theory, the willingness-to-pay for interventions that help AI safety people who are struggling with mental health challenges (and are thus less productive than they otherwise could be) should also be high.
After doing some fake math, I’d guess that the ecosystem should be willing to pay at least millions of dollars (maybe tens of millions?!) for something that reduces the productivity drag by a percentage point.
^
Ironically, one of the hallmarks of AI safety work is that problem diagnosis is enticingly easy, whereas solution generation is much harder.