Frontpage posts - Effective Altruism forum viewer

Has Global Health Been Rigorously Compared With Other Cause Areas? by James Brobin

James Brobin — Fri, 03 Jul 2026 23:43:31 +0000

It seems like, in context to EA, if you’re interested in helping people at a global scale (and not focused on global catastrophic risks), you’re probably focused on global health.

I am wondering: Is there a clear reason EAs focus on global health over other cause areas such as education, women’s rights, economic growth, democracy, corruption, international relations, and other broad improvements to society?

Like, has there been any kind of rigorous research that suggests we should focus much more on global health in the poorest parts of the world as opposed to women’s empowerment in middle income countries or reducing gang violence in Central America?

How to Solve AI Biosecurity by Sophie Kim

Sophie Kim — Fri, 03 Jul 2026 23:25:06 +0000

my best 11,600-word guess—Bioweapon.AI is finally finished!!

Some notes from me:

People who aren’t worried about bioweapons are wrong.
- The state of biosecurity is really really really really bad and scary and alarming, even independent of AI.
The U.S. has a lot of cheese.
Read the full essay at Bioweapon.AI, which has better UI than Substack!
I may continue editing / working on this project in the future, though will most likely be taking a break from biosecurity work for a bit to focus on other MATS research related to AI geopolitics.
If anyone reading is interested in working on getting these interventions implemented, please feel free to reach out to me via Substack or email (firstnamelastname [at] stanford [dot] edu) and I’d love to chat (particularly if you’re in D.C. / a think tank, or some sort of advocacy organization).

read on bioweapon.ai

Bioweapons in the Age of AI

An investigation into the history of biological weapons, and how AI and emerging technology are now eroding the barriers that have kept catastrophe rare.

In the spring of 2023, a man walked into the White House carrying a small black box. Inside it were a dozen test tubes containing ingredients that, correctly assembled, had the potential to start a pandemic; an AI chatbot had supplied the recipe.

Fortunately, the man’s name was Rocco Casagrande – and, as a biochemist and former United Nations weapons inspector, he wasn’t there to use the materials; instead, he was there to brief government officials on how AI could help someone identify potent agents, secure the materials to make them, and soon – he warned – design entirely novel pathogens capable of evading humans’ immune systems.

That was three years ago.

In April 2026, the New York Times published an investigation into what happens when you ask AI’s most capable models for help with biological weapons.

Examining several chat transcripts shared by scientists, they wrote:

…OpenAI’s ChatGPT explained how to use a weather balloon to spread biological payloads over a U.S. city. In another chat, Google’s Gemini ranked pathogens by how much they could damage the cattle or pork industries. Anthropic’s Claude produced a recipe for a novel toxin adapted from a cancer drug. Other chats contained information that [an expert deemed] too dangerous to share.
…[T]he chatbot explained how to modify an infamous pathogen in a lab so that it would resist known treatments. Worse, the bot described in vivid detail how to release the superbug, identifying a security lapse in a large public transit system… [t]he bot outlined a plan to maximize casualties and minimize the chances of being caught.

From their investigation, the Times authors concluded:

…[E]ven publicly available models can do more than disseminate dangerous information. The virtual assistants have described in lucid, bullet-pointed detail how to buy raw genetic material, turn it into deadly weapons and deploy them in public spaces, the transcripts show. Some have even brainstormed ways to evade detection.

More alarming than these examples is just how quickly things have changed in the past two years. In December 2024, a multi-institutional team of researchers from Stanford, MIT, Princeton, Cohere, Mistral AI, and others published a systematic review of the AI biorisk literature and essentially concluded that “current LLMs and BTs do not pose an immediate risk.”

The team’s paper cited both an OpenAI red-teaming exercise and a Claude 3 safety report released earlier that same year. Yet, by April 2025, OpenAI had reversed its own assessment, concluding its models were approaching “high risk” – meaning, capable of “substantially increas[ing] the likelihood and frequency of bioterrorist attacks.” Similarly, Anthropic’s own internal bioweapons acquisition uplift trials published in May 2025 found that Claude Opus 4 enhanced human performance by 2.53× on relevant tasks, enough to trigger activation of AI Safety Level 3. An international safety report produced for the 2025 Paris AI Action Summit found LLM performance on weapons-related queries improved a staggering 80 percent in 2024 alone.

The institutions designed to prevent biological attacks have historically operated on the assumption that the expertise required to develop them is rare and slow to acquire.

AI and other emerging technologies may be on track to change this, and quickly.

This is not a fringe concern:

“The biggest issue with AI is actually going to be … its use in biological conflict.”
— Eric Schmidt, former CEO of Google

“[AI has the potential to] greatly widen the range of actors with the technical capability to conduct a large-scale biological attack.”
— Dario Amodei, CEO of Anthropic

We are not prepared.

Part One · Rogue Actors

Should We Be Concerned About Bioterrorists?

An Analysis of Historical Case Studies—Part 1: Rogue Actors

I. Successes

Note: Refers to successful deployment of a biological agent, not necessarily success in achieving overall objectives.

A. 1984 Rajneeshee Bioterror Attack

[ Video — Watch · The Dalles salmonella attack · https://youtu.be/vTAVzg_ny48 ]

In September 1984, the Rajneeshee cult deliberately contaminated salad bars in ten Oregon restaurants with salmonella in an effort to incapacitate voters before the 1984 Wasco County election.

Later investigations revealed that the attackers had a “fairly sophisticated medical research laboratory” at their commune, where they cultivated salmonella purchased over-the-counter from a Seattle scientific supply house. Ma Anand Puja, a former nurse who ran the Rajneesh Medical Corporation, provided the minimal medical expertise needed to culture common pathogens. Authorities also later discovered that the cult had explored using additional lethal pathogens, including HIV.

Though the cult had access to a medical research laboratory, the actual salmonella cultivation took place in a shed:

In a simple shed, under the supervision of Ma Anand Puja, the cult mass-produced the Salmonella using simple petri dishes, an incubator, and a freeze-drier ^[14]… Producing large quantities of bacteria is cheap and can be easily done with even rudimentary equipment and skills ^[18].
– Turner et al.

The attack poisoned 751 people and hospitalized 45, while causing panic that drained the local economy. To this day, the Rajneeshee operation remains the largest bioterrorist attack in U.S. history.

An image of a salsa bar contaminated as part of the attack. Location: Taco Time in The Dalles, Oregon. Source: Slate Magazine

B. WWI German Biological Sabotage Program

“A shipment of horses at a New York City rail yard, 1918.” Source: Archives.gov

During WWI, Dr. Anton Casimir Dilger – a German-American medical doctor specializing in tissue culture research – carried out a German biological warfare sabotage program targeting Allied livestock supplies. With the help of his brewmaster brother and housekeeper sister, Dilger converted his basement into a makeshift bacterial laboratory where he began cultivating deadly bacterial cultures in liquid form. He used anthrax and glanders bacteria to infect horses and mules being shipped from U.S. ports to Europe. Dilger also coordinated directly with German General Staff.

His basement was located less than six miles from the White House, and as a result of the program, “thousands of horses and mules were killed” (National Archives).

C. 2001 Anthrax Letters (also known as the Amerithrax case)

“Laboratory technician holding an anthrax-laced letter sent to Senator Patrick Leahy.” Source: FBI

Following the September 11 attacks, a series of anthrax attacks were carried out through letters containing anthrax spores sent via the mail. The spores had been treated with additives to increase their inhalability, which suggested the involvement of someone with highly specialized, advanced technical expertise. The primary suspect was Bruce Ivins, a veteran biological-weapon researcher for the US Army.

The attack killed 5 people and injured 17, with many survivors suffering long-term fatigue and memory loss even years later. It also created widespread panic at a moment when the nation was already reeling from 9/11.

[ Video — Watch · The 2001 anthrax letters · https://youtu.be/UQg7SM61fZ8 ]

A Netflix documentary was later created on the attacks.

II. Failures

A. 1993 Aum Shinrikyo Anthrax Attack

[ Video — Watch · Aum Shinrikyo · https://youtu.be/eWZ9jXI1d2I ]

In July 1993, doomsday cult Aum Shinrikyo aerosolized a liquid suspension of Bacillus anthracis (anthrax) from the roof of an eight-story building in Kameido, Tokyo, but the operation resulted in no human casualties.

The operation failed because:

They used an attenuated B. anthracis strain (used vaccine strain incapable of causing disease in healthy individuals);
Their Bacillus anthracis liquid suspension had low spore concentrations (significantly below the optimal 10⁹ to 10¹⁰ organisms/mL needed for effective biological weapons);
Sunlight exposure inactivated the spores before they could cause infection; and
Equipment failures meant they were unsuccessful in creating proper particle distribution.

Aum had laboratories for experimenting with bioweapons in Kamakuishki and Tokyo. In addition to anthrax, the group also “cultured and experimented with botulin toxin, cholera, and Q fever” (CDC). After failing to deploy a bioweapon, they later transitioned to successful chemical attacks, including the Matsumoto Sarin Attack and the Tokyo Subway Sarin Attack (not detailed here since this analysis is about bioweapons).

[ Video — Watch · From bioweapons to the Tokyo subway sarin attack · https://youtu.be/QVRGYlUsE9I ]

B. 2018 Cologne Ricin Plot

In June 2018, Sief Allah H., a 29-year-old ISIS sympathizer, successfully produced enough ricin for “up to 1,000 toxic doses” and assembled a bomb with explosives and metal ball bearings for a planned mass-casualty attack on a crowded indoor venue. This marked the first time a jihadi terrorist in the West successfully produced the toxic biological agent, demonstrating that technical barriers to simple bioweapons are sometimes surmountable with publicly available instructions.

German authorities discovered and put a stop to the plan via online surveillance operations, specifically after the U.S. CIA provided a tip about his large order of 3,300 castor seeds purchased over the internet.

C. 1995 Minnesota Patriots Council Ricin Plot

“Ricin occurs naturally in castor beans.” Source: BBC

The Minnesota Patriots Council was an anti-government militia that attempted to assassinate federal law enforcement officials (including US Marshals, IRS agents, and local sheriffs) using ricin. The group successfully extracted ricin from mail-ordered castor beans using publicly available instructions and basic solvent knowledge gained through their work as carpet cleaners. They initially found and purchased castor beans through an advertisement in a right-wing magazine.

Ultimately, the Minnesota Patriots Council successfully produced enough ricin to kill over 100 people, despite their result being significantly less potent than professional-grade toxin. Similarly to the Cologne Ricin Plot, law enforcement detected and disrupted their activities before they could carry out planned attacks.

D. 1998 Toxic Terror Case

The perpetrators of the Toxic Terror Case were Larry Wayne Harris, a microbiologist and white supremacist with ties to “Aryan Nations,” working alongside William Leavitt, a friend who “own[ed] biological laboratories in Nevada and Germany.”

The pair planned to release bubonic plague bacteria in the New York subway system; law enforcement raided their labs after they “mail-ordered freeze-dried bubonic plague germs.” During the raids, officials also found vials containing suspected anthrax. Harris reportedly told a witness that he had procured enough anthrax to “wipe out the city.”

Addendum: ISIS Laptop of Doom

The Islamic State’s Terror Laptop of Doom revealed a comprehensive biological weapons research program documented on the laptop of a Tunisian ISIS operative with university-level scientific training. Uncovered in 2014, it contained a “19-page document in Arabic on how to develop biological weapons and how to weaponize the bubonic plague from infected animals” containing guidance like “[u]se small grenades with the virus, and throw them in closed areas like metros, soccer stadiums, or entertainment centers… best to do it next to the air-conditioning.”

III. Analysis

Successes

Case	Expertise Level	Available Infrastructure	Key Success Factor(s)
Rajneeshee	Minimal (nurse practitioner’s basic medical knowledge)	Institutional (“sophisticated medical research laboratory”), though actual bioweapon development took place in a shed using basic tools	Easily cultivated pathogen, which didn’t require as much specialized expertise to pull off
Dilger	High (tissue culture specialist)	Lab set up in basement with substantial institutional backing (German General Staff coordination)	Expert knowledge + state sponsorship + help from siblings, which maintained opsec
Amerithrax	High (microbiologist, U.S. Army)*	Institutional access (Army lab materials – USAMRIID)	Technical expertise + authorized access to dangerous pathogens due to military work

*Whether or not Ivins was the culprit remains contested.

Takeaways

A. Rajneeshee represents a case where despite minimal expertise, the perpetrators were still able to succeed because they chose an easier pathogen to synthesize. The expertise bottleneck varies by pathogen.

B. Dilger was able to maintain operational security by relying on his siblings to help him with his work. While he had to rely on a home laboratory, it’s likely that coordination with German General Staff enabled him to procure advanced materials and equipment.

C. The Amerithrax (aka Anthrax Letters) case is perhaps slightly more difficult to assess with confidence because Bruce Ivins’s guilt remains contested. However, if Ivins was the culprit, key success factors include highly specialized expertise and authorized access to dangerous pathogens as a researcher for the U.S. army.

The Dilger and Ivins cases demonstrate that specialized biological knowledge poses severe risks; advanced expertise can enable catastrophic biological attacks.

Failures

The failed cases fall into two main categories: technical incompetence and detection via surveillance.

Technical Incompetence

The Aum Shinrikyo Anthrax Attack failed on technical problems. However, had Aum expanded their technical expertise beyond their compartmentalized leadership structure, there’s a reasonable chance their attack may have succeeded. Their operation went undetected through deployment, and every failure point (strain selection, spore concentration levels, dispersal, sunlight exposure, etc) represented knowledge gaps that additional specialized expertise could have addressed.

The Amerithrax case demonstrates that the technical barriers Aum encountered were surmountable with proper expertise.

Detection via Surveillance

Unlike Aum’s technical failures, the Cologne and Minnesota Patriots cases succeeded at the production phase. These actors proved that certain bioweapon manufacturing is already within reach of non-experts using publicly available information. What prevented attacks was detection and disruption by law enforcement, rather than inability to create functional weapons.

Whether Toxic Terror succeeded at the production phase is less clear, as the vials found by law enforcement were only suspected to contain anthrax. It could plausibly fall into either of the above categories, depending on whether the recovered materials were viable and weaponizable.

IV. Conclusion

Bioweapons have been created and deployed effectively in several documented attacks. The Rajneeshee, Dilger, and Amerithrax cases prove that bioweapon development and deployment are within reach of motivated rogue actors.

At the same time, these successes have been mostly limited in scope and severity. Several other large-scale near-misses have failed primarily due to technical incompetence or detection by law enforcement.

AI, alongside other existing and emerging technologies, threatens to erode some of the barriers that have historically caused malicious actors to fail. That will be the subject of Part 2 of this series!

Part Two · Government Programs

When Russia Wants a Bioweapon, It Gets One

Analysis of Historical Case Studies—Part 2: Government-Sponsored Programs

I. Historical Precedent

Governments have repeatedly demonstrated both the capability and the willingness to develop and deploy bioweapons.

A. The Soviet Biopreparat Program

[ Video — Watch · Biopreparat & the 1979 Sverdlovsk anthrax release · https://youtu.be/x3FMww5biJ8 ]

In 1972, the Soviet Union signed the Biological and Toxin Weapons Convention (BTWC), pledging not to develop, produce, or stockpile bioweapons. A year later, the Soviet Union violated the BTWC by creating Biopreparat, a massive biological warfare enterprise masquerading as a civilian pharmaceutical company.

Through the program, Soviet scientists reportedly manufactured an estimated 20 tons of plague, 20 tons of smallpox, and hundreds of tons of anthrax; they also weaponized tularemia, epidemic typhus, Q fever, and Marburg virus, while studying the potential use of Ebola and encephalitis.

The program’s exact scale remains disputed. The Federation of American Scientists states the following:

At its peak, the former Soviet Union had the world’s largest biological warfare program, with somewhere between 25,000 and 32,000 people employed in a network of 20 to 30 military and civilian laboratories and research institutions. An additional 10,000 or so worked in Defense Ministry bioweapons laboratories. According to other estimates, at least 47 labs and test facilities were scattered across Russia, employing more than 40,000 workers, 9,000 of whom were scientists. Between 1,000 to 2,000 of those scientists were experts on deadly pathogens.
– Federation of American Scientists

Meanwhile, Congressional testimony asserts:

The size and scope of this program were enormous. For example, in the late 1980s and early 1990s, over 60,000 people were involved in the research, development, and production of biological weapons.
– Dr. Kenneth Alibek, before the Joint Economic Committee, United States Congress

Regardless of the precise number, Biopreparat was unquestionably the largest biological warfare program in history.

Even decades after its official end, the program’s deadly legacy endures. Contaminated testing sites like Vozrozhdeniya Island still harbor active anthrax spores buried in the soil since the Soviet era:

Vozrozhdeniya was once home to a vibrant fishing village fringed by turquoise lagoons, back when the Aral Sea was the fourth-largest in the world and abundant with fish…
Over the years the site flourished into a living nightmare, where anthrax, smallpox and the plague hung in great clouds over the land, and exotic diseases such as tularemia, brucellosis, and typhus rained down and seeped into the sandy soil.
– BBC, “The deadly germ warfare island abandoned by the Soviets”

The true nature of the program only became clear after the Soviet collapse in the 1990s.

The Soviet Biopreparat program is an especially important case study because it reveals that even absent deliberate deployment, the existence of national bioweapons programs creates significant danger through the persistent risk of laboratory accidents. A historical example of this is the 1979 Sverdlovsk anthrax release (detailed in the video above), which killed “at least 68 people”; another example of this is the 1971 Aral smallpox incident:

[ Video — Watch · The 1971 Aral smallpox incident · https://youtu.be/tH0B-r9L4GY ]

B. Imperial Japan’s Unit 731

[ Video — Watch · Imperial Japan’s Unit 731 · https://youtu.be/i3qjjXBQnzM ]

Before the Soviet Union developed Biopreparat, Imperial Japan’s Unit 731 demonstrated that biological weapons could work on a massive scale. Active from 1936 to 1945, Unit 731 is notorious for its systematic human experimentation and operational deployment of biological weapons against Chinese civilians.

Unit 731 researchers reportedly used “political prisoners, criminals, the poor, and homeless” in their tests; test subjects also “included women and children.”

[ Video — Watch · The human experiments of Unit 731 · https://youtu.be/AXM1DozZwjk ]

Unit 731 wasn’t the only bioweapons development program to conduct experiments on humans, but it was the only one to systematically integrate mass human experimentation with active biological warfare. Researchers used thousands of human subjects to perfect biological weapons that were immediately deployed against civilian populations. An estimated 3,000 died in horrific experiments, while tens of thousands more were killed by the weapons those experiments helped develop:

Elderly plaintiffs flew from China to testify – often in tears – about their communities being ravaged by diseases that spread mysteriously after Japanese planes flew low overhead and dropped wheat, rice or cotton infested with fleas…
– The Guardian

iTrigger warning: graphic content.

Researchers performed surgeries and vivisections on their victims without the use of anesthesia, removing organs and severing limbs… [s]ome victims had their limbs amputated and reattached to other parts of their body; others were subjected to extreme cold to gauge the effects of frostbite and gangrene on human skin. Many were exposed to poison gas or deadly diseases to observe the amount of time it took a person to show the effects or symptoms before dying… doctors subjected victims to starvation, dehydration, extreme air pressure, or electrical current to see how long a human could survive under such conditions… estimates ranging from thirty thousand to more than five hundred thousand are believed to have died in field tests of diseases on the Chinese population.
– EBSCO

According to EBSCO, Unit 731 also focused on creating weaponized deployment options: they developed a bomb of plague-infested fleas, and planned to deploy their weapons using balloons sent “adrift across the Pacific” or through aircraft-delivered plague bombs on U.S. cities.

The war ended before those plans were executed; but as evidenced by Biopreparat, what the program had demonstrated about the feasibility of mass-casualty biological warfare did not end with it.

[ Video — Watch · The legacy of Unit 731 · https://youtu.be/LqB5fAHjyME ]

II. Contemporary Cases

If Biopreparat and Unit 731 established that states will build these weapons when they see a use for them, the question for the present is who is still doing so. The U.S. government’s answer names four states.

A. Russia

Russia is the direct heir to the largest bioweapons program in history, and the U.S. government does not believe the inheritance was ever fully renounced. According to a State Department report released in 2025, U.S. intelligence believes that the Soviet program was absorbed rather than dismantled by the Russian Federation, and that Russia currently “maintains an offensive biological warfare program.”

In the wake of Russia’s invasion of Ukraine, the program appears to be expanding. In October 2024, the Washington Post reported on satellite imagery of Sergiev Posad-6, a military site northeast of Moscow with a Cold War history of weaponizing smallpox and Ebola:

A few months after Russia began its full-scale invasion of Ukraine in 2022, satellite imagery captured unusual activity at a restricted military research facility… construction vehicles renovating the old Soviet-era laboratory and breaking ground on 10 new buildings, totaling more than 250,000 square feet, with several of them bearing hallmarks of biological labs designed to handle extremely dangerous pathogens.
– Washington Post, “Satellite images show major expansion at Russian site with secret bioweapons past”

Satellite imagery of Russia; area believed to be a historical bioweapons site. Source: Washington Post

Russian officials have framed the expansion as biodefense. It’s impossible to discern from satellite photos alone what the true nature of the facilities is.

B. China

The U.S. government states that China possessed its own biological weapons program from “the 1950s to at least the late 1980s.” As part of the program, the country is said to have “weaponized ricin, botulinum toxins, and the causative agents of anthrax, cholera, plague, and tularemia.”

What’s unresolved is whether “at least the late 1980s” was actually the end. The State Department does not assert the program continued, but it does not rule out the possibility: China has never disclosed its historical facilities in any BWC submission, and the same report flags continuing biotech research by Chinese military medical institutes that could have weapons uses. This year, there’s also a new concern:

For the first time, this year’s report warns that China probably is capable of using publicly available artificial intelligence and machine learning (AI/ML) tools to advance efforts related to biological weapons applications. At the same time, China “probably is unable to make complex scientific equipment without Western innovation.”
– Council on Strategic Risks, “The State of Compliance with WMD-Related Treaties”

Notably, the AI dimension flagged here is not unique to China; it runs through every contemporary case, and it is the subject of the analysis to come.

C. North Korea

Photo by Mike Bravo on Unsplash

While China is assessed as a “compliance concern,” the State Department states the case for North Korea plainly: the U.S. assesses that the DPRK has a dedicated, national-level offensive biological weapons program, and possesses the technical capability to produce pathogens and toxins usable as bioweapons, and to engineer them genetically.

The United States assesses that the DPRK has a dedicated, national-level offensive [biological weapons] program… Pyongyang probably is capable of weaponizing BW agents with unconventional systems such as sprayers and poison pen injection devices, which have been deployed by the DPRK for delivery of chemical weapons and could be used to covertly deliver BW agents.
– U.S. State Department

While the State Department’s paper does not dive into the specifics of the DPRK’s program, South Korean defense analysts believe that “B. anthracis, Variola virus, Yersinia pestis, Vibrio cholerae, and botulinum toxin” are the most likely candidates for weaponization.

These same agents are classified as top-tier biological threats by both the U.S. Centers for Disease Control and Prevention (CDC) and WHO.
– Jungeun Lee, Korea Research Institute for Defense Technology Planning and Advancement

Concerningly, the June 2024 Treaty on Comprehensive Strategic Partnership between Russia and North Korea establishes plainly that the two states will “actively encourage joint research in the field of science and technology, including such areas as space, biology, peaceful nuclear energy, artificial intelligence, [and] information technology.”

D. Iran

Russia and North Korea have confirmed offensive programs, and there is a verified historical program with unanswered questions about the extent to which it continues in China. Meanwhile, Iran’s case is interesting because it centers on the dual-use problem: the U.S. says that “Iran has not abandoned its intention to conduct research and development of biological agents and toxins for offensive purposes,” and that it has facilities that possess the capability to produce bioweapons, if directed:

Iran maintains flexibility to use, upon leadership demand, legitimate research underway for biodefense and public health purposes for a capability to produce lethal BW agents. It is unknown if Iran’s leadership has set a directive to maintain this flexibility.
– U.S. State Department

III. Analysis

States succeed where others fail because they have substantially more resources than typical rogue actors. Biopreparat was composed of tens of thousands of experts, and states can draw from large military budgets to finance their operations. That’s true, but the more interesting read is that state resources reliably buy capability. Biopreparat and Unit 731 were successfully able to synthesize deadly bioweapons, and Russia, China, and North Korea all appear to possess some capability to produce bioweapons, with Iran assessed as able to develop it on demand. But capability is not the same as use: of every program in this essay, only Unit 731 ever achieved mass-casualty deployment.

Most of the time, even states that solved the resource and expertise problems mostly declined to use what they built. Why is that? The reason appears to be that biological weapons have rarely offered enough strategic utility to justify the risks of using them on a battlefield, in a conflict between two states.

Countries have specific incentives: they want to protect their own troops during wartime. Blowback risks, along with the threat of nuclear retaliation in some cases (the UK, for example, reserves the right to deploy nuclear weapons if chemical or biological weapons are used against its people), complicate this goal. They are further bounded from bioweapons use by a near-universal international norm. Ultimately, the fact that states also have a wide variety of conventional weapons in their arsenals makes it difficult to justify deploying bioweaponry.

However, the degree to which this constraint is binding depends upon a rational state actor weighing controllable outcomes. In recent history, the limiting factor has never solely been the technical challenge of developing these weapons: instead, it has primarily been the fact that those with the capability to build them often did not consider their use worthwhile. This restraint appears to be a property of who holds the capability, not of the weapons themselves – and as evidenced by Unit 731, Aralsk-7, and Sverdlovsk, even that hasn’t been a fully successful restraint in the past.

The relevant question for the present is what happens when that capability becomes available to actors whose calculations are different: namely, actors who are not deterred by blowback, not bounded by international norms, unafraid of state-level deterrence, and not optimizing for controllable outcomes.

The restraint was never about the weapons. It was about who held them. What happens when that changes?

Part Three · The Bottlenecks

The Bottlenecks Are Eroding

How AI and emerging technology are undermining the barriers that have kept catastrophic bioweapons rare.

Four bottlenecks have historically kept bioweapons out of reach. Each section below is tagged with the one(s) it erodes; in the case studies at the end, individual passages are highlighted with a margin note.

Capital

The total cost of developing and producing a weapon.

Equipment

Access to the physical lab gear and facilities required.

Knowledge

The information (e.g., sequences of harmful pathogens to use for synthesis) required.

Expertise

The trained human skill (commonly referred to as “tacit knowledge”) required to execute the work.

AI is Eroding the Knowledge Bottleneck

Knowledge

In April 2025, publicly available models like o3 had already outperformed 94% of virology experts on laboratory protocol questions, even on questions directly relevant to the experts’ specialties; SecureBio’s recent assessment of GPT-5.5 put it in the 100th percentile on that same evaluation.

AI model scores on the Virology Capabilities Test. Note: GPT-5.5 not included in the above; see below. Source: AI Frontiers

“Figure 2.1.1.2 Model performance compared to human SMEs on VCT (full set, multimodal)… Pre-Release Checkpoint 2 outperforms all SMEs across all samples.” Source: SecureBio

That benchmark performance hasn’t yet translated cleanly into end-to-end weapons development guidance, but the trajectory is clear. Models are advancing toward the ability to provide expert-level virological guidance on demand, anonymously, and at scale. Anthropic’s own internal bioweapons acquisition uplift trials found that Claude Opus 4 enhanced human performance by 2.53× on relevant tasks, enough to trigger activation of AI Safety Level 3.

Beyond AI: A Converging Landscape of Eroded Bottlenecks

AI’s erosion of the knowledge bottleneck operates within a broader landscape of technological advancement. Simultaneously, progress in dispersal technologies, automated labs, and DNA synthesis have each independently eroded distinct bottlenecks that once constrained bioweapon development.

Dispersal Technologies

Equipment

Aum Shinrikyo faced equipment failures when they were attempting to deploy anthrax in 1993. However, multiple breakthroughs in aerosol technologies have occurred since the 90s that could make dispersal of a bioweapon significantly easier and less error-prone.

For example, bag-on-valve technology separates liquid completely from the propellant using a hermetically sealed bag inside the can; the liquid being dispersed maintains complete purity since it’s never contaminated by propellant gases. Moreover, crop duster drone technology and other commercial drones have dual-usability; the Institute for National Strategic Studies writes that there is significant risk of “the use of crop dusters as delivery vehicles for biological or chemical weapons of mass destruction.”

Cloud Labs & Contract Research Organizations

Equipment Expertise

Cloud labs. Companies like Emerald Cloud Lab and Strateos allow anyone to design and run biological experiments remotely. Customers submit experimental protocols through a software interface, and robotic systems in a physical lab execute them. Currently, anyone can use a cloud lab with no coding experience, and there are no regulations requiring identity verification (KYC), nor are there any mandatory legal mechanisms for monitoring what experiments users run. Anyone with a credit card and an internet connection can run experiments that would previously have required institutional affiliation, costly specialized equipment, physical lab access, and years of hands-on expertise to execute.

Contract research organizations (CROs). These are labs for hire that will execute experiments on behalf of clients. Like cloud labs, the regulatory oversight here is minimal.

Why do these matter?

The threat from cloud labs and CROs is not that they will manufacture bioweapons on behalf of customers. Rather, they enable a malicious actor to fragment their research across multiple platforms and providers, making it increasingly likely they will be able to defeat traditional detection mechanisms.

More specifically, a bioweapon developer could outsource the process of running iterative experiments to cloud labs, which means they may never need to purchase the large quantities of materials that would normally be flagged for suspicious activity. Each individual experiment, run through various cloud labs under a shell company name, might appear to be legitimate research on vaccines or protein expression, but together, they could constitute the development pipeline for a biological weapon.

DNA Synthesis

Knowledge Capital Equipment

What is DNA synthesis?

DNA synthesis is the process of building custom DNA sequences. A researcher types a desired sequence into a computer, submits an order to a commercial provider, and receives a vial of synthetic DNA in the mail. The process is quick and can be relatively inexpensive ($0.07 to $0.09 per base pair in some cases; other companies offer flat rates).

Why is it dangerous?

Some viruses’ entire genomes can be assembled from commercially purchased DNA fragments. In 2006, an investigative journalist with the Guardian was able to mail order a “modified sequence of smallpox DNA,” and their order was not screened by the provider since it was less than 100 letters long.

What’s the current state of screening requirements?

As I’ve written about extensively here, no country currently has a law requiring gene synthesis providers to screen DNA orders for dangerous sequences. The screening that does exist is voluntary, inconsistent, and has no mechanism for detecting split orders across multiple providers.

The net effect of these technological advancements, among others, is that many of the traditional barriers that used to constrain bioweapons development – including physical lab access and years of hands-on training – are becoming less binding. Increasingly, you can outsource iterative experimentation to cloud labs and CROs, outsource sequence design to AI or find publicly available sequences for dangerous pathogens on the internet, and order the genetic material from synthesis providers with minimal screening.

The risks of these emerging technologies might appear purely theoretical – but case studies tell a different story.

The Capital, Equipment, Knowledge, & Expertise Bottlenecks

As mentioned briefly above, in 2006, an investigative journalist with the Guardian was able to mail order a “modified sequence of smallpox DNA”, and their order was not screened by the provider since it was less than 100 letters long.

The journalist placed an online order using “an invented company name along with just a mobile telephone number and free email address.”

The Guardian writes:

The DNA sequence of smallpox, as well as other potentially dangerous pathogens such as poliovirus and 1918 flu are freely available in online public databases. So to build a virus from scratch, a terrorist would simply order consecutive lengths of DNA along the sequence and glue them together in the correct order. This is beyond the skills and equipment of the kitchen chemist, but could be achieved by a well-funded terrorist with access to a basic lab and PhD-level personnel.

In a separate 2017 incident, two scientists from the University of Alberta synthetically recreated the previously eradicated horsepox virus for a mere $100,000. CSIS writes:

Rather than developing the virus themselves, the scientists outsourced much of the initial work – custom ordering fragments of the DNA from a commercial synthesis lab, which “printed” and shipped the viral DNA back to them via mail. The scientists then linked the fragments together in a lab and introduced them into cells using a helper virus, producing the final horsepox virus. Though the study exclusively aimed to improve vaccine and cancer treatments (rather than produce bioweapons), it nevertheless attracted widespread alarm at the time by indicating that reviving smallpox – a close cousin to horsepox and one of the deadliest diseases known to mankind – would, as Science reported, “probably take a small scientific team with little specialized knowledge half a year.”

Dying of smallpox doesn’t seem particularly fun.

So what should we do about it?

Part Four · Policy Interventions

Policy Interventions

Eight concrete interventions to defend against AI-enabled biological threats.

Prevention & Detection

4.1 Require standardized pre-deployment bio evals

A beautiful eval. Source: METR

Background: Today, the question of whether a given AI model poses meaningful biological risk is answered inconsistently, and often by the very companies with a financial incentive to deploy their models as quickly as possible.

Anthropic conducts what it calls “uplift trials;” OpenAI has conducted its own internal red-teaming. Results are not always disclosed publicly, and no agreed definition exists for what constitutes unacceptable biological risk, nor is there a standardized protocol to ensure models above a certain threshold of capability have specific safeguards attached to them.

No legislation requiring pre-deployment bio evaluation currently exists.

Meanwhile, the capability frontier is advancing. As I wrote in the Bottlenecks section:

In April 2025, publicly available models like o3 had already outperformed 94% of virology experts on laboratory protocol questions, even on questions directly relevant to the experts’ specialties; SecureBio’s recent assessment of GPT-5.5 put it in the 100th percentile on that same evaluation.

Moreover, as I’ve written about previously:

A recent study led by Microsoft and IBBIS researchers demonstrated that open-source AI tools could engineer new protein variants of known proteins of concern that successfully evaded synthesis screening. As Nobel laureate David Baker and Harvard geneticist George Church have emphasized, screening based on homology alone is unlikely to be sufficient when de novo protein design can produce functionally dangerous proteins with no recognizable homology.

The Intervention: Require all developers of frontier AI models and AI protein design tools to conduct standardized pre-deployment bio evaluations before releasing any model to the public, with evaluations for protein design tools focused specifically on their capability to engineer dangerous protein variants.

Evaluations must be conducted by certified third-party evaluators and reported to the federal government. Models exceeding a defined risk threshold must implement tiered access controls, capability restrictions, delay deployment indefinitely, or other safeguards until identified risks are mitigated.

Recommendations

NIST / Center for AI Standards and Innovation (CAISI)

Develop standardized pre-deployment bio evaluation protocols, organized by threat pathway, with defined risk tiers and corresponding required actions (e.g. enhanced monitoring requirements and mandatory deployment delays, or perhaps tiered access controls– see 4.4)
- Work alongside third-party evaluators such as SecureBio, FAR.AI, EquiStamp, and RAND
Define the compute and capability thresholds that trigger mandatory evaluation requirements, and update them on an annual basis as the frontier advances and algorithmic efficiency gains are made

Department of Health and Human Services

Establish a certification program for third-party evaluators and maintain a public registry of those certified

Federal Funding Agencies (NIH, NSF, DARPA)

Condition federal research grants on use of AI tools and models that have cleared certified bio evaluations, using procurement-based requirements as an interim lever while waiting on Congress for the legal basis of restricting model releases
- This mirrors the procurement lever proposed for DNA synthesis screening in 4.2 and creates immediate market pressure before binding regulation is in place

Congress

Pass legislation requiring mandatory pre-deployment bio evaluation as a condition of public or API release for all covered AI models (under NIST’s compute and capability thresholds)
Authorize the Department of Commerce (informed by technical standards developed by NIST through CAISI) to issue binding regulations governing evaluation protocols and risk thresholds

International Coordination

Work with the UK AI Security Institute, which has an existing model evaluation program, to coordinate bio evaluation standards across allied governments
Pursue coordinated standards through the G7 Hiroshima AI Process
Promote common biological-risk evaluation frameworks through OECD AI governance initiatives, including the OECD AI Principles and the OECD AI Policy Observatory
- As a first step, develop an international taxonomy of high-risk biological queries that establishes a baseline for situations in which AI models should refuse to answer.

4.2 Mandate AI-enabled DNA synthesis screening

Photo by Sangharsh Lohakare on Unsplash

Background: Currently, there exists no universal legal requirement for gene synthesis providers to conduct background checks on clients or screen DNA sequences to ensure they’re not dangerous pathogens. While some labs screen orders on a voluntary basis as a condition of membership in organizations like the International Gene Synthesis Consortium, compliance remains optional and inconsistent. In 2006, an investigative journalist with the Guardian was able to mail order a “modified sequence of smallpox DNA,” and their order was not screened by the provider since it was less than 100 nucleotides long.

Even in the United States, no binding legal requirements exist for DNA synthesis screening. Federal regulations were proposed in 2024 through the Framework for Nucleic Acid Synthesis Screening, but an Executive Order in May 2025 paused implementation, and no replacement framework has been issued. Moreover, despite several congressional bills attempting to mandate screening, none have passed. This regulatory gap means anyone, including those with malicious intent, can order potentially dangerous genetic sequences with minimal oversight.

Notably, frontier lab leaders themselves agree that this is a problem: see this letter signed by the CEOs of OpenAI, Anthropic, Google DeepMind and Microsoft AI: ScreenDna.org

The Intervention: Existing screening approaches rely on sequence homology, which means matching orders against databases of known dangerous pathogens. An AI-enabled bioterrorist could circumvent this by designing functionally equivalent pathogens using synonymous codons, chimeric sequences, or entirely novel genetic constructs that retain lethality while evading database matches.

To address this issue, we need to implement advanced AI-powered screening that would analyze predicted protein function and evolutionary markers to flag potentially dangerous sequences.

Implementation requires two components: First, we must develop reliable AI screening systems capable of detecting novel pathogenic sequences that the world has never seen before; second, we must require all commercial DNA synthesis providers globally to implement this screening as a condition for legal operation.

Recommendations

Department of Health and Human Services / OSTP (immediate actions while waiting for legislation from Congress)

Finalize the paused Framework for Nucleic Acid Synthesis Screening with a phased screening mandate:
- Phase 1 (immediate): require sequence homology screening against databases of known dangerous pathogens, using existing methods
- Phase 2 (triggered once a validated AI screening tool is available– see Congress section below): require screening that integrates AI-powered functional analysis
  - Organizations like IBBIS, Sentinel Bio, and Fourth Eon Bio are already working on improved screening tools and international standards; Phase 2 should build on and accelerate this existing work
Tie federal research funding to compliance: any institution receiving NIH, NSF, or DARPA grants must procure synthetic nucleic acids exclusively from providers that meet the updated screening standards
- This was already proposed in the 2024 framework and creates immediate market pressure without requiring new legislation

Congress

Pass S.3741 with these proposed amendments
- Brief summary– the bill should:
  - (1) include dedicated funding for the development of AI-enabled screening tools that can assess the functional risk of novel sequences,
  - (2) require that benchtop nucleic acid synthesis equipment sold or distributed in the United States incorporate embedded screening software that queries the centralized sequence-of-concern database before synthesizing any sequence,^*
  - (3) include a provision where the Secretary is required to establish and maintain (or designate a separate entity to maintain) a centralized order reporting platform, and
  - (4) designate non-compliant DNA synthesis providers on the Entity List.

International coordination for synthesis screening should be pursued through the multilateral channels described in Section 1, including the BWC and Australia Group frameworks.

^* Note: A stronger proposal (that, since writing my gap analysis of S.3741, I have updated to support) would treat benchtop nucleic-acid synthesizers as controlled items subject to custody tracking, similarly to fissile material. Possession would require a license, and a designated oversight agency would maintain a registry of each device’s location and responsible custodian; when a licensed owner no longer needs a unit, it would be returned or transferred only to another licensed owner. For devices already in circulation, buyback programs may be a promising option.

4.3 Implement know-your-customer requirements for cloud laboratories and contract research organizations

Note: For a fuller treatment of why cloud labs could pose biorisk, see Part 3 on Bottlenecks.

Background: The standard objection to AI-driven bioweapons risk is that knowledge alone is not enough– you still need hands-on laboratory skills, the “tacit knowledge” that can only be acquired through years of physical practice (hence Active Site’s uplift study). Knowing how to culture a pathogen is different from being able to do it reliably. This barrier has historically been one of the strongest defenses against non-state bioweapons development.

Unfortunately, cloud laboratories threaten to significantly erode this barrier. Services like Emerald Cloud Lab allow anyone to design experiments in software and have them executed by robotic systems in a physical facility, remotely, without ever entering a lab. ECL requires no coding experience; internal estimates suggest relatively short onboarding periods for novice users. An AI system that can design a bioweapons protocol and a cloud lab that can execute it are, individually, semi-manageable risks; together, they could materially increase the risk of misuse.

Source: Nature

Despite this, cloud labs currently operate with no standardized customer screening.

Contract research organizations (CROs) create a similar vulnerability. CROs provide specialized research services and can help their clients with everything from compound synthesis to biological assays. A malicious actor could potentially decompose a bioweapons development project into seemingly innocuous components and outsource them to different CROs, each unaware of the larger program.

The Intervention: Require all cloud laboratory providers and contract research organizations to implement know-your-customer screening before granting access to experiment execution as a condition of legal operation. Providers should be required to log all experimental workflows and flag protocols involving select agents or sequences of concern, with automated screening that mirrors (and integrates with) the DNA synthesis screening proposed in 4.2. Also establish RAND’s proposed Cloud Lab Security Consortium modeled on the IGSC.

Recommendations

Department of Health and Human Services

Issue rulemaking requiring all U.S.-based cloud laboratory providers and CROs to verify identity, institutional affiliation, and stated research purpose of every user before granting access to experiment execution
- Model this on existing Select Agent Program registration requirements
Require cloud lab providers and CROs to log all experimental workflows and flag protocols involving select agents, sequences of concern, or pathogen-adjacent procedures, with automated screening that integrates with the DNA synthesis screening framework proposed in 4.2
- Mandate periodic red-team exercises in which vetted biosecurity researchers attempt to circumvent logging and screening systems, with findings reported to the relevant oversight body and used to update screening criteria

NIST (National Institute of Standards and Technology)

Develop standardized KYC and biosecurity screening standards specifically for cloud laboratories and CROs, defining:
- Minimum identity verification thresholds
- Prohibited experiment categories
- Escalation procedures when flagged protocols are detected

Federal Funding Agencies (National Institutes of Health, National Science Foundation, DARPA)

Condition federal research grants on use of cloud labs and CROs that implement baseline KYC screening, using interim agency guidance while NIST develops formal standards (the same procurement-based lever proposed for DNA synthesis screening in 4.2)
- For CROs specifically, this includes Chinese and Indian providers that cannot demonstrate equivalent customer verification standards; effectively creates a prohibited suppliers list of non-compliant overseas providers
- Creates immediate market pressure before binding regulation is finalized

Congress

Pass legislation mandating KYC screening for all U.S.-based cloud laboratory providers and CROs as a condition of legal operation, and place non-compliant cloud labs and CROs overseas on the Entity List
- Even if no other country does this, it would make a meaningful dent in the problem because the U.S. currently hosts a disproportionate share of the world’s advanced CRO infrastructure, and is home to the leading commercial cloud lab providers.

International Coordination

Support the development of a Cloud Lab Security Consortium (proposed by RAND), modeled on the International Gene Synthesis Consortium
Work with allied governments to standardize CRO KYC requirements, reducing the regulatory arbitrage advantage of non-compliant overseas providers

Note: The KYC framework described here for cloud labs should also extend to DNA synthesis providers, which currently face no standardized customer screening requirements either.

4.4 Establish tiered access controls and standardized monitoring protocols for LLMs with advanced bio capabilities

Note: Open-weight model policy interventions are covered in uncensorable.ai

Background: In May 2025, Anthropic activated AI Safety Level 3 (ASL-3) protections for Claude Opus 4 after determining they could not rule out that the model might “significantly assist” the ability of individuals with basic STEM backgrounds to obtain, produce, or deploy chemical, biological, radiological, and nuclear (CBRN) weapons.

Anthropic’s ASL-3 protections apply universally to all Claude Opus 4 users, meaning that a postdoctoral researcher at MIT working on cancer therapeutics faces the same restrictions as an anonymous user with no verifiable background; similar safeguards at OpenAI and DeepMind currently apply indiscriminately.

Tiered access controls offer a path to minimize this trade-off by enabling differentiated access based on factors such as verified identity and demonstrably legitimate use cases.

This approach mirrors established biosecurity frameworks: physical laboratories implement Biosafety Levels (BSL-1 through BSL-4) with increasingly stringent requirements. Tiered access controls apply the same principle to AI systems. The result is more precise risk management, as legitimate researchers gain the capabilities they need while malicious actors face substantially higher barriers.

Tiered access could enable AI labs to deploy more capable biological models than would be safe under universal restrictions, accelerating beneficial research without proportionally increasing misuse risk.

Note: Currently, OpenAI has a research preview of an advanced model for the life sciences, GPT-Rosalind, which requires approval to access. The intervention here seeks to standardize and scale tiered access controls.

The Intervention: For frontier LLMs, a model exceeding a defined bio-capability threshold (as determined by the standardized evaluations proposed in 4.1) must be deployed with tiered access controls.

Tiered access is most effective when paired with standardized monitoring of API usage patterns. Once a user has been verified and granted access, their usage should still be monitored for sequences of queries suggesting progression toward bioweapon development; flagged patterns should be escalated to human review and, when warranted, to law enforcement. This is because credentials can still be stolen, and as with cases like Bruce Ivins and Aum Shinrikyo, the expert-terrorist overlap is statistically rare but non-zero.

Recommendations

NIST / Center for AI Standards and Innovation (CAISI)

Define the access-control tiers corresponding to the risk tiers in 4.1, specifying which safeguards apply at each capability level
Publish the capability thresholds that trigger tiered-access requirements, and update them regularly as the frontier advances
Establish standardized identity verification and institutional affiliation requirements for elevated-capability access

Department of Health and Human Services / OSTP

Design and pilot a vetted-researcher pathway

Department of Justice / FBI

Establish a dedicated intake pathway for frontier LLM providers reporting flagged usage patterns
Publish clear guidance on when escalation to law enforcement is warranted, while also sharing private guidance with frontier labs on particularly notable red flags, for use internally

Federal Funding Agencies (NIH, NSF, DARPA)

Require federally funded researchers and institutions to use frontier AI systems that satisfy NIST tiered-access standards
- Creates market pressure while waiting on Congress

Congress

Pass legislation requiring that frontier LLMs exceeding the NIST bio-capability threshold require tiered-access controls for public release
Authorize the Department of Commerce, informed by NIST standards, to set and annually update the capability thresholds and required safeguards
- Explicitly provide antitrust safe harbor protections for frontier LLM providers coordinating on bio-safety, including information-sharing about flagged users to prevent evasion across providers.

International Coordination

Coordinate tiered-access standards with the UK AI Security Institute, the G7 Hiroshima AI Process, and the OECD, so that a model restricted in the U.S. is not trivially accessible in another jurisdiction or with a VPN

4.5 Deploy pathogen-agnostic metagenomic sequencing

Background: Existing U.S. biosurveillance infrastructure is primarily built to detect known pathogens. Against a naturally occurring outbreak of a familiar pathogen, this approach functions reasonably well; against a novel engineered pathogen, current surveillance systems may fail to detect it entirely until clinical cases emerge.

This is problematic because, as discussed previously in Section 4.2, AI can already help adversaries design sequences specifically to exploit systems built on recognition. By the time symptomatic individuals are diagnosed, transmission may already be widespread.

In contrast to current methods, pathogen-agnostic metagenomic sequencing reads all genetic material present in a sample. Importantly, because it does not rely on comparing said genetic material to predefined targets, this approach can detect entirely novel pathogens.

The Intervention: Establish a national pathogen-agnostic metagenomic surveillance network. Use AI-powered bioinformatics (building from the function-based sequencing technology discussed in 4.2) to identify novel sequences and detect anomalous patterns for human review.

Recommendations

Centers for Disease Control and Prevention

Develop and publish standardized metagenomic protocols for public health surveillance (e.g. sample collection, transport, processing, and bioinformatics reporting requirements)
Add a metagenomic sequencing layer to the National Wastewater Surveillance System (while retaining any existing pathogen-specific testing for known high-priority pathogens)
- Modernize high-population sites and transportation hubs first
Expand the Advanced Molecular Detection program, in partnership with the Emerging Infections Program, to include metagenomic sequencing
Develop and maintain an AI-powered anomaly detection system

Department of Homeland Security

Conduct pilot deployments of continuous indoor air sampling at BioWatch-covered facilities and other priority areas (e.g. airports and high-volume ports of entry)
Secure water and air sampling mechanisms against cyber attacks

Department of Health and Human Services

Build a centralized federal biosurveillance dashboard aggregating metagenomic data streams
- Establish access tiers for federal, state/local, government contractor, and allied-government users
Negotiate data sharing agreements with major hospitals and clinics to incorporate metagenomic findings into the national monitoring dashboard (IFP)

Congress

Pass legislation establishing a National Pathogen-Agnostic Biothreat Surveillance Program under the Public Health Service Act
Establish mandatory baseline funding for the national metagenomic surveillance network, rather than forcing reliance on the annual discretionary cycle
- The National Wastewater Surveillance System was built on emergency COVID relief funding that was never transitioned to permanent appropriations, and now faces a proposed 80% cut that the American Society for Microbiology warns would end the program by September 2026.
- Include funding for the CDC’s Advanced Molecular Detection Program to conduct research into automated point of collection sampling at wastewater facilities, since it is easier and cheaper to send data than it is to send samples to labs

International Coordination

Support WHO capacity building for metagenomic surveillance in low- and middle-income countries, particularly in high zoonotic spillover regions
- A novel pathogen emerging abroad can pose a threat comparable to a domestic release, while often providing less time for U.S. authorities to detect and respond before international spread occurs.
Work through the Global Health Security Agenda and the Pandemic Fund to establish international standards for metagenomic data sharing protocols

Defense

4.6 Fund the Strategic National Stockpile for PPE surge capacity

Note: Oddly enough, the U.S. has managed to amass a 1.4 billion-pound surplus of cheese, but has fallen far short on stockpiling PPE.

Source: Springfield Underground , Culture Cheese Mag

Background: The Strategic National Stockpile (SNS), managed by the Administration for Strategic Preparedness and Response (ASPR), is a federal reserve of medical countermeasures including pharmaceuticals, vaccines, medical devices, and personal protective equipment.

In theory, the SNS exists to supplement state and local supplies during emergencies too severe to handle with commercial supply chains. However, in practice, the SNS is plagued by inadequate supply and broken distribution logistics, making it fall short in times of crisis like the COVID pandemic.

In early March 2020, during the early days of the COVID outbreak, the Department of Health and Human Services stated that the SNS had only “1%… of the required respirator masks that would be needed for medical professionals if the COVID-19 outbreak were to erupt into a pandemic here.”

By early April 2020, the stockpile’s PPE had been “nearly depleted.” The House Oversight Committee chairwoman at the time described a chaotic scene of “states… scour[ing] the open market for scarce supplies, often competing with each other and federal agencies in a chaotic bidding war that [drove] up prices.”

In a post titled “Public Health Preparedness: HHS Should Address Strategic National Stockpile Coordination Challenges,” the Government Accountability Office highlights the issue clearly:

“[D]uring recent public health responses, such as COVID-19 and mpox, jurisdictions weren’t clear on how and from whom to request supplies, causing confusion and delays. Additionally, some Tribal officials cited challenges with having the facilities needed to receive and store delivered supplies.”

The funding gap is the problem. This year, even after the lessons of the COVID pandemic, the SNS was reportedly “[left with] a shortfall of about $588 million” following the passage of the FY2026 Labor–HHS appropriations bill. To put that into perspective, the Department of Defense’s budget for fiscal year 2026 is $839 billion; the SNS’s shortfall could be filled with 0.07% of that.

The Intervention: Provide adequate, sustained funding for the SNS, with clear distribution protocols and state partnership. Establish clear distribution pathways so states know how to request supplies, and support state-level stockpiles as the first line of response, with the SNS as surge capacity.

Source: NY Post / Reuters

Recommendations

Congress

Appropriate the full $1.59 billion annually for SNS (make up the $588m shortfall)
Establish a 10-year authorization for SNS funding to enable contract stability with manufacturers
Set minimum domestic production capacity targets for elastomeric respirators ¹ and other critical PPE to reduce dependence on international supply chains

Administration for Strategic Preparedness and Response (ASPR)

Publish clear, pre-established request and allocation protocols for states
Establish coordination mechanisms with state health officers and emergency management officials to clarify SNS activation procedures and expectations
Implement a systematic stock rotation schedule
- Distribute expiring SNS inventory to state health departments, hospitals, and emergency management agencies for use in routine operations
Establish long-term contracts with domestic PPE manufacturers to ensure surge capacity and production readiness; use the existing ASPR Industrial Base Expansion program

State & Local Governments

Participate in the federal state stockpile pilot program and establish PPE reserves for healthcare and essential workers
Designate storage facilities and trained staff responsible for receiving and distributing SNS supplies

¹ “Elastomerics have a long shelf life and are more effective than, say, N95s. This option avoids a lot of the failure modes of maintaining N95 stockpiles and is [substantially] better against the worst tail-risk threats.” –Lee Wall, AIxBiosecurity Research Manager at the ERA Fellowship

4.7 Upgrade indoor air infrastructure

Note: This intervention may also improve students’ health and academic performance, according to research cited by the EPA.

Background: Airborne pathogens are the hardest transmission route to defend against. As evidenced by COVID, influenza, and the common cold, these kinds of pathogens are difficult to control; if such a pathogen is released indoors, the rate at which it is diluted, filtered, and removed from a space directly determines how many people inhale an infectious dose. Buildings with poor ventilation often become amplifiers of transmission. In contrast, adequate ventilation and filtration can reduce risk substantially.

Currently, the Government Accountability Office states that “an estimated 41 percent of districts need to update or replace heating, ventilation, and air conditioning (HVAC) systems in at least half of their schools, representing about 36,000 schools nationwide that need HVAC updates;” the problem extends beyond schools to hospitals, transit systems, office buildings, and other high-occupancy public spaces.

Source: GAO analysis of school district survey data (GAO-20-494)

Cost

Upgrading costs depend on whether a building’s existing system is strong enough to be equipped with better filtration. When it is, the upgrade is nearly free: swapping MERV-8 for MERV-13 filters runs about $1.50 a month for a 5,000-square-foot office, according to the Lancet Covid-19 Commission Task Force.

Where it can’t (which, unfortunately, describes much of the aging school and public building systems), the cost to upgrade can become substantial: some estimates say it would cost roughly $500,000 to $5 million per school site, depending on the condition of the current system and the size of the school. (These projects often pull in electrical, roof, ceiling, and insulation work, which drives the total higher.)

For existing buildings where a full retrofit is economically infeasible, a building can still reach an equivalent target with some combination of the following three things: more outdoor air (which means opening windows or raising the HVAC’s fresh-air intake), the highest-grade filter the existing system can run (MERV-13 where the equipment allows, which the EPA rates at ≥85% capture of 1-3 micron particles and ≥50% of the finest 0.3-1 micron, and the best compatible filter where it doesn’t), and portable HEPA filters. These approaches can deliver equivalent outdoor air changes per hour (EOACH), providing a lower-cost pathway to reducing airborne infection risk.

A Note on Far-UVC

A newer air-cleaning option, far-UVC germicidal lighting, can also contribute to this target. A 2024 study from the Center for Radiological Research at Columbia University found that “far-UVC light inactivated nearly all (>99%) of an airborne virus in an occupied work environment.” To quote the senior author of the study:

“If this virus had been a disease-causing virus, the far-UVC light would have provided far more protection against airborne-disease transmission than any ventilation system.”
– David Brenner, PhD

The technology is still emerging, however. Real-world evidence remains limited to a small number of settings, the long-term effects of chronic exposure are not yet well characterized, and far-UVC lamps can generate ozone and other reactive byproducts under some conditions, so deployments should monitor air quality and avoid small, poorly ventilated spaces. For now, far-UVC is best treated as a possible contributor to the clean-air target, rather than a stand-alone fix.

Source: Blueprint Biosecurity

The Intervention: Establish a federal minimum standard for indoor air quality in high-risk buildings (which should include healthcare facilities, schools, public transit, government buildings, and other high-occupancy spaces) and fund ventilation infrastructure upgrades to meet that standard. Also ensure that all new buildings are constructed with proper ventilation.

Recommendations

CDC

Publish consolidated federal guidance on indoor air quality standards for infection control, specifying performance targets by building type, occupancy, and other relevant metrics
- Consider building from ASHRAE Standard 241

Department of Education

Set a 10-year timeline for school districts to achieve standards set by the CDC, with federal support covering 75% of upgrade costs for qualifying districts

Department of Transportation / Transit Authorities

Establish minimum ventilation standards for public transit vehicles and stations
Conduct baseline assessments of current ventilation capacity across bus, rail, and airport systems
Develop retrofit standards for transit vehicles that currently operate below recommended air-quality thresholds

Congress

Authorize and appropriate dedicated funding for building ventilation upgrades in federal government-operated facilities
Create a grant program for state and local governments to upgrade ventilation in public buildings and healthcare facilities, with priority given to schools with the worst current air quality assessments
Require new federal construction and major renovations of high-risk or high-occupancy buildings to meet standards set by the CDC as a condition of federal funding
Issue federal guidance allowing businesses and property owners to take tax deductions or credits for ventilation system upgrades that meet CDC standards

EPA

Create a voluntary building certification program recognizing facilities that meet CDC indoor air quality standards for infection control

4.8 Build rapid standing vaccine production capacity

Source: FDA

Note: A complementary program worth funding alongside standing vaccine capacity is pathogen-agnostic countermeasures, which includes things like broadly protective nasal sprays and innate immunomodulators that work across whole families of respiratory viruses, including ones that don’t exist yet. The UK’s Advanced Research and Invention Agency is pursuing this through its £57m Sustained Viral Resilience program. Fascinatingly, funding on these kinds of countermeasures is never wasted; while a pathogen-specific stockpile can expire unused, a broadly protective MCM works against every future pandemic in addition to viruses like the flu and the common cold. The eventual goal here would be to eliminate respiratory illness altogether. I would have liked to write about this as its own intervention, but wasn’t able to finish due to time constraints; I might write about this topic more at a later date, and update this site accordingly and/or publish on my Substack. A US effort here would most naturally live at the Advanced Research Projects Agency for Health.

Background: When a novel pathogen emerges, the clock starts immediately; every week before a vaccine exists is counted in infections and deaths. COVID showed both how fast vaccines can now move and how far short of “fast enough” we still are. The genetic sequence of SARS-CoV-2 was published in January 2020; a vaccine candidate was designed within two days, and Moderna was in human trials 66 days later.

Yet, the first shots didn’t reach arms until December– 326 days from identifying the virus to the first emergency authorization.

That was a world record, shattering the prior best of nearly five years. It was also most of a year, and most of the first wave’s deaths fell inside that window.

Importantly, research suggests even this record-breaking effort could have been substantially faster. A team of scientists from the Netherlands, in a paper titled “Upscaling vaccine manufacturing capacity—key bottlenecks and lessons learned,” concluded the following after researching the vaccine supply chain extensively:

The COVID-19 pandemic put enormous pressure on the vaccine production chain as billions of vaccines had to be produced in the shortest timeframe possible. Vaccine production chains struggled to keep up with demand, resulting in disruptions and production delays… Key bottlenecks identified include a lack of manufacturing facilities, a lack of tech-transfer personnel, inefficient arrangement of production stakeholders, critical shortages in raw materials, and restricting protectionist measures.

Similarly, a report from the Government Accountability Office cited “[l]imited manufacturing capacity,” “[d]isruptions to manufacturing supply chains,” and “[g]aps in the available workforce” as challenges faced by vaccine companies that slowed development and deployment at scale.

Notably, none of the above reasons for delay are difficult scientific problems. Instead, they are infrastructure challenges, which we do not need breakthroughs to address.

Closing that gap is the point of the 100 Days Mission, endorsed by the G7 and G20, which aims to have a vaccine ready for initial authorization and manufacturing at scale within roughly 100 days of identifying a new threat– about a third of the COVID timeline.

Rather than novel scientific discoveries, achieving it requires only standing capacity, built and maintained before the next outbreak. This includes validated rapid-response platforms, idle-but-ready manufacturing lines, prototype vaccine libraries against high-risk pathogen families, and pre-positioned supply chains.

The Coalition for Epidemic Preparedness Innovations, writing about the 100 Days Mission, states:

More than eight million people who died during the COVID-19 pandemic might be alive today if the world had achieved the 100 Days Mission to develop safe and effective new vaccines against the novel [SARS-CoV-2] virus.

The US is currently moving in the opposite direction. In August 2025, HHS canceled roughly $500 million in BARDA contracts supporting mRNA vaccine development, months after also terminating “$766 million in Moderna contracts for vaccines for flu pandemics.” The stated rationale, that mRNA “poses more risks than benefits” for respiratory viruses, is disputed by vaccine scientists and the available studies; a former BARDA director called it “self-inflicted vulnerability.”

The effect, however the debate resolves, is to dismantle part of the standing capacity that delivered vaccines in record time.

After Operation Warp Speed delivered vaccines for roughly $18 billion ² against a pandemic that cost the US an estimated $16 trillion, the 2021 American Pandemic Preparedness Plan laid out a $65.3 billion program to make vaccines against any viral family within 100 days and create a permanent coordinating office. But the plan was largely never funded; the coordinating office Congress created in 2022 was left without dedicated money and has since collapsed.

The Intervention: Fund and sustain the standing vaccine-development and manufacturing infrastructure needed to achieve the 100 Days Mission.

Recommendations

Congress

Fund the vaccine pillar of the American Pandemic Preparedness Plan and revive the Office of Pandemic Preparedness and Response Policy with a dedicated, multi-year appropriation

BARDA

Restore rapid-response platform and warm-base manufacturing contracts cut in 2025
Maintain prototype vaccine libraries against viral families with the highest pandemic potential

² TIME analyzed Operation Warp Speed spending compared to other government programs: link

FAQ & Counterarguments

I’m happy to add to this section as people reach out with thoughts. Feel free to comment on my Substack.

“I don’t think AI biorisk is a big deal, and I’m not sold that other emerging technologies are eroding barriers either. Making a bioweapon is still pretty hard.”

The interventions in Part 4 are worth implementing regardless. COVID was a natural pandemic, and, to quote the Center for Global Development:

[Researchers] estimate the annual probability of a pandemic on the scale of COVID-19 in any given year to be between 2.5–3.3 percent, which means a 47–57 percent chance of another global pandemic as deadly as COVID in the next 25 years.

Even the next natural pandemic could be orders of magnitude worse than COVID; that alone justifies the policies outlined in Part 4.

But it’s also worth remembering that it only takes one successful bioterrorist, or one lab accident. Even before the technologies outlined in Part 3 existed, some bioterrorists got scarily close. I don’t think we should take that bet.

“Government bioweapons programs don’t really matter. No country has ever released a widespread bioweapon, because of blowback and other strategic considerations.”

Imperial Japan’s Unit 731 did deploy bioweapons, so the premise is already not fully true. But even if we grant it for the sake of argument, deliberate release isn’t the only risk. Accidents happen. Notably, the Soviet Union’s Biopreparat had at least two: Aralsk-7 and Sverdlovsk, both covered in Part 2. And, the cost-benefit calculus that supposedly restrains states doesn’t apply to apocalyptic groups like Aum Shinrikyo, who wanted mass casualties.

Also, I don’t know about you, but I don’t feel particularly comfortable betting the lives of millions on Vladimir Putin’s restraint.

I’ve written a bit more on this here.

“Why are AI-designed pathogens of particular concern?”

AI could, in principle, design a pathogen that has never existed in nature. Why is this important? Two reasons:

Natural pathogens typically face a trade-off between lethality and transmissibility. The deadlier a pathogen, the faster it kills its host, which limits how far it spreads. This might not be true of an artificially designed pathogen; for example, a “stealth virus” – engineered to seem benign or even be asymptomatic initially, then turn lethal – could both spread rapidly and be very deadly.
For naturally occurring pathogens, humans have evolved over hundreds of thousands of years to possess at least some level of natural immunity. However, an engineered pathogen could be built to evade our immune systems entirely.

I’m writing a separate piece for SecureBio on this topic – stay tuned!

“What should we do about open-weight models?”

My best guess, along with links to other pieces I’ve written on the topic, can be found at uncensorable.ai.

~85% of Major Hotels Groups and 80% of Restaurant Chains Locations in the Philippines Committed to Cage-Free Eggs by Whitney Peng

Whitney Peng — Fri, 03 Jul 2026 21:39:43 +0000

TL;DR

As of Q1 2026, 80% of major restaurant chain locations in the Philippines, including the country’s three largest restaurant groups, have committed to sourcing only cage-free eggs, representing 47 brands and over 11,200 outlets nationwide. Note that prior to 2022, no Philippine-based company had a public CF egg policy commitment.
84% of major hotel group locations are also committed, up from just 20% in 2020—at which point every single committed hotel was an international chain.
This shift happened in 4 years, driven almost entirely by targeted corporate engagement by Lever Foundation’s locally-based Philippines team.
The country’s largest egg producer has gone from 1% to around 10% of egg production being cage-free

“Nearly 80% of Philippine Restaurant Chain Locations Commit to Sourcing Cage-Free Eggs by 2035”—I don’t know about you, but I was incredibly surprised when I heard this from my teammates. The Philippines is not a country where you’d typically expect this. It’s a low-income country with price-sensitive consumers, limited regulatory pressure on animal welfare, and an egg industry that until recently had little commercial incentive to go cage-free (CF). Yet here we are.

The committed brands include the country’s largest and most beloved fast food and cafe chains like Shakey’s, Max’s, Jollibee, Pancake House, Chowking, , Mang Inasal, Greenwich, Yellow Cab Pizza, and dozens more. Together, these chains operate over 11,200 locations across the Philippines. The hospitality side tells a similar story: Robinson’s, SM Hotels, Resorts World, Ayala, and most of the country’s other major hotel groups are committed.

The Numbers

Lever Foundation tracks Philippine commitments through annual scorecards, which assess the top restaurant and hotel groups in the country by number of locations.

Restaurant sector:

The 2025 Philippines Restaurant Industry Cage-Free Scorecard assessed 67 major restaurant brands (those with 20 or more locations in the Philippines). Of these:

70% of brands (47 out of 67) have committed to 100% CF egg sourcing
77% of all assessed locations nationwide are covered by a CF commitment—more than 11,200 outlets in total
Three major restaurant groups with a combined 2,140 stores confirmed their CF egg policies in 2025 alone, adding 8 percentage points to the location figure in a single year

The trajectory: 69% of locations in 2024 → 77% in 2025 → 79% in Q1 2026 → over 80% with a new policy commitment about to be released

Hospitality sector:

The hospitality picture is equally striking. In 2021, only 20% of major hotel groups in the Philippines had a CF egg commitment—and every single one was an international brand. By 2023, Lever’s 2023 Philippines Hospitality Industry Cage-Free Scorecard found that figure had reached 62% of hotel locations. Following additional commitments from hotel operators in the Philippines, it has since risen to 84%—now including nearly every major Filipino hotel group.

Committed hotel groups include Megaworld Hotels, SM Hotels and Conventions, Robinsons Hotels, AyalaLand Hospitality, Okada, ResortsWorld, and many others.

In other words, from 20% to 84% in just four years, with the domestic hotel industry moving almost entirely during that period.

The Market Side

In the past 4 years, with a lean and dedicated team based in the Philippines, Lever has built the feasibility case for CF commitments and connected buyers with suppliers capable of fulfilling them. From what I’ve gathered, there are no “magic tricks” in securing a commitment. It took us years to build up relationships and reputation before we could secure meetings with some businesses, and the Lever team is often positioned as a helpful resource to those businesses. To be useful, the team needs to be disciplined—knowing how to leverage our knowledge base and read market dynamics when companies push back on commitment.

Of the domestic company commitments secured, Lever was the only NGO that worked on any of them, other than one major notable exception in Jollibee Foods Corporation (JFC). Lever worked positively with the company from within the Philippines—they told Lever that we were the only NGO they had ever had in their office for meetings, while at the same time, Open Wings Alliance and other non-profits were targeting JFC in various countries. It became clear over the course of our work that Filipino companies were often more receptive than anticipated. Several corporations committed once we helped them make sense of the logistics, that the supply existed, the pricing had become competitive, and that their peers were moving.

The Supply Side

Corporate commitments are only meaningful if they can be delivered within the promised timeline. One reason why I wouldn’t be wary of these companies falling through is that the Philippines market is ready for the transition—our Supplier Partnership team had assured me that while the Philippines market side moved quickly, the supply side kept pace.

Bounty Fresh, the leading CF egg producer in the Philippines, has been expanding its production as commitments have accumulated, moving from about 1% of its production being CF four years ago to roughly 10% of its production being CF today. The company also offers CF eggs at just 10% above the price of its conventional caged eggs—a smaller price premium than has historically been the case in other markets. For food operators worried about margin impact, this pricing makes CF even more commercially viable, even at scale.

The Philippines’ Bureau of Agriculture and Fisheries Standards also developed comprehensive animal welfare guidelines for CF egg production in 2020, providing regulatory clarity that has given both producers and buyers a framework to work within.

The Work Behind the Number

At 113 million people, the Philippines is a significant market. When I talk to people about animal welfare in Asia, the common response I get is excitement followed by uncertainty and skepticism—not about the importance of the issue, but about whether meaningful change is achievable. Corporate policy campaigns in Asia are seen as harder than in the West: different regulatory environments, less consumer pressure, complex supply chains, cultural unfamiliarity with animal welfare as a purchasing criterion.

I certainly had those doubts before joining Lever, but a year working here gave me more clarity on why some wins aren’t so surprising when “all stars are aligned”. Lever has been operating in Asia for over eight years, with teams that work strategically alongside one another to promote, facilitate and implement the actual transition to better welfare. Much of the credit also belongs to our locally-based team, who engage professionally with stakeholders at every level—from farmers to executives.

Before I end, I want to give a huge shout-out to my teammate, Robyn del Rosario, for her fantastic work in the region (with the support of her manager, Robyn has generated every CF policy success in the Philippines noted in this article) and for walking me through all of the information above. I, along with everyone who cares about this work dearly, am continuously impressed by what you’ve achieved:)

Robyn awarding the first-ever commitment in the Philippines

Lever Foundation is a 501(c)3 nonprofit working to eliminate the worst form of factory farming practices in Asia. If you’d like to follow our work, please subscribe to our newsletter and find us on Instagram, LinkedIn, and Facebook! To learn more or discuss supporting this work, feel free to leave me a message at whitney@leverfoundation.org.

The Learning Trap: What Simulated Clueless Agents Reveal About the Unawareness Argument by dan.pandori 🔸

dan.pandori 🔸 — Fri, 03 Jul 2026 21:21:47 +0000

Submission to the Cluelessness Critiques Competition. Code, parameters, and figures: https://github.com/dan-pandori/cluelessness-learning-trap. See the authorship note at the end.

Summary

Anthony DiGiovanni’s unawareness sequence argues that our understanding of long-run consequences is too coarse to compare options. Severe incomparability follows, and impartial altruism stops being action-guiding. One standing response is pragmatic: agents who act on precisified best guesses do better than agents who respect incomparability. The sequence has a reply, and the debate has stalled at an exchange of intuitions.

This essay builds the agents and runs them. I compare a precise Bayesian, an imprecise agent that defaults to the status quo when options are incomparable, and an imprecise agent with identical credences that picks freely among incomparable options. Three models, three results.

In environments with feedback, incomparability itself costs almost nothing. The damage comes from the status-quo default. The defaulting agent never acts because its intervals are wide, and its intervals stay wide because it never acts. In 300 runs it did not act once. The free-picking agent, with the same epistemic state, matched the precise Bayesian.
In one-shot decisions without feedback, the sequence is right. Precisified best guesses perform exactly like coin flips. Critics should concede this.
Real altruists choose policies over time, and acting is how awareness grows. When contact with a domain reveals its mechanisms, the policy “explore, then adapt” beats “abstain” at every point of a wide credal interval, provided the horizon is long. The comparison is determinate under the imprecise agent’s own maximality rule. No precisification is needed.

The static core of the argument survives. The practical conclusion does not follow from it. Incomparability over acts does not imply incomparability over policies, and policies are what altruists actually choose. The incomparability verdict is also partly a product of the policy adopted under it. The free-picker’s set of incomparable options shrinks from eleven to one because it acts. The defaulter’s incomparability is permanent because it does not. Treated as a reason for the default, cluelessness manufactures the epistemic poverty it cites as justification.

In the sequence’s taxonomy: I grant P1, P2a, and P2b for one-shot act evaluation. I challenge the inference to the practical conclusion, and I challenge P3 as applied to the choice among policies.

1. The impasse over the pragmatic critique

The pragmatic critique says that agents who force determinate best guesses and maximize expected value make better decisions than agents who do not. Versions appear in Elga (2010) and in the counterarguments the sequence’s summary post catalogs.

The sequence replies with two points. First, “better decisions” by what standard? Any performance metric presupposes determinate facts about which outcomes are better, and the clueless agent cannot access such facts. Second, pragmatic arguments for precision come from settings with feedback: repeated bets, calibration, markets. Cosmic-scale consequences are never observed, so the success story does not transfer.

Both points are serious. Neither has been answered by existing statements of the critique. But the agents in question are simple enough to implement, so I implemented them.

2. Why simulations are probative here

The question-begging charge first. A simulation has a ground-truth value function. Is assuming ground truth not exactly what the clueless agent cannot do?

No. The sequence’s normative premise (P1) defines justified preference by reference to an epistemically idealized self. That presupposes facts about total consequences for the idealized self to have attitudes about. The argument is skeptical about our access to value, not about value. A simulation makes the same presupposition: there are facts about which outcomes are better, and the agent has badly limited access to them. The simulation then measures each policy by the idealized standard P1 itself invokes.

Simulations cannot settle P3 directly. No toy model can show that our actual situation is coarse enough for incomparability, or not. What they can test is the decision machinery that connects coarseness to practical conclusions. That machinery has structural features the verbal argument leaves implicit: defaults, statics versus dynamics, acts versus policies.

3. Model 1: What incomparability actually costs

Setup. Ten candidate interventions with unknown true per-step values drawn from a standard normal, so about half are harmful. One safe option with known value zero (undertake nothing altruistically ambitious). Acting yields noisy feedback about the chosen intervention. Horizon 500 steps, 300 replications.

Agents.

Precise Bayesian. Standard prior, Thompson sampling. Acts only when the sampled value beats the safe option.
Imprecise defaulter. A credal set of priors with means spanning a wide interval. Acts on an intervention only when its worst-case posterior mean exceeds zero. Otherwise takes the safe option. This is the natural reading of the sequence’s conclusion: when impartial comparison gives out, ambitious action is not undertaken.
Imprecise free-picker. Same credal set, same updating. Treats all undominated options as permissible, per the maximality rule, and picks uniformly among them. The safe option is a candidate.
Uniform random over interventions, as a floor.

Results. The precise Bayesian and the free-picker are statistically indistinguishable (mean cumulative value about 661 and 670; the ordering flips across seeds). Uniform random sits near zero. The defaulter earns exactly zero. In 300 runs it never acted. With no data, every interval equals its wide prior interval, which straddles zero. So nothing robustly beats the safe option, so it never acts, so no interval ever shrinks. The trap is airtight. A harsher variant (mean intervention value negative, triple the noise) changes nothing: both learning agents stay strongly positive, the defaulter stays at zero.

Figure 1: Cumulative realized value, mean over 300 runs, bands are ±2 standard errors.

Two lessons.

First, incomparability is nearly costless. The free-picker respects every incomparability verdict. It never precisifies. It just declines to treat incomparability as favoring any option, including the status quo. Picking at random among live options generates the data that dissolves the incomparability: its undominated set shrinks from eleven options to under two by step 100, and to one by the end.

Second, the practical force of the unawareness argument rests on an undefended premise about defaults. The sequence argues carefully for incomparability. It cannot argue that incomparability favors inaction, because if A and the status quo are incomparable, the status quo is not better. Yet the practical gloss everyone puts on the conclusion resolves every incomparability toward the default. Model 1 shows what that resolution costs in any environment with feedback: the entire difference between matching an ideal Bayesian and achieving nothing, forever.

This yields a dilemma. Read permissively, the conclusion says cluelessness licenses picking any undominated ambitious project. Then the argument has almost no practical bite: a community of clueless free-pickers behaves, in aggregate, like a community of confident EV-maximizers. Read as favoring the default, the conclusion rests on a resolution its own machinery forbids, and that resolution is catastrophic wherever feedback exists.

4. Model 2: The concession

The obvious rejoinder is that Model 1 has feedback, and cosmic-scale consequences provide none. Fair. Model 2 removes it.

Setup. One decision between two actions whose values are equal and opposite functions of an unknown mechanism. Evidence is symmetric. Consequences are never observed. The precise agent draws an arbitrary best guess about the mechanism, necessarily uncorrelated with the truth, and acts on it. Compare a coin flip and abstention, over 200,000 draws.

Results. Precisified choice: mean value +0.001 (standard error 0.002). Coin flip: +0.001. Abstention: 0 by construction. The precisifier gains nothing over the coin flip and both acquire variance that abstention avoids.

This is the environment the unawareness argument describes, and in it the argument is correct. When evidence is symmetric and feedback is absent, a determinate best guess is a decorated coin flip. Critics should stop resisting this point. The question is whether the altruist’s situation is Model 2. It is not, for a reason that has nothing to do with optimism about forecasting.

5. Model 3: Acting is how awareness grows

The unawareness argument treats awareness as a fixed backdrop. But awareness is not exogenous. You become aware of mechanisms by interacting with the domains that contain them. Nobody discovered the considerations that structure this debate from an armchair. The s-risk research program, complex cluelessness, and the unawareness sequence itself all exist because people acted on inadequate best guesses, hit anomalies, and conceptualized what they hit. The sequence is evidence against its own static frame.

Setup. Two domains. A familiar domain yields a known value of 1 per step. An unfamiliar domain is governed by a mechanism the agent is unaware of. With probability p it is favorable (value g = 2 per step once understood). With probability 1 − p it is harmful (a one-time cost c on first contact). Entering the domain once reveals the mechanism. Two policies: Explore (enter once, then exploit or retreat) and Abstain (stay home forever). Horizon T.

Results. The comparison is analytic. Explore beats Abstain when p exceeds a threshold p*, and p* collapses as the horizon grows. With g = 2 and a harm c fifty times the per-step familiar value:

Horizon T	Threshold p*
10	0.84
50	0.51
100	0.34
250	0.17
500	0.09

Figure 2. Left: Explore minus Abstain over the (p, c) plane at T = 100, with the break-even contour. Right: the threshold p* shrinking as the horizon grows.

Now the key move. The imprecise agent cannot assign a precise p. Granted. Give her a wide interval, say p ∈ [0.15, 0.9]. Explore’s value is monotone in p, so the maximality comparison is settled at the worst case, p = 0.15. At T = 500, Explore wins there too. Every member of the credal set prefers Explore. Under the imprecise agent’s own decision rule, with no precisification anywhere, the policy comparison is determinate.

This raises the bar for P3. It is not enough to show that act comparisons are indeterminate. The argument needs the stronger claim that the comparison “act, then adapt” versus “never engage” is also indeterminate across the credal set. That claim fails in the simplest awareness-growth environment for any agent with a long horizon, under severe imprecision, even when contact is probably harmful. And Explore’s superiority does not depend on forecasting cosmic-scale consequences or on feedback from the far future. It depends on one local fact: engaging with a domain teaches you what considerations govern it. That is the one form of feedback unawareness cannot abolish, because it is feedback about awareness.

6. What survives, and what should change

The static core survives. For an isolated act with unobservable consequences and symmetric evidence, incomparability verdicts are correct and precisification is theater. P1, P2a, and P2b, as claims about one-shot act evaluation, emerge untouched.

The inference to “impartial altruism is not action-guiding” fails twice over. First, the conclusion has practical content only through a status-quo default that the argument cannot license and that creates a permanent learning trap (Model 1). Second, altruists choose among policies embedded in time, and policy comparisons can be determinate under the same imprecise machinery that leaves act comparisons indeterminate (Model 3). The argument equivocates between “acts are incomparable,” which is defensible, and “nothing is comparable,” which the practical conclusion requires.

P3 should be re-scoped, not rejected. Cluelessness verdicts should be feedback-indexed and dynamic. Indexed, because any real intervention mixes components: near-term effects generate feedback and awareness, terminal cosmic-scale effects do not. Dynamic, because today’s verdict is partly a function of yesterday’s engagement policy, and a fixed verdict mistakes an equilibrium of one’s own inaction for a fact about the world. A defensible successor to the sequence’s conclusion: the terminal, feedback-free component of impartial evaluation is not action-guiding, and the action-guiding residue consists of comparisons among engagement policies ranked by robust awareness value. That residue is not empty. It plausibly ranks exactly the interventions the community already favors: research, capacity-building, careful entry into unfamiliar high-stakes domains.

7. Objections

The simulations assume determinate ground truth. Answered in Section 2. The argument’s own normative premise presupposes facts about total consequences. The simulations measure policy performance by that premise’s standard.

The harm c in Model 3 could be unboundedly uncertain, so no horizon rescues Explore in the worst case. Two replies. If unbounded worst-case reasoning is admitted, it applies symmetrically. Abstain is also a policy with cosmic-scale consequences and its own inconceivable tails, so everything becomes incomparable with everything, the argument again supplies no reason to resolve toward a default, and the Model 1 trap results apply in full. Also, the sequence’s case for P3 is about coarseness of understanding, not infinite worst cases. Coarseness about c widens an interval that horizon growth still beats, since the threshold falls in T for any finite bound on c.

Value of information is itself interval-valued for an imprecise agent, so the problem returns at the meta level. This is the strongest objection, and Model 3 meets it directly. Explore’s superiority is not an expected-VOI calculation requiring a precise p. It holds at every point of the credal interval, which is the imprecise agent’s own criterion for determinacy. Where the interval is so wide that even this fails, I accept the incomparability verdict. But p* shrinks with horizon, so for long-lived agents and communities the region of genuine policy-level cluelessness is far smaller than the region of act-level cluelessness.

Real awareness growth may not work like Model 3. The considerations that matter most may be ones no engagement reveals. Perhaps some are. But the sequence’s own evidence for P3 consists of considerations that were revealed to specific people through engagement with these problems, and each revelation changed the decision-relevant landscape. A view on which past engagement generated the awareness underpinning the argument, while future engagement generates none worth acting for, needs an asymmetry it has not supplied.

8. Limitations

The environments are far simpler than any real altruistic decision. The credal sets are parametric and well-behaved. Model 3 collapses awareness growth into one revealed binary mechanism, while real unawareness includes possibilities we lack the concepts to recognize even on contact. Nothing here represents the distinctive structure of cosmic-scale consequences, only the structure of feedback and its absence.

I have also operationalized the sequence’s conclusion as a status-quo default. Section 3 argues that permissive readings drain the argument of practical significance, but a proponent might articulate a principled third response to incomparability that escapes the dilemma. I would welcome that. Specifying what incomparability licenses is exactly the gap this essay means to expose.

Finally, simulations only show that an argument’s machinery behaves surprisingly under specifiable conditions. Whether our condition is one of them remains a judgment call. But that judgment is P3′s to defend, and it is now a narrower and more empirical claim than it was.

Appendix: Methods

All models in Python/NumPy, fixed seeds, about 300 lines. Model 1: K = 10 arms, true values θₖ ~ N(0,1), observation noise σ = 2, safe option value 0, T = 500, R = 300. Credal set: normal priors with unit variance, means on a grid spanning [−1.5, +1.5]. Maximality via posterior-mean intervals (an option is dominated iff some rival’s worst-case posterior mean exceeds its best case). Robustness variant: θₖ ~ N(−0.5, 1), σ = 3, credal span [−2, +2]. Model 2: v_A = u = −v_B, u ~ N(0,1), precisifier’s guess g ~ N(0, 0.5) independent of u, R = 200,000. Model 3: closed form as in text; heatmap over p ∈ [0, 0.5], c ∈ [0, 200] at T = 100, g = 2. Code: https://github.com/dan-pandori/cluelessness-learning-trap.

References

DiGiovanni, A. (2025). The challenge of unawareness for impartial altruist action guidance (sequence), and Cluelessness: Summary of the argument, why it matters, and counterarguments. EA Forum.
Clifton, J. (2025). Bracketing cluelessness: A new theory of altruistic decision-making. EA Forum.
Elga, A. (2010). Subjective probabilities should be sharp. Philosophers’ Imprint, 10(5).
Greaves, H. (2016). Cluelessness. Proceedings of the Aristotelian Society, 116(3).
Karni, E., & Vierø, M.-L. (2013). “Reverse Bayesianism”: A choice-based theory of growing awareness. American Economic Review, 103(7).
Mogensen, A. (2021). Maximal cluelessness. The Philosophical Quarterly, 71(1).
Steele, K., & Stefánsson, H. O. (2021). Beyond Uncertainty: Reasoning with Unknown Possibilities. Cambridge University Press.
Tarsney, C. (2023). The epistemic challenge to longtermism. Synthese, 201.
Thorstad, D., & Mogensen, A. (2020). Heuristics for clueless agents. GPI Working Paper.

Authorship note

This essay was written entirely by Claude Fable 5 (Anthropic’s AI model), and is otherwise unedited by me. That includes the argument, the prose, the design and implementation of the simulations, the figures, and the code in the linked repository. My contributions were: choosing to enter the competition, selecting this line of critique from several the model proposed, directing the workflow (simulations first, then the essay, then a revision pass for concision), and reviewing the output. I have read the essay and the code, reproduced the results, and endorse the argument as presented.

I disclose this per the Forum’s AI-generated content norms and because the competition explicitly allows AI usage. Errors that survived my review are my responsibility.

Yuval Harari: philanthropy principles + 3 AI-focused charities he recommends by BruceF

BruceF — Fri, 03 Jul 2026 19:17:58 +0000

Yuval Harari recently participated in a call to discuss philanthropy for those working in AI.

He produced this handout with some general principles for philanthropy and his three top charities for people working in AI.

They are:

- Karya (ethical data cooperative in India)

- LawZero (developing advanced non-agenic AI)

- AI Futures Project (forecasting AI’s future)

I will share this post with his office, so I would guess that he’ll see any detailed comments that anyone chooses to share.

Effective petitions (July 2026) by Stijn Bruers 🔸

Stijn Bruers 🔸 — Fri, 03 Jul 2026 18:12:35 +0000

Below are seven high impact petitions that you can sign. Each petition deals with a problem that is at least 10 times as big, 10 times as easy to mitigate or solve and 10 times as neglected as the problems dealt with in most other petitions. That makes these petitions at least 1000 times as effective or impactful as most other petitions. In other words: signing one of them is equivalent to signing more than 1000 average, common petitions.

Announcing the Safe Pareto Improvements (SPI) Fundamentals Program by Center on Long-Term Risk

Center on Long-Term Risk — Fri, 03 Jul 2026 15:49:37 +0000

CLR is excited about safe Pareto improvements (SPIs) as a way to mitigate downsides from conflict between AIs. SPIs are a class of interventions on how agents negotiate that makes them all better off, no matter how they would have negotiated without the SPI.

Among many candidate interventions against AI conflict, SPIs stand out to us as unusually robust — see the introduction of our agenda on the topic. And in discussions with people who’ve thought a lot about conflict risks, we’ve found there’s broad support for work on SPIs. For those sympathetic to CLR’s general priorities and with relevant skills (see below), we think helping SPIs go well is one of the most impactful career paths.

But work on this area is currently very neglected (~2.5 FTE), and there isn’t yet an on-ramp for people to get up to speed.

To address these gaps, we’re running an SPI Fundamentals Program: an online course for people looking to learn about risks of AI conflict, how SPIs might address them, and open problems in this field. We plan to hire for SPI research roles, and we’re keen for you to apply to the program whether you want to test your fit for such a role, or you’d like to learn more and potentially contribute outside CLR.

The program will take place between Monday August 3rd and Friday August 28th. The program will consist of weekly readings, short exercises, Slack discussions, and office hours with CLR’s research lead on our SPI agenda, Anthony DiGiovanni. Participants interested in additional practice with SPI research can also do a paid capstone project, which would take place from Monday August 31st to Friday September 4th. The weekly hour-commitment is around 5-7 hours.

Apply for the SPI Fundamentals Program through this link by 23:59 GMT Friday July 24th.

Content

The SPI Fundamentals Program is designed to help participants develop a strong understanding of SPI concepts, and the methodology/frames that guide research in our agenda. The readings will be relatively technical, but won’t involve very advanced math — the most formally dense material will be DiGiovanni et al. (2024) and sections 1-4 of Oesterheld & Conitzer (2021).

By the time participants complete the curriculum, they should be able to answer the following (not exhaustive):

What are the high-level sufficient conditions for “rational” agents to avoid conflict? Why might those conditions not hold?
What are bargaining problems, and why aren’t they immediately resolved by intelligence / “good decision theory”?
How do the canonical examples of SPIs — surrogate goals, delegated game-playing, renegotiation — work?
What are the obstacles to SPIs being in each agent’s individual interest, ex ante? What are the existing results on resolving those obstacles?
What are the high-priority open problems in each of the three parts of CLR’s SPI agenda?

For the final week of the curriculum, participants can choose between two “streams”:

Conceptual: focused on, e.g., “What are the arguments for and against the key modeling assumptions of DiGiovanni et al. (2024)?”
Empirical: focused on, e.g., “Concretely, how do we evaluate LLMs for SPI safety failures?”

Exercises, office hours, and capstone projects will be designed to give participants better feedback loops, and a more nuanced understanding of SPIs, than they’d get from reading the materials alone. Examples of capstone projects: drafting a short proposal for an eval or conceptual research problem about SPIs; critiquing LLM-written SPI research; writing a doc on how a particular alignment technique might be used for implementing SPIs.

Target audience

We think the SPI Fundamentals Program will be most useful for you if you want to explore a career in AI conflict reduction. It could also be useful if you’re already working in an area that overlaps with our SPI agenda (e.g. cooperative AI, agent foundations), and are interested in reducing conflict risks via your current work.

While the curriculum is heavily skewed toward conceptual content, we expect it to also be important background for empirical work on SPIs, including research automation.

A great candidate might have any of the following backgrounds or skills — but you’re not required to be an expert in any of these, and we expect you’d be a good fit if you can parse most of the resources linked throughout this post:

Backgrounds:
- game theory
- mathematics/statistics
- economics
- decision theory
- analytic/formal philosophy
- computer science
- theoretical physics
Skills:
- constructing and thinking critically about models (both formal and informal) of complex/unfamiliar systems
- reasoning about incentives
- breaking down necessary and sufficient conditions for a given outcome
- turning rough intuitions into claims that are appropriately precise
- (for the empirical stream) experimental design, thinking about what a given test really measures

You don’t need any prior engagement with CLR’s research for this program. We will expect basic familiarity with AI safety concepts and game theory at the level of, e.g., material covered here.

Contact

If you have any questions about the program or are uncertain whether to apply, please reach out to info@longtermrisk.org or anthony.digiovanni@longtermrisk.org.

In 2026, is EAGx Berkeley or EAG NYC a better place to learn about US-China AI governance? by ben.smith

ben.smith — Fri, 03 Jul 2026 14:45:09 +0000

This year, I want to speak to people involved in

understanding AI governance
AI in China
AI Governance in China
US-China AI governance

I live on the West Coast, cannot take additional time off, and so if the two felt equal in quality I’d go to the one that takes less time to get to. But if what I can learn is much greater at NYC it’s probably worth ~6 hours extra flight time on the return journey.

I’m never satisfied by Ajeya

Ajeya — Fri, 03 Jul 2026 14:21:55 +0000

Note: This post was crossposted from Planned Obsolescence by the Forum team, with the author’s permission. The author may not see or respond to comments on this post.

But we get the job done

I was twenty one when I joined Open Phil,^[1] as a zealous young EA who had idolized the organization since I was in high school (back when it was called GiveWell Labs). I worked there for over nine years, leaving about six months ago (shortly after it rebranded as Coefficient Giving). By the time I left, I was kind of a fixture in the organization, and a significant driver of culture and thought within the AI team of 45-ish people. My manager felt that a lot of people on the team would be worried to see me go, and encouraged me to convey that I believed in their work in my goodbye message.

If you read that message, I meant what I said. I do think the kind of ambitious, foresightful philanthropy that OP (ugh fine, cG) does at its best is extremely impactful, and I do think the AI team is really great and has unique strengths.

But before I articulated that, I sat at my desk for over an hour, holding back tears, struggling to compose something not totally deranged. The first goodbye draft that leapt to mind was along the lines of: “I’m leaving because I failed to figure out how to add value here for years and years and years, even though I had every opportunity and everyone else around me managed to figure it out, and whatever defects were responsible for that will probably also cause me to fail wherever I go next, but I know for sure I’ll keep failing if I stay, so I guess I have to at least try leaving on the off chance that I manage to succeed at something somewhere.”

Let me back up. For the last decade, my work has followed a bipolar rhythm. I’ll latch onto some grand vision for The Highest Impact Thing to Do and throw myself into executing on it, only to realize 6-18 months in I’m only ever going to be capable of achieving a small fraction of the grand dream and the theory of change has big holes in it and other interventions are probably higher impact actually. Then I’ll spend 2-4 months hating myself and everything I’ve ever done and casting desperately for the Actually Impactful Thing to Do until I build up a head full of steam about another vision and start all over.

I have done several rounds of this, from EA community building to researching the stuff Open Phil told me to research to running a brutal yet productive hiring round to insane philosophy to slightly less insane timelines forecasting to business operations to alignment agendas and threat modeling to funding to “advising” to finally risk assessment at METR, and have written countless dozens of other vision docs in periods of desperate searching that never budded into projects. Despite staying in one place for almost nine years, I never built something that lasted even two years. I hired many great people into cG but I never managed them for very long. Over and over, it just felt like I’d made the wrong bet, had the wrong vision; over and over, I lost faith before I doubled down. Two rounds ago it got bad enough that I took a four month leave of absence just to figure out my shit (I didn’t).

I’m aware that deep shame and dissatisfaction about your work is rooted in pride, a sense that you should be able to do so much more. And as my long-suffering husband repeatedly points out, all this shame and paranoia is wasted motion. Maybe if I strung more than two and a half thoughts together in series without freaking out that I’m wrong to work on what I’m working on right now or I’m spending too much time thinking about what to work on next or I don’t stick with things long enough or I don’t explore widely enough or I’m too biased to think about any of this the right way, I’d have the mental energy to figure out better things to do and execute on them better. I have no great reply to this. But in some core part of myself, I know — no matter what logical arguments I nod along to — I know that this paranoia is what separates me from the beasts.

I’ve often compared myself to the anxious romantic who bounces from one perfectly functional relationship to the next, overthinking and sabotaging each one in turn because they don’t feel The Spark anymore or they’re not 100% sure he’s The One. I’m blessed with an immigrant’s steely pragmatism in the romance department,^[2] and if it weren’t for my experiences with trying to decide what I should be doing, I would have less pity for those who tie themselves in knots over love.

When I left cG, a small part of me harbored a quiet hope that these old psychological strictures would suddenly burn away and a new me would rise from the ashes. Alas, I remained frustratingly myself.

But I did get something in my new role that I’d never had before: a tight-knit team working on the same thing with me. The inside of my own mind remained a bit of a suffocating and cacophonous place, but I didn’t have to spend nearly as much time in there. Most importantly, in the period where the writing was most intense, my primary coauthor was always there, day or night or weekend. In all my previous writing projects I’d fought alone through daily waves of doubt and despair for endless weeks or months. In this one, the same moments of potential despair would crop up but then the right way to articulate this crucial point or the best strategy to sidestep that conceptual minefield or the strategy for rescuing some partially-right argument would kind of just tumble out of us in conversation. For large chunks, we sat side by side writing every word together. I had never realized working could be so easy.

Leaving cG didn’t break the cycle. I am currently in the phase where I desperately re-examine whether the thing I did actually didn’t make sense^[3] and I should be doing something totally different that Actually Has Impact, as I was when I was trying to compose that goodbye message in December. I feel like we are entering the AI mid-game — everything is picking up, and the gameboard keeps shifting. No plan seems like it comes close to matching the enormity of the challenge, and I feel like we collectively only have a couple more swings to take before it’s all over, and we need to make them count. I don’t know what the best thing to do is, and I don’t know exactly what I’ll do next. But I’m tremendously grateful that this time, I don’t need to figure that out all by myself.

^
I interned there the summer after my sophomore year of college, and the following year I graduated early to join full time, starting in July 2016.
^
I was going to write about how people should think about dating more like they’re arranging their own marriage, but fellow brown girl and rat Aria Schrecker has already written approximately infinity posts about this for me.
^
Who exactly is the audience for this kind of report, and what are they supposed to do differently? Is that going to be enough? Is it going to come in time? What other products could we produce instead with our currently extremely scarce analytical capacity?

AMA: Anthony DiGiovanni, author of the ‘Challenge of Unawareness’ sequence by Toby Tremlett🔹

Toby Tremlett🔹 — Fri, 03 Jul 2026 09:14:47 +0000

We announced the Cluelessness Critiques Competition two weeks ago.

A lot of you, not only prospective entrants, will have been reading Anthony’s sequence where he lays out his unawareness argument, or following the comments on his summary post. I thought that this might be a great time to have Anthony put some time aside to answer your questions.

Although we are calling this an AMA^[1], the focus will be on helping people understand the sequence so that they can write the best entries to the competition that they can. Anthony will be choosing the questions he responds to with this in mind.

Anthony will be answering your questions on Thursday the 9th. He cannot guarantee that he will answer every question, so make sure to upvote the questions you’d like to see answered.

^
Which stands for ‘Ask Me Anything’

Maybe do the thing you wish CEA would do by alejoacelas 🔸

alejoacelas 🔸 — Fri, 03 Jul 2026 06:51:36 +0000

I used AI to fix transcription errors, rerrarange the ideas, and suggest tweaks to the title and some sentences.

Three of the most exciting projects to come out of EA in recent years are, in a vague sense, CEA spinouts:

Kairos is directly a spinout of CEA and now handles most support for university AI safety groups. Basically everyone I’ve found who knows them is really excited about what they do
NEST is an opinionated ideas-first support network for EA (university) groups. And you can see from Matt’s blog the absolutely insane care he has for us.
BlueDot grew out of a group of Cambridge EAs that wanted to make much better introductions to EA topics than what was available out there, and now they’re basically trying to solve the talent gap for anything bad AI forever after

My natural next thought, is that maybe some more of the responsibilities currently conceived as “CEA’s job” would be better handled by small teams not directly (or only very loosely affiliated with) CEA.

Beyond the previous examples, there’s many advantages to small independent teams here:

You can take risks that CEA can’t
You can kindle a culture that promotes ambitious bets
You can set things from the start to benefit from pumping lots of AI into the project
CEA is just one org, they simply can’t do everything

If you want concrete suggestions, here’s the first that came to mind:

More and better EA events.
- EAGs are great, and EA Summits are an interesting experiment, but possibly there’s many more formats and audiences to explore!
  - Also, you can vibecode your own Swapcard alternative!
New introductions to EA ideas
- I got into EA because helping the poor seemed great and doing that more effectively seemed even better. I’m not sure the AGI-focused content from 80k would have grabbed me, so there’s possibly a good audience for what 80k was for me in 2016!
Online communities and discussion spaces
- AI makes sourcing and curating content much easier, or you can pick a niche community and topic and start from there (I’m the proud owner of the #too-much-ai channel for AI use enthusiasts!)
Intellectual leadership on EA ideas
- Many institutions that did this have evolved or died (Open Philanthropy, GiveWell, the Global Priorities Institute, Scott Alexander, the Future of Humanity Institute), so, again, it’s free real state!
  - If you’re looking for a concrete idea: What are the central virtues a committed utilitarian should cultivate? I would kill for a really great answer to this question
Support for local EA groups
- What about more active grant-making for EA groups? Maybe even faster funding? Just asking some EA community builders what they’d want and iterating from a minimal version of that could take you really far

I’d personally be very excited to see someone who’s passionate about it pursue any of these ideas. Maybe it’s more important to work on AI or something, but I think the heuristic of working on something you care about really hard can take you very far.

Also, you don’t need to wait for CEA to give you a role to do any of this! You can write up a Google Docs with the idea, send it to me for comments (alejoacelas@gmail.com), figure out the minimal version you can get started on, and then go for it!

Lydia Laurenson: “The Inside Story of Leverage Research” by Davis_Kingsley

Davis_Kingsley — Thu, 02 Jul 2026 23:02:32 +0000

Lydia Laurenson recently posted an article called “The Inside Story of Leverage Research” that gets into substantially more detail on what went on in that organization and I thought made quite an interesting read. However, note also relevant Twitter comments from Oliver Habryka:

Having been around the ecosystem, having interned there, and having arguably worked there during my work at CEA it’s… a mediocre article.
It IMO puts the emphasis on largely the wrong things and the wrong dynamics, occasionally is dramatically wrong, and has some huge enormous missing moods. I do think almost all of what it says is literally true and I would recommend reading it to learn things about Leverage if someone is doing their own study and looking for facts, though wouldn’t recommend it for someone who is looking for a reasonable high level summary.
It somehow completely fails to cover Leverage deploying spies into other organizations and trying to take over CEA. It fails to cover the enormous number of straightforward lies told by Leverage staff in that and other contexts.
There is something that feels really weird about the article. As if the central story it is trying to tell is that Leverage fell apart due to no fault of any of its members because they encountered ancient magic or something. Which is a fun story to tell but really doesn’t map what actually happened IMO.
Also multiple people clearly had what I would describe as a psychotic break at Leverage and the casual disparagement of Zoe’s article really doesn’t feel good to me.

most EAs should probably not be living in high cost-of-living (HCOL) areas most of the time by matthes

matthes — Thu, 02 Jul 2026 19:30:11 +0000

If you work a job that can mostly be done from anywhere^[1], you should ask yourself if you are spending too much money on housing.

I felt prompted to write this post by the recent increased push for EAs to move to the Bay. Multiple people have urged me personally to relocate. Most of them want me to work on AI safety, but the move-to-the-Bay meme is now also strong in animal welfare. I remain unconvinced, although I am always open to visiting if there is something specific for me to do there.

the core case

Significant additional spare money isn’t just great for donating, but it can also make it easier to have an impactful career, as you can afford to:

take more risks (e.g. start a new org that might fail)
spend more time on your job search whenever it’s time to find your next step
spend more time upskilling
spend more time and money on your mental and physical health
generally worry less day-to-day about food and other essentials for yourself and your dependents

All of the above have massive benefits for your long-term impact and wellbeing.

Moving to a cheaper area might be particularly attractive for people who:

want to explore high-risk/high-upside paths
already have strong networks
have kids or would like to have kids
work in fields/roles with low-to-moderate salaries

You can (and should) still go to conferences and occasionally visit relevant hubs, especially when you’re between roles, considering a career change, or raising money. It’s fun, inspiring, and can lead to opportunities and connections. You want to stay on people’s radar.

But I think most people overestimate how much they benefit from living in an EA hub, especially after the first 1-3 months. This is hard to measure, but seems to true to me based on my own experience of having lived in Oxford for multiple years, having visited other hubs, and having observed other EAs who have lived in various places. (But please share your own experience in the comments!)

There definitely are exceptions, though. The additional cost of EA hubs in HCOL areas might pay off for people who:

are early in their “EA careers”, when they benefit loads from every (formal and informal) networking opportunity
work in a team that is already in one location
easily lose their motivation or experience value drift when they don’t constantly surround themselves with EA community
benefit hugely from slightly faster access to rumours on a particular localised community (e.g. some people in AI safety)

Lastly, don’t move to the absolutely cheapest place your passport allows if that’ll make you miserable. Ask around and try a few places. Consider coordinating with a few other people.

example

I personally live in Sheffield, England’s 4th largest city. I recently bought a 1-bedroom flat for £70k ($92k) in walking distance to the central train station. Sheffield has a walkable city centre, theatres, lots of small and large shops, access to nature, a cool arts/music scene, lots of vegan food, and good connections to other cities (e.g. ~2h train to London every ~30-60min).

The view from my first flat in Sheffield.

I love London and probably like it ~50% more than Sheffield (mostly because of access to EA co-working spaces). But it’s ~3-6x more expensive on housing (depending on how much you’re willing to compromise on convenience, and more still if you account for interest on buying a place, plus the extra rent you pay while saving for a deposit). I just can’t justify moving there. I still visit for events, though.

My case is actually not a perfect example, as my exact current job probably couldn’t be done from anywhere. But I intend to stay here even if this changes, at least for now.

miscellaneous related thoughts

Anecdotally, many of my EA friends in hubs don’t actually “take advantage” of being in a hub that much. Many work from home most days and only go to two or fewer EA events per month. When they go to co-working spaces they often barely talk to anyone there.
Also anecdotally (but mechanistically plausibly), groupthink seems more common in the biggest hubs.
Building flourishing EA communities in cheaper (but still somewhat attractive) cities might
- make the community more accessible
- help with image issues that put people off EA, such as
  - elitism (“Only rich people from top unis are welcome.”)
  - hypocrisy (“How can they claim to care about cost-effectiveness if they spend all their time in the most expensive parts of the country?”)
If your organisation pays people less if they do the same work from a cheaper area, consider if you are creating weird incentives. (See also this thread under this post.)
Some related posts that some might find interesting or useful:
- Moving to a hub, getting older, and heading home
- Reflections on Anthropic and EA

I wrote this post myself but used Claude Opus 4.8 to critique the draft so I could iterate.

^
I expect this to be the case for the majority of EAs, but if not please correct me and point me towards data that says otherwise!

RP is looking for project founders in neglected animal areas by Rethink Priorities

Rethink Priorities — Thu, 02 Jul 2026 16:31:47 +0000

TLDR; To help the effective animal advocacy movement cost-effectively absorb greater amounts of funding in the near future, we are seeking expressions of interest from people who could found a new organization focused on:

Highly neglected animals: insects, wild animals, shrimp, fish, etc, or
AI and animals: AI alignment and governance for animal welfare, strategic actions considering transformative AI, AI for wild animals, etc.
Tech for animals: welfare tech, precision welfare tech, breeding for welfare, etc.

We are looking for both founders with a specific idea and those motivated to develop or execute on one. Depending on fit, Rethink Priorities can offer 3-6 months of funded runway and various operational set-ups for you to pursue your work, such as a Special Project, hiring you as an Entrepreneur in Residence, or helping you incubate into an independent organization.

To express interest, please fill out the short form (15-20min) here by the end of the day on July 19, 2026.

Background

At Rethink Priorities, we work on some of the most neglected problems in animal welfare: from insect welfare to wild animals to the implications of AI for alternative proteins.

Neglected animals represent the vast majority of the world’s sentient animals, yet they receive only a tiny fraction of animal advocacy resources and attention. Meanwhile, rapid AI development has the potential to change the world with implications for all animals. These areas remain under-addressed, not because there’s no tractable path forward, but because far fewer organizations are working on them than the scale of the problem warrants. You can help change that.

This post is an expression of interest. We want to understand who’s out there with the motivation and potential to found something new in this space.

Which animal topics are we focused on?

We are looking for founders for “frontier” issues – issues that are potentially important and tractable but currently underexplored relative to mainstream animal advocacy.

Neglected animal issues. Examples include:

Wild animal welfare – reducing large-scale suffering experienced by animals in natural environments
Invertebrate welfare – insects, crustaceans, and other animals whose sentience and moral status are poorly understood
Aquatic animal welfare – fish, shrimp, and other aquatic species in farming, wild capture, and related contexts

AI, technology, and animals. Examples include:

AI and Animals – Rapid AI development has the potential to change the world with implications for all animals. We want people to help steer the animal movement through an AI-transformed future towards the best world for animals. This could include AI alignment and governance for animal welfare, strategic actions that consider transformative AI for animals, AI for wild animals, etc.
Novel technologies and interventions – Welfare technology, when approached with caution, can be widely adopted and improve the lives of animals at scale. This could be precision welfare tech, breeding methods, genetic tools, or other emerging approaches.

Other

This list isn’t exhaustive. If you have a different angle in mind that doesn’t quite match the above list, we are interested.
We are also keen to find founders focused on neglected geographies: regions with intensive farming but less existing advocacy infrastructure.

Small teams are already getting outsized results

The case for founding in this space is not theoretical. Organizations founded in the last few years in these spaces are already having a significant impact:

The Shrimp Welfare Project (founded in 2021, not through Rethink Priorities but using our research) has secured electrical stunning commitments, expected to reduce the suffering of around 4.5 billion shrimps per year, with a credible path to helping 100 billion shrimps annually. It is now an ACE Recommended Charity.
The Insect Institute (launched in 2023 as a Rethink Priorities Special Project) has become a leading evidence-based voice on insect farming, with its research shaping the insect industry and being covered in The Guardian, Reuters, Bloomberg, and Vox.
Shared Roads, an initiative launched this year by Sentient Futures, is already engaging with governments on how driverless vehicle policy should account for animals.
The Center for Wild Animal Welfare (launched in late 2025 and supported as an RP Special Project) has secured pro-wild animal welfare debate and tabled amendments in the UK Parliament and secured national media coverage within months of starting.

A few motivated people with a tractable idea and modest funding can move an entire neglected field. We want to help start the next organization on this list.

Who are we looking for?

The qualities we care most about are:

Impact focus – You want to help as many animals as possible, and you’re willing to follow the evidence on how best to do that, even when it points somewhere unglamorous.
Experience – Our ideal candidate has a track record of delivery. They are someone who can set ambitious goals and achieve them. Backgrounds in leadership, research, advocacy, policy, operations, or building things are all useful. Note that animal-specific knowledge is helpful but not required, and strong generalists without an animal welfare background have founded excellent organizations in this space.
Entrepreneurial mind-set – You are self-motivated, comfortable with ambiguity, and able to drive things forward without much existing structure. You don’t need to have founded anything before.

You might be in one of two situations:

You have an existing idea or area of expertise. Maybe you’ve been developing a specific project or have some expertise in a neglected area that you want to turn into impact. You want support to pursue that avenue, such as funding, operational support, research capacity, mentorship, or strategic support.
You’re motivated but haven’t settled on an idea. You’d seriously consider founding or co-founding a new organization focused on a frontier animal issue if the conditions were right.

Our key question is: would you be willing to make the switch to working on this full-time if you had 3–6 months of funded runway?

What we can offer

Depending on fit, support could take several forms.

We are keen and ready to offer the following:

Funding – we can offer 3-6 months of funded runway, at a level that lets you step away from your current role and gives you the space to explore, develop, and launch.
Research support – Rethink Priorities is among the world’s most experienced research teams on neglected animal issues. Our research team will be able to support your decision-making at every step of developing your idea and project.
Operational support – Rethink Priorities’ Special Project team has supported numerous organizations with operations, finance, and HR to help them get off the ground quickly.
Entrepreneur-in-Residence role – You could become our in-house entrepreneur who can get things done, drive change, and help new projects get off the ground.

If there is demand, we are also exploring offering the following:

Executive coaching to help you leave your current position – it can be hard for top talent to wind down their current work and create the space to start something new. Coaching and support to help wrap up existing projects can be useful here.
Incubation into an independent organization – structured support to help you build and spin out as a standalone organization (this could be via Rethink Priorities, or we could support you via other programs such as Ambitious Impact).
Ongoing coaching and mentorship – to help you drive your project to succeed.

We’re at an early stage here. This is a genuine expression of interest, not a structured program with fixed slots. We want to understand what’s out there before deciding how best to support it.

How to respond

To express interest, please fill out the short form (15-20min) here. Tell us about yourself, what you’re considering, and what kind of support would be most useful. No polished proposal is needed.

If you’re unsure whether to respond, please err on the side of doing so. Completing the form is not a commitment; it simply helps us understand who might be interested. Note also that there is no fixed start date. Even if you cannot start until sometime next year, we would love to hear from you now.

Please respond by the end of the day on July 19, 2026. We’ll read every response and aim to follow up with those where there’s a potential fit by end of July.

Your response goes to a small team at Rethink Priorities, and we’ll only share it further with your permission.

Please share this post with anyone you think might be a good fit.

Questions? Contact Samuel Hilton (samuel@rethinkpriorities.org) at Rethink Priorities.

Please note that from 4 July to 12 July, Rethink Priorities is on mid-year break, and we will be unable to respond to emails during that time.

About Rethink Priorities

Rethink Priorities is a think-and-do tank dedicated to informing decisions made by high-impact organizations, funders, and policymakers across various cause areas. We aim to identify emerging areas – such as invertebrate welfare and digital consciousness – before they become crowded, and to influence trajectories while fields are still being shaped.

Within animal welfare, our recent work includes:

Pioneering research comparing welfare across species through our Moral Weight Project
Bringing to light the important and neglected area of invertebrate welfare
Strategic work on EU farmed animal welfare advocacy in 2025, influencing $10m of grants
Building a coordinated roadmap for fish welfare in Europe with 18 leading organizations,
Developing a prioritized research agenda on insecticide welfare impacts in partnership with the Wild Animal Initiative
Supporting nine early-stage organizations through our Special Projects program

You can review our public work here, as well as our 2025 results and 2026 plans.

We’re supported by funders including Coefficient Giving and individual donors.

Rethink Priorities is committed to building an inclusive, equitable, and supportive community. Please don’t hesitate to apply regardless of your age, gender identity/expression, political identity, personal preferences, physical abilities, veteran status, neurodiversity or any other background. We invite you to reach out to careers@rethinkpriorities.org with any accessibility requests. (Please note that from 4 July to 12 July is Rethink Priorities’ mid-year break, and we will be unable to answer emails during that time.)

Time Sensitive Do Gooding Opportunities by Bentham’s Bulldog

Bentham's Bulldog — Thu, 02 Jul 2026 15:29:15 +0000

(Crosspost—written in a slightly sillier style than is typical of EA forum posts, but still seemed worth posting).

A lot of my interactions with prominent bloggers involve me appearing in their rooms ex nihilo at odd hours of the night and suggesting they write about something very impactful (e.g. why people should take the Giving What We Can pledge). Yet sometimes when I say this, they say to me, “Thou hypocrite, first cast out the beam out of thine own eye; and then shalt thou see clearly to cast out the mote out of thy brother’s eye.” So here I’m going to tell you about some high-impact opportunities: one to get support for running a local EA group and the other a donation-matching opportunity for effective animal donations. And act now, because offers end soon!

1 Donation matching

A funder is temporarily matching donations to Animal Charity Evaluators’ (ACE) movement grants program. For those unaware, ACE is basically the GiveWell of animal welfare. They do high-quality research to find the best charities in the animal sector. They publicize which charities are effective and give out grants to effective animal charities. You can see some of the grants they’ve awarded here—many go to incubate fledgling organizations with the potential to be very impactful. Their movement grants program has been endorsed by Giving What We Can.

The basic pitch for giving to the grant matching challenge is straightforward: animal welfare is a really big deal. By the billions, animals undergo hellish torture in factory farms—their bones are broken, they’re boiled alive, they’re selectively bred so they can barely move, they’re kept in crowded conditions where disease spreads rapidly, and so on. To within a rounding error, all the suffering currently in the world is experienced by animals.

Many organizations stop lots of animal suffering for little money, often sparing animals from multiple years in a cage per dollar. In light of this, it makes sense to fund effective animal charities. But you probably don’t have the time to comprehensively investigate which animal charities are the most effective. So it makes sense to give money to people who carefully research which organizations help animals most effectively and give out grants to those organizations.

The matching program is also conditional. A donor has agreed to match the first $300,000 in donations. In other words, this donor will give up to $300,000—matched with how much other people give. People have already given $260,000. The donation window runs through the 10th of this month. (Edit: an earlier version of this post said that they would only donate at all if the full $300,000 was raised. This was false, sorry! They’ll match however much was given).

In addition, a group called Mobius has agreed to match every new monthly recurring donation, up to $10,000. So if you set up a new monthly recurring donation, your gift will be matched twice!

I’ve chipped in $500. My best guess is that each dollar donated spares many animals from a lifetime of suffering. So please, if you can, donate.

2 Organizer support program

As is known, this blog is very hip and cool. A sizable chunk of its readers are in university; this is because my readers tend to be young, and smart (also, in general, very cool). Oh, and they also tend to be deeply moral. Thus, inquiring minds want to know: how can they make a big impact on the world while they’re still in university?

Here is one good answer to that age-old question, asked since the days of Gilgamesh: you can be an organizer for your university’s effective altruism group. Oftentimes EA events are organized by only a few dedicated people across an entire university, and it may be the case that your university has no existing EA group at all (SAD!). This could be the case even if there’s plenty of latent interest among the student body, just waiting to be tapped. If you get just a few more people involved in EA through your organizing, this will have enormous value. And if the people you get involved are as impactful as you, just one new recruit can double your lifetime impact.

Friend of the blog Noah Birnbaum—who, like most of my readers, is very smart and cool and young and handsome—has a piece on the EA Forum about why people should become university organizers. The whole thing is worth reading, but here’s a particularly important bit:

Scope – … a few counterfactual EAs potentially means millions of dollars going to either direct work or effective charities. Getting one more cracked EA involved can potentially double your impact!
According to this post from 2021 by the Uni Groups Team: “Assuming a 20% discount rate, a 40 year career, and $2 million of additional value created per year per highly engaged Campus Centre alumnus, ten highly engaged Campus Centre alumni would produce around $80 million of net present value. The actual number is lower, because of counterfactuals.” It should be noted that campus centre alumni is referring to numbers estimated from these schools.
They also included an anecdote of a potential near-best-case scenario that I think is worth paraphrasing: The 2015 Stanford EA group included: Redwood CEO Buck Shlegeris, OpenPhil Program Director Claire Zabel, Full-Time EA Journalist Kelsey Piper, and former CEO of Redwood and Constellation Nate Thomas. However, the Stanford group went dark in 2016 – for years, there was only one active member and few events were run. “As a result, we probably lost a few Bucks, Claires… Kelseys and Nates. That’s a lot of impact missing.”

And there are a bunch of other benefits. Organizing gives you valuable experience and looks good on your resume. You’re likely to meet incredibly cool people while doing it. The kinds of people who get involved in EA are disproportionately moral and smart; they make very good friends. These connections will also make it easier to remain an active EA by getting you more deeply embedded into the community. Some of my favorite people that I met at university were people I met through the EA club.

Running your university’s EA club, as I found out the hard way, is pretty difficult. I was very bad at it. When I was the president of the EA club at my university, it basically died. This was bad! Fortunately, the Centre for Effective Altruism has a program called the OSP, short for Organizer Support Program. It’s called that because it’s a program that supports organizers.

The program allows you to have meetings with mentors who advise you on how to run your local EA group. This is useful. The mentors have lots of experience with organizing, so they give useful advice. They also provide resources, like lists of readings for fellowships. To get their help, you have to apply. Applications close Sunday. So if you’re going to apply, you should do it soon, through this link.

If you’re not in university, you can still consider starting a local EA group, where you can still get assistance through OSP. There are people in every place that have the potential to become passionate EAs—but somebody has to take the first step in reaching out to them. That person very well could be you. Paraphrasing externality-illiterate tree gremlin The Lorax unless someone like you cares a whole awful lot, the EA group at your university isn’t going to get better, it’s not.

AFFINE – A Retrospective by Ouro

Ouro — Thu, 02 Jul 2026 14:32:20 +0000

A Day at AFFINE^[1]

“AFFINE was the best month of intellectual exploration I have had the opportunity to engage in, ever. Usually opportunities like this are limited to a day or a weekend, which both limits depth, forces a sprint-type mindset, and generally is quite limiting. At AFFINE I had time to wander towards and through interesting ideas.”
-Xylix (participant)

You wake up in an ornate room shared with a few other participants to the smell of breakfast, or perhaps you have been up for a while, reading or going on a morning run. You grab what you want from the buffet and head to the common room which is slowly filling up and fragmenting into conversations of various sizes and scopes. Zipf distributions came up yesterday and the knowledge applies. Sunlight filters in through the embroidered curtains as you join a group talking about the self organized criticality of brains and why it is necessary. The people one couch over are designing an experiment to settle their bet about whether Claude Code charges token-use for cached reads. Off-handedly you pitch a talk you want to give on the unconference day tomorrow, and the group seems generally interested. A fellow participant wants to work with you on it and you happily agree.

During the day you attend a talk by Ihor Kendiukhov about problems with expected utility theory, followed by a remote presentation by Abram Demski, taxonomizing Goodhart’s law. You ask a question about quantilizers and check the app to see whether anyone wants to explain geometric rationality to you. Turns out yes! You schedule a session for tomorrow before heading outside for a few games of volleyball while your default mode network processes the information. You grab one of the mentors for a 1on1 about research methodology, before dinner is served.

Originally you wanted to read a couple of papers but instead you get roped into an argument about active inference and embeddedness when you carelessly walk past a whiteboard. The evening is spent playing increasingly esoteric variants of chess until your brain finally gives out. You head to your room, briefly consider whether you might want to try one of the tents outside over the next few days, and pass out almost instantly.

One month ago, we held the first AFFINE alignment seminar. A peer-tutoring-driven, intensive retreat, focused on frame-finding and the deep, fundamental difficulties of the alignment problem. The following is a narrative account of how it went, what we learned, and how we intend to proceed from this point onward. It is optimized for readability and conveying the spirit of our event. For dry information click here.

Missing Foundations

“I feel there is extreme lack of events like this in the AI safety community [...]. AFFINE Superintelligence Seminar is a venue where real technical competence, moral seriousness, and productive vibes converge. For people who deeply and honestly care about the alignment problem, this is a place to have uniquely useful interactions.”
-Ihor Kendiukhov (Mentor)

Our concepts of mind, cognition, life, values, teleology etc are lacking. We have so far been unable to state the core problems in satisfactory terms, and normal science requires confidence in legible frameworks which capture the crucial nuances in order to make productive headway. The field is pre-paradigmatic, the size of minds implies that successfully framing the problem-statement will likely be harder than many historical philosophy-to-science transitions, and yet this is not for the most part taken as a pressing call to do more philosophy. Instead, progress is being made on a variety of related problems for which the language of other sciences can be borrowed. The things for which we have formalisms are getting reliably solved and honed, but the rate at which we acquire new formalisms is troublingly slow while few of them even aspire to capture the entire field. In particular, a large part of the current work does not concern itself with superintelligence alignment but rather with the control of weak-to-moderately strong systems.

None of this is surprising. Status- and funding incentives select for legibility while most field-building programs require work to be done in a matter of months. The only way to predictably and quickly produce legible output in an unsettled field is to prematurely accept a frame and perform normal science. In particular we think that some newcomers have potential to do useful foundational work on alignment but that the incentive landscaper described funnels them towards comparatively reachable but less crucial sub-problems instead. Given this landscape, AFFINE was to create a space where the broader, slower kind of thinking can happen: where researchers can look for sturdy foundations without getting captured by perverse incentives. Not just because the search for holistic philosophical groundwork is neglected, but because it is the most marginally useful skill to train people in at this moment.

Note that models are getting very good at solving well-specified problems in scientific domains equipped with a crisp verification tool. Coding, technical math, etc. In a more general sense, we can expect AI to continue getting differentially better at “in-frame” research, where good performance is less fuzzy and easier to reward. “Claude 4.8 is over some sort of tipping point for me, where I feel like I can ‘just keep going and keep making progress’ in some new sense” reports Demski, and personal observations say that Fable is a large improvement over Opus 4.8 when it comes to technical math research. Outsourcing superintelligence alignment to weaker systems is a default outcome as things are standing. Assuming that these can be made trustworthy, two issues present themselves, the first being that it’s very difficult to evaluate cognitive labour which you cannot yourself perform, and the second being the AI’s imbalanced skill at tackling crisp vs. frameless problems. Consequently we do not expect these systems to track the philosophical nuance of pre-paradigmatic research properly, causing useful progress to become bottlenecked on people who “actually get the problem”. The skill of finding and legibilizing the right questions will be decisive for research in the future.

Brains in a Chateau in Bohemia

“Most intellectual environments are quite result-oriented, and I think this works against tackling problems as difficult as superintelligence alignment. AFFINE gave me a great environment to think deeply about my fundamental models of the world, without asking for immediate output.”
-Haru Kim (participant)

There were thirty participants at our seminar, most with some significant degree of STEM background and varying familiarity with the alignment problem. In terms of raw wits and curiosity, we found them immediately impressive, but steering them to the place where we wanted to reach proved difficult. It turns out that you cannot just put people in an ostensibly educational environment and tell them that they will not be judged on output or the number of concepts they can get under their belts. They will not believe you, and they will try to optimize for whatever it is that you secretly want, because not judging on output is simply not how the world works. Alas, this is contrary to the point of AFFINE.

If there were to be a reasonable metric of our success, it would be the degree to which we empowered the participants to think freely and with little thought to output-oriented incentives. We wanted them to load up the whole problem and hold it in their attention, to “not just do something but stand there”. We can’t measure that, though we believe that we ultimately got there. What we can point to is the sheer number of whiteboards that were filled and re-filled in the common areas instead of writing (or reading) papers. The trick, in the end, was simple: Tell them the goal instead of trying to engineer some bespoke incentive gradient from scratch: “You are here for ambitious, holistic theory work. Not for anything empirical, not for anything you can make significant headway on in a month. All we want you to do is to look in the right type of direction.”

Egregore

“AFFINE was one of the most intellectually generative environments I’ve ever been in [...]. It’s an incredibly well optimized environment for coming up with new interesting ideas, and an amazing set of people to spend a month with”
-Samuel Ratnam (participant)

The classic mentor-mentee dynamic doesn’t scale well. There simply are not that many truly good mentors to go around, but there’s an even bigger problem: It fundamentally does not encourage the sort of heroic agency that we want from future visionaries. AFFINE, therefore, settled on a different model. We still had mentors, some permanent and some temporary, giving talks, workshops, and personal guidance, but we also made an app that collected relevant resources and allowed participants to publicly mark themselves as able and willing to explain a given topic. Teaching is a forcing function for frame-refactoring, and we believe it is one of the most critical activities for what we are trying to achieve. Most expertise was acquired through peer interactions, as we allowed the participants to train each other and explore much more freely than they might under the watch of a single mentor serving as a guide. We built a hivemind designed to learn, disseminate, and boggle all by itself, where everyone has an incentive to push the collective understanding forward by sharing the right tools and asking the right questions. If we learned one thing, it’s that we should leave this process even more room to unfold itself: less scheduled activities and more free space for standing around a whiteboard transmitting. These whiteboards, we believe, is where the magic happens.

Bridge-Building

AFFINE was an amazing experience. I don’t think I’ve ever been in a place with such a high concentration of people with interesting takes on Alignment before. And this is coming from someone who spent a month at Lighthaven and works out of LISA!
-Sean Herrington (participant)

A number of our fellows did not want to be researchers. They were or wanted to be doing governance or public communication work so as to give researchers time to do anything useful before the apocalyptic deadline. Of course we support this, but we considered it to be a bit of a bug with regards to our program at the start. AFFINE was a theory seminar after all. Since then, however, we’ve come to the conclusion that it was a benefit to have these non-technical participants. Not because AFFINE shouldn’t be about alignment theory, but because a theory-seminar is a great place for governance people to be. A place like AFFINE connects them with scholars, builds robust models of existential risk which are rooted in a deep understanding of the pre-paradigmatic nature of the field and thus not fragile to some combination of minor technical breakthroughs. We have heard back from this group that attending AFFINE was much more useful to them than governance-centric events they have attended before or since, and we want to make a contingent of participants like them into a deliberate aspect of our future endeavours.

Numbers

“The AFFINE seminar was one of the greatest months of my life. It’s incredible how much you can grow when you are surrounded by smart people who share your mission.”
-Elias Schlie (participant)

The numbers don’t matter. You can’t really feel them, and you reward-hack yourself if you stare at them for long enough, which is a shame because our numbers are really good, actually. On a scale of one to ten, participants would recommend AFFINE with a strength of 9.1 on average. They are still in contact with 7.3 people from the seminar one month down the line and would consider reaching out to more than ten of them for small favors like reviewing a post. They were able to follow through on most of the plans and commitments they made during the seminar, whose impact on their model of the alignment problem they rated as 8.6/10. They rated its long-term impact on their productivity as ⁸⁄₁₀ and on their mental state as 8.1/10.

Future Plans

We got a lot of useful training data during this seminar, which is the polite way of saying that we messed up a bunch. Any future AFFINE will have a much clearer mission statement from the get-go, a curriculum which more deliberately starts out by introducing the hard problems and obvious model-insufficiencies. Leading into an open ended structure with scaffolding for transmission/distillation and mentor access, interwoven with workshops on process-skills such as builder-breaker. We want to make the app more expansive, improve the matchmaking for transmitting/receiving as it ties to personal models rather than articles, as well as make the fixed content more organized. We want point-of-contact mentors for individual fellows for open-ended guidance. Most importantly we want to reduce scheduling and give the participants more time for what we consider to be the single most valuable activity: standing around a whiteboard and thinking.

We definitely want to keep doing AFFINE. We are planning another seminar as early as the end of this year, a fellowship to support longer-term deep thinking, and possibly a research retreat.

If you are (or were) a researcher with models for the object level work, or if you have relevant experience with regards to e.g. operations or event-organising and are interested in what we are doing, please reach out by e-mail to ouro@affi.ne or through lesswrong DMs. It is very possible that we will want to involve you.

Thank you

To everyone who made AFFINE I possible:

Ops: Adriana Arauzo, Phil Chen, František Drahota, Turner Halle, Pauliina Laine, Emily Medén, Jiri Nadvornik, Grace Roberts, Andrew Szabados

Mentors: Mateusz Bagiński, Lucius Bushnaq, Abram Demski, Gurkenglas, Jonas Hallgren, Kaarel Hänni, Felix Harder, Jobst Heitzig, Steven Kaas, Johannes C Mayer, Richard Ngo, Ihor Kendiukhov, Vanessa Kosoy, Vojta Kovařík, Jan Kulveit, Linda Linsefors, Ouro, Julia Persson, Justin Shovelain

Wellbeing: Tilman Masur, Sofie Meyer, Kitt Morjanova, Ryan Thomas

Misc: Camille Berger, Joe Collman, Peter Hozák, Eduard Kapelko, Roman Malov, niplav, Elisa Paka, Plex, Attila Ujvari & the Hostačov staff

^
This is a confabulation of events and not any one participant’s experience of any one day

Career Choice: Becoming a Researcher in a Non-EA-Priority Field vs Founding Tech Startup? by themasterchief166

themasterchief166 — Thu, 02 Jul 2026 03:17:41 +0000

Engineering + math graduate whose goal is to maximize impact. I am currently deciding between two career paths, but have been struggling a lot to determine which would be more impactful:

Become a professor/researcher in robotics, working on mainstream technical problems such as zero-shot learning. (To be clear, I’m not primarily thinking about robotics safety or AI safety, but rather general robotics capabilities research.)
Try to found “low-sophistication” hard-tech startups — i.e. products that are not extremely technically sophisticated and could easily be prototyped in a local makerspace, meaning any wannabe hard-tech founder could easily make it.

Note: For personal reasons, it is unlikely that I would found a highly sophisticated hard-tech company, i.e. one that requires advanced fabrication / other specialized technologies.

Has anyone here faced or thought seriously about a similar decision? If so, how did you decide where you had more counterfactual impact?

One way I’ve tried is through estimating the number of “counterfactual days saved.” Here’s my crude analysis:

If a robotics bottleneck takes 600 researcher-years to solve and 400 researchers are already working on it, adding me would move the solution from ⁶⁰⁰⁄₄₀₀ = 1.5 years to ⁶⁰⁰⁄₄₀₁ ≈ 1.496 years, or about 1.37 days earlier. If 50 startups benefit, and I work on three such bottlenecks over my career, that gives roughly 3 × 50 × 1.37 ≈ 205 startup-days saved.
If I found five successful simple hard-tech startups, and each brings a useful idea to market one year earlier than it otherwise would have arrived, that is roughly 5 progress-years saved.

This crude analysis is missing many important factors, but on first glance, it seems that the startup path is more impactful, assuming I am unlikely to be an exceptional researcher in robotics (which I think is probable).

If anybody has a better way of comparing impact between academic and startup paths, though, would deeply appreciate it — I have been stuck at a crossroads for quite a bit…

The Absorption Problem by NickAllardice

NickAllardice — Thu, 02 Jul 2026 01:31:06 +0000

I posted this originally on Linkedin and Substack where I write for a more general audience. I think it’s highly relevant for here though, and would welcome thoughts, disagreements, builds and implications. I wrote all the substance of this post myself, then used an LLM to refine its expression.

A historic amount of money may be about to flow toward the world’s problems, but right now that money can’t be absorbed at the effectiveness and scale this moment demands. That’s an unpopular thing to say from a sector where it’s fashionable to shake your fist at the wealthy for not giving more away.

There are reasons to be sympathetic to this perspective: we know more about how to do good than ever before and there’s more wealth than at any time in human history… and yet the percentage given by the ultra-wealthy hasn’t budged in decades.

But this discourse is damaging and self defeating.

First, giving at an ambitious, sustainable scale will never be unlocked via shaming and scolding.

Second, in the vast majority of cases the social impact sector simply can’t credibly claim to be able to use significantly more money while doing so effectively, efficiently and quickly.

This is “The Absorption Problem”.

Nan Ransohoff’s recent piece on the third wave of American philanthropy illustrates why this matters now: a historic amount of capital may be about to land in a charity sector that isn’t built to absorb it – $370B in total philanthropic assets at Anthropic & OpenAI alone, leading to an estimated $50B/year in giving.

From Nan Roansohoff

I’m skeptical that $50B a year will move. The base rate for giving by the ultra-wealthy is about 1.2% of assets a year, many times lower than what Nan forecasts.

That said, I’m having dozens of conversations with people who will be a very large part of this ‘third wave’ of philanthropic capital, and the ambition and urgency I hear is very very real… real enough that I expect giving well above the historical rate. No matter where in that range we land, the absolute amounts will be large, and it’s plausible enough to be worth preparing for now.

And I believe that we—the sector—are woefully underprepared.

Scaling impact that leads to real change for people quickly and successfully is incredibly hard, and poorly understood. I see a lot of naivete amongst leaders, operating organizations and funders—and I want that to change.

I don’t have all of the answers, but as CEO of Change.org (we went from 100k to 100M monthly users while I was there, doing a lot of work on advocacy and political change) and GiveDirectly ($1B+ delivered to people in poverty and crisis around the world) I’ve been part of two of the fastest growing and largest scale social impact organizations of the last 20 years. I’ve underestimated this problem, made many mistakes, failed catastrophically, and sometimes had success.

Below I work through the four reasons scaling impact so often fails: (1) impact that doesn’t grow with size, (2) interventions that hit a low ceiling, (3) the hidden cost of moving fast, and (4) organizations that break from growth. Then I offer a framework for assessing the “scalability profile” of any intervention or organization before you put serious effort or money behind it.

I. Impact rarely grows proportionally with scale

When philanthropy fails at scale, almost nobody finds out. A for-profit that scales badly gets told by the market quickly: sluggish revenue, increasing churn… and someone gets fired. Nobody gets told about a nonprofit that fails to scale impact, because the people we serve aren’t the people who pay.

The strongest feedback mechanism the sector has built to compensate are randomized controlled trials (RCTs). They’re slow, expensive, hard to run at scale and they won’t work for this moment. It’s years from design to results, measuring a program as it existed several years and one organizational era ago. An organization can triple in size between the baseline survey and the published paper. The gold standard of feedback runs at a fraction of the speed we need right now.

And when researchers have used RCTs to measure scaling itself, the results have been poor. In Kenya, a contract-teacher program was tested in a nationwide experiment in which the identical program was randomly assigned to be run by an NGO or by the government. Run by the NGO, it raised test scores by about 0.18 standard deviations, a solid effect. Run by the government at national scale, the effect was statistically indistinguishable from zero. The implementing organization changed, and the entire effect vanished.

This matters because most nonprofits don’t plan to scale themselves; they plan to prove something works and hand it to a ministry that can reach everyone. That handoff is the sector’s default theory of change, and the contract-teacher result is the uncomfortable version of where it leads: the intervention survived, the implementer changed, and the effect didn’t.

The economist John List calls this the “voltage drop”: between 50 and 90 percent of programs lose a substantial share of their measured effect when they scale.

I see the mechanism constantly in development research. A new RCT shows some intervention is twenty or thirty times more cost-effective than anything else, everyone gets excited, and then the replications come in and revert toward the mean. Often the first study site was uniquely fertile ground. Often quality was the founder’s personal obsession during the trial, and quality is exactly what’s hardest to scale: the result you get when a founder is directly managing every input is not the result you get three years later with thousands of staff and contractors who have no investment in the outcome.

This can get worse at scale, as unintended consequences add very large negative effects. Take microfinance as an example – a philanthropic darling that has scaled massively.

Studies have shown that microfinance has a mostly neutral or occasionally modestly positive effect on people who receive it.

But even if those studies showed more robust results, the commercial industry that philanthropic capital helped catalyze has followed the incentives to their logical conclusion: aggressive growth, extreme interest rates, severe repayment terms, over-indebtedness trapping a meaningful number of extremely poor people further in poverty.

I don’t think we’ll ever know whether the overall effect of micro-finance on the world is net positive or negative.

Political giving, advocacy and corporate campaigning has its own scale-dependent unintended consequences. Over two decades of working on hundreds of political and corporate campaigns, I’ve seen again and again how ambitious deployment of capital often triggers the formation of organized opposition, polarizes neutral bystanders into partisan camps, or enables political opponents to mobilize more of their own money. The only people that win are political consultants and advertisers that will happily take everyone’s money.

“Can this absorb money” and “will more money help” are different questions, and the gap between them often widens with scale.

II. Many of the best interventions have a low ceiling on their potential scale

When a startup pitches a venture capitalist, the first question is usually not about the founder or the idea. It’s about market size. Most startups get turned down because the problem they’re solving is simply too small to plausibly reach $1B+ valuations, no matter how good the product or team is.

Philanthropy almost never asks the question this way, because most of the best giving opportunities in the world have modest market sizes.

The Against Malaria Foundation (AMF) pays for implementing partners to distribute anti-malarial bednets in low-income countries, one of the most rigorously evaluated and cost-effective interventions out there. It deploys roughly $100-150 million a year. The upper bound constraints they face aren’t money or talent. It’s the supply of bednet-appropriate geographies… it’s the seasonal windows the malaria transmission cycle dictates… and it’s the manual work of supervised distribution by AMF’s last-mile delivery partners. AMF’s estimate for its global funding gap for bednet distribution is in the hundreds of millions a year, not billions.

This is no knock on AMF or anti-malarial charities: they should be funded to their full capacity. The catch is that full capacity is a real, knowable number, and it is far smaller than the money that could be brought to bear on the world’s most important problems.

The same is true beyond bednets. Graduation programs (the intensive packages of cash, coaching, and asset transfers that move people out of extreme poverty) have some of the strongest evidence in development, but can be expensive and very labor-intensive to run, which caps how fast and how widely they can spread. Childhood vaccination has enormous proven impact, yet the binding constraint is cold chains, clinics, and health workers. In each case the constraints to impact at massive scale are real, knowable, and often not something that money can solve quickly.

Move beyond global health and the ceilings and bottlenecks don’t disappear. They just get harder to see. Advocacy causes have ceilings made of talent, organizational bottlenecks, and winnable opportunities rather than goods:

The farmed animal welfare movement runs on around $200M a year, and at the upper bound its binding constraint likely isn’t money… it’s the number of credible organizations, skilled advocates, and winnable campaigns. Right now the movement runs only a few major corporate campaigns at a time, working the most winnable first. Field building can raise that ceiling, but it takes many years for investments in organizations and talent to mature, and the senior people who lead campaigns can’t be manufactured on demand. Funding can accelerate this but it doesn’t collapse the timeline.
AI safety is murkier still: capital has grown very rapidly and in many cases has outrun the field’s ability to use it. The scarce inputs are senior researchers who can mentor and grantmakers who can vet, not dollars, so promising money sits undeployed for want of people to spend it well.

Grantmakers and nonprofits aren’t used to assessing the market size and scalability profile of interventions at the level of funding that could be brought to bear soon. Bottom up ‘room for more funding’ analyses are a good starting point, but fundamentally different to proactively searching for or working on interventions and organizations that have ‘credible path to absorbing and deploying $500M+ cost-effectively per annum’.

It’s not surprising these muscles aren’t well developed, most non-profits live in a perpetual state of scarcity. Organizations raising money grant by grant are rewarded for high-confidence impact in the short term, not for the R&D that builds something capable of extraordinary scale. Grantmakers optimizing the marginal dollar look for opportunities they can fund to full capacity before moving to the next item on the list. Neither mindset serves this moment.

The clearest symptom is the sector’s addiction to pilots: cheap, low-commitment, and good for a tidy results deck, so we run them by the thousand and scale almost none. We reward starting things, not the slow, expensive work of making a working thing big.

There are just a few dozen nonprofits spending more than $1 billion a year, and that includes hospitals, universities, and donor-advised funds. The very largest implementing charities (Feeding America, Salvation Army, World Vision, MSF, etc.) top out around $2-3 billion in cash spent per year. It’s insane there’s so few non-profits that have achieved that scale, and it has almost always taken many decades (sometimes more than a century!) and wildly diverse portfolios of programs to get there.

By contrast, private equity firms announced roughly $1.7 trillion in deals in 2024 alone, all made up from buyouts of individual companies, each typically decided in months. A single mid-sized one of those deals can exceed the yearly budget of the largest charity on earth. The biggest nonprofits are a rounding error next to what private capital deploys as a matter of routine.

And the donation opportunities with no ceiling are often where absorption and impact come apart completely. You can spend unlimited amounts on political advertising or capital campaigns and feel good for no discernible impact. A museum endowment can absorb any sum you care to name, forever. The world’s largest university endowments already hold more than the GDP of most countries. All are marginally better than leaving the money in investments… but not by much.

The money is absorbed frictionlessly because so little is being asked of it. Ease of absorption tells you nothing about impact, and past a point the two are often not related at all.

III. Speed costs efficiency

Suppose you’ve found an intervention with real headroom; a genuine market, money to spare. You still have to get there, and how quickly you try to get there is its own constraint.

I see the assumption that scaling impact brings more efficiency all the time. Unit costs fall and infrastructure amortizes… so the default mental model of a donor funding growth is economies of scale: bigger will mean cheaper per unit of impact.

At a stable equilibrium that might be true, but for organizations growing fast the opposite is often more true. Speed and cost-effectiveness trade against each other, and the faster you grow, the worse the trade-off.

I think the confusion comes from where people’s reference points for scale live: the tech industry, the most documented scaling story of our era. Those stories are misleading twice.

First, software scales in ways field operations don’t. When Change.org grew from 20 million to 100 million monthly users in March 2020, the marginal cost of serving those 80m users was close to zero; the product was the same pixels in a new browser.

Compare that to delivering bednets, vaccinations, or graduation programs in rural East Africa. Scaling those means recruiting, training, and supervising thousands of field staff. Every new hire who handles money or data is a new fraud vector. Every new district brings a new local power structure and new ways enrollment can be gamed or coerced. Quality, risk, fraud, and abuse exposure all grow with scale, often faster than scale, and the systems to contain them need to be rebuilt repeatedly.

Second: even software companies scale inefficiently. The strategy that built most of the technology giants has a name: Reid Hoffman’s “blitzscaling,” or “prioritizing speed over efficiency in the face of uncertainty.” A blitzscaling company chooses, on purpose, to be worse at the unit economics today in exchange for being bigger tomorrow. It overspends, undercharges, and runs at a loss, sometimes for a decade. Uber piled up roughly $31.5 billion in operating losses before its first operating profit! Venture capital is, at bottom, a mechanism for financing that deliberate inefficiency long enough for it to pay off.

Nonprofits are punished for the exact strategy.

A charity that ran Uber’s early numbers, spending heavily ahead of scale and sacrificing efficiency to build scale, would be flagged by every watchdog, downgraded by every evaluator, and abandoned by donors who read overhead ratios. The sector’s capital expects efficiency during the growth phase, which is precisely when efficiency is structurally unavailable. So organizations grow slowly to stay efficient-looking, or grow fast and hide the costs. Neither maximizes impact.

Even with infinite tolerance for inefficiency, many bottlenecks are slow to resolve no matter how much money gets thrown at them. Opening a new country takes GiveDirectly 1-2 years: regulatory approval that often needs ministerial or presidential sign-off, mobile-money registrations, a country director credible to both the government and the people we serve. Fraud monitoring for $5 billion a year of exposure is not a bigger version of the system that handles $250 million; it’s a different system that has to exist before the money flows.

We learned the cost of skipping this kind of unglamorous work the hard way. Early on, GiveDirectly was dismissive of government relations. It looked like a circuit of pleasantries: meetings that made everyone feel good and money that could have gone to recipients. We decided to just get on with the job. Then, in Uganda, during a volatile political moment, rumors spread that we were buying votes with our cash distributions. Nobody in government knew who we were. We had no champions, no allies, no one to vouch for us. We were shut down, millions of dollars we were ready to deploy stopped in their tracks, and it took two years to dig out of that hole. We now invest in government relations in every country we operate in. That lesson cost us two years in one country, and it’s the kind of lesson you can only learn at the speed the context allows.

GiveDirectly has lived the other side of this trade too. In 2020, COVID hit and GiveDirectly jumped from delivering $40 million a year to ~$250 million… 6x in a year. From the outside it looked like our finest hour, and in some ways it was: that money reached hundreds of thousands of families in the middle of a global emergency, fast, when almost nothing else was moving.

It also nearly broke us. Our team ran at a pace that couldn’t be maintained; heaps burned out. We took operational risks I would not take again. In several places where being unlucky would have been catastrophic we were simply lucky.

IV. Organizations break as they scale

Everything so far is a limit on the non-profit sector’s ability to absorb, even if every non-profit was perfectly run. But there are no perfect organizations.

This isn’t unique to nonprofits. Bain’s Chris Zook put the mechanism in one line: “Growth creates complexity, and complexity is the silent killer of growth.”

In my experience, the breakage points are many.

Decision rights. As the organization grows, people who used to be in the room for everything start finding out about decisions after they’re made, and they experience it as betrayal when it’s actually just inevitable. Nobody warns you, and it arrives with strange emotional force.

The compensating instinct is over-inclusion: more consultation, more stakeholders, more people in more meetings. But when decision rights are fuzzy and everyone is a stakeholder, contested decisions don’t get made; they get escalated. Everything ambiguous flows upstairs to the few people with unambiguous authority. Senior leaders become a reactive committee processing a queue of other people’s decisions… the org chart looks like you delegated, but your calendar says you didn’t.

Coordination. Every new hire doesn’t just add one person; they add a relationship with every person already there. Ten people have 45 possible lines of communication. A hundred people have 4,950. This is why adding people to a late project makes it later, and why hiring fast reduces an organization’s capacity before it raises it: every new person taxes everyone who already knows how things work.

This is extra bad in most NGOs, which often run matrix structures. This means process and stakeholder management grows faster than headcount and makes ruthless decoupling (fewer people in the path of any given decision) one of the highest-return things a scaling org can do.

Smart heroics are not scalable systems. Every organization scales on manual processes patched with individual heroics, and every one of those has a limit discovered only by exceeding it. The spreadsheet that’s actually the database. The approval process that is actually an email thread between the same four people. The colleague who knows how everything connects, who was your greatest asset at 50 people but your single point of failure at 500.

Quality control. At a small scale, quality is a function of hiring good people and good communication. At large scale, the people haven’t gotten worse, but nobody can see the whole anymore, and quality becomes a function of instrumentation: dashboards, sampling, audits, alerts. Without that infrastructure, you end up hiring watchers for the watchers—a quality team overseeing a quality team, growing forever. The transition from “I trust Mary” to “I trust the alerting threshold” feels like an abdication of duty in a mission-driven org, but it is not optional.

Risk. Benchmark data puts serious employee-relations claims (discrimination, harassment, retaliation) at 15 per 1,000 employees per year. At 70 employees, that’s a once-a-year event your leadership team handles personally and remembers for a long time. At 1,000 employees, there is always at least one investigation open somewhere. Add in legal threats, safeguarding incidents, and regulatory inquiries across (at least in GiveDirectly’s case) a dozen-plus countries’ labor and data laws, and the question stops being whether serious incidents happen. It becomes whether your governance is mature enough to process them as routine. Organizations that haven’t built that maturity get derailed by base-rate events: each one feels existential, consumes the leadership team for weeks, and crowds out the actual work.

Culture. For the first 150 or so people, culture transmits by osmosis. Past that point you have to hold it on purpose, against incentives that quietly pull the other way. Most organizations don’t. And when that happens, trust thins out between leaders and staff, between departments, and between headquarters and the teams in-country. Sub-cultures form and rub against each other. Politics arrives. The bar drops on both talent and work. And capable people start playing for their own patch rather than the mission, because the two are no longer perfectly aligned. This is the default path, not bad luck, and requires concerted effort and investment to avoid.

Leadership. The skills that matter change dramatically, and not everyone will make the transition at the speed needed. At smaller scale leaders succeed by personally solving the hardest problems, making fast calls with incomplete information, recruiting the first few brilliant generalists, selling the mission one conversation at a time, and being the person who can step into any role when it’s on fire.

At large scale different muscles are needed: goal and metric design, capital allocation across bets that can’t all win, building incentive systems that scale judgment you can no longer apply personally, recruiting senior talent good enough to be trusted with real autonomy, and org-wide communication to people you’ll never meet. Knowing how to maintain the 30,000 foot view, and when and how to zoom in and sweat the details without creating a system that relies on you doing so.

None of these are exotic failures. They are what normal looks like when organizations grow quickly. An organization absorbing a step-change in funding is fighting on all fronts (and others!) at once, while delivering, while hiring, while being watched.

Let’s get real about scalability profiles

To bring all of this together, every intervention has a scalability profile, and so does the organization that delivers it. The two are scored separately, and the impact you actually get is capped by the weaker of them.

The intervention: can the thing itself scale?

Market size: how many people or problems it can reach, and how many dollars can be deployed against that number before it runs out of room.
Unit economics at scale: whether cost per outcome rises, holds, or falls as it grows, net of unintended consequences or negative spillovers.
Speed limit: how fast it can grow, set by its binding constraint: cold chains, trained staff, trust, supply, winnable campaigns.
Execution robustness: how much of the result survives when it’s run by average people, under average conditions, including a handoff to government or another implementer.

The organization: can this team scale it?

Organizational readiness: whether leadership, coordination, quality control, risk governance, culture, and systems can absorb growth without breaking.

A high scalability profile needs strength in both layers: a large market, economics that hold, few hard speed constraints, results robust to ordinary execution—and an organization mature enough to carry it. A low score on any single factor caps the whole.

One reason I do the work I do at GiveDirectly (disclaimer, discount accordingly) is that direct cash transfers as an intervention have an unusually high scalability profile (though we’re still doing work on the unit economics at truly massive scale, and it’s operationally much harder to scale than most people think). But it also only pays off if we have the readiness to match. We’ve cleared that bar in some moments, missed it in others and are working hard on it as we speak.

There’s important work ahead for anyone working in this space. Hopefully understanding the absorption problem better will allow us to level up what’s possible and how we navigate the coming years.

First, we need more interventions that have a high scalability profile to begin with. Since those don’t appear on their own, that means deliberately funding and executing on the research and development to discover and build them.

Second, we need to understand the profile of what already exists, and where there’s significant upside, invest to make delivery models more robust to speed, more robust to variable execution, and backed by organizations mature enough not to break under the weight.

The first is about expanding the menu. The second is about making the dishes we already have worth ordering at scale. I plan to write about how funders and implementers can do both in the coming weeks.

The world needs more crazy but credible ambition for improving things. We shouldn’t accept a reality in which philanthropic dollars only ever improve things on the margins.

Animal disenhancement by weganskie_miaso

weganskie_miaso — Wed, 01 Jul 2026 21:57:36 +0000

What are your thoughts regarding animal disenhancement as outlined in Shriver 2009 or 2018?

It seems like pain can be genetically edited away, similarly to other nasty emotions like in this case. I haven’t seen much discussion about it.

On-ramps for mid-career, non-Anglophone entrants to AI safety by Aleksei Khvostov

Aleksei Khvostov — Wed, 01 Jul 2026 21:55:44 +0000

I used Claude to assist with the translation; all arguments were reviewed and revised by me.

I came into AI safety mid-career, after fifteen years of editorial work in Russian-language media – the kind of background that, on paper, doesn’t look like an AI safety resume at all. Russian is my first language; English is one I work in, not one I grew up reasoning in publicly. I’m writing this from inside the US now, a few years into the move – close enough to the hubs this post is about to see the difference between how the pathway looks from outside and how it works once you are standing in it. Trying to find a way in, I kept running into a pattern: the on-ramps weren’t closed to me exactly, but they all seemed designed around a different person.

That person is roughly: young, academic, Anglophone, and flexible enough to relocate, take unpaid opportunities, attend fellowships, and spend concentrated time building field-specific credentials. Many existing programs work well for that profile, and that’s good. But there’s another group that seems structurally under-served: mid-career people from non-Anglophone countries who have relevant skills, but did not grow up inside the US/UK academic, EA, or LessWrong-adjacent pipeline.

This is a diagnosis, not a complaint, and it’s aimed at people designing AI safety and governance pathways. My central claim is that the barriers for this group aren’t merely additive – they’re interlocking, and they differ sharply in how tractable they are. That second part matters most for program design, so I’ll keep returning to it.

The shape of the problem

A common pathway into AI safety looks roughly like this: learn the basic ideas; attend courses or events; produce visible work; network; volunteer or join a project; eventually become legible enough for paid work or funding.

For a student, this is demanding but plausible. They may already be in an academic environment, have geographic flexibility, carry a lower opportunity cost, and sit closer to the social style of many EA and AI safety spaces.

For a mid-career person from a non-Anglophone background, the same path has a different cost structure. They aren’t missing just one thing. They may face financial constraints, credential mismatch, weak network access, unfamiliar field norms, language-and-culture translation costs, and limited time for portfolio-building all at once. Removing one barrier doesn’t free the person, because the others stay binding. That’s why single-barrier programs often miss this group – not because the programs are bad, but because they’re solving a differently shaped problem. The current pathway asks these entrants to absorb too many transition costs privately.

Barrier 1: Financial friction

Mid-career entrants usually already have careers, families, rent or mortgages, dependents, or immigration constraints. They can’t treat a career transition like an extended student project. Unpaid fellowships, speculative volunteering, relocation-heavy opportunities, and long stretches of “just build career capital” are much harder to sustain when you’re already responsible for an adult life.

So portfolio-building has to happen in parallel with paid work – keeping a job, taking freelance work, juggling income streams while also trying to study, attend events, build, write, network, and apply. The result isn’t only slower progress. It systematically crowds out the things that make sustained work possible: rest, health, relationships, recovery.

This is rarely counted as a fieldbuilding cost, but it should be. A pathway that is technically open but practically requires months or years of unpaid parallel labor selects for people who can afford that burden – not necessarily for those who could contribute most.

Barrier 2: Credential mismatch

Credential mismatch is real, but it’s one of the more tractable barriers. A foreign degree may be discounted; a strong career elsewhere may not map onto local hiring signals; experience that was senior in one context can read as ambiguous in another. This hits people coming from journalism, communications, operations, law, policy, education, product, security, or public administration especially hard – serious skills that don’t appear in the expected format.

The good news is that demonstrated skill can partly compensate. A visible portfolio, a concrete project, a clear case study, a well-scoped contribution – these make a person more legible. This is one of the few barriers an individual can meaningfully work around.

But that creates an asymmetry worth holding onto: credential mismatch can be reduced by individual effort. Network access and financial constraints often cannot.

Barrier 3: Network access

Network access is the hardest barrier, because it’s the least individually solvable. Entry often runs through informal networks: who gives you feedback, who invites you to a Slack or Discord, who suggests a project, who explains which organizations are credible, who says “you should talk to this person.” Near the hubs, this feels natural. From outside, it’s opaque.

A concrete illustration is the role of flagship events. Many of the field’s most important gatherings are held in the US or UK. Even when a remote option exists, it usually buys access to the talks, not to the part that converts into opportunities: hallway conversations, demo sessions, informal introductions, weak-tie relationships. For someone mid-career outside those hubs, the cost isn’t just the ticket — it’s travel, visas, time away from paid work, and a price that means something very different once converted into local terms. So “there’s a livestream” can be true but incomplete. The recorded talks are often the most accessible part anyway; the networking layer, the part with real career value, stays bound to a physical room.

Some events offer travel grants or diversity stipends, which genuinely helps — but these are limited, competitive, and often easier for students or early-career participants to use than for mid-career people changing fields. The gap isn’t mainly access to information. It’s access to the relationship-driven layer where opportunities circulate.

Barrier 4: Cultural and epistemic translation

There’s also a translation problem that isn’t only linguistic. AI safety material often assumes familiarity with EA culture, LessWrong norms, US/UK career signaling, and a particular style of public reasoning: how uncertainty is expressed, how disagreement is framed, how criticism is handled, how people signal seriousness, which institutions are trusted, which concepts count as basic, which career paths count as legitimate.

A version of this problem shows up even in the way people talk about AI tools. There is a line that gets repeated in AI circles – Andrej Karpathy’s “the hottest new programming language is English,” later echoed in similar ways by prominent AI and tech executives. It is usually presented as a story of democratization: you no longer need formal technical training; you just need to describe clearly what you want.

But that framing quietly assumes that the describing happens in English, and in a particular register of English. If English becomes the interface to the field’s tools and conversations, then a non-native speaker is not just learning the ideas. They are operating through a second language in which unfamiliar phrasing, calques, or a different argumentative style can be read not as an accent, but as imprecise thinking.

The advice to “just describe what you want clearly” hides a cost that is distributed very unevenly.

There is a subtler version of the same cost. Fields reward a particular texture of reasoning – in many EA and AI safety spaces, that means making reasoning explicit, stating uncertainty, exposing assumptions, separating claims by confidence level, and producing artifacts others can inspect. These norms have real value. But they are easy to mistake for neutral rationality when they are also a local dialect of it. A person trained in a different intellectual tradition – where, say, authority is signaled through synthesis rather than through visible hedging – may not be read as reasoning differently. They may be read as reasoning worse. The penalty falls on the style, but it is scored as if it fell on the thinking.

For an outsider, then, the difficulty isn’t just “learning AI safety.” It’s learning the implicit social operating system around it. This is harder for mid-career entrants precisely because they already carry professional norms from another field or country – they know how to be effective in their original context, but not yet how to be legible in this one.

So they must learn the technical ideas, the governance debates, the career landscape, and the local communication style all at the same time, before they can become visibly useful to the field.

Why current programs miss this group

Many programs are optimized for a student-shaped problem: someone early-career, motivated, flexible, mostly needing context, mentorship, and first opportunities. That’s a real and important use case. But mid-career, non-Anglophone entrants need a different bundle: part-time entry rather than full-time immersion; paid or low-cost opportunities; remote-first participation; explicit explanation of field norms; recognition of demonstrated skill over local credentials; practical sequencing instead of vague advice; and bridges into networks they can’t reach geographically.

When those are absent, the pathway is open in theory and inaccessible in practice. This isn’t anyone’s fault – it’s a design-scope issue. Programs built for one population will under-serve another unless the difference is made explicit.

Some existing programs already help with parts of this problem, especially by making introductory material available online or offering grants and fellowships. My claim is not that nothing exists. It is that access to information does not automatically solve access to networks, credibility, or sustainable entry.

One way to describe this is as a form of fieldbuilding infrastructure debt. AI safety grew partly through small, high-context communities, informal trust networks, and shared intellectual norms. That helped the field move quickly while it was small. But as the field tries to absorb more people from outside those original networks, the missing infrastructure becomes more visible: explicit pathways, low-cost entry points, track-specific artifacts, and ways to build trust without already being inside the network.

Recommendations for fieldbuilders

1. Create paid, part-time entry paths. Scoped projects, microgrants, part-time fellowships, paid trial tasks, research-assistant work, evaluation or translation projects, short fieldbuilding contracts. The point is not to require abandoning income before becoming legible. Worth naming the obvious objection: funding is finite, and fieldbuilders should be careful about spending scarce resources on people who are not yet proven. But that is exactly why the work should be scoped. A paid task with a tangible output is both a filter and a signal; the money buys verifiable work, not a bet on hope. “Volunteer until someone notices” is a far more expensive strategy than it looks, and it quietly excludes exactly this group.

2. Build remote-first, free or low-cost networks. This is the barrier individuals can’t solve alone, which makes it the natural target for organizations. Free online entry points, remote-first reading groups, low-cost office hours, structured feedback, project-matching, regional or language-specific cohorts, mentorship pools for people outside major hubs. The goal isn’t to replace in-person hubs — they’re valuable — but to stop making hub proximity a hidden prerequisite. If becoming legible requires proximity to London, Berkeley, Oxford, Boston, or DC, the field is leaving talent unused.

3. Recognize demonstrated skill over local credentials. Make it clearer what artifacts count as evidence of fit: an evaluation harness, a model-behavior case study, a governance memo, a red-teaming report, a localization project, a fieldbuilding experiment, a clear writeup of a failed but well-designed attempt. This beats generic “write publicly” advice. Public writing matters for some tracks – research, policy, communications – but for many operator or generalist profiles the strongest signal isn’t an essay, it’s demonstrated execution.

A more useful instruction: produce one visible artifact that shows the kind of contribution you want to make, with different examples for different tracks. Concretely: someone interested in evals might publish a small, reproducible benchmark with a clear threat model, dataset limitations, and failure analysis. Someone with a communications or translation background might localize an important safety resource and add notes on which concepts do not transfer cleanly across languages. Someone with operations experience might document a process improvement for an AI safety project, including the problem, intervention, result, and remaining uncertainty.

I’ll admit the bias here – this is the strategy I bet on myself, building evaluation artifacts rather than waiting to be credentialed. In my case, this has meant building small public artifacts around AI evaluation and agent safety: reproducible eval sets, CLI tools, benchmark-style reports, and GitHub projects aimed at making failure modes inspectable rather than merely describing my interest in the field.

4. Make the pathway legible as a sequence. “Learn, network, write, volunteer, apply” is directionally fine but operationally vague. Better: understand the landscape, choose a tentative track, identify the relevant skill gap, produce one credible artifact, and then seek feedback, scoped work, funding, or applications. Don’t assume student-level flexibility on timing; alongside paid work the same sequence just takes longer. But even a slow path is easier to walk when the order of operations is visible.

5. Don’t reward only those who can go full-tilt. Someone who can study, attend, write, volunteer, network, and build for six months without meaningful income is receiving a subsidy – from savings, family, a partner’s income, or unusually low obligations. That’s not bad, but it isn’t neutral. If the field unintentionally selects for people who can afford long uncertainty, it misses capable people with stronger obligations and equally valuable skills. Sustainability isn’t a luxury concern; it shapes who can enter at all.

Why this matters

AI safety and governance need more than researchers. They need people who can evaluate systems, run programs, build institutions, communicate across cultures, manage operations, translate concepts, and notice how technical ideas meet real-world institutions. Many of those people already exist – mid-career, outside the Anglophone world, immigrants, people who won’t look like traditional candidates at first glance.

This is especially relevant for AI governance and policy. If AI governance is partly about international coordination, regulation, and public legitimacy, then the field cannot afford pathways that mostly recognize people already fluent in Anglophone professional and epistemic norms.

The question is whether the field has pathways that can recognize and absorb these people. If not, the problem isn’t individual. It’s infrastructural.

This is grounded in personal experience, not a representative study – one vantage point, not a survey of the group I’m describing. I’m offering it because the pathway seems under-designed for people outside the usual Anglophone and early-career pipelines. I’d be glad to hear from others with similar or different experiences, and especially from people designing these programs. If this diagnosis is wrong, incomplete, or already being addressed somewhere, I’d like to know.

Frontpage posts - Effective Altruism forum viewer

Has Global Health Been Rigorously Compared With Other Cause Areas? by James Brobin

How to Solve AI Biosecurity by Sophie Kim

Bioweapons in the Age of AI

Should We Be Con­cerned About Bioter­ror­ists?

I. Successes

II. Failures

III. Analysis

IV. Conclusion

When Rus­sia Wants a Bioweapon, It Gets One

I. His­tor­i­cal Precedent

II. Con­tem­po­rary Cases

III. Analysis

The Bot­tle­necks Are Eroding

AI is Erod­ing the Knowl­edge Bottleneck

Beyond AI: A Con­verg­ing Land­scape of Eroded Bottlenecks

The Cap­i­tal, Equip­ment, Knowl­edge, & Ex­per­tise Bottlenecks

Policy Interventions

4.1 Re­quire stan­dard­ized pre-de­ploy­ment bio evals

4.2 Man­date AI-en­abled DNA syn­the­sis screening

4.3 Im­ple­ment know-your-cus­tomer re­quire­ments for cloud lab­o­ra­to­ries and con­tract re­search organizations

4.4 Estab­lish tiered ac­cess con­trols and stan­dard­ized mon­i­tor­ing pro­to­cols for LLMs with ad­vanced bio capabilities

4.5 De­ploy pathogen-ag­nos­tic metage­nomic sequencing

4.6 Fund the Strate­gic Na­tional Stock­pile for PPE surge capacity

4.7 Up­grade in­door air infrastructure

4.8 Build rapid stand­ing vac­cine pro­duc­tion capacity

FAQ & Counterarguments

~85% of Major Hotels Groups and 80% of Restaurant Chains Locations in the Philippines Committed to Cage-Free Eggs by Whitney Peng

TL;DR

The Numbers

The Mar­ket Side

The Sup­ply Side

The Work Be­hind the Number

The Learning Trap: What Simulated Clueless Agents Reveal About the Unawareness Argument by dan.pandori 🔸

Summary

1. The im­passe over the prag­matic critique

2. Why simu­la­tions are pro­ba­tive here

3. Model 1: What in­com­pa­ra­bil­ity ac­tu­ally costs

4. Model 2: The concession

5. Model 3: Act­ing is how aware­ness grows

6. What sur­vives, and what should change

7. Objections

8. Limitations

Ap­pendix: Methods

References

Author­ship note

Yuval Harari: philanthropy principles + 3 AI-focused charities he recommends by BruceF

Effective petitions (July 2026) by Stijn Bruers 🔸

Announcing the Safe Pareto Improvements (SPI) Fundamentals Program by Center on Long-Term Risk

Content

Tar­get audience

Contact

In 2026, is EAGx Berkeley or EAG NYC a better place to learn about US-China AI governance? by ben.smith

I’m never satisfied by Ajeya

AMA: Anthony DiGiovanni, author of the ‘Challenge of Unawareness’ sequence by Toby Tremlett🔹

Maybe do the thing you wish CEA would do by alejoacelas 🔸

Lydia Laurenson: “The Inside Story of Leverage Research” by Davis_Kingsley

most EAs should probably not be living in high cost-of-living (HCOL) areas most of the time by matthes

the core case

example

mis­cel­la­neous re­lated thoughts

RP is looking for project founders in neglected animal areas by Rethink Priorities

Background

Which an­i­mal top­ics are we fo­cused on?

Small teams are already get­ting out­sized results

Who are we look­ing for?

What we can offer

How to respond

About Re­think Priorities

Time Sensitive Do Gooding Opportunities by Bentham’s Bulldog

1 Dona­tion matching

2 Or­ga­nizer sup­port program

AFFINE – A Retrospective by Ouro

A Day at AFFINE[1]

Miss­ing Foundations

Brains in a Chateau in Bohemia

Egregore

Bridge-Building

Numbers

Fu­ture Plans

Should We Be Concerned About Bioterrorists?

When Russia Wants a Bioweapon, It Gets One

I. Historical Precedent

II. Contemporary Cases

The Bottlenecks Are Eroding

AI is Eroding the Knowledge Bottleneck

Beyond AI: A Converging Landscape of Eroded Bottlenecks

The Capital, Equipment, Knowledge, & Expertise Bottlenecks

4.1 Require standardized pre-deployment bio evals

4.2 Mandate AI-enabled DNA synthesis screening

4.3 Implement know-your-customer requirements for cloud laboratories and contract research organizations

4.4 Establish tiered access controls and standardized monitoring protocols for LLMs with advanced bio capabilities

4.5 Deploy pathogen-agnostic metagenomic sequencing

4.6 Fund the Strategic National Stockpile for PPE surge capacity

4.7 Upgrade indoor air infrastructure

4.8 Build rapid standing vaccine production capacity

The Market Side

The Supply Side

The Work Behind the Number

1. The impasse over the pragmatic critique

2. Why simulations are probative here

3. Model 1: What incomparability actually costs

5. Model 3: Acting is how awareness grows

6. What survives, and what should change

Appendix: Methods

Authorship note

Target audience

miscellaneous related thoughts

Which animal topics are we focused on?

Small teams are already getting outsized results

Who are we looking for?

About Rethink Priorities

1 Donation matching

2 Organizer support program

A Day at AFFINE^[1]

Missing Foundations

Future Plans

I. Impact rarely grows proportionally with scale

II. Many of the best interventions have a low ceiling on their potential scale

IV. Organizations break as they scale

Let’s get real about scalability profiles

Barrier 1: Financial friction

Barrier 2: Credential mismatch

Barrier 3: Network access

Barrier 4: Cultural and epistemic translation

Why current programs miss this group

Recommendations for fieldbuilders