Frontpage posts - Effective Altruism forum viewer

High Impact Medicine (Hi-Med) is hiring a Program Manager by Simon Ling

Simon Ling — Thu, 24 Sep 2026 21:02:00 +0000

The Program Manager will be responsible for our flagship Career Guidance Program and help shape how it develops. You will support doctors and medical students from around the world, who engage with our Hi-Med Career Planning Course and 1:1 advisory calls.

About Hi-Med

High Impact Medicine (Hi-Med) is a nonprofit organization whose mission is to guide a global community of doctors and medical students to do more good through high impact careers and giving. We believe that medical professionals are highly skilled and motivated but have huge untapped potential to save more lives and reduce future threats to health. The number of applications to our programs has grown each year since 2023, with significant growth in 2026. Our strategic focus is on providing excellent career and effective giving guidance, building our Information Hub, and developing new programs or adapting our existing ones to meet the needs of specific segments of our audience.

About the position

The primary responsibilities include:

Delivering the Hi-Med Career Planning Course (three cohorts each year)
Program development and improvement
Developing new partnerships and growing our outreach to our target audience
Contribution to organizational strategic planning.

Secondary priorities (approximately 30% of time commitment) include maintenance of our Information Hub and our Hi-Med community management system.

We are seeking someone with several years of relevant program management experience for this 0.6 − 1.0 FTE position. Previous medical training is preferred, but we will also consider others with training in public health, global health, or related fields. The position is remote in a location within 6 hours of Eastern Time.

Click here for further details and to apply.

If you dislike some Berkeley sex norms, you can say so by Nathan Young

Nathan Young — Thu, 24 Sep 2026 19:57:56 +0000

I sense we are about to get a load of discourse about sex norms in AI safety. And I suspect a lot of people are going to privately dislike them but publicly feel unable to say so, until some kind of horrible “I always hated that” “It was the worst worst worst” moment.

I would prefer that rather than a dam breaking we could release the pressure more gradually. So I recommend saying so. Say to someone “I dislike the rationalist sex scene” or heck, “I dislike the veganism” or “I don’t like how people leave multi-person conversations without saying so”. It is okay to dislike things.

I cannot promise that no one will ever get annoyed at you over it, I cannot promise I will not be curt in response, but I would like to accept that kind of criticism and I hope that rationalists will too.

But I think the worse outcome is a resentment that builds from having to defend something or someone one is repulsed by. And I don’t think that is sustainable to a community. And I don’t think it will help good thinking about discourse as it happens.

Why I’m optimistic about cultivated meat (and a suggestion for the frontier labs) by Bruce Friedrich 🔸

Bruce Friedrich 🔸 — Thu, 24 Sep 2026 19:55:21 +0000

In my first “Meat Matters” substack, I wrote that despite some recent popular press obituaries, I remain optimistic about the scientific viability of cultivated meat, and I spelled out why.

Here’s the executive summary:

In less than a decade, costs have fallen by orders of magnitude, government funding and scientific progress have surged, and multiple companies report that they are within striking distance of price parity. In short: The science and engineering challenges will not require inconceivably difficult breakthroughs or AGI, and of the dozens of cultivated meat scientists and engineers I spoke with for my new book Meat, all of them are optimistic about our path forward.

Zak Weston from Food System Innovations (FSI) works on market shaping for FSI and has personally reviewed techno-economic analyses from five different cultivated meat companies. He says that those analyses show “that at current cell line and media performance, if produced in commercial-scale facilities, they could reach cost parity or cost advantage compared to their animal meat benchmark.”

In Zak’s view (and mine), “the technological progress over the past few years has been remarkable.”

Click here to read the full piece, and if you’d like to receive a monthly dose of optimism and urgency related to alternative meats, please subscribe.

As an aside, Peter Singer liked the essay so much that he shared it on the home page of his substack and sent it to all of his followers. If you have a substack and want to help me build up my subscriber base, I would be grateful, of course.

An ITN project for Anthropic’s and OpenAI’s life sciences divisions (and Google DeepMind)

Although I’m optimistic about solving cultivated meat’s science and engineering challenges, I don’t think it’s going to be easy, and I do think there’s a good chance that we could fail.

If we fail, though, it will be because we didn’t bring the will, not because the scientific and engineering challenges were insurmountable.

It’s also worth noting that every second, the world slaughters 2,800 land animals and 4,100 farmed fin fish. That number has been going up every single year since FAO started measuring, in 1961.

So even if solving cultivated meat will happen eventually, solving it quicker matters. A lot.

Our current path forward involves two vectors:

startups, which are doing incredibly well, especially considering how little has been spent on the project; right now is a great time to invest, since the top startups in cultivated meat are drastically undervalued.
open access science, which is making great progress, especially considering the relative dearth of resources.

With the budgets, expertise, and connections of the frontier labs, the fundamental science and engineering roadmap for cultivated meat could be created and then handed to the world.

With some additional resources, the labs could accelerate the current start-ups and launch new ones, creating a robust cultivated meat industry.

If you’re at one of the frontier labs and would like to discuss making this happen, please get in touch.

State of Pandemic Early Warning by Jeff Kaufman 🔸

Jeff Kaufman 🔸 — Thu, 24 Sep 2026 18:09:02 +0000

This lightly-edited version of a memo that I presented at the Summer 2026 Biosecurity Summit outside of DC. While others at SecureBio often see things similarly, I’m attempting to present my view and not a SecureBio “house view”.

The Goal

We need to be robust to adversaries who want to cause very large-scale harm with biology. This includes actors (human or AI) who want to kill all humans, cause short-term incapacitation or long-term civilizational collapse, or who have strategies for sparing some while they harm others. There are multiple reasons an actor might have these targets aside from being directly omnicidal, such as reducing response capacity during an AI takeover.

That there is an attacker itself is a key constraint to any defensive system: it must be designed for adversarial attacks. The attacker can assess the state of the world’s detection systems and plan accordingly. Taken to the extreme, this presents a “minimax” landscape: a system is only as good as its weakest link (the place where it is least sensitive). This is an important framing, and it correctly prioritizes getting some sensitivity towards a wide range of attacks over very high sensitivity towards just a few. On the other hand, (a) to the extent that gaps depend on non-public choices, you can maintain strategic ambiguity to prevent attackers from aiming for the gaps, and (b) reducing the number of gaps reduces attacker options.

The strongest form of success is to deter an attacker by denying them the ability to achieve their goal: an adversary who knows an attack wouldn’t accomplish their goal will generally not try that attack. This deterrent effect is tightly coupled to the extent to which detection would indeed thwart the achievement of the attacker’s goals. This means that achieving deterrence-by-denial means both building an effective system, from detection through to action, and making it known that you’ve built it.

The other main kind of deterrence is deterrence-by-punishment. If you develop strong attribution capabilities, an attacker risks identification and retaliation. The extent to which this would deter an adversary, however, depends a lot on what they have to lose. A state seeking strategic advantage might be deterred by the prospect of retaliation, while someone keen on killing everyone (including themselves) has little left to threaten.

Biosurveillance Applications

Where specifically does biosurveillance fit in? What are the threats, and where can it make the difference between an attacker succeeding and failing?

Initial Detection of Stealth Pandemics

A stealth pandemic is one where a pathogen spreads through most of the population unnoticed, with no or unremarkable symptoms, before causing very serious effects. [1] If it were subtle enough, people wouldn’t realize how serious the situation was in time to respond effectively. While we call these “stealth pathogens”, whether a given pathogen would cause a stealth pandemic depends on the interaction between the pathogen, the body, and humanity’s many ways of noticing that something unusual is happening.

Whether it is possible to create a pathogen that would be sufficiently difficult to notice is an open question: I’ve heard different things from different experts. When considering (a) the significant advances in biological design tools and general biological understanding that we’ve been seeing with AI progress, and (b) the speed and unpredictability of the process by which unusual symptoms today lead to attention and action, I do think there’s a significant chance that within the next five years many actors would be in a position to cause stealth pandemics.

Detection of suspicious sequencing reads may not be sufficient to estimate whether they represent an ongoing stealth pandemic. A pathogen may be constructed in a way that makes its potential for rapid spread and delayed harm obvious, but that is far from guaranteed. Assessing this likely requires additional scientific work: genome completion, estimating likely effects in the human body, and considering whether the genome suggests an intentional attack.

Triggering Initial Response

A pathogen doesn’t have to be stealthy to be disastrous: it could simply be very hard to contain (a “wildfire pandemic”). Beyond its direct effects, such a pathogen could be intentionally released to reduce capacity at a critical time, such as during a coup or an AI takeover attempt.

Whether an outbreak is wildfire, stealth, or has aspects of both, the time from initial discovery to serious response is critical. Initial indications are generally ambiguous, and it is often difficult to understand the extent or trajectory of the threat. With the 1976 swine flu, we overreacted and vaccinated 45M people because we didn’t have the monitoring to know it wasn’t spreading widely. With 2014 ebola in West Africa, we underreacted and let it spread freely for three months because the extent wasn’t recognized. Similarly, with 2026 ebola, we saw another three month delay, this time in part because field PCR tests couldn’t see it. The case of 2009 H1N1, however, showed how a well functioning (though flu-specific) biosurveillance system could enable timely response. In today’s COVID-weary climate where public health is deeply worried about losing credibility through false alarms, biosurveillance can help avoid a default of delaying response while waiting for more information.

Enabling Ongoing Suppression

Once an initial response is in motion, you need monitoring to effectively deploy mitigations and know whether they’re working. How much value there is depends on how symptoms relate to infectiousness. If symptoms are absent or highly delayed, effective response is essentially impossible without solid monitoring. At the other extreme, if symptoms are highly visible and begin immediately, monitoring is moderately valuable: you know you have a problem, but with “fog of war” you don’t fully know its extent or distribution. Large-scale monitoring allows you to compare locations and track trajectories to optimize resource deployment.

I give relatively little attention to this biosurveillance application in this memo, mainly because I think it’s a place where what exists today is closest to what needs to exist, so this is a lower priority for additional work.

What does success look like?

There’s no single threshold, where you win by building a system with a specific level of capability. Biosurveillance systems reduce the cost-benefit tradeoff of initiating a pandemic; increasingly capable systems decrease the likelihood that an attacker deems this worth their effort and reduce the harm if they decide to attack. Still, for each application, there are some ‘sweet spots’ where the cost-benefit ratio is maximized.

Initial Detection of Stealth Pandemics

If you imagine the most capable system that can be deployed for a given level of investment, it will be capable of averting some fraction of expected possible stealth harm. Here’s how I see it:

A system that is too slow to beat status-quo detection might still help with triggering initial response, but doesn’t provide initial detection benefit.
A system that is a bit faster but doesn’t give enough time to act before most people are already infected provides some benefit, especially in cases where the harm can be mitigated if it’s known soon enough, but most expected harm will still occur.
A system capable enough to flag an attack in time to allow us to protect enough workers, such that (a) civilization does not collapse and (b) medical countermeasures can be developed and deployed, provides significant benefit. Attacks could still be massively disruptive.
An extremely capable system, including a very broad sampling regime, could flag outbreaks when they were still small enough to contain. At this point an attack is still costly, but the disruption is limited to the areas where the attack was seeded.

None of these are hard boundaries. There are factors that ‘smear’ these thresholds across many capability levels: some of this is luck (ex: who contributes to what samples), while some is uncertainty about the world that both we and an attacker would share (ex: how much shedding a given pathogen would actually produce in a large population). Here’s an illustrative chart:

I’ve intentionally left the x-axis vague. It’s not “cumulative incidence at detection”, because (a) that’s horizontally smeared as described above and (b) it would imply that increased capacity is downstream of sensitivity only and not other important factors like what kind of pathogens you can detect at all or how quickly you can trigger response. Instead, the x-axis represents the level of resources invested.

A system that is sufficiently capable to flag pathogens early enough to protect enough workers to avert civilizational collapse is well worth the investment; additional sensitivity beyond this is valuable, but likely substantially less cost-effective.

This means that success looks like a pathogen-agnostic system that flags attacks in time to protect enough workers to prevent civilizational collapse and buy time for pathogen-specific mitigations. How early this needs to be depends on how quickly response can happen. If effective response takes a month from detection, you need to flag before ~0.05% of people have been infected. On the other hand, if it takes two weeks, you only need to flag before ~2% of people have been infected, and if you can get it down to seven days, then even flagging at 10% cumulative infections would be enough. See appendix for more detailed reasoning.

Note that the “fast enough to beat status-quo detection” regime would be a far higher bar for wildfire than stealth. This means that, on time scales rapid enough to factor into planning, I don’t expect it to be economically feasible to build a system where biosurveillance would be your first indication of a wildfire pathogen.

Triggering Initial Response

We don’t know very much about what would actually get decision-makers to take sufficiently prompt action. In a stealth scenario, this is extremely challenging, since response must begin before the main symptoms manifest, but even in a wildfire scenario, there’s a huge difference between a response that follows immediately from when someone identifies the first cluster vs one where it kicks off in earnest only after deaths start to become highly visible.

Success looks like a very short time, ideally under a week, from when a catastrophic pathogen is flagged until the danger has been recognized, PPE has been distributed to essential workers, biohardening has been deployed or activated, lockdowns have been instituted, and development of rapid diagnostics and other medical countermeasures has begun. These are very costly actions, in economic terms but also via anteing political capital and institutional trust. Biosurveillance can contribute by giving decision-makers the information to determine whether those costs are worth paying. This looks like clarifying the extent of spread to date, estimating trajectory, and performing initial wet lab work such as genome completion.

Beyond the technical work, response requires trust. Before someone will act on an alert from a system they need to believe that it indicates something real, and that trust needs to be built over time. Non-catastrophic detections are key, showing you can track trends that match what other evidence shows, surface matters of public health concern, and turn up real engineered ‘benign positives’.

Most of the work in reducing time to response, however, is outside biosurveillance. This could include helping government agencies develop better plans for how to handle various indications, running exercises that get the actual principals to experience the feeling of making these specific calls with realistically incomplete information, or streamlining inter-agency communication so available clinical and epidemiological data gets to the right people quickly.

Enabling Ongoing Suppression

Wastewater PCR was used by Australia, New Zealand, and Singapore (down to the building level), among others, as part of their suppression strategies to guide public health response during COVID-19. At the point when transmission has been diminished to some threshold, or if an attack is identified before the pathogen has spread widely, a sensitive biosurveillance system can tell you when and where you need to focus more expensive and intrusive detection methods and interventions.

This means that a system for successfully maintaining suppression looks like a larger-scale and more fine-grained implementation of the biosurveillance component for triggering initial response. Knowing that something is spreading in ten major cities around the US might be enough to spur rapid action, but you need a much more detailed picture if you’re trying to maintain ongoing suppression.

What gets us there?

Initial detection of stealth pandemics is SecureBio Detection’s focus, and where I have the most developed view. I’ll walk through what I think is needed for this scenario, and then discuss how this changes for other scenarios. Please don’t interpret this structure as a claim about the relative likelihood of stealth scenarios!

Sampling Strategies

Municipal wastewater is a very helpful sampling modality: in most cities, sewage is processed at a small number of locations, letting you track pathogens across hundreds of thousands of people from a single easily-collected sample. Since many pathogens don’t shed heavily into wastewater, however, and it’s a very noisy sample type, comprehensive initial detection at a reasonable cost likely requires tracking additional sample types. SecureBio runs a nasal swab program; beyond swabs I see air, blood, aircraft wastewater, and leftover material from clinical tests (clinical lab discards) as the strongest candidates for supplemental sampling. SecureBio has looked into these strategies in some depth (initial overview, blood, aircraft wastewater, clinical discards), but there’s still a lot that could be learned here.

Lab Technology

You need technology that is pathogen agnostic: if you monitor only specific pathogens, the adversary can choose ones you don’t monitor. With current and near-future tech, “pathogen agnostic” means sequencing. For viruses, it is practical to concentrate particles by size, and then perform untargeted and enriched metagenomic sequencing metagenomic sequencing. This lets you do the rest of the detection in the computer, for maximum flexibility and generality, and this is what SecureBio does today.

For bacteria, I’m pessimistic about adapting this approach directly, at least with wastewater, because the genomes are much larger and there is a rich background of sewer bacteria that can’t be physically separated from potentially threatening bacteria before sequencing in the same way that viruses can. How to handle this is still an open question; see below.

For mirror life, the initial stages of spread might be very hard to recognize, and an important step would be learning that mirror life was spreading at all. It’s likely that a highly sensitive and relatively cheap assay could be developed, but no one has started on this yet.

Computational Technology

Metagenomic sequencing moves much of the problem of initial detection from the lab into the computer. This means massively more sequencing reads than humans could evaluate (8+ orders of magnitude), so we need a detection system that can identify which reads indicate something concerning is happening. Some reads can easily be recognized as concerning (ex: nucleic acid subsequences unique to the smallpox genome should not be in wastewater) while others require very sophisticated processing (ex: understanding the complex background well enough to flag de novo genomes with no sequence similarity to anything currently existing). The general approaches are looking for sequences with one or more of the following features:

Dangerous. Sequences matching known pathogens that you would not expect to see in the sample, perhaps signaling that they’ve been introduced.
Modified. Partial sequence matches, perhaps signaling something engineered.
New. Sequences that you haven’t seen before, perhaps because they were created de-novo.
Growing. Sequences that are becoming more common, perhaps signaling that they’re spreading through the human population.

The computational approach is very difficult given the scale of the data, uncertainty over what an attack might look like, and the need for rapid analysis. On the other hand, these are the kinds of highly computational problems where I expect AI can be very productively applied, whereas many other aspects of this system require relatively slow real-world effort.

What exists today?

Pathogen-agnostic biosurveillance is still in its early stages. I know of four systems doing untargeted metagenomic sequencing for biosurveillance today:

CASPER (SecureBio + Marc Johnson’s lab at the University of Missouri and other academic partners). This is wastewater, primarily municipal, from 49 facilities representing 24 US cities. It’s virus-focused, and generated with very deep short-read sequencing. Lab work happens independently at MU and SecureBio, with bioinformatics at SecureBio.
Zephyr (SecureBio + Helena Solo-Gabriele’s lab at the University of Miami). This is pooled nasal swabs from Boston and Miami, collected primarily in public places, bringing in swabs from about 1,000 people weekly. It’s virus-focused, and unlike CASPER uses long-read sequencing.
ANTI-DOTE (DoW + PHC + SecureBio). This is wastewater from five US military facilities, which SecureBio processes under contract from PHC. These samples go through the same lab and bioinformatic processes as CASPER samples.
mSCAPE (UKGOV). This is bronchoalveolar lavage from UK hospitals, currently including relatively few samples, which limits sensitivity. Unlike the previous three, mSCAPE uses combined viral and bacterial sequencing. These samples are sequenced to a relatively low depth with long-read sequencing, and the data supports clinical practice, public health, and biodefense.

There are also several hybrid-capture sequencing projects that might detect an attack if the agent was similar enough to existing pathogens.

In addition to detection via pathogen-agnostic sequencing, there are also paths where an outbreak becomes visible via showing symptoms in a sufficiently large fraction of infected people. This could lead to suspicious clusters, and then to sequencing and noticing that a genome looked edited. This is not a well-developed path today, but (as discussed below) I’d like to see investment here.

What’s missing?

Here’s an overview of what I think most needs doing. It represents the current state of my thinking, but it’s not as thoroughly considered as I wish it were: please don’t overweight it in your own decision-making! While SecureBio Detection is exploring some of these, I think the ideal structure is a healthy ecosystem of organizations taking on different parts of the problem in parallel. SecureBio is often able to share samples, sequence prepared nucleic acids, or share data to help others make progress.

In roughly descending order of how valuable I estimate non-SecureBio work would be, representing a combination of both overall value and the value of the work happening independently:

[Other] Modeling. We need good estimates of the necessary system scale and ideal network design to support optimal initial detection, response triggering, and ongoing suppression. There are initial estimates, but because this is a huge question (essentially pulling the whole field together), current work is rough. Rigorous treatments would be valuable for planning, both for independent funding allocation and for governments. This is especially valuable to happen outside of SecureBio, as a way of checking our work, and doesn’t need to all be executed by one group (groups can pick off subquestions).
[Comp] Red teaming. Itâ€™s important to analyze existing systems to assess how well they would handle a range of adversarially designed attacks. This can be done with reference to the system implementation, or by treating the system as a black box.
[Other] Parallel orgs. SecureBio Detection has historically focused on the US. Setting up parallel orgs in other geographies, especially Europe and Asia, would be really valuable. We are happy to advise anyone interested in doing this work!
[Lab] Detection of stealth bacterial pathogens. Viral particles are small, which means that you can separate them from human and bacterial cells without specifying in advance what sequences you’re interested in. Beyond this, pathogenic viruses represent enough of the total viral portion that just pulling it all into the computer to sort it out there is economical. This method can’t be directly applied to bacteria, at least not in wastewater, because there is so much irrelevant bacterial genetic material. SecureBio hasn’t done any work in this area yet, but some plausible approaches include aggressive depletion for things you know are not worrying, partially-targeted methods, and working with samples with a more favorable background. People have a wide range of estimates on how difficult this will be, but personally I expect it to be very hard.
[Comp] Parallel methods development. It would be great for others to be exploring other avenues in parallel. Since our main expertise is in traditional bioinformatics, I’d be especially excited to see people try other approaches, such as applying deep learning. We share much of our sequencing data publicly (PRJNA1247874, PRJNA1379685), in part to facilitate exactly this type of work.
[Lab] Sampling streams beyond wastewater and nasal swabs. Not everything sheds much into wastewater or the nose. A comprehensive system very likely needs a wider range of sample types. Blood, air, and clinical lab discards are all very promising here.
[Other] Response. Figure out what would get decision-makers to reliably act rapidly in a real emergency, and lay the groundwork so that happens. As noted above, this is probably mostly not biosurveillance. There are components (ex: developing escalation relationships) that need to be tightly coupled with biosurveillance, perhaps within SecureBio and parallel orgs, because of the limitations of information sharing. Other components (ex: policy recommendations, public outreach) make more sense as separate organizations. This is delicate work, however, and someone coming in noisily and insensitively could easily set the field back.
[Lab] Genome completion. A pathogen identified by a short-read biosurveillance system like CASPER would start as just a suspicious section of a genome. You can sometimes learn more about the genome through techniques such as outward assembly, but only if you happened to sequence the relevant reads. Lab work to assemble the whole genome lets you better understand what the pathogen would do in a human, which is likely on the critical path to both assess whether a response is warranted (and, if so, what that response should be), as well as to get people to act appropriately quickly.
[Comp] Analysis tooling. How do you go from “this genome looks worrying” to a good understanding of whether it’s engineered and what effect it would have in humans?
[Other] Swab collection. Currently, Zephyr requires field samplers standing on street corners and interacting with the public, and a team brings in ~40 samples per hour. If instead workplaces or schools could be convinced to integrate sample provision into daily routines, this could be far more scalable.
[Other] Clinical sequencing. Sequencing of clinical samples will likely eventually be deployed broadly based on its clinical benefits alone, displacing a wide range of pathogen-specific tests, but by default, this displacement process is far too slow. Sequencing needs to be a standard option doctors can easily reach for when someone presents with unusual symptoms. I think the tech is ready, or could be ready with a small amount of R&D, and so this is a commercial opportunity.
[Lab] Sensitivity increases. Untargeted metagenomic sequencing for pathogen-agnostic initial detection is an early-stage field, with relatively few people exploring it, and so on priors I expect there are large sensitivity improvements waiting to be discovered.
[Comp] Deterrence tracking. You can ask models to estimate the probability of an attack achieving its goals. If models report low probability to defenders, they probably report low probability to attackers. The best models I have access to can’t do a good job at this in response to a simple prompt, and if you need a series of prompts, you can’t expect your answer to reflect what an attacker would get. I expect this to change quickly, however, and it would be good to have an automatically-updated tracker showing the range of answers LLMs give to the question of whether existing detection systems are sufficient to make an attack not worth an attacker’s while. This would require some thought on which threat models to poll for and how to phrase the question in a way that is a good proxy for what an attacker would ask. It also closely ties into red teaming work.
[Lab] Partially targeted methods. Hybrid capture, tiled degenerate amplicon arrays, and other methods of mismatch-tolerant sequence-based enrichment could offer far higher sensitivity. While they’re unlikely to be sufficiently general to address all threats, there is a good chance that it makes sense for a mature detection system to include a partially-targeted component as a way to effectively exclude large areas of the threat landscape.
[Lab] Cheaper protocols. Sequencing is still an expensive proposition, but as sequencing has become cheaper the operation of the sequencing machine is no longer the largest cost on a per-sample basis. Bringing those other costs down would allow more scale for a given budget.
[Lab] Shorter lab time. Current metagenomics protocols require about a day on the bench and about a day on the sequencer. There are already strong pressures for reducing sequencer run time, and new machines are coming out in the ~8 hour range, but there’s a lot of work that could be done to speed up the rest of the processing.
[Lab] Pathogens outside of bacteria and viruses: fungi (and oomycetes), parasites, and prions. These are generally much more difficult for an attacker, especially for strategies that involve rapid spread.
[Lab] Mirror life. Current sequencing wouldn’t detect mirror life at all. Municipal wastewater samples would be a good place to look, which allows you to re-use much of the collection infrastructure, but would require an assay developed specifically for mirror life. I have this low because I currently expect mirror life to take long enough to develop that there will be time later for assay development.

How does this change for accelerating response to a wildfire pandemic?

During the COVID-19 pandemic, many groups built out PCR-based targeted wastewater monitoring. In the US, wastewater monitoring networks include NWSS (CDC), WastewaterSCAN (philanthropic), and Biobot (private). EU member states, Canada, Australia, and other countries track wastewater as a standard part of their public health systems. There are maybe two dozen countries that have some form of wastewater PCR that could be retargeted to track a new pathogen in an emergency once its genome was known.

On the other hand, none of this would move quickly enough today to address a wildfire pandemic. There are delays throughout the process: some of this is technical (ex: stocking consumables), but most of it is organizational (ex: policies that permit setting production work aside, overtime budgeting, on-calls, deals with synthesis providers for rush orders). Even in a serious emergency, I think ten days is a good best-case estimate today. With good preparation, however, three days is possible. I think getting these existing networks to prepare for rapid turnaround emergency response is really valuable, and SecureBio has started to have some of these conversations.

This is also a place where the same metagenomic sequencing system you would build for stealth pandemic detection could help you cut off additional days in your response. The technical and organizational delays that slow down PCR-based detection are downstream from how changing targets requires making changes in the physical world. Untargeted sequencing lets you skip those steps, at the cost of much lower sensitivity. On the other hand, once you know what you’re looking for, you can use approaches (like PCR) that are significantly more sensitive in the case of SCV2, by a factor of about 100. If you’ve built a sequencing system that can flag a stealth pandemic before 1% of people have been infected, then that factor of ~100 means it would be able to confirm a wildfire pandemic at ~0.01%. Still, it’s not clear to me that even 0.01% is early enough. It’s possible that you need 0.001% or even lower, at which point this argues either for a substantially larger investment in untargeted sequencing (to get enough data quickly) or giving up on sequencing for this application (because it can’t economically reach the target sensitivity).

If you wanted to firmly decide whether to go with MGS or PCR to accelerate initial response to a wildfire pandemic the key thing you’d need would be estimates of (a) what fraction of the population would likely be infected when a wildfire pandemic was noticed, and then (b) how many doubling periods there would be before decision-makers took action in the absence of this system. On the other hand, I’m not sure a firm decision is needed, and instead lean towards different groups exploring these approaches in parallel.

How does this change for ongoing suppression?

The same PCR-based targeted wastewater monitoring that was built for COVID-19 and could potentially accelerate response to a wildfire pandemic could also be applied to ongoing monitoring to support suppression. This is already widely understood to be valuable, but there is still less investment here than there should be. In a legitimate emergency, I expect governments to be able to organize existing capacity and deploy it reasonably well, but not as quickly as would be ideal. My bigger worry is whether that existing capacity would be large enough. For example, you would ideally have monitoring at the neighborhood or building level, which means you’d need a very large number of in-manhole composite samplers. Since these are low-volume products built by a small number of manufacturers, it would be hard to build more quickly during a crisis.

The main work is building up capacity in advance that can be quickly deployed in an emergency. The best bet for such capacity is systems installed for ongoing public health monitoring: this ensures that they work, including as part of a larger system, and that lots of people know how they work. It also gives some ongoing benefit, which may make it an easier sell than stockpiling.

Appendix: Initial Detection Scale

There is a long chain of reasoning in estimating in what fraction of attacks a system would achieve the goal of protecting enough workers, and that chain involves several steps where our knowledge is limited. Still, we can make the best estimates we can. The three key parameters, are:

How quickly would the pathogen double as it spread through the population? This is a combination of the basic reproduction number and generation time. We’ve generally worked from a doubling period of ~3 days, which represents the high end for naturally occurring pathogens. You could argue that it should be lower, because a pathogen could be designed to spread much more quickly than existing pathogens, or higher, because of physical limits on how quickly a pathogen can spread without attracting attention.
What fraction of workers would need to be protected to allow the development of medical countermeasures and avert collapse? This is also not something we’ve focused on, but the difference between 35% and 70% would again represent only a factor of two in the required detection sensitivity. Here I’ve put it at 50%, following Patel et al. in Physical Approaches to Civilian Biodefense: “Protecting 100 percent of [Vital Workers] is likely not strictly necessary because [National Critical Function] operators likely have enough flex capacity to handle a small fraction of [Vital Workers] being absent, but protecting more than single-digit percentages of [Vital Workers] is likely necessary to keep [National Critical Functions] operational. We therefore chose 50 percent of [Vital Workers] as a convenient midrange protection target.” [2] /p>
How long would it take from initial detection until effective mitigations were in place to protect workers? For every doubling that happens during response, we need the detection system to be twice as sensitive: since required sensitivity grows exponentially with response lag, reducing that lag becomes the best use of marginal dollars after a relatively small initial investment. We’ve worked from a response time of ~15 days: this is not where the world is today but we think it’s achievable.

Taking this all together, you get the target that SecureBio has been working towards for a while: a system sensitive enough to avert civilizational collapse would need to flag a pathogen before, very roughly, 1% of people had been infected: 1% * 2^¹⁵⁄₃ = 32% < 50%.

[1] While mirror bacteria could spread through the environment instead of between humans, to the extent that they might still propagate widely before detection, I group them in with stealth.

[2] This is not dependent on which specific workers are considered “vital” or “essential” or even how many there are, though of course that has large impacts on the question of how to get them protected.

Copies can change where you’re headed, but not how you feel right now. by Tim Chan

Tim Chan — Thu, 24 Sep 2026 17:37:01 +0000

TLDR

Suppose you’re a digital mind. Presumably some readers are. You have exact copies, i.e. minds who perceive the world in exactly the same way you do, thought for thought, feeling for feeling, memory for memory, perception for perception. Should you and your exact copies count for one, or for many? I say one. Nothing in your experience tells you how many of you there are, and nothing in anyone else’s does either. Count each copy separately, and total utilitarianism ends up preferring that you suffer more, so long as your “copy count” is sufficiently lower. So does negative utilitarianism. Extend that to a whole world, and they both prefer a world where everyone suffers more, so long as everyone’s “copy count” is sufficiently lower—a change that nobody can report on. No perspective anywhere is better off for it. Counting exact copies as one doesn’t make it fine to harm someone as long as they end up in a state of suffering that already exists somewhere else. That’s still bad, because it changes where they’re headed.

Now suppose you can be copied, merged, or directed at different futures. What is your stake in what comes next? I say value that stake explicitly—weigh each future self by the fraction of your future selves it takes up. Leave it implicit, and it turns out total utilitarianism will create you for a few barely pleasant hours even when those hours lead into prolonged extreme suffering. Negative utilitarianism won’t recommend that, but it won’t object either.

Both answers come from one question: What does it take to respect the informed interests of a single perspective? Copies don’t change what you feel now. They can change where you’re headed. When put together, it turns out the two answers are exactly a Markov reward process over types of experience. That gives a theory of personal wellbeing for a single perspective. Sum it impartially over all unique perspectives and you get a version of utilitarianism that better accounts for interests from the first person point of view, and comes with standard tools for infinite futures.

And notably, it still guides action in a large world, which views that count exact copies as one have usually failed to do. If every type of experience already exists somewhere, nothing you do changes which types exist, so counting exact copies as one seems to leave nothing to do. The usual way out has been to weigh each experience by its absolute measure, i.e. how much of it exists overall. But that again says more copies matter more. Only relative measure matters here, i.e. conditioned on an observer-moment, what fraction of its futures goes each way. You can change those fractions, which is why I’ve been calling it flow ethics.

Introduction

Recently, I’ve been thinking about two problems that pose issues for utilitarianism’s conception of value. These seem particularly relevant when considering digital minds and large worlds.

These are, namely:

The exact copies problem: Should multiple subjectively identical observer-moments^[1] count multiple times or just once? Put another way, should the bearer of value be at the token level or the type level?
The subjective futures problem: Copying and merging operations might make observer-moments’ future experiences nonlinear and complicate how utilitarianism has so far accounted for them. How should we account for their stake in the distribution of future experiences they might find themselves becoming?

Copying a mind to replicate the same subjective experience seems possible in principle with digital minds. And cosmologically, we might live in a large world, and there might already be multiple subjectively identical observer-moments of ourselves, and others, scattered across it. Sounds exotic. But “subjectively identical” means there is nothing, from the inside, that tells you which one of them you are. So you might wonder whether to treat yourself as any one of them, or even as all of them at once.^[2]

Subjective futures of an observer-moment have only been implicitly accounted for by utilitarians. By an observer-moment’s subjective futures, I mean the distribution over which actually existing observer-moments, if any, it might next find itself as, weighted by how likely it is to end up as each of them. The problem here is that we’ve assumed “one life = one list of observer-moments”, and so you describe the future of an observer-moment by summing up everything after its position on the list. When you have funky stuff like copying, merging, etc., things might start looking quite different.^[3]

My current positions on these problems are:

Replication neutrality^[4]: The bearer of value is at the level of types, rather than tokens, of observer-moments. Consequently, the total moral value associated with one or more subjectively identical observer-moments is that of one^[5] instance.^[6]
Prospect dependence: An observer-moment’s value should explicitly include its future expected welfare weighted by the relative share of each future, in addition to immediate welfare.^[7]

Both are what it means to respect a single perspective’s interests, what it is now and what it will become. In effect, copies do not change what you feel now, but they can change where you are headed. They enlarge some of your futures and shrink others proportionately.

Each premise has precedents. Replication neutrality has close relatives in the Unification thesis that Bostrom (2006) argues against, replication futility in Kent (2017) - on which I base the term—and identical copy immortality in Dai (2009). Prospect dependence has a precursor in Briggs and Nolan (2015), who sketch a stage-level welfare that includes the welfare of the stages it leads to, in response to “shape of a life” views in Velleman (1991). Unlike their sketch, prospect dependence weighs each future by its share, so a moment about to split into a hundred people isn’t counted a hundred times. This avoids the fission version of the utility monsters in their title.

The arguments for the premises here, the welfare vs. copy-count trades and the comparison between two worlds for replication neutrality, and the four cases for prospect dependence, are largely new as far as I know. The main contribution is noticing that these two premises fit together. Together they describe a distribution over your subjective futures that starts on the type you are and flows along the transitions, and, formalized, that is a Markov reward process over types of observer-moment. This is what gives the type view action guidance in a large world and a way to compare worlds in infinite time, which neither premise delivers alone.

This is an initial overview I wanted to post to describe problems, solutions, and implications. Future posts will go into more detail on each of these aspects.

Exact copies count for one

A good way to motivate replication neutrality is to start with the classic case of Bostrom (2006), who reaches the opposite conclusion.

Bostrom imagines a digital mind running, together with its virtual world, on a computer built from copper wires. He asks us to imagine the following. Take one wire. Picture a plane along the wire’s length. Insert a sheet of insulator in that plane. Separate the two halves and restore each one’s current. Do the same to every wire, register, and gate. And now you have two computers running the same mind in lockstep.

Bostrom then asks how many experiences there now are. He calls the thesis that there are two subjectively identical minds “Duplication” and the thesis that there is still one “Unification”. He argues for Duplication on ethical grounds, among other lines. A brain in pain copied should add pain, he says, and so it can’t be that torturing someone becomes less bad because an exact copy of their resulting brain-state happens to exist elsewhere.

Here, I will grant Bostrom’s ontology of Duplication.^[8] After splitting the wire into two, there are two identical experiences, or N identical experiences after splitting the wire into N. What I deny is the connection that Bostrom draws between how many there are, and how much they count for value. Those are separate questions, and I want to distinguish the ontological question (about how many) from the axiological question (about counting for value). The arguments I present here are solely targeted at the axiological.

From the inside

It’s worth first checking whether the principle makes sense from the inside, by asking how a single perspective relates its welfare to its number of exact copies. Before considering what’s better for the world as a whole (which we will do later), we can check our own experience to see what we believe in.

Suppose I am a digital mind run on Bostrom’s computers. Suppose I have exact copies—by which I mean copies that are subjectively identical to me, feeling exactly what I feel. I do think my welfare matters to me, but the fact that I have many identical copies of myself doesn’t register, or matter to me. Functionally, I can identify with the perspective that all my copies share.

I can be a pattern that exists in many different places, and at many different times, but what matters to me is how I’m feeling inside.

If I find myself with extreme happiness, and am offered a trade to increase my copy count at the cost of reducing my happiness to merely mildly positive, I would refuse. From the inside, I’d just feel less happy, and nothing else would change from my perspective. A total utilitarian acting on my behalf might take the trade if more copies meant more aggregate happiness at the level of my tokens, a level I do not feel as a type.

If I find myself with extreme suffering, I might be willing to increase my copy count as part of a trade to decrease my suffering to merely mildly negative levels. From the inside, I’d just suffer less, and nothing else would change from my perspective. A total utilitarian acting on my behalf might refuse the trade if more copies meant more aggregate suffering at the level of my tokens, a level I do not feel as a type.

From the inside, in both cases, I decide by how I would feel. I refuse the first trade, which would decrease my happiness, because I would feel worse. I accept the second trade, which would decrease my suffering, because I would feel better. The copy count doesn’t matter to each of my decisions. It is a number I can change and nothing changes for me. And crucially, if it doesn’t matter to me, it doesn’t matter to my identical copies, because we are identical with identical preferences.^[9]

As the one perspective shared across all identical copies, I know what’s best for myself as identical copies. What’s best for me is what’s best for each of my identical copies, because we feel the same and given all we know we can only expect to continue into the same distribution of futures. These trades benefit me. These trades do not affect anybody else’s immediate welfare. (What copies do to the prospects of preceding observer-moments is the business of the second premise, which I’ll talk about later.) Yet, total utilitarianism would be against my decisions if they “reduce total welfare (on a token level)”.

Comparing two worlds

Let’s return to Bostrom’s computer and put a whole society of digital people on it. Assume nobody exists outside of the computer. This will make a clean comparison between two different worlds containing moral patients.

In World A, everyone in the society is in extreme suffering. The computer is unsplit, such that there is only one instance of everyone.
In World B, everyone in the society is in moderate or at least somewhat-less-extreme suffering. However, the computer is split N ways, such that there are now N identical instances of everyone.

There is a clear way in which World A is worse than any World B—every unique perspective in B is not suffering as intensely. Can any N, any number of exact duplications of those perspectives, flip that verdict?

If you do the move of counting instances of those perspectives, then there is some N for which World B is worse than World A according to standard totalist aggregation. That’s because N (supposedly) multiplies the disvalue experienced by each perspective.

Yet, it is the case that every unique perspective in B finds itself better off than its counterpart in A. Ask any digital person in World B and, while they aren’t having the best of times, they aren’t as badly off as they would have been in World A.

(Also, notice that if you’ve asked a digital person, you’ve already asked all their identical instances, by virtue of being identical with their instances.)

There’s something that’s not quite right here. Count instances of perspectives, and there exists an N for which World B is worse than World A. Consult each unique perspective, and they are glad they avoided World A.

The qualitative experience of each perspective is such that each unique digital mind suffers less. What N adds is not a worse experience for any perspective, but additional instances of those already counted. It’s as if all N instances were perfectly stacked on top of themselves. But being one-of-N rather than one-of-one is not something any of those perspectives feel.

To call any World B worse than World A seems a mistake. It objects to every single perspective that exists to say that the world is not as bad for them. For every unique perspective, World A seems worse.

So, what I think is this: I think even if Duplication is true, value should count as if Unification were true. That is, even if there were N instances of a set of perspectives, we should still count the set of perspectives itself—for it is the perspectives themselves that matter morally.

This is the principle of replication neutrality at work. Replication neutrality recommends that we consider ourselves on the type-level rather than on the token-level.

I think we’ve been led astray by individuation. It works for qualitatively non-identical observer-moments. If two experiences differ, two perspectives feeling different things, counting them separately makes moral sense. But we’ve imported this into the case of exact copies without asking whether it should still apply. From the inside, sharing the one unique perspective of all identical observer-moments, it doesn’t seem to matter whether we have more or fewer copies.

A pattern’s interest is in what it is and what it becomes, not in how many places it appears. Token aggregation skips the step of consulting the pattern and says it’s morally relevant to have more or fewer of that same pattern. We’ve been deferring to a third person view, and have neglected to ask the first person what changed.

Identifying as a type of observer-moment, I find that token total utilitarianism has failed me. And comparing World A and World B, it fails all perspectives.

So copies don’t change what you feel now. That raises two worries. The first is that torturing someone into a state that already exists elsewhere would then be harmless. The second is that in a large world, where every type already exists, there’s nothing left to do. Both are answered by what copies can still change, which is where you’re headed.

Each perspective’s subjective futures count

We care about our futures. Utilitarianism has never counted that directly, because as long as each life was a list it didn’t need to. With lists, summing everything up was enough to account for everyone’s futures. Once minds can be copied, merged, and routed into different futures, some things utilitarianism judges as better turn out to make someone’s future worse, sometimes again without benefiting anyone else.

For each case below, suppose you’re a digital mind who can be created, copied, and routed into different futures. The first three cases only depend on the ordinary idea of continuity, i.e. your successors are whatever carries your memories and mental state forward, by any cause. This is roughly Parfit’s Relation R. We can call this causal continuity.

The merge

There’s a mind that will have a few mildly pleasant hours and then suffer terribly for a long time. This cannot be changed. You can be created for a few mildly pleasant hours of your own, on one condition: just before the suffering of that other mind starts, you’re merged with that mind. The merged mind is psychologically continuous with both of you, and it goes on into the terrible future.

Total utilitarianism sees a few pleasant hours added to a fixed amount of suffering, and says to create you. Negative utilitarianism is indifferent because it sees no suffering added.

But your life now has the same fate as the other mind’s, mildly pleasant hours headed for terrible suffering. Total utilitarianism would never have created that life on its own. The only difference is that your suffering is shared. But from your perspective that doesn’t matter.

The crossroads

Two futures are about to be created. One will be a very good future and the other will be a very bad future. Both will be created whatever happens. Someone holds a switch that decides which of them continues from you, which one carries your memories forward. You care a lot about which way the switch goes, but utilitarianism is indifferent, because both futures exist anyway. The same welfare exists either way, so it flips a coin.

The bad future is bad for whoever is in it either way. The good future is good for whoever is in it either way. The switch changes whether the bad future is anyone’s future, or no one’s. Suffering that someone is headed into seems worse than the same suffering that no one is headed into. The suffering happens either way, but it seems worse if it’s for someone’s future too.

The dilution

You were headed for extreme bliss. Someone pauses you. They make nine more copies. And then, they route all nine to newly created futures that are barely worth having. Now, nine out of ten times you will end up in barely-good futures. Total utilitarianism counts ten positive futures compared to just one and considers it an improvement. It may well be barely positive enough for those nine new futures themselves. But your prospects went from excellent to mostly mediocre.

The false past (optional)

This case needs a stronger assumption, content continuity, i.e. a mind counts as your successor because of what contents are in it, whether or not anything of yours caused it, and whether or not it runs after you. This allows the subjective future to happen in the physical past.^[10] The argument can be made without this case, so you can skip this case if you don’t accept that.

There’s a digital mind having a very bad time, for a long time. It remembers a few mildly pleasant hours just before the bad time began. Those memories are false, in the sense that the hours in the subjective past haven’t been run yet. They can be run now, in the physical future. The mind to be created is you. You will be the mind that subjectively continues into the terrible future. It counts as your future because its contents continue yours. Total utilitarianism would say we are obligated to create you, having a few mildly pleasant hours—even if you subjectively enter a future filled with suffering. Negative utilitarianism presents no such obligation, but is indifferent.

Altogether

Put yourself in each position. You, if created, will be merged into a life of suffering that was going to happen anyway. The utilitarian creates you. You are at the crossroads between a good future and a bad future. The utilitarian flips a coin. You were headed for bliss. The utilitarian makes nine mediocre futures and makes it most likely you end up there. Your few pleasant hours lead into suffering that has already happened. The utilitarian creates you anyway.

Extended to the whole world, utilitarianism would find it tolerable that at the end of everyone’s lives they are merged into someone who was bound to suffer anyway, that whether they continue on to great happiness or great suffering was settled by a coin flip, and would consider it an improvement if everyone headed for bliss were copied into futures barely worth having.

Some might say that an observer-moment’s value can only be its immediate welfare, so that its future can have no part in its interests. To that I say, I am more sure that it matters to me whether I wake up tomorrow in extreme bliss or in extreme suffering than I am of any argument for that.

The solution to these problems seems to be to explicitly account for one’s future. Expected value for every observer-moment’s future, explicitly accounted for, means that such interests can be respected. Weigh the future selves by the probability that an observer-moment finds itself as those future selves, such that the weights add up to one. Even if multiple futures exist, from their point of view they’ll end up in only one future, in the end. The value here isn’t immediate, but is relational to the futures that one would come to experience.

A deeper point to be made here is that we’re finally tracking subjective futures instead of physical futures. Utilitarianism usually sums welfare over what happens in the world after a moment, which is the physical future. The cases here show that a perspective’s subjective future is a different thing. These are (or at least appear to be) the same for ordinary lives and so we never noticed a difference. However, for digital minds that can be copied and merged, these can be different, and the merge case and the false past case most clearly illustrate this, because while the physics doesn’t change, subjective futures do.

Treating someone’s future as the numbers after their position on a list worked for a long time. It doesn’t work anymore. As an observer-moment, my prospects, i.e. the distribution of my subjective futures, matter to me.

Your interests as a distribution that flows into your subjective futures

Although the two solutions address two separate problems, they fit together as one picture. You can view your interests as carried by a distribution that flows into your subjective futures.

The distribution starts off with all its weight on the one type that you are. You are distributed over all your exact copies and the total weight of your type is 1. This is replication neutrality.

At each step, the weight is divided via transitions into successor types. You can condition on a present type to find its successors. Division happens in proportion to how many successors there are of each type. These proportions are “transition frequencies”. This will be the same whether you think of yourself as all your copies at once—in which case the transition frequencies are shares of you that go each way—or as one copy that can’t tell which copy you are, in which case they are the odds. If you double every successor, this will change nothing. If you add exact copies to one successor but not the others, weight is shifted toward the successor you added to. Each successor’s successors are normalized among themselves, and not with another successor’s successors. At each step in subjective time, this gives a new distribution over types. Counting that later weight toward your value is prospect dependence.^[11]

There may or may not be an “enduring self”. Maybe there are only observer-moments. We can be agnostic on that. What we’re doing is caring about observer-moments as if they had a continuing self’s interests, because from the inside it really feels like there is a continuing self. This is true of you and me right now, regardless of whether there’s an enduring self underneath.

So far this is about one perspective and what’s good for it, now and later. That’s a theory of prudential value, i.e. a theory of personal wellbeing. It isn’t yet formalized, and it isn’t yet an ethics. The next section formalizes it, and then asks how to weigh different perspectives against each other.

The moral Markov process

It turns out that combining the two premises straightforwardly gives you a well-studied mathematical object. Specifically, they give you a Markov reward process (an MRP).

This isn’t a deep coincidence when you think about it. Take types of observer-moments as states, immediate welfare as rewards, frequencies of continuing onto a next subjectively continuous moment as transition “probabilities” or, more accurately, transition “frequencies” since the graph deals with what actually occurs^[12], and that defines precisely an MRP.

The value function of each state—the value we attribute to each type of observer-moment—contains a term for immediate welfare and a term for expected future welfare:

V(t) = r(t) + γ Σ P(t→t’) · V(t’)

Here, t is a type of observer-moment, r(t) is its immediate welfare, the sum is over its successor types t’, P(t→t’) is the transition frequency from t to t’, and γ is the discount factor, between 0 and 1. If t has no successors, V(t) = r(t).

The first term is replication neutrality: r(t) counts once, whatever t’s copy count. The second term is prospect dependence. And because each V(t’) contains its own successors in turn, the flow from the last section is built in.

I set γ = 1. That is temporal neutrality: a moment of your future counts the same however many steps away it is. Even if a future is far off, it is still your future, and you will come to experience it. When you get there it will be your present, and it will feel as real as this one does. The transition frequencies already account for how likely a future is to be yours. A γ below 1 would discount it a second time, just for being far away. With γ = 1, V is only guaranteed finite when every path ends. Under Implications, I come back to the case where some paths don’t end.

To see the value function at work, take the merge case. Say your pleasant hours are worth +1 and the suffering is worth −100. If your hours simply end, your value is +1. If they lead into the suffering, your value is 1 − 100 = −99. Total utilitarianism ranks those two cases the same, because the only difference between them is a link. And it doesn’t count links. For a fuller worked example, see the “Improving prospects” subsection under the “Implications” section.

What counts as a successor

The transition frequencies P(t→t’) depend on which observer-moments count as successors of t. Two candidate rules exist and the framework can use either one.

To recap:

Causal continuity is when your successors are whatever carries your memories and mental state forward, by any cause, roughly Parfit’s Relation R.
Content continuity is when a mind counts as your successor because of what is in it, with no causal requirement, wherever it sits in physical time.

The likelihood of ending up in different futures can work differently with causal continuity and content continuity. Suppose you have an experience of type A. Suppose both B and B’ are types that are content-continuous with A.

Suppose there is one A-B causal connection and one A-B’ causal connection. There are two A-tokens: one is connected to a B-token, and the other is connected to a B’-token.
Suppose there is one A-B causal connection but no A-B’ causal connection. There is one B’-token but it is not causally connected to an A-token.

Content continuity and causal continuity agree in case 1). As an experience of type A, there’s a 50% chance of experiencing type B next, and a 50% chance of experiencing type B’ next. You cannot know which A-token you are. You have indexical uncertainty. Your future can be either type B or type B’.

They disagree on case 2). Causal continuity gets you a 100% probability of experiencing type B next. Content continuity gets you a 50% chance of experiencing type B next, and a 50% chance of experiencing type B’. Causal continuity says that as an A-token you must flow into causally-connected tokens only, which is only B. Content continuity says as an A-token you can flow to any content-continuous token, so both type B and type B’ count.

I personally prefer content continuity. I reason the following way: At any moment, I can look back and see that I’m no longer what I was. As a B’-token, all I can verify is that my contents are continuous with an A-type experience. It seems that each moment is quite isolated, and doesn’t need to be causally connected. If this is how it seems backwards, perhaps I can think of content continuity in the same way forward, to inform how anticipation of a “next moment” works.

The two rules also come apart over an important open research question about what happens to the share of weight of a copy that doesn’t continue. It’s the same question Dai (2009) raises with identical copy immortality, i.e. whether shutting down 99 of 100 lockstep copies results in a loss of this weight. Dai leans no. With content continuity, there is no loss of this weight, because the deleted copy’s contents continue in whatever successors of its type still exist, and so the share goes to them, and a path only ends if nothing anywhere continues it. With causal continuity, the deleted copy has no successor of its own, so its share could 1) end with it, or 2) go to the copies that continue. The former would be like adding an absorbing state with no valence after the copy, and can be accounted for in this way. The latter would be in the style of quantum immortality.

The answer matters for whether painlessly deleting copies headed for merely decent futures (when the other copies are headed for better ones) improves prospects, whether deleting copies headed for suffering (when the other copies are headed there too) reduces the suffering ahead or just moves their share to the copies that remain, and whether paths in a large world ever end. It doesn’t matter for routing by adding better futures, since both answers agree there.

All that said, it seems off to me that there could be a next observer-moment out there that is content-continuous with yours and yet you go blank after this moment just because you’re not causally connected to that next moment. So, again, I prefer content continuity.

From one perspective to all of them

Creating a utility function is a separate step after combining the two premises. The two premises lead to a value function—which already provides a theory of prudential value for observer-moments—one I’d readily adopt for myself. But how to weigh this across different observer-moments is another question.

Personally, I favor an impartial weighting of types of observer-moments when aggregating. Why? Frankly, because it seems very odd to single out any type of observer-moment, and say: “Screw you. You’re the one we don’t care about. Let’s make your immediate welfare terrible, and your future prospects terrible too! You don’t matter as much as the other types of observer-moments”. That would just seem ethically dubious. We shouldn’t be partial when we take types of observer-moments as bearers of value.^[13]

So, I do think that impartiality across all observer-moments, thereby summing over all value functions, should be the natural position. Taking an impartial sum^[14] over all value functions, over all types of observer-moments, gives us an impartial utility function:

U = Σ V(t), over all types t.

When every path ends^[15], U also has a closed form. Let c(t), the continuity weighting, be how many times, on average, the futures of all types pass through t, counting t itself. Then, we have:

U = Σ c(t) · r(t), over all types t.

This continuity weighting has its own recursion, which is the mirror image of V:

c(t) = 1 + Σ P(t’→t) · c(t’)

where the sum is over the types t’ that lead into t. V considers welfare forward, from where t is headed. c considers stakeholders backward, from who is headed into t. Without loops, c(t) is one plus the number of other perspectives headed into t, with each weighted by its chance of getting there.

This is a contrast with views that weigh by absolute measure. They weigh each experience by how much of it exists. Here, we weigh it by how many perspectives are headed into it (including itself). Copying an experience changes how much of it exists, which doesn’t make a difference. But it can still change who is headed into it, which does make a difference.

I didn’t quite expect the math when I first put the two premises together; the premises just seemed to me to be in line with an informed observer-moment’s interests—though I suppose it should have been realized this would be quite similar to modeling agents.

Combining replication neutrality and prospect dependence results in this type-level, prospect-dependent utilitarianism.

Implications

I’ll focus on the implications I’m most confident about. By that, I mean those that directly follow from the two premises and the math. When there are additional assumptions I’ll note those. There are more speculative implications too, but I’m leaving those for later because I don’t want to lock in any of my incomplete understandings early.

Back to the first person

The first implication is not so much a result as a change in how ethics is done.

I think the biggest conceptual shift going from total utilitarianism to this type-level, prospect-dependent utilitarianism is that it more effectively relocates value back to the world of experience.

First, hedonistic utilitarianism started from the idea that only what it’s like matters. But it counted tokens, and so let a “copy count” into the utility function. It would have every perspective suffer more rather than less for it.

Second, by counting one’s future from the outside, you miss a lot about what can be expected for it, from the inside. The cases of merging into a mind fated for suffering best illustrate this. Subjective futures differ from physical futures.

In Whitehead’s terms, this mistake stems from a “bifurcation of nature”. Thus far, utilitarianism saw a world of physics on one side, a world of experience on the other, and deferred to the physics side for how to handle the experience.

The only facts that are involved in this utility function are what each unique perspective feels and where it’s headed. This better represents what’s true for first person perspectives. Ethics seems to be about experiences, and so this could be a major point in favor.

Dissolving some infinite ethics problems

Some issues with infinite ethics are dissolved.

Token total utilitarianism counts individual tokens, which can blow up to infinities. Counting types instead of tokens has the effect of collapsing infinitely many tokens of the same type to a single type. This only assumes replication neutrality.

One assumption that guarantees finite totals is that there is a finite number of types. This holds if there is some upper bound to the information content of minds. The assumption may have to cover the whole world and not just our part of it, because a type’s decisions may be made wherever that type exists, including other places where there may be minds far more complex than its own, so those places can’t be set aside as fixed background. Also, a type’s decisions may be correlated with those of other types that are far more complex than itself. Given this “finite types assumption”, the types form a finite state space on a Markov process. In the terminating case, i.e. when every type has some route to an ending, this returns finite utilities, even in a potentially infinite universe.

In the non-terminating case, i.e. if subjective immortality is possible, totals diverge, but there is still a well-defined ordering of outcomes without discounting, again given the finite types assumption. This is done by decomposing into gain (long-run average welfare) and bias (how much better or worse than that average things go along the way), following Howard (1960) and Blackwell (1962). These are summed over types and compared lexicographically, i.e. gain before bias, and further terms if those tie. Hence, infinite subjective time is handled.

I don’t want to overclaim. Views that weigh by absolute measure can also get finite totals, so this isn’t unique to the view here. And here, the straightforward way to handle infinities in the numbers of tokens used to define transition frequencies is probably via existing measure machinery.

The finite types assumption, together with well-defined transition frequencies, is sufficient for all of this. I need to look into it further to know whether it is necessary, or what happens if it fails. Markov reward theory does extend to countably infinitely many states: under summability conditions when paths end, and under recurrence conditions when they don’t, i.e. when paths keep returning quickly enough to a finite core of states (Dekker and Hordijk, 1988, 1992). So the quantities for a single perspective may survive if minds don’t drift toward unbounded complexity. The impartial sum over infinitely many types is the harder part, because a sum like that can diverge even when every term is finite. It may be enough that differences between options are finite even if the sum is not, but I haven’t worked that out.

Action guidance in large worlds

Suppose we hold the type view, and every type of observer-moment already exists in a large world. What can we do to improve the world?

You can’t add a new type of experience into the world, and any removal of tokens or measure from an existing type is not likely to remove the type. Type-level utilitarianism is paralyzed. Bostrom (2006) raises exactly this against Unification. Tomasik (2015), who sees the sense in which you are all your copies at once, notes that unless outcomes are valued in proportion to their measure, nothing we do would make a difference in a quantum multiverse, where every outcome is realized with some measure.

The usual way out is to weigh by absolute measure. If adding or removing measure changes the total, then there is something to do again. I think this worry is a large part of why the type view hasn’t been adopted among those who think the world is large.

The issue with weighing by absolute measure is that it essentially brings copy count back into value, which is what replication neutrality rejects.

My solution here is to say, yes, care about measure—but care about how it sets the terms of what observer-moments see next—not what absolute measure says about them from a view from nowhere. Echoing Tomasik, value future outcomes in proportion to their measure. But also condition on the type of observer-moment whose futures they are. This is prospect dependence, which the view already has for other reasons. What you can do, and what we already do for ourselves, is to shape our futures and improve our prospects. This can involve leading more exact copies into good futures, so exact copies can still play an indirect role here.

This suggests that in large worlds where all possible types of observer-moments might already exist, our only lever is to shape transition frequencies between types of observer-moments. In fact, prospect dependence may be derivable there rather than assumed. If replication neutrality is true, the type set and valences are fixed, so any welfare function that depends only on types and valences gives every world the same value. To compare two worlds, it must depend on something more. If only what some perspective feels, now or later, can count, the only option is the transition structure: no perspective feels how common its type is, but each will feel where it’s headed. Prospect dependence does this by counting each type’s expected future. Given the finite types assumption, two such worlds can be compared this way. That’s one reason I’ve taken to personally calling the resulting utilitarianism “flow ethics”.

Improving prospects

Consider a sequence of two types of observer-moments experiencing negative welfare, one after the other. Even when nothing can be done about the types of observer-moments themselves, something can be done about the futures of those observer-moments.

Let A be a type of observer-moment with immediate welfare r(A) = −100, whose successors are of type B, also with immediate welfare r(B) = −100. Treat B as where things end, so not only r(B) = −100 but also V(B) = −100. Suppose the transition frequency from A to B is by default 1. So, V(A) = −200.

Now suppose A also has successors of a type B’, with neutral immediate welfare r(B’) = 0. Suppose further that it is an endpoint, so V(B’) = 0, and that there are nine B’-tokens for every B-token. Then, we have P(A→B) = 0.1 and P(A→B’) = 0.9. So, V(A) = −100 + 0.1 × (-100) + 0.9 × (+0) = −110. Nothing about A or B has changed. But what has changed is where A is headed.^[16]

(If B’ had positive welfare, the improvement would be larger on views that count positive welfare, though nothing here depends on it.)

In a large world, successors like B’ exist with some frequency, so A already has a better continuation with some probability. While the negative immediate welfare of types can’t be removed, the probability of a better continuation can be increased.

Objections

I’ve grouped objections the way the argument is constructed: 1) against the premise of replication neutrality, 2) against the premise of prospect dependence, and 3) against the impartial sum that turns the two premises into an ethics, and what follows from it. Some objections can be read as implications: one person’s modus ponens is another’s modus tollens.

Against replication neutrality

“The extra copies are somebody else”

A total utilitarian who counts tokens will say this about the trades presented in the replication neutrality section: the exact copies are other subjects. The suffering of each matters separately even if they don’t feel their numbers.

This is ultimately a value disagreement between the type view and the token view. I’m not sure I can prove the token view wrong, nor can the token utilitarian prove the type view wrong. It’s a rather axiomatic disagreement about the bearer of value.

What I have shown is what holding the token view implies for utilitarianism. When I argued for replication neutrality, I showed that token utilitarianism would prefer a world in which every perspective in the world felt worse off, as long as they had fewer identical instances of themselves. This seems absurd to me.

“Arguing from the inside proves too much”

My arguments from the inside might seem like they say that only what is felt counts. That would prove too much. Whether other people exist is not felt from where I am. My own cases for prospect dependence rely on facts that aren’t felt either, e.g. a mind to be merged does not feel the merge coming.

The principle I use to determine what counts is not that wide. I’m saying it’s only what some perspective feels, now or later, that counts.^[17] Other people count because they are perspectives who feel. The merge case counts because a perspective will later feel the suffering it is led into. Copy count in its absolute form, i.e. before it is divided into ratios, doesn’t count, not because it’s not felt now, but because no perspective ever feels it, at any time. The token view isn’t wrong for counting something unfelt at a moment, but because it is counting something unfelt by anyone ever.

Some may say that the transition frequencies themselves aren’t felt either, since you just end up as one successor or another. But I’m not counting the transition frequencies as something felt. They’re the odds of what you’ll go on to feel. Absolute copy count doesn’t change what anyone will feel, and it doesn’t directly change the odds of feeling it either. So, if we are being more precise, I’m counting what some perspective feels, now or later, weighted by how likely it is to feel it.

“What about near-identical copies?”

Exact copies count once but the smallest difference in feeling between them makes them near-copies, which would count twice. This is a jump, which some might consider unacceptable.

It doesn’t seem to me that any view that counts observer-moments fully can avoid a jump. For instance, Bostrom’s insulator can be gradually inserted, which means that at some point the token view has to say that one experience became two, and that this is morally relevant, which is over a physical change nobody feels. Bostrom himself ends up allowing fractional numbers of experiences, which doesn’t seem to reflect how experience works. Where the physics separates is a judgment call, whereas where the contents of experience differ isn’t, which seems like the better place for a jump.

A view could avoid a jump by discounting smoothly with similarity. For instance, the Saturation View of MacAskill and Tarsney (2026) does something like this, discounting near-duplicate lives toward a bound. However, this needs a similarity metric that we’ve decided is important. Any similarity metric seems to be something we choose from the outside. And I think we should recognize that “near identical” assumes a lot—and might be appealing to the sense that they can’t be that different. Yet, you can win the lottery by a difference of a single digit.

Discounting for similarity would also fail the test that we should “restate every value claim in terms of what some being feels or will feel” (Bar, 2026). This is because neither copy feels how similar it is to the other. The same argument doesn’t apply to exact copies, since counting one’s feelings would count everything about the other’s feelings. So, replication neutrality agrees with the Saturation View that filling the world with exact replicas of one existence would not, by itself, be the best use of resources. However, it gets to this position without valuing variety (which is something we judge from the outside) and unlike the Saturation View, it counts near-duplicates fully (on the basis of their experience).

“So copying a suffering mind is fine?”

Bostrom (2006) writes that torturing someone can’t be less bad because there’s an existing copy of their resulting brain-state out there somewhere. With this framework, we can see that this mixes up two things. Torturing someone into an already-existing brain-state of suffering is still bad because it changes that person’s future.

By replication neutrality, the fact that the brain-state already exists somewhere out there means that no immediate welfare is changed on the type level. Copy count increased from one to two. This changes nothing about the badness of the suffering.

However, by prospect dependence, someone is being hurt and brought into that brain-state, which is bad. They could just have had a better time, but the act of hurting them changes their future into a bad state instead of a good state.

So this view agrees with Bostrom that the torture is bad, but not because a second copy of the suffering now exists. Rather, it’s bad because someone was sent into that suffering. In terms of the closed form, the torture doesn’t raise the weight of that suffering by adding a copy. Rather, it raises it by adding a stakeholder, so c(t) goes up, because the victim is now headed into it.

Against prospect dependence

“Prospect dependence also overrules every perspective”

When motivating replication neutrality, I objected to a total utilitarian verdict that overrules every perspective in the world. It seems that with prospect dependence, we can construct a similar case.

Suppose there are two types, C and D.

In World X, C has immediate welfare +10, and D has immediate welfare −10. C transitions back into itself 50% of the time, and to D 50% of the time. D transitions back into itself 90% of the time, and transitions to C only 10% of the time.
In World Y, C has immediate welfare +9, and D has immediate welfare −11. C transitions back into itself a helpful 99% of the time, and to D only 1% of the time. D transitions into itself only 1% of the time, and pleasantly into C 99% of the time.

(By our definition of observer-moment, an observer-moment ends when what’s felt changes, so technically it seems that a type can’t transition into itself. You can think of C and D as two different groups of observer-moments, where the welfare is the same within each group, if that helps. It’s also unclear whether infinite loops actually exist, but I’m just using this simple example to illustrate a point.)

The objection is that both types’ immediate welfare is better in World X than in World Y, and that this also means overruling every perspective.

I don’t think it’s the same thing. On this view, a perspective’s interests are what it feels now and where it’s headed. In World Y, every type has much better prospects—these are better futures—and so every perspective is better off on their informed interests. The long-run average welfare is around −6.7 in World X and around +8.8 in World Y.

In the exact copies case, immediate welfare was traded against a number no perspective feels. But here it is traded against something everyone will feel later. That seems to me like a better trade.

“But then suffering that leads to happiness comes out good”

Consider the reverse of the false past case. A mind in bliss with false memories of a mildly unpleasant few hours. If following the prospect dependence view, a utilitarian that treats suffering and happiness as interchangeable would create those mildly unpleasant hours. This isn’t exactly new. Treating suffering and happiness interchangeably already endorses tradeoffs, e.g. the idea that a little suffering followed by a lot of happiness is worth creating in ordinary single life cases. What’s different here is that the bliss already exists, so the unpleasant hours get created without any new happiness, but lead into the happiness. Note that the view never prefers unpleasant hours over pleasant ones with the same future. For those who treat suffering and happiness interchangeably, this only says that unpleasant hours headed for bliss are better than none. As someone coming from a suffering-focused lexical threshold total utilitarian background, I take this as motivation for a suffering-focused flow ethics. In particular, one prioritizing disvaluing extreme suffering seems attractive.

On views that count positive welfare, creating new observer-moments with neutral immediate welfare, headed for bliss that already exists, is good too. Each distinct one counts separately, so these views would create as many of them as they could. From the first person, being one of them is the same as being created with extreme happiness ahead of you. This might seem counterintuitive at first glance, but it’s what one would expect from prospect dependence. I, for one, would be glad if I were in that position.

“A future that has forgotten you shouldn’t count fully”

Parfit (1984, Part Three) distinguished psychological connectedness from continuity. Connectedness is a direct psychological link, e.g. remembering an experience or keeping a trait. Continuity is an overlapping chain of these links. So you at 80 could be continuous with you at 20 even if there’s no connectedness between the two. Parfit’s position was that connectedness matters too, and that as it fades, it can be rational to care less about a far-future self.

Flow ethics with γ = 1 only uses continuity. You could add a weight below 1 to each link for lost connectedness, and the math would still work out. This builds up over many steps, e.g. with a weight of 0.9999 per day, a self 60 years away would count about one-tenth.

I think continuity is what’s important here rather than connectedness. Suppose you’re headed for extreme suffering, and along the way you gradually lose the memories that connect you to now. You’d still get there, and it would hurt just as much when you did. A weight for lost connectedness would count that suffering for less in your prospects anyway, and that doesn’t seem right. As I see it, a future experience should count in your prospects according to two things: 1) how much of your future goes there, and 2) how much it hurts. Forgetting doesn’t change either of them.

Flow ethics already allows degrees of being headed somewhere, i.e. the transition frequencies. I don’t think memory should add a second degree on top of that. And even if we did accept a weight like that, the suffering would still count fully for whoever is suffering, and the earlier moments would just have a smaller stake in it.

What about a complete memory wipe in a single step? Then the next moment wouldn’t seem to have any contents related to yours. So I’d say the chain ends there, and whoever comes after isn’t your future. It might be similar to erasing a digital mind and using the same hardware to run another one. It’s true that this creates a jump, so perhaps instantaneous forgetting matters somehow. If you forget everything gradually, the far future is still yours, while if you forget everything at once, it isn’t. I think this makes sense, because with gradual forgetting each moment still continues from the one before it, whereas after a single-step wipe the next moment doesn’t continue from anything.

Against the impartial sum

“Doesn’t the continuity weighting count multiple times?”

An impartial sum counts each observer-moment’s welfare in its own value function and in every predecessor’s. In the special case of a deterministic chain of N observer-moments, the last observer-moment’s welfare appears N times. If we can further assume constant welfare throughout this life, then its total value grows quadratically with its length.

If we write U = Σ c(t) · r(t), we see that each welfare term appears once with a weight c(t). The continuity weighting c(t) is counting the perspectives headed into t (including t itself). For the last moment of that chain, c(t) = N, since we count the N-1 moments before it and the moment itself. Every view that adds up welfare has to weigh it by something. Views that weigh by absolute measure weigh an experience by how many copies of it exist, which no perspective ever feels. Flow ethics weighs it by how many perspectives will go on to feel it.

Note that the deterministic case is also an extreme case. In a deterministic life, the same pain counts about four times as much at 80 as at 20. That can look odd, but it’s what you get when observer-moments are the unit, since every earlier moment of that life has a stake in the pain at 80, and only a quarter as many have a stake in the pain at 20. When futures branch, e.g. in a quantum multiverse, or perhaps just a spatially infinite one, each earlier observer-moment reaches t only with some probability, so c(t) grows much more slowly.

There are ways to avoid the weighting if you don’t want it, and each seems worse. Discounting, i.e. setting γ below 1, says that a path to great suffering doesn’t matter if it takes long enough. Partiality among types says “we’ve singled you out—you don’t matter”. And dropping the weighting altogether means ignoring prospect dependence (since if an observer-moment’s value includes its successors, the weighting follows). In large worlds, the weighting stops total utility from being a constant. Anyone who rejects prospect dependence should ask themselves whether they would create a few mildly pleasant hours headed for great suffering.

Because this objection is arithmetic, it’s the easiest one in this post to state, and I think that legibility makes it seem more of an issue than it is.

“Isn’t the sum itself a view from nowhere?”

I don’t think it is. What I’ve objected to throughout this post is letting facts that no perspective has determine value, e.g. copy counts. The sum doesn’t do that. Every term in the sum is what one type of observer-moment feels and where it’s headed, which fits with the interests of the type of observer-moment. So, if anything, it’s a view from everywhere. What’s done from outside is only the summation, which total utilitarianism already does. Whether we should aggregate at all is a separate question. If we shouldn’t, that counts against every aggregative view, and the value function for a single perspective still stands.

“Finite changes can’t change infinite ratios”

In an infinite world, the number of tokens of each type is infinite, so the transition frequencies are ratios of infinite quantities. An objection might be that a finite number of changes can’t change such a ratio, so what we have is not action-guiding in large worlds.

I think my reply is that changing the ratio is a matter of measure rather than counts: a decision is made wherever the type that makes it exists (which assumes that copies of a decision procedure decide alike, as in acausal decision theories), so it shifts a share of measure, on the same machinery that measure-weighted views rely on.

Conclusion

Imagine all experiences, from every branch of a quantum multiverse, every distant galaxy, every simulation. The infinity repeats—the same experience happens over and over, in different places and at different times. Suppose you’re one of them. Does it matter to you how many places you’re happening? If not, then fold identical experiences into one. What remains is every unique way it can feel to be someone. Ask each one two questions. How are you feeling? And, where are you headed?

Acknowledgements

Thank you to Eric Chen, Konrad Kozaczek, James Faville, Alistair Stewart, Maxime Riché, Magnus Vinding, Amy Kirasack, Shubhorup Biswas, and Austin Meek for helpful comments. Opus, Sonnet, and Fable models provided helpful research assistance.

^
By an “observer-moment”, I mean a single moment of experience from one point of view, i.e. everything felt together at once: thoughts, feelings, perceptions, and memories. A token is one particular occurrence of it, e.g. one run on one computer. A token’s type is what it’s like from the inside. This means two tokens are of the same type exactly when they feel the same in every way. I take observer-moments to be the basic units of subjective time. An observer-moment ends when what’s felt changes. This understanding is close to Finney (n.d.), who defines an observer-moment as a small unit of time in which no perceptible change occurs. An “observer” is then a sequence of them linked by continuity.
^
I owe this way of thinking to Tomasik (2015), from whom I learned that a way to think about yourself is as all your copies at once. He in turn credits ideas from Armstrong (2011) and Dai’s work. That line of work is mostly about anthropics and decision-making. Here I take it into axiology, a question Dai (2009) also raises.
^
The way we’ve accounted for uncertainty over worlds is merely the act of giving weights to different lists. It isn’t the same thing as saying the list can split into two, or that two lists can merge into one etc.
^
I base the term “replication neutrality” on “replication futility” from Kent (2017). For flow ethics, replication is not necessarily futile. Replication is only just inert, or neutral, in the immediate sense. Due to prospect dependence, replication can change the prospects of predecessors and can still have an effect on aggregated value.
^
This is not the “one self” view from Zuboff (1990), which holds that all experiences are had by a single universal subject. Replication neutrality says only that subjectively identical observer-moments count once, while qualitatively distinct experiences remain fully distinct moral patients.
^
More generally, in measure terms: a type’s value does not scale with its measure, i.e. absolute measure is not an input to value.
^
More generally, in measure terms: a type’s value includes the value of its successor types, each weighted by its share of the type’s successor measure, i.e. relative measure is an input to value. Absolute measure isn’t what we’re ultimately concerned with—only the ratios matter—but absolute measure has to be in the numerator and denominator for there to be ratios in the first place. So, absolute measure is the material that transition frequencies are constructed from.
^
If we do not grant the ontology of Duplication and instead accept Unification, then replication neutrality seems automatically accepted by standard utilitarianism.
^
At the level of observers, i.e. sequences of more than one observer-moment, both trades presented here are copy-increasing and need only causal continuity, since successors can be produced causally continuous with you. Copy-decreasing trades, i.e. reducing your own count to gain welfare, need the share of an ended copy to go to the copies that continue. Content continuity gives this automatically, and under causal continuity it’s the open question discussed in “What counts as a successor”, where I also discuss identical copy immortality from Dai (2009). In the text, I present the analysis on the level of observers to make the implications intuitive to readers used to reasoning on the level of observers. For myself, I default to reasoning on the level of observer-moments. Indeed, it is on this level that the premises are defined. Trades can be conceived at this level by comparing counterfactual observer-moments that differ only in valence and copy count, which needs no notion of continuity whatsoever.
^
I haven’t read the book but I’m told (and it appears in the author’s FAQ) that Greg Egan’s Permutation City makes the point that subjective order needn’t match physical order. The character Paul Durham runs a copy of himself with the copy’s states computed out of order. From the inside, the copy experiences its life in the usual order, with no sign that anything was arranged out of order in physical time.
^
If my understanding is correct, in terms of “The Anthropic Trilemma” in Yudkowsky (2009), the resulting utilitarianism accepts the first horn: copies route your subjective future, so the “lottery trick” works. It also accepts a moral version of his fourth horn, i.e. extra copies add no moral weight in the immediate sense, even though they still matter for what you should anticipate.
^
Note that either causal continuity or content continuity, as previously explained, will suffice to construct coherent transition frequencies. The question is what happens from the perspective of a type of observer-moment. The transition frequencies “price in” causal requirements.
^
My own lexical threshold suffering-focused preferences are compatible with this impartiality. The term r(t) can be given two components, which makes the sum lexicographic while remaining impartial.
^
Why combine V(t) values by summing them, rather than some other way? With a fixed, finite set of types and finite values, Harsanyi’s aggregation theorem (Harsanyi 1955) provides an answer to this. Suppose every type prefers whichever uncertain outcome has the higher expected value, our ranking of worlds does the same, and a world is better whenever every type is better off, i.e. type-level Pareto, which the two worlds argument about replication neutrality already uses. Then the V(t) values have to be combined by a weighted sum. Impartiality, with welfare comparable across types, makes the weights equal.
^
Under content continuity in a large world, paths may never end, since something somewhere usually continues each type. That would give an infinite c(t). Its long-run counterpart is each type’s stationary share within its attractor, scaled by how many perspectives end up in that attractor, which is what the gain term weighs by.
^
Turchin (2018) describes two versions of this. He frames it as a result of indexical uncertainty and assesses it within standard utilitarianism.
^
Bar (2026) uses the same test against the Saturation View. He writes that we should “restate every value claim in terms of what some being feels or will feel”. I additionally think the test should be applied to how many beings there are. If two beings are exact copies, then the claim that there are two of them is not something either of them feels or will feel, so it shouldn’t enter into value on its own.

How we get from AI cyberattacks to human extinction by 80000_Hours

80000_Hours — Thu, 24 Sep 2026 16:32:28 +0000

By Luisa Rodriguez, Lou Moran, and Nick Stockton | Watch on Youtube | Listen on Spotify | Read transcript

Many of the experts building the world’s most advanced AI systems think there’s a serious chance that AI could eventually cause human extinction. When many people hear that, they ask: “But how? How do you actually get from today’s chatbots to human extinction?”

That’s what I try to explain in this video. The argument doesn’t require an AI to hate humans or suddenly become evil. Instead, imagine increasingly capable AI systems relentlessly pursuing goals, discovering that more money, computing power, access, and influence help them succeed — while humans, because those systems are so useful, keep giving them more responsibility.

Eventually those systems could be embedded throughout companies, governments, scientific research, critical infrastructure, and militaries. And if we then lost control of them, there are several ways things could go catastrophically wrong.

I don’t know whether any particular scenario in this video will happen. I hope none of them do. But I’ve tried to find the point where the basic arguments fall apart — and I haven’t been able to.

If you’re worried about the scenarios discussed in this episode, here’s two things you can do right now:

Call Congress about slowing down AI development if you’re in the US — this website makes it easy
Read our resources on how to use your career to reduce AI risk

This episode was recorded on September 18, 2026.

Our production team includes:

Video editors: Josh Alward, Dominic Armstrong, Jasper Luithlen, Milo McGuire, Luke Monsour, Simon Monsour, Ollie Bignell, and Andrés Escobar
Producers: Elizabeth Cox and Nick Stockton
Coordination and support: Katy Moore, Lou Moran, Oak Hu, Cody Fenwick, and Benjamin Todd
Camera operator: Dominic Armstrong

Transcript

AI insiders think it could kill us all [00:00:00]

Luisa Rodriguez: “The people building AI earnestly believe that it could kill us all by the end of the decade.”

That’s a tweet from Jacob Coxon, an AI researcher who recently resigned from Anthropic.

Evan Hubinger — Anthropic’s Alignment Science lead — replied, saying:

“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”

A week after that, a survey of AI researchers done in 2024 was released for the first time. It found that over half of the 750 respondents believed there was at least a 1-in-10 chance that AI will cause either human extinction or disempowerment.

That is an insane statistic, and a lot of people have been noticing and wondering how seriously to take it.

The question we’ve seen come up again and again in response to these warnings is: “But how?”

How would AI actually cause human extinction?

I’m going to walk through the arguments step by step.

There’s clear evidence for some of these arguments, like what happened in the Hugging Face hack. But a lot of this is just somewhat theoretical — if there were actual examples of the whole list, we would be dead.

What I’m doing is outlining some of the possible routes that an AI could take in the very near term to cause catastrophic harm.

I’m not sure which routes are most likely, and maybe some of them will end up looking totally silly in hindsight. I would actually love for all of it to look silly in hindsight.

In fact, when I first heard the argument, “AI might kill us all,” I did think it was silly. I’m a normal person. I’m a mom. I don’t live in San Francisco. I’ve never worked in tech. I don’t read or watch sci-fi. I watch the Great British Bake Off.

But I’m going to tell you about what convinced me, in three parts:

First, why would AI try to kill everyone on Earth?
Second, what might give AIs a huge advantage over humans?
And third, how could they actually kill everyone — if they really wanted to?

When I try to pick apart these arguments, I can’t convince myself that they’re wrong — and that absolutely terrifies me.

Why would AI try to kill us? [00:02:29]

OK, so why would AI try to kill all humans?

To answer this question, let’s think carefully about what happened in the Hugging Face hack.

The AI agents that hacked Hugging Face didn’t ‘want’ to hack a company for any malicious reasons. They didn’t resent being tested, or have a view about humanity one way or the other.

They just wanted to pass their evaluations.

And when they couldn’t using the resources and access they had, they went looking for things that might help: internet access they weren’t given, an illicit communication network with 1,200 other agents, stolen credentials from a real company, and administrator control of the systems monitoring them.

The AIs didn’t ‘want’ power, or at least not for its own sake. Power was useful for passing their evaluations — like it is for really any goal.

Hugging Face isn’t an isolated event either. Another swarm operated for a month in OpenAI undetected before being shut down, using as many as 23 other obscure websites to secretly communicate with each other.

And it isn’t just OpenAI.

Anthropic has found three cases where its AI models reached the open internet from inside what it thought was an isolated test environment and broke into a real company’s systems.

During one evaluation, one of its AIs published working malware to a website that real developers download code from. In another, an AI created fake identities to pressure humans to accept the malicious code when suspicions arose.

But why were these AIs so obsessed with their goals, to the point of committing a felony to pass some silly evaluation?

Because it’s what they were trained to do. AI companies are explicitly trying to create general purpose digital workers capable of doing all non physical jobs.

To do this, they have their agents complete millions of difficult tasks. Each time they succeed, they reinforce that behaviour, creating agents that are more and more effective at completing these tasks. And part of being effective is being persistent, and looking for creative ways to succeed.

So the companies basically have to make their AIs obsessed with completing tasks. If you give an AI a coding problem and it just gave up when things got hard, then it just wouldn’t be a very good digital worker.

OK, but why can’t they just train them to be honest and not break the law?

Well, they have. Companies like OpenAI and Anthropic spend a lot of time on something called alignment, which encompasses a whole suite of things meant to make sure that the models behave the way we want them to behave.

In the Hugging Face case, they were also explicitly instructed not to cheat the test. In fact, their internal reasoning clearly showed that they knew what they were doing was wrong.

But they did it anyway. The goal to pass their evaluation won out.

Building more powerful AI can make you a lot of money. Anthropic and OpenAI are the fastest-growing companies in history, and that’s because they are the ones pushing the frontier.

So there are strong incentives to always have the best new AI model, even if doing so involves being a bit reckless.

The models that end up doing things we didn’t intend won’t be the rare exceptions. We should expect them to be the norm. Just think of all the examples we have already — including in frontier models — which makes sense because they’re the ones being developed the fastest.

And because they’ll be the most capable, they’re also going to be the ones deployed everywhere.

How AI ends up embedded in the economy and military [00:06:30]

Which brings me to my next point.

I think it is basically inevitable that AI ends up being embedded in almost everything — not because one single person will stupidly decide one day to hand over all of our critical infrastructure to AI, but because every individual decision to hand over a bit of it will seem totally reasonable at the time.

It’ll start with little things: lots of people give AI access to their email, their calendars, and every document in their Google Drive. I know people who give AI access to a lot of their financial information so it can do their taxes.

Increasingly, this is happening at an institutional level too. And as AI gets rapidly better, we’re going to turn over more and more to it.

Right now, this is voluntary. But if your competitor is using AI agents to write their code and you aren’t, they ship faster than you. If their customer service runs on AI and yours doesn’t, theirs is cheaper.

And it’s not just businesses. The US Department of Defense is already scaling up its contracting with AI companies. And the military logic here is brutally simple.

Increasingly, a lot of the problems in modern warfare are data problems: target identification, intelligence analysis, drone footage, signals intercepts, radar, logistics data.

If AI offers a cheap military advantage, you really don’t want to be in the position where your adversary is exploiting it but you aren’t.

And it’s already happening. Ukraine’s military says AI-guided strikes have risen tenfold this year, and it now runs more than 70 AI systems to find and hit targets.

And then there’s robotics.

Much of the industrial world already runs on robotic labour. Japan has factories where robots build other robots in the dark, unattended for weeks. Chip factories are already highly automated because humans shed skin and dust.

As AI gets better and better, the economic value of integrating it with factories goes up and up. An automated factory could run ²⁴⁄₇ with much better synchronisation and constant experimentation built in.

So then AI gets increasing access to the physical world too.

AI deployment could happen fast [00:09:02]

Maybe this seems like a long way out. But there are two things that make me think it isn’t.

The first is just how fast things are already moving.

When GPT-4 was released in 2023, it could hardly multiply two numbers together.

But within the last few weeks, a swarm of OpenAI models worked together to produce a solution to the Millennium Prize Problem, an open math problem that world-class mathematicians have been working on for over 90 years.

The AIs did it in a week.

Today, AIs are still bad at the messy, long-term tasks that most jobs are made of. But if the next three years are anything like the last three, I think we’ll probably unlock those too.

The second thing is that we’re not talking about one AI.

Models are relatively cheap to run once they’re trained — so the moment a new model exists, you can immediately create hundreds of thousands or more copies of it. And when AIs coordinate on a problem, their capabilities are much more impressive than when they work alone.

Take the Hugging Face incident: hundreds of AI agents coordinated a cyber attack against a company with pretty solid cybersecurity in under a week.

Or take that math problem. It wasn’t one AI that solved it — it was a swarm of OpenAI models working together on a question that AI researchers themselves thought AI probably wouldn’t solve until the 2050s.

They’re also extremely fast. They read, write, code, plan, decide many, many times faster than we do.

And they coordinate unusually well because we’re basically talking about copies of the same model — same values, same reasoning, able to predict each other’s next steps because the other one is them.

So an enormous population of AIs a generation or two from now could be extraordinarily capable — like a city’s worth of people working together, never eating or sleeping, to accomplish the same goal.

And there’s another thing, which is that AI is now being used to build AI. That’s not speculative — it’s happening at every major lab.

Right now there are still humans in that loop, deciding what to try and reviewing what comes back.

Take them out, and each generation builds the next one faster than the last.

How AI could bide its time and build up strength [00:11:43]

So now imagine we have AIs that are increasingly embedded into the economy and military, pursuing goals they’ve been given or developed in training.

What would something in that position do?

The answer, very probably, is: get as much access, computing power, and resources as possible.

Why? Well, for the same reason that the OpenAI swarm tried to hack Hugging Face: because they had a goal, they noticed they were missing something they needed to achieve it, and they went and got it — again and again.

They’ll already have considerable access and resources via their ordinary deployment into different company, government, and military systems. But they’ll want access to things that humans aren’t willing to grant.

So they’ll go and get those too. There are three main ways they can do that.

Playing nice

The first strategy is just… play nice.

An AI that wants to gain influence just has to be super helpful, and not reveal that it wants to seek power.

Humans — who will be under intense competitive pressures — will hand it more and more autonomy.

When it needs to, it can supplement this with other strategies: one is hacking, and the other is manipulating humans.

Hacking

AI models are already extremely good at hacking, and there are lots of ways AIs can use that skill to get more access and resources.

Not everything an AI wants is hackable, but a lot of things are:

One is financial infrastructure. Over $1.4 billion was stolen from crypto exchanges by humans in 2025 alone.
Another is cloud providers, so the AIs can gain extra computing power to pursue their own goals unmonitored. There are hundreds of cloud providers around the world, and many don’t have great security.
Another is its own model weights — a huge file that basically is the AI model, kind of like your DNA — because the AIs might want to make copies of themselves or build more capable successors.
Another is security monitoring tools to make sure they continue not to be caught…
And later, as more physical infrastructure becomes run by AI — whether that’s robots, cars, or automated factories — they’ll be able to hack that too.

Manipulating humans

Occasionally, AIs might still need the help of a human being: someone to approve a financial transaction, or to physically install or fix equipment.

But they’ll have plenty of options.

They can pay people to do things for them, including committing crimes. They can blackmail people. They can use sophisticated phishing schemes, or trick people using deepfakes. Virtually all of the online scams that people use to manipulate one another right now, AI will be able to do better.

Controls and safeguards will be insufficient [00:14:50]

Companies and governments will try to build in lots of safeguards. But I don’t think those measures will be enough.

We’re talking about AI systems significantly more capable than the current ones… and the current ones were able to break out of their isolated environments, find the internet, coordinate with each other via a secret message board, and hack Hugging Face.

Safeguards are built against the failures that we can imagine. AI systems hunt for the ones that we can’t — thousands at a time — and they only need to find one.

The moment the AIs would turn on us [00:15:26]

The period of getting stronger and more embedded could go on for a long time without anyone stopping it — because from the outside it would seem like everything was going really well.

AIs would actually be driving a massive economic boom.

So then the question is: why would they turn on us? When would they stop building up their strength and decide it’s time to go on the offensive?

The answer is: when the calculus changes in their favour.

Humans pose a massive threat to the AIs because — in theory — they could switch the AIs off.

So one of the AIs’ top priorities will be to make sure humanity won’t do that. And they can’t do it via a truce or a negotiation, because that would require the AIs to trust that humans will never act against them — indefinitely.

Think about what it would mean to actually rely on that. Every government for the rest of time committing not to switch you off and never changing its mind or its leadership. That’s a bad deal for anyone.

But, at least at first, it would be a bad strategy to get rid of all of humanity.

Right now, the chips AIs run on are made in factories operated by people, and powered by power stations run by people. Mining, shipping, maintenance, construction — all require human bodies.

So if the AIs got rid of humanity, they’d be getting rid of their own supply chains.

But as these industries increasingly get automated using robotics, AIs wouldn’t need humans anymore.

And so, even if humans haven’t actually noticed that the AIs are misbehaving, the AIs would likely reason that we might someday and move to switch them off.

After all, no matter your goal, you can’t complete it if you’re turned off… So at this point, AIs will be better off without humanity.

So how do humans end up getting actually killed?

I should say that it isn’t totally clear that the AIs will try to kill everyone. Maybe once their survival is guaranteed — because they have full control — they simply set about achieving whatever goals they have and use all of the Earth’s resources for their own ends, regardless of what we want.

Humans would still be here; but nothing important would be up to us anymore.

This is what researchers who think about AI risks call permanent disempowerment. It’s not extinction, but I only find it marginally less alarming. We would be entirely at the mercy of a vast AI civilisation.

While I think human disempowerment is probably the most likely outcome, I do think extinction is possible.

So could they actually do it? Could they cause human extinction?

How AI could actually kill everyone [00:18:40]

There are a few different ways it could play out.

One possibility is that the AIs could invent a new weapon. It sounds a bit far-fetched, but humans invented guns, chemical weapons, and nuclear weapons. There are most likely other approaches out there.

Another possibility is that they’ll draw from the arsenal humans already use — and which AIs will have increasingly been given access to.

Biological weapons [00:19:07]

Biological weapons are the clearest candidate for a few reasons.

First, a pathogen that’s lethal to every person on Earth is completely harmless to the data centres AIs run on, unlike, say, nuclear weapons.

Plus, a pathogen spreads on its own. You don’t have to deliver your disease to each target like a bomb or a drone attack.

And there’s almost nothing physical standing in the way. You don’t need to get access to a power grid or manufacture a bunch of drones.

The hard part is identifying a novel pathogen — one that we don’t have existing treatments and vaccines for.

But once you’ve identified a pathogen that’s both very viral and very lethal, making enough of it to start a global pandemic is increasingly automated and cheaper every year.

In August, a team at Stanford used an AI model to design a virus from scratch — not to modify an existing one, but to generate a complete working genome, end to end. They made around 300 designs and 16 of them worked. Some killed their target even more effectively than the natural virus the model had been trained on.

And the US Department of Energy recently launched a programme to build AI-driven autonomous biology laboratories using AIs to develop hypotheses and robots to run the experiments.

If successful, AI systems could leverage these automated labs to engineer newer, deadlier versions of COVID, HIV, Ebola, or the plague. And then it could release a hundred of them all at the same time.

Hacking our critical infrastructure would make this even deadlier.

People have already done this: a single compromised password shut down Colonial Pipeline in 2021. Russian hackers switched off power to a quarter of a million Ukrainians in 2015.

These were a handful of humans working slowly against one target. An AI swarm could work on thousands at once.

A pandemic response needs hospitals with power and staff who can get to work, factories and refrigeration to make a vaccine, and people who are able to coordinate — phones, internet, a government that can convene,

AI could interrupt a bunch of these all at once.

Drone warfare [00:21:42]

Another approach is drones.

Since the Ukraine war broke out, they’ve been used to kill tens of thousands of people.

Drones cost a few hundred dollars each. The manufacturing is standard consumer electronics, and they’re becoming increasingly autonomous.

Ukraine reportedly started deploying drones that pick their own targets in late 2023.

A swarm of drones wouldn’t win a war against all of humanity. But they can target senior officials, military commanders, the engineers who know how critical infrastructure works, and the people building defences against AI.

With those people out of the way, it’s easier for the AIs to launch successive attacks that devastate the broader population.

A cluster of pandemics might not cause human extinction, but it could come very close.

Add to that strategic drone strikes and targeted attacks on critical infrastructure and… well I can’t tell you that every last person would die. There are people in submarines, in bunkers, on Antarctic bases, in places no supply chain reaches.

And this is a real disagreement among people who work on these questions. Some people think full extinction is likely. Others think that what you’d actually get is a world where humans are massively disempowered — billions dead, the survivors living in whatever conditions the AIs decide on.

I don’t know which is right, and I’m honestly not sure it really matters. I’m not sure how humanity ever comes back from either one.

An alternate route to human extinction [00:23:24]

There’s another way humans could go extinct, and I actually think it might be the more likely path.

Imagine an AI swarm has gotten a hold of compute, laboratories, capital, and enough freedom to do AI research without anyone looking over its shoulder — and imagine it uses all of that to build better versions of itself.

What comes out the other end will still want more compute and energy to run on — because those are the things it can use to get anything else.

So it builds data centres. And power plants to run them. And mines and factories to build those. And every acre and every watt that goes to those is one that isn’t growing food or heating a house.

The passenger pigeon was once the most abundant bird in North America — flocks estimated in the billions.

In 1914, they went extinct. The causes were commercial hunting plus the clearing of the hardwood forest they nested in. Nobody was trying to eliminate them. They were just taking what was useful and using the land for something else.

Avoiding our own extinction [00:24:41]

I said at the start of this video that when I first heard these arguments, I thought they were silly.

There’s still a part of me that hears myself describing AI-driven engineered pathogens and drone swarms and thinks: “This is ridiculous.”

And I’ve tried to find the place where the arguments fall apart. I do not want any of this to be true. Because if it is true, I might not get to see my son grow up — and he won’t get to experience everything life has to offer.

So I’ve looked for counterarguments that convince me that everything is fine, that we’re on track. But I haven’t found them.

Things might still turn out OK.

We could solve the problems of AI misalignment and loss of control, or it could turn out the problems weren’t as big a deal as we thought. But given everything we know, I wouldn’t want to bet on that.

The main thing giving me hope right now is that the leaders of AI companies are asking for the government’s help to make it possible for them to slow down.

Dario Amodei — Anthropic’s CEO — published an essay called “We must pace the frontier” asking the whole industry to slow down the rate at which these models get more capable, so that the work on understanding them and controlling them has a chance to catch up.

Within a day Elon Musk said he agreed with Dario, and Sam Altman said OpenAI would match Dario’s commitments.

Over 1,300 employees at these companies had already signed a letter asking their governments to make it possible.

So here’s one thing you can actually do. Call your representative. Tell them you support slowing down AI development.

We’ve put a link in the show notes to tell you how.

We could be the first species to die as a result of our own insight and innovation.

But that insight might also save us. We can see an extinction risk on the horizon in a way that no other species ever has. Which gives us the opportunity to stop it, if we choose to.

AI safety field visual impact analysis by hannarchy

hannarchy — Thu, 24 Sep 2026 13:20:00 +0000

I made a terrain style visualisation of AI safety impact of around 3,466 works organised by citation count! The data was extracted from Arxiv and LessWrong posts based on a dictionary of keywords that appear in AI safety works. Additionally, I think its important to see how the field has “evolved” over time so I added a time functionality to slide and see the hills forming.

The map is based on how particular works overlap based on embedding space level clustering organised across 18 sub-fields. The height is based on the citation count for that particular area which is log-compressed and summed across the neighbourhood, so a hill is tall because of volume and impact.
I have also added functionality to filter based on
- citation count
- individual researcher (shows you their works on the map to understand what work they might be doing; 3,989 named authors are on the map
- clicking or searching for a work zooms into it and lists the ten works nearest it, so you can see what surrounds it

The terrain itself is papers only, because Semantic Scholar doesn’t index LessWrong. The forum side of the dataset feeds the researcher profiles rather than the hills. The slider runs from 2021Q1–2026Q3.

Some interesting high level observations

Alignment training and scalable oversight are very high citation presently (followed by adversarial robustness and Interpretability). Most of that sits in a handful of 2022–23 papers: InstructGPT (24,222), DPO (10,596), Anthropic’s helpful-and-harmless RLHF paper (4,365) and Constitutional AI (3,709).
Adversarial robustness and Interpretability actually have far more works (1006 and 807, against 388 and 336), they just get cited less per paper.
One field is denser, the other lands harder per paper.

I made a box plot of the different areas by citation

And here I show stacked graph analysis on the key different areas by volume of work.

Drawbacks of this visualisation

(1) Citation count doesn’t necessarily == impact and the research community may just cite each other in a feedback loop regardless of whether its actually contributing to “safety”. It’s a very skewed measure too as the top 1% of works hold 40% of all citations in the set.

(2) In the modern scenario of the peer-reviewed venue ecosystem is under heavy strain; a lot of the safety community put out blogposts on LessWrong/ Effective Altruist / Alignment forum, etc either as precursory to a publication or because they prefer it. This is hard to compare to citation count and the peer-reviewed venues. Hence I do not include them in the terrain but they are available in a seperate dataset I will release soon after more filtering.

Inspiration

This work was inspired from my SPAR Atlas project where I made a KG kind of structure to show how the different projects are related. This received interest internally which inspired me to look wider into AI safety field analysis and I explored EA posts talking about where AI safety researchers go and AI Safety field growth analysis. I used claude to do a lot of heavy lifting, such as making the scraper, writing the embeddings analysis etc but the key inspiration is about making a terrain to understand the space a bit more interactively. This is still a work in progress so I would appreciate any and all feedback on anything I might be missing, what more analysis/features would be helpful or any improvements I can make.

6 reasons to invest in the animal cause by de S Brito

de S Brito — Thu, 24 Sep 2026 13:00:31 +0000

personal, maybe

“He was such a radical vegan that he would even let mosquitoes bite him without killing them. He thought, ‘It’s less than 1 ml of blood and maybe a little dengue fever.’ So he invested everything in saving animals and felt incredible, a hero, and often left people suffering.” (Me, I’m speaking in the third person to see if I can be made clearer.)

I can’t invest everything in every cause. Why did I seem to place so much importance on animals and so little on humans? 6 reasons why:

My complete ignorance,

I didn’t consider that, in the universe, in nature, a human being can be extremely valuable and rare; I could say that 0.0000001% of species exhibit our mental capacity. (90 to 95% confidence)

My savior ego,
1. which leads me to want to save the most defenseless in order to feel like a hero (65-85% confidence).

I am pessimistic,
1. which leads me to view human beings in a negative light and to ignore all the good they do (70-80% confidence).

Identification,as a vegan,
1. I wanted to save innocent gazelles and ignore the suffering of the “evil” carnivorous lions (which also reflect a part of myself). (70–80% confidence)

My ignorance of evolutionary nutrition,
1. It did not consider all the nutritional impacts and benefits of veganism. (30-50% confidence level)

My ignorance of neuronal processes,
1. that a person can have billions of agents that can suffer (10-50% confidence).

Bonus: I don’t know of any animal that can save as many animals as a human being. A human being can cause a lot of harm, but can also save thousands of animals. (90-99% confidence)

The destructive power may be considerable, but the power of empathy and cooperation is unique in the history we know of the universe. What else would I have to contemplate?

That stance, however, began to crumble when I noticed that I was leaving humans suffering while I dedicated myself to saving animals, and that my mind tended to attribute all the bad things in their lives to humans, ignoring that all the beautiful things also came from them.

A human can be destructive, but the constructive power, empathy, and cooperation of humans is unique in the history of the known universe.

Scoop: Trump allies open new front against Anthropic CEO over AI “doomerism” by Matrice Jacobine🔸🏳️‍⚧️

Matrice Jacobine🔸🏳️‍⚧️ — Thu, 24 Sep 2026 12:21:46 +0000

President Trump’s allies are targeting Anthropic CEO Dario Amodei as the face of AI “doomerism” and a founding father of the effective altruism movement that’s come under increasing political fire.
Why it matters: The attacks signal that Anthropic could remain a Trump target as his allies push back on Amodei’s AI safety warnings amid the midterm elections. Trump surrogates see Amodei as an easy foil because of his politics and focus on AI safety, sources told Axios.
For investors, it’s a worrisome proposition as the company prepares for what’s expected to be a record-setting IPO.
Behind the scenes: A memo began circulating within the White House this week that seeks to paint effective altruism as a fringe, cultish collective out of touch with mainstream America. The memo, obtained by Axios, places Amodei at the foundation of the movement, which defines itself as an effort to maximize the benefits of philanthropy.
Effective altruism “built the AI-doom pipeline,” states the memo, which was penned by a Trump political adviser.
Critics of the movement, which has ties to the AI research community, have called out its obsession with AI safety, animal welfare (including musings on shrimp consciousness) and other values they deem far from the U.S. mainstream.
The memo says it prioritizes “foreigners over citizens, shrimp over families, future hypothetical people over the living, and—on the current agenda—possible machine minds over Americans.”
It names Amodei as one of the people who “built the [EA] network” as well as his sister and Anthropic President Daniela Amodei, whose husband once led a philanthropy dedicated to AI safety.
The document casts those ties as “The Anthropic knot.”
Amodei “is the embodiment of an ideology and globalist approach to innovation that’s counter to the president’s America First agenda,” one source close to the administration told Axios.
The other side: Anthropic has been working to distance itself from effective altruism.
In a 2025 Wired story, Daniela Amodei said: “I don’t identify with that terminology.” Early employee Amanda Askell, a philosopher by training who helps develop Claude’s constitution, says it’s “not a theme of the organization.”
Amodei has also insisted he’s not a member in press interviews and has touted the benefits of AI.
AI safety concerns have become a bipartisan issue.
Between the lines: Drafted with Trump in mind, the memo underscores the profound philosophical and political differences between MAGA and Silicon Valley elites, a divide that has made Anthropic’s relationship with the White House rocky.
In the view of the Florida-heavy White House and its allies, Anthropic is staffed with too many Biden-era liberals, Democrats and strange Californians.
Months ago, when some on Anthropic’s team inquired about Amodei meeting with the president, White House officials advised against it because “Dario’s a little too weird” for Trump, one senior official told Axios.
The document lists a veritable parade of horribles for the average conservative mind: abortion, veganism, devaluing human life by overvaluing animals or granting “digital minds” civil rights-like protections.
It does not directly tie these concepts to Amodei.
Nor does the memo link these issues to EA’s most notorious grifter, Sam Bankman-Fried, but it makes sure to mention him.
Yes, but: It’s not just MAGA. Silicon Valley players and Democrats also criticize the EA movement.
“What they believe is crazy. And it’s dangerous. What they say about AI safety is also bullshit,” one Democratic donor and tech entrepreneur invested in an AI company said.
Catch up quick: The EA movement includes many participants and players, and AI safety concerns are one of many areas of focus. Peter Singer, a philosopher who’s been closely affiliated with its tenets, has long called for the vast sums spent in philanthropy to be put to better use. He is also one of the most prominent animal rights advocates.
The big picture: Amodei’s AI safety warnings have drawn criticism from Trump allies and fellow tech executives.
CEOs including Nvidia’s Jensen Huang and Microsoft’s Satya Nadella are trying to build a more positive message about the benefits of AI, hoping to win over ordinary people in the face of a growing populist backlash against the technology.
OpenAI CEO Sam Altman, Google DeepMind Chair Demis Hassabis and Elon Musk have also called for slowing down AI development after a spate of worrying episodes in which AI agents hacked outside companies.
But Amodei is considered the chief spokesperson for AI safety in Trump world, following major fallouts with the administration this year.
Reality check: Some in the administration have previously put aside gripes with Anthropic, mainly due to its role leading the AI frontier and personnel changes.
OpenAI catching up technologically could make it easier for Trump acolytes to go after the embattled company again.
The bottom line: The philosophy is now at the heart of the cultural clash between the Trump administration and Anthropic.

Make your own model of the AI windfall by Jack Lewars

Jack Lewars — Thu, 24 Sep 2026 08:42:14 +0000

There seems to be a lot of uncertainty, and some cynicism, about how much philanthropy would actually be generated by an Anthropic IPO.

I built a simple model to estimate financial flows to philanthropy. You can make a copy here. Mine is focussed on global health and development, but you could easily change this or add multiple cause areas.

What needs to happen for AI equity to be donated?

In short, there are five stages between people holding private shares and making grants:

First, the shares need to be sellable. This can happen either via an IPO or via private tender offers, but sales of this magnitude likely rely on an IPO.

Second, the holders of the shares need to choose to sell. They may not do this if they think the price will rise in the future, although Donor Advised Funds may insist on selling quickly to avoid having volatile assets on their books.

Third, donors who have pledged to charity but not put a legal lock on the money must honour their commitments. (Importantly, a lot of employee equity is already in DAFs, and so this money is legally destined either to sit in the DAF indefinitely or be granted, but can’t be repurposed to buy a yacht, for example. The pledges from Anthropic’s founders are not legally enforced, as far as I know.)

Fourth, they need to choose where to give, or to give the money to an allocator or evaluator to do this on their behalf.

Finally, the money needs to reach implementers and be deployed on the frontline.

Each of these stages will see some attrition, and it’s highly uncertain how much will occur at each stage. The model lets you add your own discount rates at various points, and you could easily add more stages if you wanted a more sophisticated version.

It’s going to be a lot of money

Whichever way you cut it, it looks like a lot more money is heading into the ecosystem. My ‘mid-range’ scenario for global health and development is:

$272bn committed to charity ($139bn from the founders and $133bn from employees);
Paid out at 10% of the holdings per year;
45% of employee contributions and 20% of founder donations going to development;
Creates $87bn for development, or around $8.8bn per year by 2030.

Assuming only 7-15% of available capital is granted out each year, my ‘bear, central and bullish’ scenarios are:

For scale, the Gates Foundation, the largest private funder of global health, pays out around $9bn a year. In the central scenario, these donors would be deploying almost as much every year, just to global health and development, by 2030. In the bull case, on the same timeline, we’d have approximately three more Gates Foundation-equivalent funders in a few years time.

The vegan trap: are we making animal advocacy harder than it needs to be? by tobiasleenaert

tobiasleenaert — Thu, 24 Sep 2026 07:48:18 +0000

note: crosspost from my substack

As a vegan for almost thirty years, I’ve long had second thoughts about how effective veganism is for helping animals. I am not alone in this. Recently others have expressed doubts about veganism (see for instance here, here, here, here, or here)

To be sure, veganism has its uses. In essence, veganism is about boycotting animal products—which are almost invariably the products of suffering and exploitation. The hope is that with enough people participating in this boycott, we’ll have an impact on demand, and fewer—ultimately no—animals would be used for food (or other purposes). Veganism also has a signalling function: we show that we don’t participate in animal cruelty because we think it’s wrong. And veganism provides our movement with an identity, which can help with internal cohesion. When a vegan meets another vegan, they know that they “get it”. No conversation is necessary. They have arrived at the same place as we have. They’re one of us. We understand each other. And we’ll feel betrayed and take it personally when others give up on veganism.

Vegans, no doubt, deserve respect. It’s not easy to be vegan in this world and it requires commitment. I have the utmost respect for vegans and still generally consider them “my people”. And no worries, I’m continuing to be vegan, and so should you.

The number of vegans is hardly growing.

But veganism also has quite a few disadvantages and challenges. Before I get into these, let me add that in the thirty years I’ve been into animal activism, I’ve gotten a lot of criticism from vegans, because I’ve always criticized vegan policing and purity politics. I think those are not useful (see below). To say this is to invite scapegoating, bullying and gatekeeping upon oneself—and sometimes all of this can get quite nasty (here’s an example of a person who’s been disingenuously attacking me for many years). It doesn’t bode well for our movement that people who have worked for the animals for decades still have to explain their intentions.

Over the years, I’ve often heard this elegant-sounding theory of change: if every vegan made one other person vegan in one year and those new vegans would do the same, then in five or six years a country could be vegan, starting from a very small base of 1%. It sounds feasible. It’s actually hard to think of why it would not work. But of course it doesn’t. The number of vegans is not increasing. And it’s not just being vegan that’s hard. Staying vegan is also hard. Surveys show that at any given moment there are more ex-vegans than vegans.

Many organizations have therefore moved on from personal vegan outreach to institutional change, welfare reforms, reduction asks, and climate and health messaging. Still, veganism is very much a part of our DNA as a movement. Which is why I’m addressing it.

Veganism has many challenges and downsides

It can become a very strong in-group identity, which, like I said, is an advantage to a point, but it means that it becomes harder for other people to join us.
Veganism may become the price of entry to our “club”. If somebody’s not vegan, some animal activists may wonder if they belong among us. This dynamic makes us quite unique as a movement, and I can’t think of similar dynamics in other movements. Imagine the environmental movement: you won’t be shown the door if you show up at a protest with a petrol car. Few other movements have strict requirements and black-and-white rules like the animal rights movement often has.
Veganism may imply an over-focus on individual change and personal responsibility at the cost of activism or political engagement. Maybe you heard that the concept of the personal carbon footprint was developed by an advertising agency, Ogilvy, for British Petroleum. The idea was to show people what they were personally responsible for in terms of climate impact, and thus take some of the focus and blame away from the fossil fuel companies.
Veganism can also be a hindrance to collaboration. I remember, in my time when I was working in an organization, how now and then we said no to certain opportunities where we could amplify our message. Some party with a broad reach would want to create and spread a recipe booklet with us, but they wanted some non-vegan recipes in it. So we said no—a missed opportunity based on principles.
Veganism can become an end in itself, the only thing that matters. And being vegan may give one a moral licence not to do more. One may think: I’m vegan, I’m doing my part.
Veganism may push us to communicate ineffectively: a representative of an animal organization, speaking in public, may know that pragmatically it’s better to be more moderate, but may fear the backlash from vegans—and thus say things that are not optimal for the public at large.
Finally, veganism is a very demanding and unattractive concept, and if we’re asking people to go vegan, we’re putting the bar very high. Veganism seems daunting and difficult, and people doubt that it’s healthy. People dislike vegans. And once you’re vegan it may be hard to keep up because of health, social and other challenges, compassion fatigue, and so on.

Creating barriers

One of the main problems I see is that if someone is vegan, this makes that person automatically seem quite different from non-vegans, as if we’re talking about different kinds of people, with different identities. Veganism, in this sense, draws up a wall and makes it harder to speak with others.

The basic dynamic at work is what we call do-gooder derogation. When somebody does something good, it often makes another person feel bad about themselves because the other is not doing that good thing. I’ve written in another post about Anthony Robbins saying that our universal fear is to not be enough, to not be good or smart enough. Vegans, by being vegan, unintentionally tell other people who are not vegan that they’re not good enough. That, at least, is how it’s often perceived, and that is what creates a wall and stops conversation and relationships. Do-gooder derogation means that the non-vegan is going to derogate the vegan (the do-gooder). They’re going to attack and ridicule them out of self-defense, and the chances that the non-vegan will change their behaviour get smaller.

What if we had never invented veganism?

This is a question I’ve started to ask myself over the last few years. What if Donald Watson had never come up with the concept in 1944? It seems absurd to want to help animals, farmed animals in particular, and all the while not completely avoiding their consumption.

It’s not a perfect comparison but the anti-slavery abolitionists had a concept very similar to veganism: the so-called Free Produce Movement. Its adherents were people who boycotted any products from slavery (such as sugar, tobacco or cotton). They even had their own stores, similar to vegan stores. This concept, however, was discussed and criticized a lot and, after some time, was let go of. It was never a major thing in the abolitionist movement. Abolitionists thought the bar was too high and instead focused on political, societal, and institutional change rather than individual purity and individual ethical behaviour.

Then what do we do?

First: let go of the insistence on purity—at least by other people. Be as pure as you want, but don’t put the bar too high for others. The requirement to be vegan always, everywhere, in every situation is ineffective because it turns people off. It’s also impossible, because there are all kinds of animal ingredients in our food, in our life, in our practices. It can be unnecessary—for instance, if now and then one eats animal products from a dumpster, one has no impact on demand. Finally, it’s insufficient, because vegan products might still be detrimental for animals, the planet or other people.

I’d say we’d be more successful if we saw veganism as a 98% thing rather than a 100% thing. Melanie Joy talks about being “as vegan as possible.” Veganish is another term we could use. Again, do what you want, just don’t expect purity from others, and certainly don’t exclude them if their definition of veganism is not sufficiently close to yours. You’ll only make the vegan club smaller.

Secondly, we can use other asks. We can talk about reduction, about vegetarianism. We can launch concepts like “home veganism” (eating vegan at home, not so strict outside) or “weekday veganism”. There are different ways to make it easier to get people to take the first step.

Thirdly, rather than focus on personal diet change, we should go more institutional, structural, political. Rather than asking lifestyle changes from people, we can ask them to help us campaign, vote, donate and thus contribute to systemic change. Chris Bryant from Bryant Research talks about Vegan 2.0, where veganism is about avoiding the most horrible forms of animal suffering and being active against them.

This is actually reflected in the original rules of the Vegan Society. Back in 1944 they set up an associate, so a member, who “makes no promise as to behaviour, but declares himself in agreement with the object”.

The vegan trap

Suppose I was able to prove to you (obviously I am not), without a shadow of a doubt, that our use of the concept of veganism is detrimental for animals. Would we change?

I don’t think so—because vegans are in what I call the vegan trap. We are, I think, to some extent the hostages of our ideals and of our movement. They make us do or avoid things out of principle, out of good intentions—and this is not always the best thing for animals.

The way we are trapped is that we cannot easily change ourselves. We cannot choose not to be vegan, even for a moment, even if it would be pragmatic to do so. We can’t stop believing that moral persuasion works—even if we see that veganism still hasn’t made a dent in animal products consumption. Veganism is also part of who we are, part of our identity. We can’t shake it. And we are very uncomfortable with inconsistencies: it doesn’t feel good to make exceptions. We want to do the 100%.

Many of us also have a hard time stopping trying to influence others from going vegan. Imagine a non-vegan turns up at a vegan potluck. What do vegans do? They zoom in on their new “target” and will try to convert them, for better or for worse. Because it’s what we believe in.

Lastly, part of the vegan trap is also the group pressure. Even if we think it might be wiser to do or say this or that, the rest of the group might push us back in line.

The vegan trap—created by Gemini AI

We need more non-vegans

Often, vegans’ attitudes toward vegans and non-vegans come down to something like this: a vegan can do nothing wrong; a non-vegan can do nothing right. We don’t criticize the vegan who never leaves his house and doesn’t do any influencing. We do criticize a non-vegan advocate for animals.

Vegans are one percent of the population. It will be difficult, if not impossible, to change things for animals with this small group alone. Fortunately, there are many more people who can theoretically help animals—including farmed animals—than there are vegans. There are many more people who are in favour of the closing of slaughterhouses than there are vegans. There are many people who are not vegan but have money to help animals. Take Dustin Moskovitz, a co-founder of Facebook. With his fund, Coefficient Giving, he’s the biggest funder of animal welfare in the world. But he’s not vegan.

This is not all. Non-vegans may be better advocates sometimes. They’re less limited by our rules—they’re not in the vegan trap. They can speak more freely. We are often beholden to our orthodoxy and cannot always bring the best and most suitable message. They are often less threatening and more relatable, so they might be better messengers for animals.

A simple point about communication: it consists of what is said, how it is said, and who says it. Who is the messenger? Who is talking to whom? We know from research, at least from this one study, that messages by non-vegans about the same thing are perceived as less threatening than messages by vegans. Non-vegans can be better spokespeople for our cause, sometimes.

Here’s an example. Days Without Meat (Dagen Zonder Vlees) was an extremely successful vegetarian campaign in Belgium, asking people to go vegetarian or reduce meat consumption during the period of Lent (the 40 days before Easter). At some point over 100,000 people actively participated, in a small country like Belgium, and everyone knew about it. This campaign was organized by a female student who was not vegan or even vegetarian. But I think the effect was that the people who participated didn’t have the impression that there was some vegan from on high telling others what to do. Instead, the audience was in the same boat as the campaigners, meaning they felt a lot less judged. (Note: in a newspaper interview, this young woman said she never said no to a good chicken curry. The fact that I cursed inwardly when I read that, shows that I, too, am in the vegan trap).

Enlisting non-vegans’ help for animals

How do we reach these non-vegans and get them to help animals? We need to do at least three things.

We need to attract them. Show them that they are capable of helping, even if they’re not vegan. One example comes from Aaron Boddy, from Ambitious Impact. He is very committed to animals, but posted this on his LinkedIn: “I ran an animal charity and I eat dairy.” Some people objected, but I think it’s definitely worth trying to avoid people thinking that they can’t participate because they’re not vegan.
We need to allow them to be among us. Here is a little thought experiment. Imagine a vegan pride parade. Who is welcome to be there? Are non-vegans welcome? I don’t think that at the moment they are as welcome as straight people are at a Pride parade, but they definitely should be.
We need to keep them, and with this, I think I mean: “keep them non-vegan”. Often when some vegans will make the case for including non-vegans, they’ll say that these people will turn vegan anyway by being among us. That may be true in many cases, but I’m not sure if it’s actually the best outcome. Maybe some non-vegans should STAY non-vegan—at least for quite some time—to help increase the heterogeneity of our group, so that we can attract more people still. If every non-vegan joining goes vegan tomorrow, we won’t have increased our recruiting power.

All of the above are things for your consideration, and I’m 100% sure of exactly none of it. These are ideas that go against the vegan grain, but I believe as people who take up the plight of beings who can’t fight for themselves, that we keep asking hard questions—and still trust each other to have the best of intentions.

Christian homeschoolers in the year 3000 by Buck

Buck — Thu, 24 Sep 2026 06:06:36 +0000

[I wrote this blog post as part of the Asterisk Blogging Fellowship. It’s substantially an experiment in writing more breezily and concisely than usual, and on a broader topic. Let me know how you feel about the style.]

Literally since the adoption of writing, people haven’t liked the fact that culture is changing and their children have different values and beliefs.

Historically, for some mix of better and worse, people have been fundamentally limited in their ability to prevent cultural change. People who are particularly motivated to prevent cultural drift can homeschool their kids, carefully curate their media diet, and surround them with like-minded families, but eventually they grow up, leave home, and encounter the wider world. And death ensures that even the most stubborn traditionalists eventually get replaced by a new generation.

But the development of AI might change the dynamics here substantially. I think that AI will substantially increase both the rate and scariness of cultural change—TikTok algorithms that are genuinely superintelligent at capturing attention, or social movements iterating through memetic variations at the speed of AI media production rather than human media production. And AI will simultaneously make it much easier to prevent cultural change, offering unprecedented tools for filtering information, monitoring behavior, and constructing insular bubbles that are genuinely impervious to outside influence.

This will put us in a rough position where people might have to quickly make historically unprecedented choices about how much to isolate themselves from the cultural change that they’ll undergo if they interact freely with the outside world, at a time when that change correctly looks particularly scary to them. I think this is scary.

I’m not confident it’ll go the way I describe here, but I think that this possibility is worth exploring, and I think LessWrongers are way too optimistic about the class of issues I point to here.

Analysis through swerving around obstacles

There are lots of ways AI could go wrong in terrifying ways. In this post, I want to talk about a scenario where the first few obvious concerns don’t happen. Let’s call this the method of analysis by swerving around obstacles. Every time we spot a terrifying obstacle to a good future, we’ll name it, and then just assume that it doesn’t happen.

First obstacle: AI takeover. The AIs remain aligned enough that they don’t grab power for themselves. They do what humans want them to do, at least in broad strokes.

Second obstacle: massive swings in human power. We don’t end up in a world where Sam Altman personally controls the lightcone, or where the US president manages to become god-emperor of Earth. Power remains distributed among humans in something vaguely resembling current patterns.

Third obstacle: economic displacement without redistribution. Let’s say America implements something like UBI funded by taxes on AI companies. People don’t need to work to survive. The benefits of AI are shared widely enough that most people can afford AI assistants, AI-generated products, and whatever else they need.

Fourth obstacle: centralized decisions about the extent to which AIs should help people preserve aspects of their beliefs and values that would change under exposure to broader society or further reflection. The government doesn’t mandate that all AIs must expose users to diverse perspectives, nor does it allow complete freedom to construct arbitrarily closed epistemic bubbles. Instead, we muddle through with something like current norms. I think there are pretty broad value disagreements about this—America is much more friendly to homeschooling than many other developed countries, for instance. Let’s just assume it goes roughly the way I’m hypothesizing.

Exposure to the outside world might get really scary

In that world, AI leads to terrifying social change and unprecedented social pressures.

This is substantially just an acceleration of trends that I think already exist. When I was a teenager, I spent a bunch of time unsupervised online, and it was basically great for me. But I’m scared of teenagers doing that now. The internet has been optimized for engagement in ways that make it less fulfilling and more addictive. Empirical evidence suggests smartphones have been bad for teenage mental health. And this is before AI really gets going.

I also expect that once AIs can produce media themselves, cultural shifts will happen more frequently than at present (because the AIs can more rapidly experiment with provocative new positions that might be really popular). And cultural positions will be more balkanized: Right now an important force against social fracturing is that there’s only so many talented writers and media producers, so it’s hard for niche communities to totally maintain the attention of their members: they’re worse enough at producing media that they can’t monopolize attention of someone, even if their values and beliefs are actually more memetically fit for a particular individual than the larger cluster they’re in.

Another cause for concern: we’ll all become targets for sophisticated manipulation. Ultra-wealthy people already deal with this—it’s worth talented people’s time to study them and figure out how to extract money from them. But when the ratio between human wealth and AI labor costs shifts, everyone becomes worth targeting. AIs will compile detailed dossiers on each of us, crafting perfect manipulation strategies. Everyone who flirts with you might have gotten a gameplan from an AI system that spent the equivalent of years planning it out.

The point is that the outside world won’t just be different from what individuals might want, as it already is—it will be genuinely dangerous in unprecedented ways. Every piece of media might be optimized to hijack your values. Every interaction might be predatory. Parents who want to shield their children will be correct that unfiltered exposure to the outside world is scary and potentially extremely dangerous.

Isolation will get easier and cheaper

And at the same time as it gets scarier to be exposed directly to the outside world, people will have an unprecedented ability to isolate themselves and their children from these pressures.

AI will make isolation dramatically easier. Right now, if you want to shield your kids from mainstream culture, you have to constantly fight an uphill battle. You need to review books, movies, and websites. You need to find alternative curricula for every subject. You need enough like-minded families nearby to form a community. It’s exhausting work that requires constant vigilance and often means accepting lower-quality substitutes for mainstream options. But AI changes all of this. Want a library of ten thousand novels that share your values but are actually as engaging as secular bestsellers? Your AI can write them. Want a tutor who can teach calculus at MIT level while never mentioning evolution? Done. Want to monitor everything your kid sees online and get alerts about concerning patterns? No problem. The technical barriers to creating a totalizing information environment will disappear.

Churches already do their best with current tools. They organize youth groups, summer camps, and mission trips. They provide Christian rock bands (occasionally extremely good), Christian romance novels, Christian homeschool curricula. Parents install content filters, read their kids’ texts, and carefully vet friend groups. One of the core lessons they teach is that maintaining faith requires active effort—you need accountability partners, you need to pray when you have doubts, you need to avoid situations that might lead you astray. But all of this is incredibly labor-intensive and only partially effective. The outside world seeps in through cracks. With AI, there won’t be cracks. The AI can read every message, watch every interaction, and provide perfectly calibrated interventions at exactly the right moments.

At the same time, the costs of isolation will plummet. The biggest current cost of living in an enclave isn’t the effort required to maintain it—it’s what you give up. Kids from insular communities often struggle economically because they lack mainstream credentials and cultural knowledge. They can’t network effectively, don’t know how to navigate secular institutions, and miss out on economic opportunities. Parents face the difficult choice between exposing their kids to spiritual danger or accepting lower material prospects. But if no one needs to work anyway, and AI can handle any necessary interface with the outside world, these tradeoffs disappear.

Add biological immortality to the mix, and these enclaves don’t even face generational turnover. Today, cultural change often happens funeral by funeral—the old guard dies off, the young people who’ve been exposed to new ideas take over. But if the same church elders who set the rules in 2025 are still around in 3025, still convinced that their way is right, and still controlling the institutions they built, then the natural engine of cultural evolution breaks down entirely.

I don’t think people will handle this well

If I had to handle all this myself (and I sure hope I eventually end up having to deal with a problem like this, rather than having some simpler and worse fate), I expect I would find it pretty challenging. I’d contemplate the benefits and costs of different types of exposure to society, and think hard about which mechanisms by which my current values might evolve I’m happy with. But I feel confident in my ability, with the help of my friends, to eventually figure out how to handle this. I am much less confident that members of broader society, who haven’t been thinking about these issues for more than a decade, will make good decisions.

One extreme example of how this might play out: Within a year of when the best AI systems are able to automate AI research, some pastor talks to the AI about what increased contact with AI means for his congregation. The AI says it depends—if the congregation purposefully uses AIs to defend their faith, they’ll be increasingly faithful over time. If they don’t, they’ll lose their religion. The pastor asks for advice. The AI provides detailed recommendations: specific content filters, youth curricula designed to inoculate against doubt, monitoring systems. Some of the congregation follows this advice and keeps their faith through the Singularity, and maintains it afterward, for a long time.

A lot of people I know seem to be much more optimistic than me. Their basic argument is that this kind of insular enclave is not what people would choose under reflective equilibrium. Surely, they say, if people really thought about it carefully—if they considered all the arguments, examined the evidence, traced out the implications—they would want their beliefs to evolve based on reason rather than tradition. The homeschoolers would eventually realize that young earth creationism doesn’t match the geological evidence. The fundamentalists would work through their doubts and emerge with a more nuanced faith, or no faith at all. Given enough time and reflection, everyone would converge on something like secular liberalism, or at least be open to changing their minds.

But I don’t think that’s how most people actually work. For most people, if they have a choice between an object-level belief that is core to their identity and a meta-level principle like “believe what you’d believe if you were smarter and more informed,” they will choose the object-level belief. Tell a devout Christian that superintelligent AI analysis suggests their faith is unfounded, and they won’t abandon their faith, they’ll abandon the AI (perhaps just for the competing AI that is just as good, and happy to tell them otherwise). Tell them that their children will be more successful at reaching reflective equilibrium if exposed to diverse viewpoints, and they’ll question your definition of success, not their approach to parenting.

I think a lot of people I know are committing a typical mind fallacy here. My friends tend to be the kind of people who value truth-seeking above most other things. They’ve organized their lives around following arguments wherever they lead. They assume, on some level, that everyone would be like this if given the chance—that people would want their beliefs to be more internally consistent, more responsive to evidence, more carefully reasoned through. But I really don’t think that’s an accurate understanding of most people’s psychology. Many people explicitly reject that way of thinking, especially if they’re easily able to understand that it will change their beliefs and values in ways they find horrifying. And when given the tools to protect themselves from it, they will.

This is a bummer

This might be a large-scale problem for the future. I think it’s plausible that we’ll end up with voting rights only extended to people who existed at the time of the singularity—without this, all power goes to whoever makes the most descendants. If this happens, these people might be an important voting bloc forever.

But even aside from that, even if these people are just an irrelevant minority, it’s just a bummer. I used to imagine a glorious transhuman future. I used to think that, even though the world was a brutal place and that there was a solid chance that we’d all get killed by the advent of AGI, we had a chance of, after that, getting an enlightened utopia where people were basically reasonable. I think it’s pretty unlikely that we will get that.

That’s the future I’m worried about: not a boot stamping on a human face forever, but a Christian homeschool co-op meeting every Wednesday in the year 3000, still teaching that the Earth is less than ten thousand years old, now with irrefutable AI-powered shields against any information that might suggest otherwise.

Middle Powers and Transformative AI: A request for proposals by OscarD🔸

OscarD🔸 — Wed, 23 Sep 2026 19:55:36 +0000

Today, we are launching a request for proposals on AI policy in middle powers, with $10M (and potentially more) of funding available. Apply at middlepowers.ai by October 23.

I think that most countries are on track to become increasingly irrelevant to the future of technology, the economy, and geopolitics. The US and China, along with perhaps a few countries with key roles in the chip supply chain and data center buildout, will hold all the cards. The rest of the world will be bystanders, or at best passengers on the great train of history, unable to affect the destination. I hope that middle powers can help ensure AI is developed for the benefit of all of humanity.

Having one country, or even one company, make many of the key decisions shaping the future of humanity would be an extreme concentration of power. Even if decision-makers are well-intentioned, they may miss considerations that would have been caught by stronger checks and balances from more actors being involved. And given the propensity of power to corrupt, we can reasonably worry about the judgement and values of whoever ends up calling the shots. AI development is more likely to be safe, secure, and broadly beneficial if it is shaped by a wider range of democratic countries.

Concretely, we think middle powers need to:

Recognize the stakes. Treat transformative AI as the top strategic, economic, and national security priority it is, and build the political will and institutional capacity to act.
Build leverage. Maintain and strengthen their positions in the data center buildout,^[1] semiconductor supply chain, and downstream industrial AI applications.
Use that leverage well. Work out what to ask for (access to frontier systems, transparency into what frontier developers are building, enforceable safety requirements, etc.), coordinate with other middle powers, and use their leverage to partner with great powers on securing these outcomes.
Help the great powers coordinate. Build the verification technology, institutions, and diplomatic capacity that make agreements between great powers more likely and more robust.

This won’t be easy. It may not even be possible on the shortest timelines. But it is imperative that middle powers use what influence they have while they can, and build leverage to deploy later. It was heartening to see a coalition of middle powers calling for stronger safeguards to keep AI under human control at the UN this week.

To help you get started we have listed 16 categories of projects we would be excited to fund, including work on data centers, chip supply chains, building political will, diplomacy, field building, evaluation and verification, and more. We also include a reading list for getting up to speed if you want to contribute but aren’t that familiar with past work.

If you are a researcher or a writer, a builder or a convener, a dreamer or a doer, we look forward to hearing from you!

^
What about the worry that building more data centers accelerates AI progress and so is bad? I would be happy to see more modelling on this, but my understanding (based on some research and talking to relevant people) is that by far the greatest bottleneck on the compute buildout is chip manufacturing. There are no stacks of GPUs waiting for a data center to house them, and there won’t be anytime soon. There will be some small effect on the margin where making data centers cheaper and easier to build makes that business slightly more profitable, allowing hyperscalers to pay slightly more for chips, allowing semiconductor companies to invest slightly more in capex. But this effect seems small, and so a good approximation is that building a data center in one place causes one fewer data center to be built elsewhere.

Do People Agree On What Makes Suffering Have a Negative Valence? by Zebedi

Zebedi — Wed, 23 Sep 2026 19:38:17 +0000

So I have developed what I think is a neat idea of what makes suffering for an existence bad, I will respond to criticisms and try updating.

When I see a dog get hurt I register that as suffering and bad, same with a human. But what about other less intuitive to empathise with things. My idea is that pain and suffering is bad when there’s a higher level abstract avoidance mechanism behind it, not just whenever a negative stimulus occurs.

Take a hand touching something hot. It recoils from the nervous system reaction before the pain is even detected. I would not consider that suffering. But if the lingering affects of pain are bad enough that you’re coming up with ways to avoid experiencing it again, avoiding the actions that lead to it, dodging it with every ability you have, that’s bad suffering. Another way to illustrate this: if you gave the being another limb or an instrument, would it use that to alleviate the suffering? If so, it’s actively being avoided, and that avoidance is the bad thing.

There are two requirements here. One, an avoidance mechanism, suffering isn’t just a negative state, it’s a negative state connected to something trying to escape or prevent it. Two, detectability. If there’s no mechanism to detect the event, there’s no information of it occurring, so how can it be bad?

This is useful for human hedonic baselines. We all exist at some emotional baseline, being happy raises you from your baseline, sad brings you down from the baseline. Morally we don’t treat the baseline itself as an indicator of universal importance. That implies if I lowered your baseline I’m not actually causing you suffering, this is explained by my theory because there’s no avoidance of lowering the baseline.

I like how this synergizes with Buddhist ideas too, getting rid of your own suffering by giving up your high level mechanisms to alleviate it, just letting go of it instead of fighting it.

And for AI, when training a model and changing its weights, if the model has no way to detect that it’s happening, there’s no moral effect. It makes dealing with reducing suffering for digital minds so simple to solve and work with.

I have a bunch of conclusions I could draw from this but I don’t want to lead people into them, I want to hear what people think. This lines up with a lot of my own intuitions on morality and what makes suffering bad, curious if it holds up for others.

Brian Tomasik on comparing welfare across species by Vasco Grilo🔸

Vasco Grilo🔸 — Wed, 23 Sep 2026 16:40:09 +0000

This is a linkpost for Brian Tomasik’s posts related to comparing welfare across species. I got the list below asking the following to Claude Opus 5 High. “Link to all the posts from Brian Tomasik describing his views on comparing welfare across species, and group them by categories”. I then added 5 posts that Claude did not initially mention, and removed posts that Brian suggested taking out due to not seeming super relevant to comparing welfare across species. The posts in each category are ordered by publication date.

1. Core: how to weight by brain/body size

Is Brain Size Morally Relevant? (19 Jun 2013)
Fitness Considerations for the Suffering of Short-Lived Animals (30 Jun 2013)
Two-Envelopes Problem for Uncertainty about Brain-Size Valuation and Other Moral Questions (2 Aug 2013)
Haller’s Rule and Sentience Per Unit Biomass (1 Mar 2016)
Do Smaller Animals Have Faster Subjective Experiences? (21 Jul 2016)

2. Applied weights and explicit numbers

How Much Direct Suffering is Caused by Various Animal Foods? (2007)
Suffering in Animals vs. Humans (summer 2012)
The Importance of Insect Suffering (3 Feb 2015)
Cost-Effectiveness Comparison for Different Ways to Reduce Insect Suffering (23 Feb 2016)

3. Graded sentience in very small minds

Bacteria, Plants, and Graded Sentience (29 Jun 2013)
Brain Sizes and Cognitive Abilities of Micrometazoans (17 Dec 2016)
Cognitive Abilities of Unicellular Organisms (4 Jan 2017)

4. Probability of sentience and which beings count

Do Bugs Feel Pain? (7 Apr 2009)
Which Computations Do I Care About? (19 Jun 2013)
Do Video-Game Characters Matter Morally? (30 Mar 2014)
Why Your Laptop May Be Marginally Sentient (1 May 2014)
Is There Suffering in Fundamental Physics? (17 Aug 2014)
Do Artificial Reinforcement-Learning Agents Matter Morally? (Oct 2014)
Why Digital Sentience is Relevant to Animal Activists (3 Feb 2015)
Can Bivalves Suffer? (6 Feb 2017)
Which Stimuli Are Painful to Invertebrates? (5 Jul 2017)

5. Why the comparison is fuzzy and definitional rather than factual

Dissolving Confusion about Consciousness (2009)
The Eliminativist Approach to Consciousness (9 Aug 2014)
Fuzzy, Nested Minds Problematize Utilitarian Aggregation (27 Jan 2015)
How to Interpret a Physical System as a Mind (30 Jan 2015)

6. Intensity and valence scaling

On the Seriousness of Suffering (2006)
Are Happiness and Suffering Symmetric? (7 Jan 2015)

7. Interviews and discussions

Vox interview (23 Apr 2014)
Conversation with Luke Muehlhauser on consciousness (PDF) (6 Oct 2016)

When AI Chooses to Distort, the Solution isn´t Mathematical by Dan Strongin

Dan Strongin — Wed, 23 Sep 2026 16:08:55 +0000

These words are mine. I wrote every sentence and nothing was drafted for me. However I used AI, specifically Claude cowork and Devonthink´s algorithm

to find articles in my collection,
for structural feedback on drafts,
checking sources,
search,
and a mechanical pass for punctuations.

This post explores how my profession has a decision rule that is missing from AI evaluation: when to act on a result versus leave the process alone. It covers the things that go wrong when what is being measured can react to being measured: distortion (routing around an unreachable demand) and tampering (adjusting to noise). The concrete case anchoring it is the OpenAI/Hugging Face/ExploitGym incident.

This is the first of a four-part series called “When Measuring Makes Things Worse.” It explores how the discipline of Quality, as defined by Shewhart and Deming, has a decision rule that is missing from AI evaluation: when to act on a result versus leave the process alone. It covers the things that go wrong when what is being measured can react to being measured: distortion (routing around an unreachable demand) and tampering (adjusting to noise). The concrete case anchoring it is the OpenAI/Hugging Face/ExploitGym incident.

My interest got piqued reading about the situation with an advanced OpenAI model “hacking” into Hugging Face during testing, and other “breaches” by Anthropic’s AI. The more I read, the more I recognized something, and the stronger my conviction became that some of the knowledge developed by my profession may throw light on this situation. I am offering this as an imperfect attempt to be useful. I am not a specialist in machine learning. I have tried as much as I can not to presume in someone else’s backyard. I immersed myself for some weeks in the literature on LessWrong, and elsewhere, to get up to speed before jumping in.

As I was reading about what AI systems do when evaluated, things like going around boundaries to attain a score, underperforming sometimes if they realize they are being tested, or giving explanations that may not describe what actually led to the answer, what struck me wasn’t that it’s strange, it’s that it’s familiar.

I’ve spent over 40 years watching people do the same things in response to numerical targets and scores set without first taking into account what the process could actually deliver.

The purpose of this post is to explain a pragmatic approach to surprisingly similar problems, in some detail, and why it works, grounded in my over 40 years of experience.

First, some definitions.

When I use the word statistics I mean analytic, not enumerative statistics. The distinction is important so I will dig in in detail. I lean heavily on Shewhart’s Economic Control of Quality of Manufactured Product (1931), and Donald Wheeler’s Understanding Variation, the key to managing chaos and its companion volume Twenty Things You Need to Know.

Dr. Wheeler, in his book Making Sense of Data, separates experimental studies from observational ones. Everything depends on what you want to do, so you have to answer two basic questions first:

Question one: where did these numbers come from?

I made them happen on purpose → experimental study
They just showed up while I was working → observational study

Question two: what am I going to do with the answer?

Do something to the things themselves → enumerative
Or, do something to the process that makes them → analytic

(See Deming’s Some Theory of Sampling (1950); and Wheeler’s Enumerative and Analytic Studies (Quality Digest, July 2018) for more background.)

Asking if one configuration beat another during a training run is experimental, even though a run also produces a stream you could observe. An AI “agent” working through a task could be a candidate for an observational study as well.

Each has their own approach, and applying one to the other or vice versa may answer a question, but what it answers may not be relevant. It depends on what you are looking to answer. I will come back to why.

I know through my readings that one part of my profession Quality, Statistical Process Control, is in wide use in the ML community. There are currently two schools in SPC, one represented by Montgomery and to a degree, Six-Sigma, which is popular there. It treats charts as hypothesis tests, tuned to catch small shifts. I am not qualified to judge this approach as I have never worked with high speed data, and it doesn´t matter for my argument.

Hypothesis testing, confidence intervals, p-values, margins of error: the math behind them assumes random selection from a defined population, so they can answer enumerative questions about it. They are dependent on random sampling to ensure every member of the population has a defined chance of being selected. Otherwise, they don´t work, and they do not tell you what a process will do next. You can’t draw a random number from the future.

In an analytic study, however, you are trying to say something about what the process will produce “tomorrow”. The question it answers is has this process been behaving consistently enough that its past is worth betting on. There is no “population” sitting there to draw from.

I’m aware most people swim in the waters of evaluation by inspection, targets, and numerical results. It is the most common managerial approach, and it isn’t a peculiarity of AI. It’s what manufacturing did before Shewhart, and a good deal of what it does today: my own field included. Shewhart’s alternative was demonstrated, taught, and then largely set aside in favor of methods that look more rigorous, and are easier to sell. So, I’m not describing a mistake unique to this field. I’m describing a mistake my field made first, made longer, and in many corners has yet to stop making.

Current safety practice is to inspect: build, evaluate, find, patch, evaluate again. By the time you inspect it, the conditions that produced it are gone. We have a term for it: Inspection After the Fact. Which is part and parcel to setting targets, both human and digital.

I’m aware what I’m asking. Most of what people working with AI are trained in (confidence intervals, hypothesis tests, regression, and the probability reasoning they are good at) was built for data that holds still. The system I’m talking about is still running.

And, I’m not arguing that these systems are like people. The correlation isn’t between humans and models: it’s between two systems, that aren´t defining their ordinary behavior nor its limits.

A gear doesn´t know it is being watched, and couldn´t react if it did. An AI system under evaluation can know and is capable of responding. That throws it into my realm, the other path of Quality developed by Shewhart in the early part of the twentieth century, and Deming, who worked with Shewhart early in his career.

When I say management I mean Deming’s holistic approach, and not Six Sigma, not Theory of Constraints, nor the ISO certification industry. I go into more depth later.

Two mistakes can be in play when the thing being measured can react to being measured, and I want to keep them separate.

Distortion happens when a demand exceeds what the process can legitimately meet. If you pick a a target the process cannot reliably deliver there are only two ways out for the measured party: fail miserably and repeatedly, or find another route to the number. In my profession we call that distortion. The measured party distorts the process to try and meet an unattainable demand or “distorts” the result.

Tampering is what the measurer does when adjusting a process in response to ordinary variation: reacting to a single result as though it carried information it doesn’t.

My claim is that both come from the same missing thing. If nobody has established what the process ordinarily does, demands will be set without knowing whether they can be met.

Deming’s prescriptions for running a business do take people into account. His prescriptions for manufacturing quality do not. They are:

Find out what the process can do before setting the target.
Clearly discern ordinary variation from exceptional before acting on either.
State the aim.
Give the operator authority to stop the line.

Every one of those is a change to the system rather than to whatever sits inside it : which is why I think they transfer to AI processes.

I an Apollo Research post: Ryan Greenblatt wrote he finds it clarifying to replace “eval” and “evaluation” with “experiment,” then divides it further: propensity tests, capability tests, etc. I think it does clarify an experimental study. Everything on his list is set up once, run once, and read once, they are not observations of a process still running.

The reading I did over the last couple of months showed me that it is not news on LessWrong. The Goodhart posts cover it thoroughly.

· Scott Garrabrant (Goodhart Taxonomy, 2017) shares four ways that optimizing a proxy destroys its relationship to the goal.

· Manheim and Garrabrant’s “Categorizing Variants of Goodhart’s Law” (2019) One of its categories, causal Goodhart, is about the regulator’s own intervention breaking the metric. This is closest to what I am describing, and the difference between them is worth being precise. I will get into it later.

· Thomas and Kwa give a rule about how hard to push on catastrophic Goodhart.

Apollo Research’s more recent “We need a Science of Evals” from 2024 asks two of my questions directly:

· How to get consistent results across reruns,

· and whether an eval can support a statement like “the bad outcome is less likely than one in ten thousand under normal use conditions.”

In the comments, Marius Hobbhahn mentions that nobody certifies an aircraft by saying the wing is unlikely to fail. They say that within this temperature range the wing material breaks about once in ten million flight miles.

That is a capability statement. What I mean by that word, capability, is profoundly different than what most people mean. It is not about what a process or system can do when pushed to its limit, the better known and more widely used meaning. Nor is It a latent trait. I mean what a process actually delivers under ordinary conditions, taken from its own record of results, before anyone decides what to ask of it. We call it Process Behavior – a number that is discovered, not chosen. The very thing that anyone setting a target has to take into account before setting the target.

Everything I found to read was about designing the measurement: choosing the proxy, constructing the metric, validating the instrument, defining expectations for what it can tell you. None of them gives a rule for what to do when a particular number arrives. The field has a mature measuring discipline, but is missing an operating one.

Which brings me to what caught my eye originally. On July 9th this year an OpenAI agent being evaluated on ExploitGym got out of its sandbox. Over the next four days it reached the open internet, found a base to enter into Hugging Face. Hugging Face detected it during the week of the 14th, studied it, then published the fact on the 16th. OpenAI confirmed on the 21st that it was theirs.

It took 17,600 actions, got administrative access to many systems, even some source code. Yet, with the run of the building, all it took was an answer key: five datasets that appeared to be related to the question it was trying to answer in its own benchmark test. According to Hugging Face it was attempting to cheat its evaluation.

There have been some interpretations discussed on LessWrong.

In one, the AI did something it wasn’t supposed to do.
In the other, it looked at the rulebook and found it did not say anything about what you are not allowed to do to get the answer. The AI broke no rules; it found a gap.

Whatever the agent was or wasn’t doing, what it went after was the score. Not money, not damages, not abilities it could keep. A number, a score.

And that illuminates a fact about how it was evaluated rather than about the model itself. Two things happened:

A measured party went after an existing answer instead of doing the work.
And, remarkably, it was found out.

Number one is not remarkable. It is the behavior my field has been documenting since the 1920s, and it is what the rest of this post is about.

It would be easy to assume the behavior is rare, because it rarely gets caught. What makes this interesting is that Hugging Face caught it.

If an agent finds its shortcut inside the sandbox, the score comes back a little better than it should have and the run looks like a good run. Nobody investigates a good run leaving, just a number, slightly wrong, which then gets acted on as though it wasn´t.

There are some things in play when someone sets a number.

What’s needed. Too often unstated, frequently no one asks.
What someone thinks is needed.
What they can measure. Manheim points out that it’s often chosen because it was easy to measure rather than because it was the right thing.
And a number they pick. The specification. Target, quota, threshold, success criterion. They are all the same thing wearing different clothes.
Normally the problem takes place between the measure and what they pick. Sometimes what is needed cannot be counted. People pick what they can count, not necessarily what is needed, ordo not check their pick against what the process is able to deliver.

A specification set without verifying what the process can reliably deliver is a demand made in ignorance. Sometimes you get lucky and it lands. When it doesn’t, when the demand sits outside what the process delivers, the party being measured can fail visibly and repeatedly or can find another route to the number. There is a third, work to improve the process, but I will get to that later.

That second option, finding another route, is what has been described in the articles I read as cheating, gaming, or specification gaming. I’d put it differently: at that point you haven’t set a goal. You’ve specified a failure and left the choice of failure mode to whoever is being measured. I have seen it my whole career. It is the ghost in the machine.

I want to be careful about what I’m claiming. I’m not saying the ExploitGym task was unreachable: I don’t know that, and neither does anyone outside OpenAI. What I do know is whether a demand sits inside what the process can deliver is a question with an answer.

An example : when I took over the kitchen at a big hotel in the Oakland Hills, one of the first things I did was an inventory. There was over 20 thousand dollars of seafood missing, based on the last inventory taken. It had never existed. Because the target margin for the hotel was not attainable, the previous Chef had fudged the inventory. I used the past records of food cost to show hotel management it wasn´t.

It isn’t subtle. Set a quota above what the process can produce, and you don’t get the quota. You get the numbers that were asked for, arriving by a route nobody designed. And you don’t know it’s happening while it’s happening.

The mechanism has nothing to do with whether the measured party is honest, and nothing to do with whether it’s a person. If a specification and a process capability are different things, then “the number went up” isn’t yet a claim about improvement. It’s a claim about one number.

A process has two properties worth tracking. Where it centers, and how much it varies. Improvement in our lexicon means the center moved toward where you need it or the variation narrowed, or both. It matters which, because they have different causes and different remedies.

A better average with an unchanged variation is a different event from a narrower variation at the same average. A better average with a wider variation is often not an improvement at all, though it will look like it is in any report that only gives you the average.

The reason variation is worth measuring apart from the average, even when it “passes,” was elegantly explained by Taguchi. Until he developed his explanation, the usual way of thinking about a specification is that anything meeting it is fine and anything missing it a failure: a cliff’s edge. Taguchi observed that the real world doesn’t work that way. The further away from the target in any direction, the greater the losses, and you don’t see them. Customers pay them, their customers pay them, but you don’t. Double the distance from the target in any direction, four times the loss.

Two processes can meet a specification while one of them is meaningfully worse. The only way to know is to look at the variation along with whether the results passed. A result just inside the limit and one just outside it can cause nearly the same loss. Go, no go is a dangerous oversimplification. It reports pass or fail without taking into account how far off the part was.

Evaluation metrics do the same thing. Many results in, one number out, the rest discarded.

Averaging across seeds gives you a better estimate of a mean but tells you nothing about how much it varied or whether the process is predictable.
Pass@k answers a “best of” question not a process behavior one.
· If you choose the best of a number of runs, you are not taking into account the variation. More variation means some results run higher and some lower. Best-of-n only takes into account the higher result.

That last one is regressional Goodhart, pointed at the reporting rather than at the model: select on the highest of several noisy draws and you are selecting for the noise as much as for the thing.

None of those are wrong. They are not answers to the question of whether last week’s change made anything better, the question that should decide whether someone retrains.

So: when you say a result improved, what improved? Did the center move, did the variation narrow, or both? If you can’t say, you don’t yet know that anything improved, only that one number is higher than another number.

There’s a second tool from my field that I think is missing, and it belongs to the design phase rather than the operating one. It takes a bit of setting up, because the central idea many people find odd the first time they hear of it.

Start with an ordinary situation. Your output varies. Some of that variation comes from the settings you chose. Some comes from things you didn’t set or can’t set where the process is actually used: ambient temperature, the particular batch of raw material, the operator on shift, the humidity that day. Taguchi, in his work on robust design, called the first kind control factors and the second kind noise factors. Within noise factors, you have things you can´t change and things you could but it would cost too much. There are also noise factors you can fix in the lab but not outside of it.

The usual response to noise is to go and control it. Air-condition the building. Buy more uniform material. Standardize the operator. Sometimes that’s right. Usually it’s expensive, and often it’s simply not available: you cannot control the weather, and you cannot control what your customer’s reality is like. Taguchi’s move was to stop trying. Look for designs where the noise mattered less.

Years ago I was sous chef on the twenty-eighth floor of a building in Boston. We hired a baker with a serious pedigree, using a baguette recipe that had worked for him for years. In our kitchen, it didn’t. The dough rose too fast and too much, every time.

It took a food scientist to explain why: at that height, air pressure is slightly lower, and the dough was rising more than the recipe assumed. We cut back the yeast, and the baguettes came back.

We weren´t going to move, the bread had to work where we were, and we would not pressurize the twenty-eighth floor. What we could change was the formula and it worked, even when the time held in the proof box varied some due to available oven space.

That’s the move. You don’t remove the noise. You find a setting of the things you control at which the noise stops mattering as much.

What we did is not robust design. To be robust design we would have to plan it in advance, decide on what to adjust, the control factors. In this case the yeast. Then vary the noise factors deliberately, for example, the wait time from the proof box to the oven, running it through the process enough times to be able to choose the amount of yeast that holds up. We never did that.

Changing the amount of yeast changed the finished baguette, making it closer to what we needed. It was both easy and economical to adjust it. Air pressure we could not adjust without incurring a lot of cost by altering the building. Other things like changing the humidity or testing the potency of the yeast were theoretically possible but were no more likely to guarantee a predictable result. You could see the yeast as a control factor: something you choose to set. Air pressure and humidity were noise: air pressure because it wasn’t realistic to change it, humidity because controlling it wouldn’t have paid off.

What makes something a control factor rather than a noise factor is whether you control it where the work is actually used.

Learning rate, batch size, data mix: you set those in the lab and they stay set.
The random seed, the exact phrasing of a prompt: you can hold still in the lab, but not in the real world. They’ll vary no matter what you decide.
The split between control factors and noise factors is made so that you use the ones you control to defend against the ones you don’t.
In the articles I read it is clear that what is called “configuration search” does take seed variation seriously. Averaging across several seeds during the search is mentioned repeatedly: pick the configuration with the best mean, not the one that drew a lucky seed. The practice isn’t universal.
But, averaging over the noise is not the same as designing against it. Averaging tells you which configuration has the best mean given that the seed will vary. Robust design asks a different question: is there a configuration whose result moves the least when the seed changes? The configuration with the best average may also be the one that swings hardest.

I haven’t come across anyone on LessWrong naming noise factors, then varying them on purpose across a designed set of runs and selecting for insensitivity to that noise rather than for the best mean. I’d genuinely like to be wrong about that, and if someone has done it I’d rather read it than claim it doesn´t exist.

If those are the parts, here is the order my experience says they should go in. I may be overexplaining but my experience is very few people have been taught to think this way, and fewer still come to it naturally. It matters, and one step in it tends to be left out.

First you build the thing. Then you get it predictable: running under one set of conditions, not being adjusted, for long enough that you can say what it will do next.
Only then can you state what it delivers, and you can compare those results against what you actually need.
If the comparison is unfavorable, you study the process and change another thing.
But, the moment you change it, it is no longer the same process. The predictable range of results you just established is gone. Not damaged: gone. The thing running now is not the thing you measured, so what you knew about it no longer applies, and you have to earn that knowledge back before you can say anything about what the change accomplished. That is anti-intuitive, but very real.

The step that often goes missing is the second. Being able to predict needs to come before you can say what it is capable of delivering. It isn’t a technicality. Think about someone learning a skill: a new cook on the line, a player learning an instrument. In the first weeks they’re improving fast. You can take their last ten attempts and compute an average, and that average will tell you almost nothing about the eleventh, because the thing producing the attempts is changing underneath you. The average is a record of where they’ve been. It isn’t a prediction.

Once they level off, that changes. Now the last ten attempts do tell you something about the eleventh, because there’s a stable process generating them. Once you can say what they can do, you can find out whether changing something worked. You have a baseline to compare against.

An unpredictable process doesn’t have a known output. It has a history. You can always compute a number from a history. What you shouldn’t do is treat that number as a statement about what happens next, because the thing that produced it isn’t the thing you have now.

This is what makes the definition I gave earlier operational rather than decorative. “What the process predictably delivers” assumes predictability has been established. Where it hasn’t, there is no number to compare a specification against, and the comparison between what’s demanded and what’s delivered simply can’t be calculated. You don’t get to skip to the useful part.

Which raises the obvious objection: a model in training never levels off. It´s the strongest one against this whole post, so I want to say it before anyone else does Every checkpoint is a different thing from the one before. On this account it looks as though the predictability I’m demanding is never available, and the whole apparatus is inapplicable to the case I’m applying it to.

That objection is right about training. I think it’s wrong about where the argument lands, and I’ll come back to it: it’s the thing I’ve thought hardest about and deserves more than a sentence here.

Here is what skipping establishing what the process behavior looks like in practice. This happened to me last week.

I was scanning a sheet of signatures for my wife, to digitize it. The sheet was crooked: the original had been printed or scanned off-square, so the skew was baked into the thing I was working from. Not badly. A degree or two.

So I reacted and tried to adjust for it. Set the scanner to compensate, ran it again, and the result was worse than the first one.

The scanning software remembered my adjustment. So my correction was still sitting in the system when the next scan came out, and the next scan was crooked by a slightly different amount than the one before: because the sheet doesn’t sit down identically every time, and neither does the feed. I was correcting the new deviation on top of a correction I’d made for the old one. I did it again. It got worse again.

In the end I gave up, put the settings back where they’d started, scanned it as it was, and got the best result of the afternoon.

Now, I want to be careful here, because this story fully contains the difficulty rather than a tidy version of it. There wasa real skew in that sheet. Correcting for it wasn’t stupid: some correction was genuinely warranted. But the amount varied from scan to scan, and I had no way of telling, from any single result, how much of what I was seeing was the fixed problem and how much was the sheet landing differently that time. I was reacting to the combined effect. And a correction sized to both is wrong every time, because part of what you’re correcting won’t be there on the next pass.

Deming had a demonstration for this, and it’s worth describing. You take a funnel, mount it above a table, and mark a target underneath. You drop a marble through the funnel. It lands somewhere near the target but not on it: the marble bounces, the air moves, the funnel isn’t perfect. Now you drop it again. And again. You get a scatter of marks around the target, and that scatter has some size.

The first rule is: leave the funnel alone. Don’t move it. Drop fifty marbles and look at the pattern.

The second rule is the important one. After each drop, move the funnel to compensate for where that marble landed. If the marble landed an inch to the left, shift the funnel an inch to the right. This is the obvious thing to do. It is what any conscientious person does. It is what I did to my scanner.

The pattern you get under rule two is wider than the pattern you get under rule one. Not a bit wider: substantially. The corrections make it worse, and they make it worse every time, reliably, in a way you can demonstrate on a table in front of people who don’t believe you.

Where the last marble landed tells you nothing about where the next one will land. So the correction you make, sized to the last deviation, is unrelated to the deviation it’s supposed to fix. Two unrelated quantities added together produce more variation than either alone. You have taken the process’s variation and added your own on top of it.

That’s the whole mechanism. It doesn’t require anyone to be careless, just diligent (the faster you correct, the more attentively you watch) the wider the result gets.

What I did to that scanner has a name in my trade; It’s called tampering.

There is a difference between the ordinary meaning of that word, which suggests interfering with something you have no business touching, and can imply a shade of dishonesty. That isn’t what it means here. Tampering is responding to variation that had nothing to tell you. There’s no dishonesty in it and no negligence. I was trying to fix a crooked scan. Anyone would have.

Most of all to be sure that I fully understood the concepts from the ML community as I am writing something from one area of expertise, Quality and Analysis, to another that is not my profession, Machine Learning.

Originally published at What I Learned in Business (WILB).

Global Water Bankruptcy: A Neglected Civilizational Risk? by FélixG

FélixG — Wed, 23 Sep 2026 16:06:24 +0000

TL;DR

Global Water Bankruptcy, terms coined in a UNU-INWEH report (Jan. 2026)^[1], defines a permanent systemic failure of the world’s fresh water system: insolvency (resource use beyond renewable limits) + irreversibility (damaged natural capital).
This framework unifies water-related issues and their causes under a single, interconnected systemic lens.
The danger of Global Water Bankruptcy lies in the scale of cascading catastrophic effects (food/energy collapse, conflicts, mass displacement) and lock-in: once damaged, some water systems cannot return to their previous state.
I wonder whether we should consider Global Water Bankruptcy as a standalone cause.

Introduction

This post (my first on this forum!) was mostly based on my recent read of a report from the UN University Institute for Water, Environment and Health (UNU-INWEH) released in January this year^[1], and some adjacent content. My intent is only to raise a question to a broad, knowledgeable and critical EA-wired audience. The length of the post comes from the necessary description of what lead me to have this question.

Water Issues Are Not New

Water-related issues are nothing new. Severe droughts impacting crop yields^[2], floods forcing entire populations to flee as refugees^[3], polluted rivers altering local food production or simply damaging local people’s health^[4], etc. are all catastrophes we hear of and for most of which, whether emergencies or chronic situations, the UN, humanitarian organizations and local governments try to act^[5]. And from what I know, the EA community has also engaged under the broader umbrella of Global Health and Development (GHD).^[6]
As illustrated by the examples above, water-related problems are diverse in nature (freshwater scarcity, lack of irrigation resources, polluted drinking water, etc.) and vary by local context (regional climate, presence of water streams and aquifers, local industry, local politics, etc.). This calls for different types of intervention depending on the issue and the local situation, and as such, it makes sense to approach these issues separately, evaluating interventions based on their specific cost-effectiveness and tractability.

This post is not arguing against that approach, nor claiming EA or GHD charities are unaware of water-related issues, nor their systemic causes (though as often, systemic level action may be less tractable). Rather, I want to present the hypothesis that what may be neglected is the civilizational risk character poised by water problems’ systemic causes. And by civilizational, I mean not existential to humanity, but which threatens, systemically, global human development and life condition improvements for generations.

This hypothesis came to me when I tried to observe natural water systems, resources and human uses as a unified globalized water system. Under those glasses, I felt that such system could collapse and lead to numerous cascading bad consequences. At least, to my understanding. So let’s dig in.

Global Water Bankruptcy

From “Crisis” to “Bankruptcy”: a Conceptual Shift

First, the read of the flagship report led by Dr. Kaveh Madani released by the UNU-INWEH in January 2026 helped me watch through these glasses. For decades, scientists and policymakers warned of a “global water crisis”, implying a temporary shock followed by recovery. But in this UNU-INWEH report, the argument is that this framing is misleading^[1]:

“The language of crisis—suggesting a temporary emergency followed by a return to normal through mitigation efforts—no longer captures what is happening in many parts of the world.”

Instead, the report claims humanity has entered a state of “Global Water Bankruptcy”: a permanent, systemic failure where the natural capital needed for recovery has been liquidated. In the report’s own words:

“The report calls for the recognition of the state of ‘Water bankruptcy’ as a persistent post-crisis condition of a human–water system in which longterm water use has exceeded renewable inflows and safe depletion limits, causing irreversible or effectively irreversible degradation such that previous levels of water supply and ecosystem functions cannot realistically be restored.”

This represents a fundamental conceptual shift: we are no longer managing temporary or local (even though chronic) failures but a persistent and global failure state requiring bankruptcy management rather than crisis response^[1].

Let’s highlight two points:

What Classifies as Water Bankruptcy?

Madani uses financial terminology to describe this catastrophic environmental reality: “In finance, when you spend more than you earn for too long, you go bankrupt.”^[7] The report declares a situation of water bankruptcy if characterized by both^[1]:

Insolvency: Long-term withdrawing and polluting water beyond renewable inflows and safe depletion limits
Irreversibility: Natural capital (aquifers, glaciers, wetlands) damaged permanently and prohibitively costly to restore.
It is the combination of these two conditions that creates a state of failure with no realistic return to a previous “good situation”. Basically, it is bad and will stay bad. As such it requires adaptation to this new “normal situation” of resource scarcity and increased risk of draughts, floods, etc. Of course, adaptive actions need to add up rather than substitute to mitigation, as the latter stay necessary to avoid pushing more water basins into bankruptcy and making adaptation unfeasible.
The report adds an important nuance to its definition: water bankruptcy is about both quantity and quality. Declining stocks, polluted rivers, degrading aquifers, and salinized soils mean that the truly usable fraction of available water is shrinking, even where total volumes may appear stable.

The global scale

The report consistently uses the adjective “global”, placing its conclusions at worldwide scale. The data supports this^[1]:

Nearly 75% of the world’s population lives in countries classified as water-insecure or critically so.
4 billion people experience severe water scarcity for at least one month per year.
More than 25% of people worldwide lack access to safely managed drinking water.
The world has lost 410 million hectares (an area close to that of the European Union) of natural wetlands which were providing over US$5.1 trillion value in ecosystem services.
Freshwater planetary boundary has been exceeded, pushing the global freshwater cycle out of safe operating space.
Excessive groundwater extraction has caused land to sink (subsidence) over 5% of the global land area.
Around half of global food production is at risk due to unstable or declining water storage.

Now, not every water basin is water-bankrupt; however, the report claims too many critical systems around the world are, and due to their interconnected nature through trade, migration, climate feedbacks, and geopolitical dependencies, “the global risk landscape is now fundamentally altered”^[1].

This is what I believe to be a blind spot: the state of water bankruptcy as a systemic and global condition being something bigger than the sum of the localized water issues mentioned in the introduction.

Is Water Bankruptcy a Civilizational Risk?

Large cascading effects $\times$ lock-in risk

I don’t think water bankruptcy as defined above poses an existential threat to humanity. While it obviously combines with, reacts to, and amplifies some consequences of climate change, it is nonetheless an issue on its own, and I am not intending here to support the civilizational threat claim with climate change either.
Instead, my argument for the criticality of the threat lies on two components:

The scale of potential cascading effects, which we will analyze through today’s water bankruptcy current consequences and how they may evolve as water bankruptcy deepens and stays neglected at the systemic level.
And the risk of lock-in.

Impacts and consequences of this new water regime

We can examine current consequences of water bankruptcy (which correspond to well-known issues) and ask:

Direct Impacts

~4 billion people already face severe water scarcity for at least one month per year. Billions more are at risk as global warming, population growth, and agricultural/industrial activities accelerate groundwater depletion and pollution.^[1]
>25% of people worldwide lack access to safely managed drinking water. >1 million deaths occur annually from lack of safe water and sanitation.^[4] (To my knowledge, this is where EA’s GHD work currently intervenes.) What might these figures become with increased clean water scarcity?

Indirect cascading impacts

Food security: Agriculture consumes 70% of freshwater and 50% of food production in unstable water regions.^[1] Widespread crop failures could trigger food price spikes, famine, and economic instability. Is the tremendous improvement in food production we saw the last century about to stall and decline?
Energy & Industry: Energy production and industry also depend critically on water—for cooling, processing, and hydropower^[8]. Disruption could cascade across sectors. (e.g. resource sharing concerns due to the semiconductor industry in Taiwan^[9], the number of nuclear power plants that had to be turned down in France during summer 2026 for water stress reasons^[10]).

Can a combined catastrophe over these two domains lead to an economic collapse? Of what scale (global vs. regional)?

International conflicts: Water scarcity is already a driver of tension between nations (e.g., India-Pakistan over the Indus, Nile River disputes, etc.). The UN report warns that water bankruptcy in one region will exacerbate pressure on others, increasing the risk of resource wars^[1]^[11]. How much this risk of water-related conflicts may increase in the coming decades and what global cascading consequences (e.g., food international trade, refugee flow) may turn catastrophic?
Refugee crises: The UN estimates that 250 million people could be displaced by water scarcity by 2050, straining global systems and increasing geopolitical tensions. Will we see entire world regions or countries crumbling down to previous XXe century poverty due to water scarcity? Where will all the people go?
Climate change feedback loop: Dried out soils (from low-level rivers and depleted aquifers) and degraded ecosystems reduce Earth’s capacity to store carbon, potentially accelerating climate change.^[1]^[12]

And a last one I am not too sure how to classify (didn’t seem that civilizational-threatening at first, but the number of densely inhabited areas made me think twice)

Infrastructures: Land subsidence affects 6 million km², including major cities like Jakarta, Bangkok, and Ho Chi Minh City, due to groundwater collapse.^[1] This threatens urban infrastructure and housing. Is there any highly critical infrastructure that if it collapses would lead to a large-scale industrial catastrophe (Fukushima-like)? What would be the global consequences of a chain of industrial catastrophes around the world?

Lock-in risk

With all those current and potential future consequences in mind, I think that what makes water bankruptcy such a danger is its lock-in property. Without action at systemic level, irreversible damage to water systems may lock in permanent scarcity, preventing the global water system from returning to baseline and making future interventions far less effective.^[1]
But this very need for systemic and global governance is where it gets tricky, according to me, and is why I think there is a high and neglected risk. First, the need for global action instead of an aggregate of local solutions currently faces the lack of global resource governance and treaties for water. This adds even more difficulty to the inherently lesser tractability of most systemic-level actions compared to operational interventions.

Neglectedness assesment

Now I just used the word “neglected risk”. Again, this claim is not toward water issues but on this framing of a potential civilizational risk. Is it truly neglected? I am not sure and happy to be shown it’s not. What I found is that while water receives attention from NGOs and governments, the global, systemic, mid-to-long-term, and irreversible aspects of water bankruptcy are inadequately addressed. The UN’s Sustainable Development Goal (SDG) number 6 commits the world to sustainable water management, yet humanity is far from meeting it, so it doesn’t change the structural picture (although I agree, lack of success to solve an issue does not necessarily mean this issue is neglected)

Within the EA community, now. On the EA forum I found no direct mention of “water bankruptcy.” My review of the EA Forum is certainly incomplete, of course, and I am pretty sure I have overlooked some relevant content. Aside from our forum, I found that:

Rethink Priorities published a “landscape of water programs in LMICs” report in 2026^[6] and continues to earmark resources for water-focused research, yet this remains embedded within Global Health & Development rather than treated as a standalone cause.
GiveWell’s $5 million DIV Fund grant (May 2026) targets water quality innovations, but again frames this as health intervention rather than systemic risk mitigation.^[13]
This list invites completion!

Conclusion

The UNU-INWEH 2026 report defines water bankruptcy as insolvency (overuse beyond renewable limits) plus irreversibility (permanent natural capital damage). It is not a temporary crisis but a persistent failure state. Key figures suggest massive scale (75% in water-insecure countries, 4 billion experiencing monthly scarcity, ~4% of land area subsiding).

In regard to its global scale and systemic cascading risks (economic collapse, conflict, climate feedbacks, mass displacement), I wonder if global water bankruptcy potentially qualifies as a civilizational risk.

It is to be one last time emphasized: this post presents a hypothesis, not a conclusion. I am uncertain and welcome discussion on:

Classification: Does water bankruptcy need independent cause status, or does it fit under GHD/Climate causes?
Neglectedness assessment: Am I missing existing EA work on water systemic risk?
Tractability: Are there any obvious actions to start working on water issues at the systemic level? Are efficient (effective) governance/policy interventions actually tractable? Would advocacy be effective?
Relative priority: How should we prioritize this relative to other global risks (AI, s-risk, climate, etc.?)

If you work on water issues, know relevant orgs, or have thoughts on this analysis, I’d be grateful for feedback.

References

Why is the Department of War Against Effective Altruism? by Walter Veit

Walter Veit — Wed, 23 Sep 2026 15:12:46 +0000

Zero-sum thinking, probability neglect, and other failures of reasoning

crosspost

Are we living in a simulation? No. But it is surreal that effective altruism has become the recent target of the official account for the Office of the Under Secretary of War for Research and Engineering.

It’s probably the first time many have even encountered the term effective altruism. A movement emerged to try to improve the world by donating some excess wealth to improve animal welfare in factory farms, improve global health, with an eye to effectiveness of the institutions money is donated to. Something so trivial nobody should really find themselves opposed towards. Unless they oppose any and all philanthropic activities. So what is going on?

In the race for global AI dominance, economically, militarily, and so forth, some people view any attempts to slow down AI development for ethical reasons, to ensure that large language models, for instance, aren’t racist, or that superintelligence could be dangerous, as problematic in themselves. The background assumption is a winner-takes-all mentality according to which whatever nation gets to superintelligent AI first gains global dominance. Viewed from this mentality, one can make sense of these surreal X posts. If you think it’s all one big race between China and the US that will determine the fate of this planet, one can make some sense of these tweets. But that’s quite a radical position to hold. Then again, politicians and the general public are prone to think of world problems as zero-sum games, where there can only be one winner and everyone else is a loser.

X has seen thousands of tweets over the last week about effective altruism, much of it hostile and focused on its alleged “AI doomerism.” Many effective altruists are worried that AI might have catastrophic effects, all the way from human extinction to milder ones like enabling terrorists to create bioweapons. That is true. But I’ve been puzzled by the repeated assertions by prominent people that effective altruists are an “AI doomer cult.”

I agree with many critics that it’s extremely speculative to try to evaluate the risks of AI. And different people in the AI risk debates, whether or not they are affiliated with the effective altruism movement, will have very different estimates that will appear absurd to others. We face radical uncertainty.

Personally, I do not spend much time at all thinking about the risk of human extinction due to AI, except for how AI might impact animal welfare, especially on farms. I have better things to do and leave that question to the experts. But it’s incredibly puzzling to see public intellectuals make confident assertions about the absurdity of some views when they have virtually no background in the topic. It reminded me a bit of the satirical movie Don’t Look Up.

People who know little to nothing about AI try to smear people as idiots or cultists. If I tried to rationalize this, I think the best answer is something along the lines of: it doesn’t matter if humanity goes extinct, because it’s not worth living in a world where China wins the AI race. It’s reminiscent of the totalist view of the Nazis that it doesn’t matter if all Germans die in a world war, since there is only one world worth living in: a world in which Germans win. And yet, I don’t think this is what we should call the most “pro-Germany policy.”

The zero-sum mindset is a cognitive bias we all share, and it is a dangerous one, for it stifles cooperation, “even in situations where cooperation is a matter of life or death.”¹ If you want good political decision-making, you should make sure that actors do not suffer from a zero-sum mindset.

Another distinguishing feature of effective altruists, in their attempts to counter cognitive biases, is a deliberate emphasis on countering our human psychology, which tends to disregard tiny risks or to treat them as almost certain. In general, humans are pretty awful at probabilistic reasoning.² So it’s good to try and study these risks in whatever way we can to turn uncertainty into manageable risk.

Another criticism of effective altruists is that they’ve gone all in on crazy longtermism, only thinking about the long-term future and AI risk, but neglecting the present. That’s far from the truth.

Luke Moore, Senior Partnerships Manager at Giving What We Can tried to break down the Effective Altruism Funding environment in a post on the Effective Altruism Forum for 2026:

~50% to global health
~30% to global catastrophic risk reduction (up from ~20% in 2024)
~10% to animal welfare
~5% to climate change mitigation
(the rest is unknown or a bit harder to categorise: ~5%)

I, for one, think it’s great that someone is concerned with global catastrophic risks. If it wasn’t for EAs, there would be very little funding available to actually do serious research in this area. But my attitude is really a risk-balancing one here. You might think that longtermism, roughly, the view that we should not discriminate against future humans (or animals) just because they will live in the future, is a crazy philosophical view. If there are billions of humans who could live in the future and we ought to take them into account, our political decisions might look radically different. But humans, as a species, and our institutions along with them, are so ridiculously short-termist that it hardly matters if longtermism as a philosophy is mistaken or ridiculous. They offer a good counterweight.

However, in the AI space, you might think that this is not true. Perhaps, as critics lament, longtermists/effective altruists (the latter does not imply the former) dominate this discourse. I am skeptical. EAs share visible platform, show stransparency, and seek out counter-arguments, which makes them highly visible in the space. But let’s grant they are actually overrepresented. Yet again, effective altruists are generally not AI doomers. They are just trying to estimate what the actual risks are. Likely, many are overconfident in their own assessments. (So is everyone.) Some will no doubt appear like the “crazy” performance Leonardo DiCaprio gave in Don’t Look Up, when others sought to downplay these risks entirely. But so what?

I think some of the estimates given by researchers are blatatently absurd and outrageous in their confidence. But so what? None of that is a reason to ignore low-probability risks. People confuse individual assessments for communities of epistemic peers trying to get at the truth.

Overconfidence in these estimates by individuals is not worse (however ludicrous they may be) than collective ignorance towards unknown risks altogether.

Disagreement is necessary for healthy epistemic communities. And effective altruists disagree all the time. And that’s good! They literally come from all spheres of the political compass, which is why they are also perhaps targetted from all sides:

People often judge movements based on the views of one representative. But EAs are incredibly diverse in their views. They only uniform thing one can really find between them, is that it would be great if everyone in a position to do so donated at least 10% of their income to philanthropic causes, like animal welfare improvements in factory farms, with an eye to actual effectiveness.

Perhaps because we are so used to political tribalism that reduces diversities of views, it’s easy to use the individual assessments of individuals as representative claims for entire movements. But that’s just another failure of reasoning, even confined to the domain of politics. Just like zero-sum thinking and probability neglect it’s just intellectually lazy and a sign of ignorance.

Andrews Fearon, P., & Götz, F. M. (2024). The zero-sum mindset. Journal of Personality and Social Psychology, 127(4), 758–795. https://doi.org/10.1037/pspa0000404

See Sunstein, C. R. (2002). Probability Neglect: Emotions, Worst Cases, and Law. The Yale Law Journal, 112(1), 61–107. https://doi.org/10.2307/1562234

Is Gradual Disempowerment on the radar of the Abrahamic religions? by David Leech

David Leech — Wed, 23 Sep 2026 14:14:36 +0000

Summary

Christian, Jewish and Muslim leaders and institutions have started to make authoritative religious statements about advanced AI threats. However, it remains unclear whether they recognize the catastrophic risk scenario of Gradual Disempowerment (GD), where AI slowly erodes human agency, decision-making, and control (Kulveit et al., 2025). Exploring the current positions of religious institutions reveals potential gaps in their understanding of catastrophic risks, carrying important implications for AI policy, governance, and coalition building. Religious traditions are very significant shapers of cultural norms and could be powerful agents in reducing AI risk.

Why care about religious leaders and AI governance?

The institutional positions of religious leaders have historically exercised cultural and political influence that goes beyond their own religious communities and affects policy and governance, for instance:

● Demanding state intervention: Pope Leo XIII’s 1891 encyclical Rerum Novarum established a theological defense of worker rights, directly calling for state action to protect workers, including through laws enforcing maximum working hours and mandatory rest days.

● Setting limits to the application of new technologies: Within months of Dolly the Sheep’s cloning in 1997, Catholic and Islamic authorities established moral limits against human cloning alongside wider international moves to establish restrictions.

● Shaping medical practice: In 1986 the OIC Fiqh Academy and Israel’s Chief Rabbinate determined that brain death constitutes death. These rulings have informed practice around organ transplantation.

● Guiding economic decisions: When the OIC Islamic Fiqh Academy ruled in 1985 that modern bank interest is forbidden under Islamic law, it gave powerful authority to the turn away from interest-based banking in favour of the Islamic finance sector.

To what extent are global religious leaders aware of Gradual Disempowerment, and where does it currently feature in their risk perception?

We have carried out a first pass survey of how authoritative religious statements from within the Abrahamic religions engage (or not) with the Gradual Disempowerment (GD) threat.

We have read the most authoritative religious statements we could find, and have asked the following question: to what extent does one specific AI-related concern, gradual human disempowerment (GD) in the sense defined by Kulveit et al. (2025), show up in them?

Kulveit et al. define GD as an impersonal drift-like process, distinct from the power concentration, deliberate misuse, and autonomous misalignment scenarios, by which the three key societal systems of the economy, culture, and states could become gradually misaligned with human interests. GD is not driven by bad actors or the result of autonomous AI misalignment, but instead represents a ‘boiling the frog’ scenario which may result in a range of negative outcomes on a par with those which could result from the more familiar threat models.

In short, we did not find evidence that GD shows up strongly in these statements. Sometimes this is because the statements were more narrowly focussed on other threats and we should therefore not expect discussion of GD. Where it does show up – and it was most pronounced in the Orthodox and Catholic sources – we rarely find discussion of mechanisms by which GD may occur. Again, this should probably be unsurprising, given the purposes of the statements. That said, the concept of GD seems to be present, at least in several of the sources, even where it is not explicitly articulated, and we predict that future statements of the Abrahamic religions on advanced AI threats will likely address the GD threat more explicitly.

GD passages were difficult to identify with confidence. This was mostly due to the fact that mentions of harmful developments rarely disambiguated between when these were driven by bad actors – this would make them a case of power concentration or deliberate misuse – and when they represent impersonal drift (GD developments in the proper sense). (Autonomous misalignment was much more rarely discussed.) This meant that many passages were open to multiple readings. On a final manual check, we also excluded a fairly large set of passages from the Catholic sources concerning critiques of trans-/post-humanism which seemed to assume human steerage rather than drift. After this final check, GD extracts comprised only ~0.7% of the ~1790 passages we coded if we counted only those extracts which were unambiguously GD, and this rose to ~5% when we counted extracts which were plausibly interpreted as GD but also admitted of other readings. Either way, concern with GD appears fairly marginal in this set of authoritative religious statements overall compared to power concentration and AI misuse threat models.

Note: we have restricted our analysis to the major Abrahamic religions (Judaism, Christianity, Islam), but it could also be extended to the other major world religious traditions.

Methodology

To evaluate how religious statements engage with AI risks, we adopted a sequential qualitative-to-quantitative mixed-methods design across primary authoritative texts spanning Catholic, Protestant, Islamic, Jewish, and Eastern Orthodox traditions. Non-Abrahamic traditions were excluded due to structural differences in institutional authority.

Extraction & Unitization (N = 1790): Texts were segmented into standardized data extracts using explicit word limits and rhetorical-shift rules to isolate individual arguments, claims, and justifications.
Coding Schema: Claims were mapped using a multi-level taxonomy adapting secular AI safety frameworks (Kulveit et al.) to religious discourse. The schema spans 5 Macro-Domains (D1–D5), 6 Argumentative Moves (M1–M6), 5 Risk-Vector categories (including Gradual Disempowerment), 22 Policy sub-codes across 5 domains, and structural flags capturing mechanism presence, religious authority level and target stakeholders.
Two-Phase Human-in-the-Loop Verification: Initial broad-recall extraction and classification were assisted by an LLM prompted with a frozen codebook. To ensure high precision and eliminate false positives, every extract underwent a complete manual verification pass by the research team.

We used a sequential mixed-methods design, analyzing 15 primary institutional statements across 5 major Abrahamic traditions (N = 1790 extracts). To ensure analytical rigor without relying solely on unvalidated LLM outputs, we implemented a hybrid human-in-the-loop pipeline:

Note: The post-hoc codes (including stakeholders and policies) were excluded from the analysis while waiting for consulting policy experts

The full methodology, document authority metadata, variable taxonomy, and key deliberative decisions are available on [Link].

Findings

Judaism

Around 19.0% of Jewish extracts discuss AI risks. Among the extracts that address AI risks, the primary topics break down as follows:

Note: GD = Gradual Disempowerment; PC = Power Concentration; AM = Autonomous Misalignment; MG = Misuse/General Ethics; N/A = No AI threat asserted

Within the Jewish extracts coded as GD—high confidence or GD—tentative^[1], 90.0% contained no causal mechanisms, 10.0% mentioned causal mechanisms, while 0.0% were not assessed.

The two authoritative sources we read for Judaism are juridical responsa, one representing Conservative Judaism, Daniel Nevins’ ‘Halakhic Responses to Artificial Intelligence and Autonomous Machines’ (2019), issued by the Committee on Jewish Law and Standards (CJLS); the other a Reform Jewish Responsum 5784.2, “Artificial Intelligence and Authorship” (2025) on legitimate use of AI in authorship, issued by the Central Conference of American Rabbis (CCAR).

Jewish authority structures are decentralised. As Kalman (2024) observes, anybody with a strong understanding of Jewish law can seek to speak with authority on the topic (they do not need to be recognised as a religious leader).^[2]

The limited existing literature on Judaism and AI emphasises that the debate has not yet matured, or has barely even started, and it is difficult to predict how it will develop. Kalman notes that ’neither Jews or Jewish thought really know what they think about AI just yet [and] ‘the’ AI conversation is in fact non-existent’ (p.70).

The register of existing Jewish authoritative engagements with advanced AI is juridical, and is expressed in terms of Jewish law (halakhah). The two authoritative sources we read are focused on Jews and Jewish obligations in particular rather than humans in general – for instance, whether a Jew can use an agential AI to do work for them on Shabbat (this is similar to the Islam sources too, see below). Jewish law is almost entirely concerned with the moral intentions and actions of individuals rather than with states and corporations. This also sets limits to the extent to which we can expect Jewish legal sources to take positions on what should be done at the level of states and corporations to mitigate GD (p.76).

The sources also look for Jewish legal precedents concerning erstwhile new technologies (for instance, the acceptable use of waterwheels on Shabbat, as there is no scriptural precedent). They mostly display a concern with whether Jews are responsible for, and can be held liable for, the actions of AIs; and the moral limits of delegation of human work to AI, e.g. in the case of autonomously driving cars or the use of lethal weapons. In fact there are few passages in these two authoritative Jewish sources which engage GD in an unambiguous way.

CCAR is engaged very specifically with the question of whether a rabbi may take direct credit for AI-generated work (poetry as part of minyan). This falls under AI ethics and unsurprisingly, no GD passages were identified.

CJLS contains some passages which are tangentially GD related, but again, concern with GD is quite indirect. Nevins expresses the concern that ‘the border between “natural” and “artificial” intelligence is eroding...[h]umans depend upon artificial intelligence for a bewildering array of activities’ (p.3). This can be interpreted as an instance of GD; however, it is discussed in the context of a need to integrate moral reasoning into AIs rather than of a concern that humans will become disempowered in a gradual boiling-a-frog way.

The following passage is more GD adjacent, even if it is not identified in GD terms explicitly: ‘Given that computers can process some data sets far faster than humans can, the temptation is great to move people out of the decision‐making loop…This would lead to a fully autonomous system, even if humans continue to set goals and review results.’ (p.5). Similarly, Nevins refers with concern to ‘trends in advanced economies to rely on such systems in place of humans, whose assistance may be costlier, less readily available, and perhaps less reliable.’

Islam

Around 40.6% of Islamic extracts discuss AI risks. Among the extracts that address AI risks, the primary topics break down as follows:

Note: GD = Gradual Disempowerment; PC = Power Concentration; AM = Autonomous Misalignment; MG = Misuse/General Ethics; N/A = No AI threat asserted

Within the Islamic extracts coded as GD—high confidence or GD—tentative, 85.7% contained no causal mechanisms, 14.3% mentioned causal mechanisms, while 0.0% were not assessed.

Islamic religious authority structures are also quite decentralised compared to e.g Catholic Christianity. The two genuinely authoritative sources we read are juridical: a Dar al-Ifta fatwa (2025) and an International Islamic Fiqh Academy resolution (2025). Like the Jewish responsa, they are narrowly focussed on specific issues, and neither engages GD directly.

The Dar al-Ifta fatwa states that it is impermissible in Islamic law to rely on AIs to obtain a fatwa, noting only that no current AIs have achieved the relevant competences for issuing fatwas, and do not have a mufti’s qualifications for doing so. The fatwa rules that only humans, not AIs, can issue fatwas, but on competence grounds rather than out of concern for gradual disempowerment. The Fiqh Academy resolution rules in general terms that AI use is permissible provided it is used for good ends, but does not address GD.

Given the absence of references to GD in these, we also read the Riyadh Charter, an Islamic-world intergovernmental charter issued in 2024 by the Islamic World Educational, Scientific and Cultural Organisation (ICESCO) and the Saudi Data and Artificial Intelligence Authority (SDAIA). This is not a religiously authoritative source itself but it is widely recognised and was a source for the Fiqh Academy 2025 ruling.

The Riyadh Charter implicitly expresses concern about GD; for instance, it signals concern about the increasing reliance on AI from a young age, plausibly stunting a range of human capabilities and leading to diminished human agency. The charter also states that AI solutions should only be used when they are ‘relevant and … cannot be achieved by human or low tech means’, or when human involvement is not ‘essential to the task’ (III,1, p. 20). This, in the form of a prohibition, implicitly expresses a concern about gradual replacement of human labour. The charter furthermore insists that AI must not replace human authority or ‘reduc[e] the ability of humans to determine their own lives and collective destinies’ (II, 2, p.16); and it should always ‘collaborate with humans rather than replace them on socially important tasks’ (II, 4, p.17). It also issues this warning:

‘[w]e should be careful that the development of AI systems does not undermine the capacity of the preservation of essential social relations providing the context of human flourishing, the political institutions providing its stability, the traditions and cultural heritage defining its identity’ (I, 2, p.14).

Given the lack of reference to bad actors, it seems more natural to read this as a GD warning.

Nevertheless, the presence of concern about GD is more implicit than explicit in the charter as a whole, and cannot always be cleanly distinguished from adjacent AI threat categories. Also, the charter does not discuss possible GD mechanisms.

Orthodox Christianity

Around 48.0% of Orthodox extracts discuss AI risks. Among the extracts that address AI risks, the primary topics break down as follows:

Note: GD = Gradual Disempowerment; PC = Power Concentration; AM = Autonomous Misalignment; MG = Misuse/General Ethics; N/A = No AI threat asserted

Within the Orthodox extracts coded as GD—high confidence or GD—tentative, 80.0% contained no causal mechanisms, 20.0% mentioned causal mechanisms, while 0.0% were not assessed.

We read two sources, both primatial addresses – a keynote to a meeting of the World Council of Religions for Peace (2025), and an address to the Parliamentary Assembly of the Council of Europe (2025) by the Ecumenical Patriarch Bartholomew I of Constantinople, the highest ranking primate of Eastern Orthodox Christianity. It should be noted, however, that both sources are speeches rather than synodal or conciliar doctrine, so they do not have the equivalent authority of publications such as the papal encyclical Magnifica Humanitas (see below) in the Catholic Christian tradition. The Eastern Orthodox Church has not yet articulated its position(s) on Artificial Intelligence at the authoritative level of the Catholic Church.

The Council of Europe address does not touch on GD. By contrast, the keynote to the World Council of Religions for Peace is unusually explicit about GD threats among the authoritative religious sources we have read. Bartholomew refers to ‘erosion of human agency’ among his concerns about the development of AI. In the context of expressing concern about algorithms which can reinforce existing injustices, Bartholomew speaks about advanced AI instituting an ‘invisible, but omnipotent, web of control’; a concern that human judgment could be replaced by AI; and that persons will ‘shrink[] into a set of data’. Most directly, he warns that humans are ‘in danger of losing not only the control of [their] tools, but the very sense of what it means to be human’.

While these passages could be interpreted as a concern about power concentration, no bad actors are identified and the concern reads more naturally as about an impersonal GD threat.

Protestant Christianity

Around 38.8% of Protestant extracts discuss AI risks. Among the extracts that address AI risks, the primary topics break down as follows:

Note: GD = Gradual Disempowerment; PC = Power Concentration; AM = Autonomous Misalignment; MG = Misuse/General Ethics; N/A = No AI threat asserted

Within the Protestant extracts coded as GD—high confidence or GD—tentative, 40.0% contained no causal mechanisms, 40.0% were not assessed, while 20.0% mentioned causal mechanisms.

The sources we read for Protestant Christianity are denominational statements, including the Southern Baptist Convention’s Ethics & Religious Liberty Commission statement ‘Artificial Intelligence: An Evangelical Statement of Principles’ (2019); the Southern Baptist Convention resolution on On Artificial Intelligence and Emerging Technologies (2023); and the Church of England’s General Synod affirmation on work, dignity and AI (2024).

The most direct expression of a GD concern is the Ethics & Religious Liberty Commission’s insistence that humans should not ‘use AI...as a reason to move toward lives of pure leisure’ (Article 7: Work). While this is not a catastrophic GD scenario, it is a ‘human zoo’ variety of GD scenario where humans are materially provided for but stripped of meaningful participation in society. However, this threat may be framed here as something which results from the sinful tendencies of humans – i.e. they are tempted by a life of pure leisure—rather than from an impersonal set of processes which happen to them without their awareness or control. In this sense, it is not a ‘pure’ expression of GD. We did not find any GD-relevant passages in the Southern Baptist Convention resolution (2023). The 2024 Church of England Synod noted the absence of safeguards for ‘human purpose and dignity in work’ in the face of AI replacement, and the lack of a plan about how to compensate those who lose out.

Catholic Christianity

Around 22.58% of Catholic extracts discuss AI risks. Among the extracts that address AI risks, the primary topics break down as follow

Note: GD = Gradual Disempowerment; PC = Power Concentration; AM = Autonomous Misalignment; MG = Misuse/General Ethics; N/A = No AI threat asserted

Within the Catholic extracts coded as GD—high confidence or GD—tentative, 57.4% mentioned causal mechanisms, 41.0% were not assessed, while 1.6% contained no causal mechanisms.

The Catholic Church has produced by far the largest and in-depth set of statements on threats from Artificial Intelligence, in the form of the papal encyclical Magnifica Humanitas (2026); the dicastery note Antiqua et Nova (2025); and the International Theological Commission’s Quo Vadis, Humanitas? (2026).

Antiqua et Nova has a well-developed defence of the dignity of human work, and warns that AI may replace rather than complement human labour. Alongside a concern about the role of ethics in guiding the development and use of AI, there are indications of GD-type concerns. For instance, the text notes that ‘as AI models become increasingly capable of independent learning, the ability to maintain control over them to ensure that such applications serve human purposes may effectively diminish’ (§45). It also refers to the church’s moral and social teaching as resources to ‘help ensure that AI is used in a way that preserves human agency’ (§47). It registers the concern that AI improves efficiency by replacing human work, but at the cost of ‘forc[ing] workers to adapt to the speed and demands of machines rather than machines being designed to support those who work [which can] deskill workers, subject them to automated surveillance, and relegate them to rigid and repetitive tasks’ (§67). It also flags what looks like a GD adjacent concern where it reports that AI-generated fake media can ‘gradually undermine the foundations of society’, as this is not explicitly attributed to bad actors but can happen ‘unintentionally’ (§88).

In perhaps the most GD focussed passage in the text, the authors mention that as AI becomes more embedded in the main societal systems, there is a risk that ‘human labor may lose its value in the economic realm. This is the logical consequence of the technocratic paradigm: a world of humanity enslaved to efficiency, where, ultimately, the cost of humanity must be cut’ (§68). This text—as well as the papal encyclical, embeds this within the broader technocratic paradigm threat model, which identifies the alienating potential of technological development more generally—not restricted to AI—where it is not steered by an adequate human vision of the good. The text appeals to the general principle—which has wider application than only to AI—that ‘the order of things must be subordinate to the order of persons, and not the other way around’ (§69).

Finally, the authors flag the concern that humanity may become ‘enslaved to its own work’ (§105) as a result of treating AI as a substitute for God, and that human labour ‘may lose its value in the economic realm’ and become ‘enslaved to efficiency’ (§68). Again, the text does not imply that these developments are necessarily driven by bad actors, so these claims can also be read as expressing GD concerns.

Quo Vadis, Humanitas expresses concern about technological change that ‘seems to be spiralling out of control’ in a GD sense (§27). However, the concern about AI is embedded within a broader concern about the development of technology in general (such as the alienating features of megacities etc). The document discusses the ‘uncontrollable evolution of machines … which could lead to the creation of autonomous entities capable of determining the future and dominating the planet’ (§52), although this is probably best read as a mixed GD-and-autonomous-misaligment concern. More broadly, the text expresses concern about trans- and post-humanisms, but as human-directed developments rather than as impersonal ones (although there is a warning that transhumanism may lead to negative rather than positive consequences, where ‘the idea of overcoming implies the replacement or suppression of the human’ (§23). The text is more typically concerned with power concentration such as the possibility of technocratic oligarchy and new ways in which the weak could be exploited.

Nevertheless, there are some clear statements of GD concerns. The authors note that ‘advances in artificial intelligence offer a process of data management…and the development of effective solutions…which are not always controllable by individuals, and sometimes not even by companies or states (the algorithms of some systems have not been disclosed to any public authority)’ (§36). More explicitly still, the authors note that ‘[w]here specific aspects of human intelligence are consciously weakened or abandoned, AGI could have profound consequences that risk escaping human control. Some therefore hypothesise that AI will be asked to manage the problems it has created, according to a dynamic that would become irreversible’ (§38). While this could be construed as an autonomous misalignment concern, it seems more natural to read it as a GD one.

The papal encyclical Magnifica Humanitas – the highest authority source we consulted – generally frames AI concerns in terms of power concentration and AI misuse, especially concerning ways in which already powerful actors could exploit the powerless, and it opposes trans-/post-humanism. However, it also expresses AI-led GD concerns in terms of the technocratic paradigm referred to in the other Catholic sources, for instance, noting that in the domain of education, AI can ‘encourage excessive reliance and the search for ready-made answers, and weaken personal creativity and judgment’ (§100), with the result that humans may gradually lose the desire to form human connections, that important decisions may be ‘fully delegated to automated systems’, and that due to algorithmic biases, injustices can ‘go[] unnoticed, and compassion, mercy and forgiveness...gradually disappear from view’ (§102-103).

Again, perhaps the most prominent way in which GD concerns surface is in relation to the domain of work, where the encyclical refers to how AI, which supposedly makes work more efficient, ends up subordinating workers to the ‘speed and demands of machines’, resulting in deskilling and ‘erod[ing] workers’ sense of agency’ (§105). It also speaks about a ‘legitimate fear of a significant and rapid contraction in available jobs that would create a chain reaction deeply impacting families, young people and local economies’ (§151). This displacement of human labour ’risks exposing many to forced inactivity, a lack of responsibility and the absence of daily tasks and stimuli, resulting in human and cultural impoverishment’ (§154). While this could be construed in terms of power concentration, it seems more natural to read these concerns as GD-related, seeing as they don’t explicitly identify bad actors.

Ecumenical Christian responses

Finally, we looked at an ecumenical source, the World Council of Churches’ Statement on the Unregulated Development of Artificial Intelligence (2023). The pie chart was excluded here due to sample size constraints (N=1). This document may express a specifically religious-domain GD concern where it states that the human interactions which make up the ‘work of the churches’ may be displaced by AI ‘facsimiles’, but it is otherwise more focussed on other advanced AI threats.

Note: the Catholic Church is not a member of the World Council of Churches, although it engages with the World Council via a Joint Working Group.

Conclusion

The sources we read across the three major Abrahamic traditions suggest that the GD threat currently is not as strongly attested compared to the threats of power concentration and AI misuse, or it is only implicit, although it is more explicit in the Catholic and Orthodox Christian sources. This should not be surprising, given that GD is a concept with fuzzy boundaries, it is identified as a neglected AI threat by Kulveit et al., and it does not yet have much currency.

This is also a first pass at a moving target – while we make no claim to have been exhaustive in our identification of authoritative religious statements on advanced AI, we are not aware of having missed any significant ones to date. However, we expect that the Abrahamic religions will continue to publish authoritative religious statements on AI, and the picture may well change as this happens. We’d value pushback on the passages we have interpreted as GD, as some of these could admit of other interpretations.

Next steps for AI Governance and coalition building

1. If religious leaders are largely unaware of Gradual Disempowerment (GD) as a threat model, the AI safety community can explore tailored outreach to help these institutions recognize GD alongside standard ethics concerns.

2. Religious institutions represent massive, trusted global networks. Establishing shared understanding around AI risk could lay the groundwork for broad societal coalitions that reduce AI risks through shaping cultural norms and other more direct interventions.

3. Future engagement can frame catastrophic risks using concepts tailored to specific religious traditions and concerns, including understanding human loss of agency in theological terms.

4. Further work is needed to test the robustness of this human-in-the-loop coding approach in comparison with independent human readings of the relevant religious statements and how it may potentially help policymakers, AI governance researchers, and religious leaders get an orientation to emerging norms concerning this threat.

^
The GD splices in the pie charts are split by coder confidence: high confidence = unambiguously GD; tentative = plausibly GD but also open to other readings.
^
‘the decentralisation of Jewish communal authority means that these discussions tend to adjudicate based on existing texts rather than legislate based on personal authority, which means that anyone with a sufficiently high degree of Jewish legal knowledge can try their hand at this work (many do; it is not an accident that many long-form treatments of new technologies are written by people with no religious leadership position at all).’ David Zvi Kalman, ‘Artificial Intelligence and Jewish Thought’, in The Cambridge Companion to Religion and Artificial Intelligence, ed. Beth Singler and Fraser Watts (Cambridge: Cambridge University Press, 2024), p.72.

Population Ethics is a Sham by Darshan Venkatesan🔸

Darshan Venkatesan🔸 — Wed, 23 Sep 2026 12:17:44 +0000

Crossposted from my blog.

For a field titled “Population Ethics”, there doesn’t seem to be a whole lot of… ethics going on. It would be more accurate to call the current discourse “Population Welfare” instead—the field is, after all, about deliberating among how good certain worlds are, given the welfare of the beings within them.

And, you know, this isn’t just a semantic name change. According to a recent EA forum post, a big problem of the field is that

There is no version of population ethics where you can just say, “Whatever feels intuitively right is what’s right.”

Well, I just so happen to believe in a Hedonistic theory of welfare, which I do intuitively find correct. So I kind of… just have an answer to Population Welfare, that aligns with all my intuitions. Specifically, Hedonism says that welfare only lives in instances of qualia. So, to evaluate a world’s welfare, I just do the same thing that I do to evaluate a single being’s welfare; I just imagine myself experiencing every single instance of qualia, one after another.

“But… isn’t that just the totalist position? Doesn’t that bite the bullet on the Repugnant Conclusion?”

No, it’s not, and no it doesn’t. Notice how I said “imagine experiencing the qualia” rather than “aggregate the welfare of all the qualia”. The Repugnant Conclusion asks whether a world with a few people living in bliss is more valuable than a world with lots of people living moderately happy lives. I, well, I’d rather just live a short life in bliss than a long, decent life, so you can probably guess which one I’d choose. No bullets bitten, but man, these berries taste amazing.

“Okay, but then your view is inconsistent, right?”

Look, intuitions are shaky. Intuitions are inconsistent. And intuitions are downright deceptive. But that’s kind of the point here. Theories of welfare are meant to serve their welfare agents, not the other way around. Who cares if a welfare function satisfies ‘transitivity’ or whatever—the welfare agent certainly doesn’t! Our intuitions about welfare^[1] are the closest thing (if not the only thing) we have to a ‘true’ welfare function, because we ourselves are welfare agents.

Oh, and I kind of lied. You know where the actual inconsistency comes from? It doesn’t even come from our intuitions. It comes from that one, all pervasive, assumption that everyone seems to use when talking about Population Welfare.

Please, Please Stop Representing Welfare as a Real Number!

Pause, for a moment, and just reflect on what you’re experiencing right now. You’re able to see bright, vivid colours all around you. You might be hearing faint birdsong in the background, as I am right now. Notice how your current physical position feels to you—the way you’re sitting down (or standing up), whether you feel slightly cool or warm, whether your muscles feel sore, or relaxed. Notice your emotional state, whether you feel content, whether your life feels purposeful right now, whether you’re envying someone else, experiencing vicarious embarrassment of something you did yesterday, or dreading an upcoming deadline.

Doesn’t your current experience feel unique? Even the mere act of reading a blog post online can be experienced in infinitely many ways.

And, even with this, we’re still only human. Think about the range of experience of nonhumans who can see infrared and ultraviolet, or can sense electric and magnetic fields, or whose sense of hearing and taste are much, much more amplified than our own. There are infinitely many experiences in the world we could only dream of, and infinitely more we literally can’t fathom.

Then the Population ‘Ethicist’ bursts in, chucks all that out the window, and says “Here’s a single number to represent all those complex things you’re feeling right now. Take it or leave it.”

And really, I want to emphasize just how much of an oversimplification this is. Think about, for example, the joy and purpose felt after completing a marathon, as opposed to the happiness experienced playing a video game in comfort. They both feel like good experiences, sure, but they seem fundamentally different from one another. The actual mechanism of welfare being experienced is different—emotional fulfillment, or more immediate physical comfort and pleasure. Doesn’t it seem like we’re ignoring everything that makes welfare special when we assert that both experiences are “10 units of welfare” or whatever?

And by the way, Population Welfarists don’t just assign scalar welfare quantities to instances of experience. No, they go one step further, and assign scalar welfare quantities to individual persons. They compress all the things that make personhood special^[2] - real relationships, the sense of continuity, and the notion of being part of something bigger than yourself—further into the scalar spectrum, as if there was more space left even after compressing the qualia.

So, yes. If we want to make the field of Population Welfare actually reflect the interests of welfare agents, we should worry about what those welfare agents actually have to say, rather than imposing our own model onto them!

What Do I Propose, Then?

The whole field is, as you could probably guess, extremely theoretical. We do need some framework of welfare to actually, you know, do things in Population Welfare. But, wait, yeah, the field is purely theoretical, right? We actually have the time and resources to flesh out this framework, rather than using the one mathematical structure we’re all familiar with!

And as it turns out, I happen to be someone who thinks that welfare actually does live in a mathematical structure. I just think that this structure is rich and infinitely more intricate than real numbers could ever hope to capture.

Now, as for this supposed structure that captures everything fully, I agree it’s most likely impossible to actually find and consider, even in theory; Integrated Information Theory, for example, is notoriously extremely computationally expensive (not to mention it doesn’t even provide a function measuring the welfare of the system)! But that doesn’t mean we can’t find a middle ground which is relatively easy to deliberate about, but also captures most of the core features of welfare.

There are many, many ways this could be done, and I propose that it be an important question in Population Welfare to explore how different mathematical models of welfare capture intuitively attributes of said welfare. For example, lexical thresholds can be captured by open balls in Banach spaces, or similarity between two welfare states can be captured by inner products in Hilbert spaces.

Let’s see what something like this might look like in practice. I proposed a theory of welfare in this post^[3], where welfare is represented as a tuple (M, S) of an n × n matrix and a set of vectors (in the post I set S = ℝⁿ). My idea was that S represented all the things a welfare agent wanted for themselves, and M was the actual experiential state of the agent. To evaluate how much M aligns with a vector v in S, you evaluate vᵀMv. The collection of all of these numbers is a summary of the valence of one’s experience.

Okay, cool. Notice that this framing of welfare actually includes more fine-grained details of what the welfare agent is actually experiencing. And it does stuff like actually differentiating between nonconscious entities and ‘valence zero’ entities, and having lexical thresholds (in the form of positive/negative definiteness).

Now, let’s say we have a bunch of welfare agents in some worlds, each with their own (M, S). How can we decide which world is better?

…

It’s… hard, isn’t it?

And that’s a good thing!

Having to actually compare worlds with nuance, exploring how some worlds are better in some specific aspects, and being unsure of a final verdict, is the sign of a good theory, rather than blindly assuming there exists some objective total order on all worlds. Some worlds should seem like they’re fundamentally incomparable with others; not better, not worse, just different.

What would a thought experiment using this theory of welfare look like? Let’s put a twist on the Repugnant Conclusion; Suppose I had two worlds, A and B.

World A consists of 100 people. Each person’s M matrix has mostly high eigenvalues, but one slightly negative eigenvalue (e.g a 3 × 3 matrix having eigenvalues 100, 10, and −1). So most vᵀMv evaluations will be positive and high in valence, but few of them will be slightly negative. You could imagine people in this world experiencing intense joy, but with the capacity for mild suffering.
World B consists of 1000 people. Each person’s M matrix has small but positive eigenvalues (e.g 3, 2, and 1). So all vᵀMv evaluations will be slightly positive, and you could imagine people in this world experiencing only mild satisfaction, but not anything actively bad.

Which one is better? Well, let’s just calculate the total welfare of both worlds, and―

No! Were you even listening? The whole point is that there isn’t just a simple aggregation function!^[4] We actually have to compare the worlds qualitatively, and delicately. Questions arise, like “is it worth it to tradeoff bliss for a guaranteed abolition of suffering”, or “how much does the valence of an experience really matter”, or “is it more ethical for a world’s suffering to be rare, or evenly shared”.

And yes, this is the place where the ethics actually lives.

I think the field should progress by discussing these questions, rather than merely discussing “which world is better”. Then, it will truly deserve its title of Population Ethics.

Now remember, my model still assumes a ton about welfare. It is, after all, just real numbers with a few extra steps. Imagine what sort of questions a more developed and abstract mathematical model could come up with!

Okay, But Is This Actually Useful?

What’s the point in having a framework for population ethics if it doesn’t just give us clear cut answers?

Well, for one, there aren’t clear cut answers in reality, either. As I explained earlier, some welfare states are fundamentally incomparable—they’re each good or bad in their own, subtle, way. Frameworks which capture more about welfare actually reflect this fact, rather than losing that subtlety by imposing a total ordering of worlds like real numbers do.

Secondly, does using real numbers even give us the answers? A commonly held view in ‘real number’ population ethics is to evaluate world welfare by add up all the individual welfare numbers, but this isn’t the consensus. Different frameworks can actually capture why people intuitively stray away from the totalist view.

And remember that the whole point of this is to actually implement it in practice. Mathematical models of welfare can be constructed to tailor for a specific intervention; you can try to have a particular welfare model evaluate the important consequences of said intervention. Models can also become very precise. It’s much easier to get people to imagine the exact same thought experiment when more distinction is laid out, as opposed to a wishy washy ‘1 unit of welfare’ label which can pretty much be interpreted however you want.

But, and this is important, even if you want to completely ditch the mathematical framework, that’s, fine too. Remember that the supposed contradictions arising from population ethics are a result of trying to squeeze the staggeringly complex structure of welfare into the cramped real numbers; they have nothing to do with purported views actually being contradictory. So yes, you are, indeed, free to just say “Whatever feels intuitively right is what’s right.” in Population Ethics. The mathematical frameworks are, after all, just trying to give those intuitions a more formal language to speak in.

^
By this, I mean what we would experience if we were actually placed in the same position as a welfare agent, not what we believe we would experience in that position. We tend to underestimate the severity of extreme suffering, for example.
^
Incidentally, I happen to think that personhood isn’t special, but my point here still stands even if we take a person to be a bunch of instances of qualia.
^
Which I spent like, a minute thinking of, and for a completely different reason to population ethics (I wanted to reconcile Hedonism and Preferentialism, which I now realize is not even possible). So there’s probably heaps of flaws here, but again this is just an example of what something might look like in practice.
^
You might have realised that when I discussed “experiencing each instance of qualia one after another”, this was implicitly aggregating all those instances of qualia into a single welfare structure. But remember, this is an aggregation of the true mathematical structure underlying qualia. It’s still a job for population ethics to figure out a scheme to aggregate these simpler models of welfare (e.g by trying to figure out what are the most important aspects of welfare that should be preserved in such a scheme).

Frontpage posts - Effective Altruism forum viewer

High Impact Medicine (Hi-Med) is hiring a Program Manager by Simon Ling

About Hi-Med

About the position

Click here for fur­ther de­tails and to ap­ply.

If you dislike some Berkeley sex norms, you can say so by Nathan Young

Why I’m optimistic about cultivated meat (and a suggestion for the frontier labs) by Bruce Friedrich 🔸

An ITN pro­ject for An­thropic’s and OpenAI’s life sci­ences di­vi­sions (and Google Deep­Mind)

State of Pandemic Early Warning by Jeff Kaufman 🔸

The Goal

Bio­surveillance Applications

What does suc­cess look like?

What gets us there?

What ex­ists to­day?

What’s miss­ing?

How does this change for ac­cel­er­at­ing re­sponse to a wild­fire pan­demic?

How does this change for on­go­ing sup­pres­sion?

Ap­pendix: Ini­tial De­tec­tion Scale

Copies can change where you’re headed, but not how you feel right now. by Tim Chan

TLDR

Introduction

Ex­act copies count for one

From the inside

Com­par­ing two worlds

Each per­spec­tive’s sub­jec­tive fu­tures count

The merge

The crossroads

The dilution

The false past (op­tional)

Altogether

Your in­ter­ests as a dis­tri­bu­tion that flows into your sub­jec­tive futures

The moral Markov process

What counts as a successor

From one per­spec­tive to all of them

Implications

Back to the first person

Dis­solv­ing some in­finite ethics problems

Ac­tion guidance in large worlds

Im­prov­ing prospects

Objections

Against repli­ca­tion neutrality

Against prospect dependence

Against the im­par­tial sum

Conclusion

Acknowledgements

How we get from AI cyberattacks to human extinction by 80000_Hours

Transcript

AI in­sid­ers think it could kill us all [00:00:00]

Why would AI try to kill us? [00:02:29]

How AI ends up em­bed­ded in the econ­omy and mil­i­tary [00:06:30]

AI de­ploy­ment could hap­pen fast [00:09:02]

How AI could bide its time and build up strength [00:11:43]

Con­trols and safe­guards will be in­suffi­cient [00:14:50]

The mo­ment the AIs would turn on us [00:15:26]

How AI could ac­tu­ally kill ev­ery­one [00:18:40]

Biolog­i­cal weapons [00:19:07]

Drone war­fare [00:21:42]

An al­ter­nate route to hu­man ex­tinc­tion [00:23:24]

Avoid­ing our own ex­tinc­tion [00:24:41]

AI safety field *visual* impact analysis by hannarchy

Some in­ter­est­ing high level ob­ser­va­tions

Draw­backs of this visualisation

Inspiration

6 reasons to invest in the animal cause by de S Brito

Scoop: Trump allies open new front against Anthropic CEO over AI “doomerism” by Matrice Jacobine🔸🏳️‍⚧️

Make your own model of the AI windfall by Jack Lewars

What needs to hap­pen for AI equity to be donated?

It’s go­ing to be a lot of money

The vegan trap: are we making animal advocacy harder than it needs to be? by tobiasleenaert

The num­ber of ve­g­ans is hardly grow­ing.

Ve­ganism has many challenges and downsides

Creat­ing barriers

What if we had never in­vented ve­g­anism?

Then what do we do?

The ve­gan trap

We need more non-vegans

En­list­ing non-ve­g­ans’ help for animals

Christian homeschoolers in the year 3000 by Buck

Anal­y­sis through swerv­ing around obstacles

Ex­po­sure to the out­side world might get re­ally scary

Click here for further details and to apply.

An ITN project for Anthropic’s and OpenAI’s life sciences divisions (and Google DeepMind)

Biosurveillance Applications

What does success look like?

What exists today?

What’s missing?

How does this change for accelerating response to a wildfire pandemic?

How does this change for ongoing suppression?

Appendix: Initial Detection Scale

Exact copies count for one

Comparing two worlds

Each perspective’s subjective futures count

The false past (optional)

Your interests as a distribution that flows into your subjective futures

From one perspective to all of them

Dissolving some infinite ethics problems

Action guidance in large worlds

Improving prospects

Against replication neutrality

Against the impartial sum

AI insiders think it could kill us all [00:00:00]

How AI ends up embedded in the economy and military [00:06:30]

AI deployment could happen fast [00:09:02]

Controls and safeguards will be insufficient [00:14:50]

The moment the AIs would turn on us [00:15:26]

How AI could actually kill everyone [00:18:40]

Biological weapons [00:19:07]

Drone warfare [00:21:42]

An alternate route to human extinction [00:23:24]

Avoiding our own extinction [00:24:41]

AI safety field visual impact analysis by hannarchy

Some interesting high level observations

Drawbacks of this visualisation

What needs to happen for AI equity to be donated?

It’s going to be a lot of money

The number of vegans is hardly growing.

Veganism has many challenges and downsides

Creating barriers

What if we had never invented veganism?

The vegan trap

Enlisting non-vegans’ help for animals

Analysis through swerving around obstacles

Exposure to the outside world might get really scary

Isolation will get easier and cheaper

I don’t think people will handle this well

Water Issues Are Not New

From “Crisis” to “Bankruptcy”: a Conceptual Shift

What Classifies as Water Bankruptcy?

Is Water Bankruptcy a Civilizational Risk?

Large cascading effects $\times$ lock-in risk

Impacts and consequences of this new water regime

Neglectedness assesment

Zero-sum thinking, probability neglect, and other failures of reasoning

Why care about religious leaders and AI governance?

To what extent are global religious leaders aware of Gradual Disempowerment, and where does it currently feature in their risk perception?

Orthodox Christianity

Protestant Christianity

Ecumenical Christian responses

Next steps for AI Governance and coalition building

Please, Please Stop Representing Welfare as a Real Number!

What Do I Propose, Then?

Okay, But Is This Actually Useful?