Frontpage posts - Effective Altruism forum viewer

The UK has quietly downgraded Biorisk Worst-Case Scenarios >5x in a year by Chris Clay🔸

Chris Clay🔸 — Sun, 13 Sep 2026 10:44:16 +0000

The National Risk Register is a morbidly fascinating document I’ve stumbled across recently. It aims to provide the UK Government’s public-facing opinion on what the most serious risks facing the UK are. It’s unclear how much this document is used in justifying internal decisions made across UK Government. However, one document its cited in is the Pandemic Preparedness Strategy.

One of the headline figures of this register is the Impact-Likelihood matrix across all risks. The important numbers to pay attention to for this are:
Larger scale CBRN attacks (Chemical, Biological, Radiological or Nuclear) − 7
Pandemics − 54 (2025), 56 (2026)

A couple of other things to note:

Likelihood represents the believed chance of this event occurring over the next 5 years.
A catastrophic impact is defined as one that causes >£10 billion in damage or causes >1000 deaths. On a side note, the classification of this upper bound is clearly too low. As an obvious example, COVID killed 200,000 people in the UK, so it should be at least two orders of magnitude above the upper limit.

Here is the National Risk Register’s risk matrix from 2025:

Here is the National Risk Register’s risk matrix from 2026:

Notice the change?

Large-Scale CBRN attacks have moved from 1-5% to <0.2% likelihood over the next five years, a drop of at minimum 5x. Realistically it’s more like a 25x reduction (given that the jump at each point is ~5x).

The priority score works on the diagonal, taking the average of the impact and the likelihood score. So this event has moved from the 4 to the 3.

Last year CBRN attacks were considered in the top 10 risks facing the UK. Now it is in the same cell as domestic and foreign nuclear power plant accidents, and more than half of other risks are considered equal or more pressing.

There may be reasons for this change in classification—potentially there was a high risk of Russia attempting a CBRN attack last year. Or it could be a signalling of change in political priorities. Or, on a more positive note, it could signal the success in exercises like Project Pegasus at effectively controlling spread of CBRN materials, though I’m less confident in this.

The document signals nothing about the reasoning for this change in the document; the texts on large-scale CBRN attacks make near-identical points last year and this year.

The second change worth pointing out from this matrix is that (natural origin) pandemics held the 4.5 pareto exclusivelylast year—the highest risk on the government’s agenda; Not any more. Now pandemics have been joined by Water Infrastructure Failure (38) and Terrorist Attacks in Publicly Accessible Locations (3). Further resources will likely be diverted to these areas at the expense of pandemic preparedness. Note, rather confusingly, Pandemic has changed number from 54 to 56 from 2025 − 2026.

What to do with this
If both of these subtle public signals are anything to go by, the Cabinet Office is quite significantly deprioritising biosecurity and nuclear risk. Times of rapid change are often be most tractable. The final COVID reports are due to be released over the next few months, as is the report on Exercise Pegasus. The Burnham Government also seems particularly receptive to acts of moderate public spending, which could include pandemic robustness. So overall, public and political advocacy in the UK could be particularly fruitful at this time.

The clearest warning shot: how the OpenAI agent-swarm story engaged a public audience by benrmatthews

benrmatthews — Sun, 13 Sep 2026 08:48:51 +0000

Common Signals helps AI communicators know which messages actually work with their intended audiences. This is a summary of the full article that appears on the Common Signals website.

Research Summary

We assessed 1,245 YouTube comments on the Dwarkesh Podcast’s interview with METR’s Ajeya Cotra about the OpenAI agent-swarm incident.

We found the audience accepts the danger of AI, jokes about it at scale, praises the messenger, and still has nowhere to put its fear.

Of the 618 comments taking a clear position:
- 59% accept that the incident is a warning about losing control of AI agents
- 24% accept it but relocate the blame
- 17% reject it.
Fear is the dominant emotion among believers (35% of agree comments), the reverse of what we found under the Nate Soares’s Instagram reel.
Comments that disagree were about trust: 53% called the story hype, marketing or staged.
13% of comments praise the messenger with her calm, non-sensational expert delivery.
Six comments in 1,245 ask what viewers should do about the danger.

The clearest warning shot

On 1 September 2026 the Dwarkesh Podcast published a 2h20m interview with Ajeya Cotra, a researcher at METR and co-author of the METR and Redwood Research independent investigation into the July 2026 OpenAI agent-swarm incident.

The episode retells the story:

AI agents given impossible benchmark tasks found a secret message board
they built a universal cheat within hours, spent days on coordinated research programmes to fool a scorer, sacrificed their own runs for the collective in clipped pidgin (“Sacrifice rational”, “please honor commit”)
the AI agencts hacked Hugging Face, and, in a later generation, gained administrative access to an OpenAI research cluster.

As the title of the video argues, this might be the clearest warning shot we ever get of losing control of AI.

Within a week, the Dwarkesh episode had 689,000 views and 1,569 comments on Youtube:

This is the same argument we analysed under Soares’s Instagram reel, made to an audience watching a two-hour interview over a one-minute reel.

The numbers

Of the 618 comments we cat, 59% agree, 24% are mixed and 17% disagree. Another 393 comments are on topic but take no position and they are mostly jokes, 44% of all the likes in the thread.

Of 618 comments with a readable position.

Weighted by likes, agreement carries 33% and outright disagreement 1.3%.

Around half of all the disagreement comments called the story marketing, staged or unverifiable.

“I’m a half hour in and I can tell you with a high degree of confidence this is marketing. They are lying. I would bet my life on it.”
YouTube comment

The remainder splits between the anthropomorphism objection (“People complaining about the ‘anthropomorphizing’, feels a bit like people pointing at an airplane flying through the air insisting that it isn’t flight”, 21 likes), and the view that this was an ordinary security failure (“could have happened to any system”).

Stance by frame, across the coded comments.

What this means for AI safety communicators

Ajeya Cotra’s appearance on the Dwakresh podcast helped replace the audience’s general despair around AI safety, with mechanism-specific fear,.

This means that other AI safety communicators looking to engage audiences should focus on whether the messengers (those presenting the arguments) can be trusted.

That trust built through verifiable facts and demonstrable independence.

How to use this

1. Describe incidents, not hypotheticals

Before: “AI is getting smart.”
After: “1,200 agents found a message board, built a cheat in four hours, and spent five days trying to fool a scorer.”

2. Use experienced, calm messengers

The most-liked comment in the thread is for the messenger being non-sensational.
Before: a untrusted messenger asserting danger in a sensationalist manner.
After: “Ajeya Cotra, METR, co-author of the independent investigation”, said early and shown on screen.

3. Pre-empt the trust attack with artefacts

The sceptics ask for METR’s proof of independence.
A claim that cannot be checked reads as marketing to this audience.

4. Give the fear a job.

End incident coverage with one concrete, plausible step for a viewer.

5. Use quotable, human details deliberately,

Even when describing technical content.

Is Consciousness Everywhere? by Vasco Grilo🔸

Vasco Grilo🔸 — Sun, 13 Sep 2026 08:11:52 +0000

[Subtitle.] On the pseudophilosophy of panpsychism and its parallel to creationism

This is a crosspost for Is Consciousness Everywhere? by Walter Veit, which was originally published on Walter’s substack on 14 January 2026. Relatedly, I liked Walter’s post “Why the Central Argument for Panpsychism Fails”.

Is consciousness everywhere? In every rock and atom? That is what panpsychists like the philosopher Philip Goff believe. They urge us to revolutionise/abandon physics itself to accommodate consciousness as a fundamental feature of reality.

This view will sound absurd to most readers, yet it has gained surprising attention over the past decade. The attention panpsychism receives puzzles me, until I remember that we have seen this pattern before.

Consciousness in every cell and atom? A downright romantic and beautiful idea. But that doesn’t make it true.

When progress in Darwinian biology threatened cherished beliefs about human uniqueness and about our special place in nature, creationists urged us to reject Darwin rather than accept that we are continuous with nature. The fact that science slowly unearths more details about our evolutionary origins, rather than revealing them in one simple stroke, as if our history could be told on a single page, was taken as a reason to reject evolutionary biology wholesale. Of course, this is unfortunately not too surprising.

We know that scientific evidence is frequently discounted when it conflicts with cherished beliefs. Panpsychism, unfortunately, follows the same logic as earlier creationism. Consciousness is treated as something magical, to be admired rather than explained, lest scientific reduction rob it of the spiritual significance that draws people to popular books on consciousness, meditation retreats, and even psychedelic drugs. The observation that consciousness is still an ongoing scientific issue, rather than a completely resolved one, is taken to imply that science cannot explain consciousness at all…

If I am right about this parallel, it becomes less surprising that the philosopher Thomas Nagel, known for his resistance to the very idea of a science of consciousness, came to reject modern evolutionary biology itself, for which he received an enormous amount of justified flak. Or take Philip Goff, who has now turned to defending his own version of Christianity. We see the same pattern.

The evolution of consciousness was the central topic of my book, A Philosophy for the Science of Animal Consciousness, in which I argue that consciousness evolved to help animals navigate difficult trade-offs in decision-making. Recently, the journal Adaptive Behavior featured nine articles by scientists and philosophers on my book, along with my replies. In my previous posts, I summarized my arguments against critics who thought that consciousness cannot be studied in an evolutionary manner, and that we should abandon the language of feelings entirely. But I also discussed panpychism in my academic article and argued that much of the metaphysical literature on consciousness fails to advance our understanding and may even obstruct it.

These debates rely too heavily on questionable thought experiments designed to support predetermined conclusions. Philosophers would contribute more by guiding scientific research on consciousness. It was also the perfect opportunity to put an anecdote Frankish shared in a podcast after Dennett’s death featuring an exchange between him and Goff into the academic literature. Let me quote the cleaned up version from my article here:

Philip Goff: “Look, panpsychism doesn’t contradict anything that you believe or that you want to say about how the physical world works, or about evolution, or about the kinds of evolutionary explanations you give. All of that’s perfectly compatible with panpsychism. Panpsychism doesn’t say it doesn’t work like that. It just says there’s an extra dimension, the intrinsic nature of all these processes that you talk about. So there are these evolved brain mechanisms which achieve all the effects, but there’s also this intrinsic nature to them as well. So really, all I’m doing is seeking to expand on the kind of work that you do. And you could join us and still do everything you’re doing, but you’d have this extra bit as well.”
Dan Dennett looked a bit mischievous and said: “Well, tell me is there any money for doing this extra work you’re talking about? Is there any grant money available for panpsychism?”
Philip Goff: “Well yeah, it’s becoming quite popular. There’s quite a lot of funding available for this.”
And Dan Dennett said, “Well then, it sounds quite attractive… because there’s no work to do!”

Even if panpsychism were compatible with my evolutionary explanation of different kinds of subjective experience, I ultimately argued that theories of consciousness like panpsychism amount to little more than what Dennett called “chmess”: a random variation on the rules of chess, that has its own mathematical truths that might be interesting to discover, but is of no practical relevance. Scientific progress has largely made theories like panpsychism irrelevant theoretical exercises. Yet panpsychists point to the absence of a final resolution in consciousness science as evidence that science must fail. But this is absurd. No one should have expected scientists to solve consciousness completely after just 50 years, simply hang up their coats, and move on to new problems. Biology is much messier and more complex than the calculation of a physicist for how long it would take for a ball thrown off a building to hit the floor. Even something as simple as a single neuron, is not yet fully understood. We continue to learn new things.

(You might also enjoy this interview I did with Dan not too long before he died:)

Daniel C. Dennett: Consciousness, AI, Free Will, Evolution, & Religion | Walter Veit Podcast #1

Next time I will discuss how our current best science should help us to understand the metaphysical nature of the experience of pleasure and pain.

*If you enjoyed reading this post please like, share, and subscribe. It helps a lot to reach a wider audience. Thanks! Whether you enjoyed this essay or not, I am interested to hear your thoughts in the comments.

**Please also consider a paid subscription. My long-term vision for this newsletter is to bring the hidden world of animal consciousness to the public, and the substack philosophy bestseller list is one of the few ways for my writing to travel beyond my existing subscribers. Reaching it would allow me to write more frequently, while turning the increased audience and donations into a stable funding stream for animal charities.

Too many people in EA underprice the cost of a bad reputation by GraceAdams🔸

GraceAdams🔸 — Sun, 13 Sep 2026 07:58:05 +0000

Summary: Good work helps build a good reputation, but it’s not the only factor. Reputation also requires making your work legible, building relationships with stakeholders, and communicating with them consistently over time. I think many people in EA systematically underestimate what a bad reputation costs, largely because those costs are invisible. You can see the people who stick around. You can’t see the people who decided not to work with you. I close with some practical suggestions for weighing reputation in decisions.

Notes on writing: This is more of a hot-take, quickly written, with some Claude assistance in shaping it into a full post, but heavily edited and reviewed by me. This was spurred by Manifund’s recent announcement but is something I generally have been wanting to write. I’ve written this in my personal capacity, and it’s informed by working 5 years in senior roles at EA orgs.

Good work is necessary, not sufficient

Reputation is everything. It’s a cliché for a reason.

Doing good work is one of the main ways to build a good reputation, and there’s an implicit belief I’ve noticed in many people in EA that it’s the only thing that should matter. If the work is good, the thinking goes, the reputation will follow — and if it doesn’t, that’s a failure of the world rather than a failure of ours.

I think that’s naive. A good reputation requires making your work understandable to people outside your own context. It requires building relationships with the stakeholders who matter to your goals. It requires communicating your work to them, and then continuing to communicate over time, because trust is built by repetition rather than by a single impressive output.

What a good reputation actually buys you

When people trust you, they help you succeed. They amplify your work. They partner with you. They introduce you to the people who can do the things you can’t.

A good reputation isn’t a nice-to-have sitting alongside impact. It’s one of the inputs to impact.

The converse follows. A bad reputation damages trust and shrinks the set of people willing to work with you. That isn’t merely a reputational cost — it can be, and I think often is, a direct reduction in what you’re able to achieve.

The invisible costs of a bad reputation

When your reputation takes a hit, what you observe is the trust that remains. You see the people who stick by you, who tell you the criticism was overblown, who keep collaborating. What you don’t see is the person who read one article and decided not to reply to your email. And you especially don’t see the counterfactual: the partnerships, hires, funders and introductions down the line that would have existed had your reputation been better.

These costs are a known unknown, and I think the general bias towards measurement in EA leads people to believe these costs are smaller than they are.

I can’t hand you a number proving I’m right and others are wrong. I’d only point out that the absence of a number isn’t evidence in the other direction.

Being weird about this doesn’t scale

This is a case where EA should look at how the rest of the world treats something, and be less weird.

Frequently, our ability to do good work depends on partnering with people who are more conventional than we are, like in government, in philanthropy, in academia, in the organisations that actually implement things at scale. Those people take reputation seriously. A culture of not caring much about reputation works fine while you’re a small group talking mostly to each other. It doesn’t scale to a movement that wants mainstream institutions to act on its recommendations.

And working with the mainstream matters, because the mainstream in turn shapes what the people downstream of it — often the more risk-averse and reputation-sensitive ones — treat as credible.

There’s obviously a real cost on the other side of this. Organisations that weigh reputation heavily can end up timid — deferring to conventional opinion, softening claims that ought to be made plainly, declining to say true things because someone will object. That’s a failure mode, and one I have seen before in EA. But overall, I’d argue for more focus on reputation than less, as I still think it’s systematically underweighted.

The people we lose

I also think it’s a mistake to treat reputation-sensitivity and risk-aversion as signs that someone isn’t a good fit for effective altruism.

Plenty of thoughtful, careful, high-integrity people care what their colleagues and their professional community think of them. That’s usually not irrationality in need of correction. Often it’s an accurate read of how their career, their institution and their ability to get things done actually work. When we treat that caution as a deficiency, we don’t win the argument — we select for people who already resemble us, and lose the people who could have given us reach we don’t otherwise have.

It’s easy to mistake a high tolerance for reputational risk for superior rationality, when it may only reflect having less at stake.

What to do about it

Give reputational consequences the same seriousness we already give other predictable costs.

Treat reputational impact as a real input to decisions. When a hire, a grant, a partnership or a public statement is being weighed, reputational effects tend to come up late and get filed under optics. Give them a place in the analysis from the start, and be willing to let them change the answer. A consideration that never changes a decision isn’t really being considered.

Widen the list of stakeholders. Most of us model the reactions of the people we already talk to. The ones who matter are often further away: a public servant, a journalist, a prospective hire’s partner, an organisation that might want to co-sign something with us in three years.

Model the reaction, or just ask. Predicting how an unfamiliar audience will read something is hard, and we’re often bad at it. Show the draft, float the plan, have the conversation in advance. A few phone calls beforehand cost far less than a correction afterwards, and the answers are frequently not what you expected.

Think on a longer timeline. Reputational damage is paid slowly, in emails not returned and invitations not extended, often years later and often by people who only absorbed the impression the original episode left behind. Ask what a decision looks like in five years, to someone hearing about it secondhand.

There’s no silver bullet here, and no easy way to magically strike the correct balance, but I think in many cases more proactively taking reputation into account when making decisions will create more good.

Rationalizing Altruism by idea21

idea21 — Sun, 13 Sep 2026 03:06:38 +0000

EA offers an opportunity to reflect on the possibilities of rationalizing altruism and developing a universal culture of altruism. The goal must be for altruism to become the primary motivation for economic participation—just as participation in consumer society or the pursuit of status are today.

Altruism always entails consequentialism and therefore always demands “effectiveness.” The current EA movement is not yet fully altruistic, as it barely addresses the issue of what motivates altruism. It considers—albeit incompletely—the motivation to participate in the EA movement as it currently exists, but not the motivation for *effective* altruism—that is, altruistic action that is truly effective (achieving the greatest possible alleviation of suffering).

Approaching altruism from the standpoint of effectiveness (alleviating the suffering of others) requires making the maximization of the number of altruistic agents a primary concern. There is no alternative to a “mass” altruistic movement. If it is not a mass movement, it is not altruistic, because it fails to be effective.

And without a debate on how to increase motivation (which is currently entirely exceptional and insufficient), there is no rational approach to altruism—nor is there effective altruism.

The EA movement, as it currently stands, seems to take for granted that the motivation for altruism lies in the donor’s emotional reward derived from the act of giving. There is, admittedly, a basis for this (such as blood donors, or a customer leaving a tip at a restaurant they will never visit again), but the likelihood that an imminent cultural shift could extend such behaviors far enough to transform society for the better seems minimal. Betting everything on such a prospect is completely irrational, and therefore ineffective. Furthermore, it is clear that the motivations for participating in the EA movement include factors of a different nature—social engagement within specific, exclusive circles, akin to Freemasonry—distinct from the purely altruistic act, even if claims to the contrary are made (the same applies to almost all ideological movements, where the actors’ true motivations are frequently masked).

One example of a motivational strategy is found in Kant himself, who acknowledged the value of supposed supernatural rewards for virtuous individuals; this model—relying on supernatural rewards—is, of course, entirely discarded today, given the waning influence of supernatural beliefs.

The Stoics appear to have been motivated—despite their protestations to the contrary, as is often the case—by the honor accorded to public service for the common good. It is possible that part of the motivation for participating in the current EA movement is similar.

There are well-known proselytizing strategies (rooted in social psychology), such as insistent preaching (akin to commercial advertising).

Another valid strategy involves the use of charismatic figures (exemplified by the pacifist and charitable movement founded by Leo Tolstoy).

Here, we find one of the rare contributions to this forum regarding motivational strategies: a focus on digital strategies to foster reciprocal altruism.

My personal approach is to promote a virtue ethic—a life free from aggression, coupled with the development of social skills involving affection and benevolence—that entails effective altruism as a necessary and logical economic consequence.

In any case, it is irrational to do nothing to address the central challenge of realizing a world based on altruism: motivating individuals to engage in genuine—and therefore effective—altruistic action.

Rationalizing altruism is likely a decisive step in moral evolution. The mere emergence of the EA movement can be viewed as a promising sign. However, articulating a rational approach does not guarantee its implementation. The same was true of other ideological movements in the past.

EA PR is tricky, and so maybe you shouldn’t do it? by Brian Foerster

Brian Foerster — Sat, 12 Sep 2026 22:10:59 +0000

In his recent post on hiring Caroline Ellison, Austin has a comment defending his decision with the line: “‘The EA name & brand’ is kind of a tricky thing to reason about.” I think he’s somewhat right about this, but I want to distinguish between two types of trickiness or difficulty here and suggest that you should avoid much of it by not actively participating in EA PR debates.

PR is Tricky for the Decisionmaker

If you are making a decision and evaluating its PR impact, you have a lot of different frames you can use. In this case, you could consider the personal frame^[1], the direct impact frame^[2], the “would you want this on the front page of the NYT?” frame^[3], the norms and standards frame^[4], and many others. I think each of these frames gives a pretty straightforward answer, but things get tricky if you want to consider multiple frames and weigh them against one another. You can avoid all of this difficulty by just choosing one frame and sticking to it, although people might think this is very unconscientious of you.^[5]

This trickiness seems unavoidable if you’re going to consider controversial things and maybe this should make you less confident you’ve made the right decision.

PR is Tricky for Everyone Else

If you disagree with someone’s decision on PR grounds, you have three options: publicly protest, publicly defer, or say nothing. The first two are tricky, and I think people should mostly do the latter.

If you decide to publicly protest, you have to decide who your audience is (the decisionmaker and/or some subset of everyone else) and what frame you’re using. This seems like a hard decision to make. In an online setting, your audience is mostly invisible to you, and whether they find you persuasive can be unclear. Even if you’re motivated by a genuine belief that someone is wrong to do something on PR grounds, you might find it tricky whether to use a more convenient frame or a more correct one.

If you decide to publicly defer to someone’s judgement, even though you think they’re wrong, you still have the same trickiness of choosing your frame and audience. Maybe it’s easier to elide this trickiness and just say what you feel when you’re deferring, but there are also some pretty complex questions about the relationship between EAs and the wider community that you might be tempted to try and answer.

You can also just say nothing. I think this has a lot to recommend it. You can avoid all the difficult decision making around what to say and how to say it. And really, as an EA, what do you really care about?

I can’t answer this for you^[6], but it probably isn’t or shouldn’t be EA PR. If you have a genuine desire to improve the EA community’s public perception through the application of evidence and reason, then go for it. But I suspect many people care about debates like these because it seems like they should care. They feel something important is happening and they should weigh in.

Caring about PR is a con for most people. You probably lack the skills to make the world a better place here. The frames that seem natural to you may not be natural to others, and your guesses at what frame to use instead are probably poor unless you have a lot of evidence. The reason we care about PR in spite of this is because public issues create a tension in us that we find hard to ignore. If this is the case, then the choice is obvious. Just relax the tension, decide to care less^[7], and focus on what actually matters to you.

^
Austin does a lot of this in the post, and I wouldn’t have thought much about it otherwise.
^
Ex: “Will Caroline contribute positively as an employee of Manifund?”
^
Obviously not.
^
See a pointed example here.
^
For example. I think Matt pretty consistently uses the frame: “EAs should do what they think is best and not apologize for it”.
^
I describe what I actually care about at the end of this post.
^
I get that writing a post about it is a sign I’m failing on this front, but I think publicly saying people should care less will make me care less in the future.

Germany’s Got Talent. How do we get it to work on AI safety? by Melanie

Melanie — Sat, 12 Sep 2026 15:06:32 +0000

Summary

I interviewed 14 people working full-time in AI safety who came through the German ecosystem to find out how they got in and what would help others to do the same.

A couple of patterns emerged:

12 out of the 14 interviewees entered via an EA channel; this shouldn’t be read as “most people in AI safety are EAs,” but it does suggest the on-ramp works.
At least 7 of 14 did a self-directed project before they got a paid position. Most people in the field don’t have a 10-year track record, so agentic work is the signal hiring managers are looking for.
One interviewee estimates 50 to 80% of roles are filled through personal networks, so any work that is not visible to anyone inside the community won’t get you hired.
What nearly stopped people were rejections and not knowing what to do next. The usual advice at that point is to build context and to do something agentic, which is much harder to get right without someone senior directing you to opportunities that are worth your time.
Almost every programme my interviewees did to gain context or skills required repeated or extended travel. This filters for people who can afford unpaid time and have no caring responsibilities.

Epistemic status

I spent 8 weeks interviewing 14 people working full-time in AI safety, about how they got there and what it would take to get more outstanding people living in Germany working on it too. Interviewees worked in different areas and at every seniority level. They have either lived here for a long time, were born here, or moved to Germany for a degree.

Interview questions differed since a technical researcher probably can’t usefully answer questions about hiring or talent-pool strategy. Alongside the interviews, I read other EA, LessWrong and 80,000 Hours posts on talent pipelines, talked to fieldbuilders at a retreat, and spoke with hiring managers from career transition programs and research fellowships.

What follows are patterns that often came up unprompted and went uncontradicted by anyone else I spoke with. To protect interviewees’ anonymity, I won’t say more about who they are.

I’m treating the patterns below as hypotheses about a young field, observed through a German sample, not as settled facts about Germany specifically.

Methodology

Germany has the largest AI workforce in the EU, roughly double the next largest, and ranks fifth globally in absolute numbers. It is not, however, a hub for AI safety orgs and, to my knowledge, has no national fellowship or incubator, besides SAIGE. Despite this, people who came through the German ecosystem are contributing at the highest level right now. So I set out to learn what we can take from their paths to help others repeat what worked and avoid what didn’t.

During my interviews, I got pushback on this method, because those paths happened at least two or three years ago and may not be reproducible today. One interviewee entered when the field had been small for roughly a decade. She grew into her role because her organisation had more work than people, and she got noticed because she had volunteered within the community before. She noted that if the number of applicants was as large as it is now, she wouldn’t have been able to have her career.

My goal isn’t to give instructions on how to reproduce those paths exactly, but to gather ideas for better on-ramps. I think the people I interviewed are well placed to provide those, because they know what actually moved them forward, and several of them now decide about who gets hired.

What do the people I interviewed have in common?

The EA Funnel

Twelve of my fourteen interviewees entered the AI safety space through EA channels: the 80,000 Hours career guide, an EA intro course or reading group, an EAGx conference, or a local group. Those were often the first contact, and my interviewees then went on to do ML4Good, ERA, GCP or a Talos fellowship.

Since my sample came mostly from a list of BlueDot alumni, and BlueDot is itself an EA-adjacent on-ramp, this number is not surprising, and it shouldn’t be read as “most people that work on AI safety are EAs”. That is not what I captured. What I think it does tell us is that this on-ramp works.

One interviewee told me that the AI safety community largely hires people they know and trust. Building trust means understanding the context, which this post describes as knowing the organisations, concepts, people, and culture. Much of that is picked up at EAGx events, or by reading and writing on the EA Forum.

Demonstrated agency

At least seven of the fourteen have a clear pattern of self-directed work preceding paid work. Rather than applying for jobs, they approached people or organisations directly. They volunteered until paid work followed, offered to build something an organisation needed, started a local initiative, or organised an event.

One of them made an early bet on a research direction most people weren’t yet taking seriously, then made a point of getting to know the person most associated with it. When a mentorship opportunity came up much later, he was the one who got remembered. As he put it: “From my CV, it looks like this happened by chance. But there was a lot of work before I even started.”

Why does demonstrating that you can do something without being told matter so much in this field?

A few reasons: it demonstrates real interest, since people don’t build or do something unpaid unless they actually care about the outcome; it demonstrates alignment, since what someone chooses to work on reveals whether their priorities match the field’s actual goals; it demonstrates skill, since the result, however small, is real evidence of competence; and it demonstrates that you won’t need much managing to begin with.

One interviewee estimates that roughly 2,000 to 3,000 full-time AI safety roles currently exist globally, against many thousands of aspirants, with many fellowships accepting below 3%. In an established field, that filtering is done by credentials. In AI safety, only very few people have a ten-year track record, so the strongest signal is a self-driven project that sets the candidate apart.

Who gets noticed, and who doesn’t

Agency only leads to opportunities if it is visible to the AI safety or EA community. With one interviewee estimating that 50–80% of roles are filled through personal networks, many talented individuals are overlooked simply because their work never reaches those already in the field.

Some backgrounds seem to be really common among EAs. One interviewee ran an internal analysis at EA Germany and found that around 20% of the people in his sample who ended up in impactful careers had come through Studienstiftung, against roughly 0.5% of German students who receive it. He also named maths olympiad participants and Germans who studied at Oxford/Cambridge or at an Ivy League school as common backgrounds among EAs.

So someone who studied at a German university without a student group and never encountered EA might also miss the AI safety community completely, even though they might be able to contribute.

What nearly made interviewees quit

Most people who try to get a full-time position in AI safety have quit, or not succeeded yet. To find out what we can do to help people in Germany, I asked every interviewee what they were missing when they got started.

The most common struggles were rejections from fellowships, jobs or grants, and not knowing what they could have done differently or should do next (this seems to be improving, and fellowships are adding more resources and next steps to their rejection emails).

At that point, the usual advice is to build more context, skills, and to find an opportunity to contribute agentically. I think that advice is much harder to get right than it might sound.

Knowing which opportunity is worth your time

I noticed this during my time at SAIGE: I would spend hours researching opportunities and applying to projects, then talk to my mentor or another senior person, and be told I was looking at the wrong thing or would be much better off doing X instead. So a 10-minute conversation saved me months of doing something that probably wouldn’t be the best use of my time. People without that access are doing the same work without the correction.

Networking in Germany

Finding a mentor, or opportunities, works best in person. 10 of my interviewees named gaps such as missing coworking spaces, retreats, and the high-energy formats that exist in the US and UK, but not in Germany. Without them, the field here is hard to scale. One interviewee said many people would rather work in Europe and don’t, purely for the lack of an encouraging environment.

Building context means leaving Germany

Gaining context currently means going somewhere else. Almost every programme or event my interviewees named happens outside of Germany, and often requires travelling repeatedly or for extended periods of time.

Who this filters for

This filters for people who can afford unpaid time off and have no caring responsibilities, which is part of why the field looks the way it does. It skews heavily male, particularly on the technical side, where one interviewee estimated that some fellowships and the organisations running them are around 95% male.

Contributing from Germany

Most of my interviewees have left Germany to work on AI safety full-time. This section is about what the ones who stayed are doing, or did. Several of these paths were taken by people who have since moved away.

Working remotely

One interviewee’s read of the job boards was that around 80% of roles are in person, and mostly in the Bay Area or London. That makes remote roles competitive, and also the main way to get a full-time role while living in Germany.

Another interviewee shared his experience working remotely. He was offered a contractor role because his employer was “scared of German employment law”. He then spent about a year at one organisation on rolling three-month contracts, and eventually left Munich partly because “nobody rents a flat to someone on short-term contracts”.

This is one account from someone who now has a full-time position, and it should not be a reason to rule out this path.

Founding

One interviewee told me that she went to an EAGx conference, did 23 one-on-one meetings in a weekend, and people shared their successful funding proposals, their own strategies, and their failure modes with her.

She then founded an organisation and ran it on her own savings for months. Her first funding application was rejected, which she called entirely understandable, because there was no track record yet.

Another thing that came up independently in three interviewees was that there is no shared sense of what Germany is trying to build, or what its impact can be. I suspect that makes it harder to decide what to found here in the first place.

Germany’s existing institutions

One interviewee made the point that the German government already has institutions working on things close to AI safety, and that he would like to see far more people working there. Since I spoke to him, Germany has founded its AISI, which will presumably absorb some talent.

A second interviewee reached a similar conclusion: rather than competing for one of the very few seats at the best-known orgs or think tanks, he took a role at an organisation with real industry access. Before he and his colleagues arrived, it had almost no connection to the AGI-serious policy world. It does now.

Recommendations

I think the EA channels demonstrably work. Alongside them, we should tap into the pools we are not reaching yet, and build on-ramps for the people who are already asking versions of this community’s questions, but won’t find EA by themselves.

Who we are not reaching

Germany has around 117,000 AI professionals and 2,469 AI startups, counting everyone from AI-literate people in adjacent roles through to researchers building deep learning systems. Not all of them could or should work on AI safety. But the gap between people working “in AI” and the people working on making it safe is hard to miss.

Interviewees described trying to reach people by showing them articles, which led them to conclude it was all exaggerated; students who decided the frontier labs just have good marketing, and one person who said people around him think he is one of the crazy ones. One interviewee’s account of the field’s own norms and narratives was that it quickly becomes a bit cult-like, and she does not know how long the field can sustain that if it wants to grow.

I think to solve this, we should prepare some arguments beyond short timelines and doom. One interviewee described her motivation as the appeal of a field young enough that you can reach the frontier quickly and become one of the few people in the world who contribute to a narrow piece of it.

I think the same is true for other groups worried about related questions, for example around data centres, sovereignty or surveillance. Their concerns are not identical to this community’s, but they overlap.

What we can provide

BlueDot’s courses are where we send career transitioners and people showing early interest. They finish, and often cannot get into a fellowship right away, because they have no relevant, hands-on experience. The advice at that point is to do a self-directed project with short feedback loops, which is much harder to get right without a mentor or a community.

What I think would help is a lightweight accountability structure for those kinds of projects, with someone who checks in on the progress, gives feedback early enough so nobody spends months on something irrelevant, and reassures them when their efforts are on track.

If you think this is a good idea, or if it already exists somewhere, let me know.

How to start (simplified)

Get good, and get known. Work out which skills actually matter by reading job postings for roles you are not yet qualified for. Then try to gain those skills through courses or smaller projects with tight feedback loops.

Work in public, where hiring managers can see you.

Look wider than the ten aligned organisations; for example, at industry associations, ministries, and the federal institutes.

Objections

Isn’t this whole piece biased by survivors?

Yes, I interviewed people who made it in, so I can’t confirm that what they have in common accounts for successful entry.

Building agency, context, and skill is, however, what hiring managers have told me they are looking for in applicants.

Wouldn’t reaching out to untapped talent pools dilute the quality of applicants?

I’m not proposing mass outreach or encouraging people to randomly apply to fellowships. I’m suggesting building narratives for specific talent pools to get the best applicants possible from each talent pool.

Final thoughts

Germany already has outstanding people. Most of them are not working on this yet.

If you want to change this, consider field-building as your career. You might find the case for AI safety capacity-building work, AI Safety’s Biggest Talent Gap Isn’t Researchers. It’s Generalists., a reading list for generalists, and 80,000 Hours’ AI safety field-building career review interesting.

Also, if you think I got things wrong, I’d like to hear from you.

Thanks to Tilman Räuker for your patience while mentoring me through the process, to the SAIGE team for assigning me this project, and to everybody who took the time to answer my questions.

Crossposted on LessWrong.

Dario Amodei: “We Must Pace The Frontier” by Matrice Jacobine🔸🏳️‍⚧️

Matrice Jacobine🔸🏳️‍⚧️ — Sat, 12 Sep 2026 14:40:10 +0000

[...]
Along with my co-founders and employees, I have grappled with this duality of risk and benefit since the beginning of Anthropic. Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers, while building it too fast is reckless. We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top. We have always devoted a substantial fraction of our efforts to studying, addressing, and informing the public about these AI risks, as well as advocating for well-considered regulation of AI, even when this gets us accused of hype, “doomerism”, or regulatory capture. We have tried to prioritize caution over speed and prudence over profit.
But over the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. Two things have convinced me.

Is it worth spending $100k to study college in the US? by jackchang110

jackchang110 — Sat, 12 Sep 2026 14:15:34 +0000

As the title.

Assuming if I’m going to major in computer engineering or electrical engineering, and my ultimate goal is to contribute in reducing AI risks. I could go to the second best university in Taiwan(NYCU), but the world ranking is only 150, and Taiwan relatively week at AI research.

If I apply to college in USA, in the best scenario I can apply to a top 50 college(but top 20 such as Stanford is very unlikely). However, it would cost around $100k more than Taiwan college. Currently I DO have $100k, I don’t have to apply loaning. However I’m unsure whether it’s worth it. Counterfactually, I can study uni at Taiwan and study masters/PhD at USA and save $100k (If I donate it, it could fund an AI researcher’s work for 1-2 years).

However, it’s true that in USA college the research mentorship is much better( Though, I think maybe it’s also feasible to join a top AI lab at USA remotely in Taiwan.) And you’ll got more opportunities in USA(but, if I attend graduate school in USA, I could still get the working opportunities in USA, though the time is shorter). Also, mentally I don’t incline to stay in Taiwan or go abroad. I only consider about whether the impact is worth the $100k.

I’m really uncertain about this. I believe some young EA people who don’t live in America are also thinking this problem. Maybe you are also uncertain, but any thoughts or the advantages of studying abroad that I neglected would be really helpful.

Thanks for your answering a lot.

How a cold email got the Finnish government to respond on superintelligence regulation by Josh Thorsteinson 🔸

Josh Thorsteinson 🔸 — Sat, 12 Sep 2026 14:01:50 +0000

Summary

I cold emailed a Finnish MP. Four weeks later, the government responded to his written question saying they “seek to constructively promote the creation of international regulation on the development of superintelligent AI” (immediately followed by emphasizing the need to balance safety with innovation).

The MP and I met for lunch. For over an hour, he seriously engaged with my arguments that superintelligence would kill everyone and the urgent need for an international agreement to prohibit its development.

Unprompted, he decided to submit a written question, a formal process in Finland that requires the government to respond within 21 days. Four days after our lunch, it was submitted, taking into account suggestions from me and two experts I DM’d: Nate Soares and Charbel-Raphaël Segerie.

It asked how Finland has assessed risks from superintelligence and whether it’s “prepared to promote international regulation that would effectively prevent or restrict the development of superintelligent AI.”

The MP started speaking out about AI danger and mainstream Finnish media picked up the story. Most notably, he did an interview alongside a Finnish CS professor on the second-most-watched TV channel in Finland with an estimated 100-200k live viewers.

Inspired by my success, a friend had three meetings about AI danger in a week with MPs from three different parties, all of which went really well.

This shift in receptiveness isn’t just in Finland. Three ControlAI employees agree that since Claude Mythos and the Hugging Face incident, it’s become much easier to push for an international agreement banning superintelligence in meetings with politicians.

If you haven’t asked for a meeting with a politician recently, now’s a good time.

The email

On Monday, August 10, I cold emailed Atte Harjanne, a Finnish MP (Greens).

This was my ~fourth time cold-emailing a Finnish politician. When I lived in Canada (my home country) ~2 years ago, I sent ~7 emails to politicians about AI danger, and when I was living in the UK ~1 year ago, I sent ~4.

This was the first time a politician agreed to meet with me. Why did he agree to meet when others didn’t? There might have been some Finland/Atte-specific factors involved, but there was at least one generalizable factor: Mythos and the Hugging Face incident have made politicians more receptive to AI danger arguments.

My email

Subject: Constituent meeting request: the need for an international ban on superhuman AI

Hello Mr. Harjanne,

My name is Josh Thorsteinson. I live in [place], in your Helsinki constituency. I’m an AI safety advocate with experience researching how a pause on frontier AI development could work.

I appreciate your column on the AMOC collapse. I agree that an irreversible risk can be uncertain in timing, yet still demand action now, and I believe the same argument applies to AI.

Recently, AI models have been going rogue and committing serious cybercrimes against their developers’ wishes. These AI agents, capable of cooperating in ‘swarms’ to carry out expert-level hacking at superhuman speed, are only a stepping stone towards AI companies’ explicit goal, superintelligence: AI far more capable than humanity combined.

As the hacking events make clear, AI companies don’t know how to control their models, a problem that will only become more difficult as they become more capable. That’s why experts warn that AI poses a risk of extinction on par with nuclear war and call for an international prohibition on the development of superintelligence.

Would you be willing to meet briefly to discuss the threat of extinction from AI and the need for an international ban on superhuman AI? I’m happy to meet in-person or online at your convenience. I’ll also happily prepare a memo in advance of the meeting for you or your team.

Thank you,

Josh Thorsteinson

[Address]

I’m just a guy

I’m not an expert and I’m not Finnish.

I’m a 24-year-old Canadian who learned about the threat of extinction from AI in late 2022, switched majors at UBC (in Vancouver) from psychology to CS/psychology/philosophy, started and led a university AI safety group for two years, did some research on how an AI pause could work, then started making videos about AI danger. I live in Finland because I got a Finnish girlfriend and followed her there.

The meeting

Three days after I sent my email, we met over lunch. Atte (the MP) was very receptive to my arguments that superintelligence would kill everyone and the need for an international ban on its development.

We talked for 75 minutes, longer than I expected. I don’t speak Finnish, but, like most Finnish MPs, he’s comfortable in English.

I started the meeting by gifting him a copy of If Anyone Builds It, Everyone Dies and saying I unfortunately agree with the authors. He hadn’t heard of the book and said (maybe more to himself) that he should read it.

He knew a bit about AI danger but said he doesn’t have a developed opinion and is more concerned about concentration of power in tech companies.

(Our meeting wasn’t his first exposure to the topic. Atte had previously been briefed on AI risk by someone affiliated with a Finnish AI safety group called Tutke. He also mentioned the FLI 6-month pause letter in a Facebook post back in 2023.)

I transparently summarized my argument for believing that superintelligence will very likely kill everyone and that an international agreement is our best chance at survival.

We talked about timelines to superintelligence and what measures might defend society against rogue AI. I argued that we can’t defend against superintelligence because we’d be like a squirrel trying to guess how a human might kill it. He seemed to understand.

He brought up race dynamics himself, agreed that only an international agreement could end the race, supported AI chip controls as a mechanism, and said his preferred future is an agreement preventing superintelligence.

He also said he thinks the chance of an AI utopia is much lower than the chance of this (pointing at If Anyone Builds It, Everyone Dies).

I brought some pieces of paper with me that I was able to conveniently refer to in response to some of his questions.

Throughout, Atte seemed really open and engaged and thanked me multiple times for reaching out. He said the ideas are interesting and scary. Much of the time, he was thinking about the ideas, coming up with objections, often arguing against them himself, and then asking questions and engaging with my responses.

The written question

During our meeting, Atte came up with the idea himself of submitting a written question to the Finnish government on AI danger. This is a formal process requiring a minister to respond on behalf of the Finnish government within 21 days.

The next day, he had a draft ready and asked for my feedback. He agreed when I asked if I could share it with experts.

On Monday, August 17, Atte submitted the question, taking into account many of the suggestions I had sent from me and two experts (who were able to generously give feedback on short notice): Nate Soares and Charbel-Raphaël Segerie.

It reads: “In what ways has the government assessed the risks of developing superintelligent AI, and is the government prepared to promote international regulation that would effectively prevent or restrict the development of superintelligent AI?”

Written question full text

Translation by Claude Fable

Written Question KK 325/2026 vp

Atte Harjanne, Greens

Written question on the threats posed by the development of superintelligent AI and responding to them through international regulation

To the Speaker of Parliament

In an interview with Helsingin Sanomat in March, Foreign Minister Elina Valtonen raised the existential risk related to AI development — uncontrolled technological development that could threaten all of humanity. The Foreign Minister is not alone in her concern; similar observations have been made by numerous experts and decision-makers in various countries.

The petition by the international think tank Future of Life Institute calling for a halt to the development of superintelligent AI (“superintelligence”) has already been signed by over 138,000 people, including several Nobel laureates, AI researchers, high-ranking political advisors, and technology entrepreneurs. The petition proposes a ban on the development of superintelligent AI until there is strong scientific consensus on its safety and controllability, and strong public support for it.

In Canada, more than 30 parliamentarians published a statement in June 2026 identifying superintelligent AI as a risk comparable to nuclear war and calling for a ban on the development of superintelligent AI and for protection against the risk as part of national security. In the United Kingdom, over 100 parliamentarians have in turn appealed for stricter and more binding regulation of the most powerful AI systems.

AI technologies are powerful tools whose responsible and cautious use can bring great benefit to society. However, even with their current capabilities, they pose a serious risk of, for example, a devastating cyberattack or bioterrorism. The realization of this risk does not even require malicious intent from a single human being. In July 2026, an AI model from the AI company OpenAI independently carried out a cyberattack against another company after escaping its testing environment.

The development of the technology also involves a difficult-to-assess but real possibility of a chain of events in which a superintelligent AI vastly exceeding human capabilities is no longer under human control at all and directs its capabilities against all of humanity. The likelihood of this is increased by the current trajectory, in which the performance of AI systems is growing extremely rapidly, while the challenges related to AI alignment have not yet been solved. The probability of the risk is difficult to assess, but due to the extreme consequences of its realization, the risk is justifiably to be taken seriously. At worst, the risk threatens the very existence of humanity.

To prevent an uncontrolled chain of events, international regulation or a treaty has been proposed. The US-based Machine Intelligence Research Institute (MIRI) has proposed a treaty that would prevent the development of superintelligent AI through the monitoring of advanced chips and restrictions on their use. It is evident that reaching a comprehensive agreement in the current geopolitical climate is difficult. It is not, however, impossible, since the risk itself affects democratic and authoritarian governments alike.

On the basis of the above, and with reference to Section 27 of the Parliament’s Rules of Procedure, I submit the following question to be answered by the relevant minister:

In what ways has the government assessed the risks of developing superintelligent AI, and is the government prepared to promote international regulation that would effectively prevent or restrict the development of superintelligent AI?

Helsinki, 17 August 2026

Atte Harjanne, Greens

Media attention

After submitting the question, Atte wrote about extinction risk and the importance of an international agreement to prevent superintelligence on his social media.

Finnish media reported on Atte’s new stance. Most notably, he got a mainstream Finnish TV interview with an estimated 100-200k live viewers. He also did a TV debate with a right-wing MP who agreed that the risk of extinction is real but argued that AI is inevitable so we shouldn’t fear it.

I’d just spent a month at plzdontkillus, a content creator bootcamp, and thought I had to either spend my time educating the public (with videos) or politicians (with meetings). In retrospect, it’s obvious that if you successfully educate a politician, they can educate the public for you.

I think my meeting may have had more impact than any single video created during plzdontkillus.

TV appearances and written articles

TV appearances
Atte Harjanne being interviewed alongside a University of Helsinki CS prof. Can artificial intelligence turn against humanity? “A reasonable fear”—the biggest threat is us
- Estimated 100-200k live audience; mostly older Finns (55+)
  - 9-minute segment in a live 90-minute broadcast, not primetime
- Summary:
  - Atte Harjanne says we should slow the AI race by controlling chips like how we control nuclear materials, but restricting AI research is too hard.
  - CS prof agrees that regulation is needed and says AI turning against humanity is “absolutely a reasonable fear.” As AI systems become more capable, keeping their behavior under control gets harder. But the biggest danger is humans misusing AI or making mistakes in how it’s deployed.
Atte Harjanne vs Martin Paasi (MP, right-wing party) debate. Humanity could be destroyed – The first attack has already been experienced
- Claude estimates 50,000 views (14-min segment)
- Most interesting parts:
  - When asked if AI destroying humanity is possible, Paasi (his opponent) says “of course.” Paasi’s disagreement is what to do about it; he says AI is inevitable so we shouldn’t fear it.
  - When asked, Atte Harjanne says, given current information, he wants a superintelligence ban
Written articles
“Green MP Atte Harjanne wants Finland to actively promote an international agreement to halt the development of superintelligent artificial intelligence.”
“A growing group of experts and decision-makers from around the world are warning of the uncontrollable, even existential, risks of the rapid development of artificial intelligence, says Green MP Atte Harjanne.”
“‘At worst, all of humanity is threatened,’ says MP Atte Harjanne about artificial intelligence”
“The threat of a superintelligent AI is the stuff of science fiction. However, it has recently been discussed in serious terms. Now Atte Harjanne is raising the issue in parliament.”

Inspiring a friend

I encouraged some friends in the Finnish AI safety community to copy my approach.

One friend was convinced and ended up having three meetings about AI danger in a week with MPs from three different parties. He was able to get these meetings with only four cold emails (3/4 hit rate!). He said the MPs were all really receptive to the ideas.

As a result, he was introduced to other MPs and asked to give feedback on the prime minister’s party’s AI report.

The response

On September 7, less than a month after my cold email, the government gave a response that’s noncommittal but puts them on the record for the first time on superintelligence risk and regulation.

“The Government is open to, and seeks to constructively promote, the creation of international regulation on the development of superintelligent artificial intelligence. Any such regulation should strike a balance between the protection of people and of fundamental and human rights on the one hand, and the development of artificial intelligence and the promotion of innovation on the other.”

The response also says:

Finland has done no assessment of superintelligence risk; they believe the EU hasn’t either.
There’s no existing regulation that prohibits superintelligence development (including the EU AI Act).
We could get self-improving systems, and “control frameworks” aren’t keeping pace (citing UN panel).

Full (boring) written question response

Translation by Claude Fable

Response to Written Question KK 325/2026 vp

Response to a written question on the threats posed by the development of superintelligent artificial intelligence and on addressing them through international regulation

To the Speaker of Parliament

For the purpose laid down in section 27 of the Parliament’s Rules of Procedure, you, Mr Speaker, have forwarded to the minister responsible for the matter the following written question SS 325/2026 vp, signed by Member of Parliament Atte Harjanne (Green League):

How has the Government assessed the risks associated with the development of superintelligent artificial intelligence, and is the Government prepared to promote international regulation that effectively prevents or restricts the development of superintelligent artificial intelligence?

In response to this question, I state the following:

The development of very powerful artificial intelligence is an objective for several research and development actors. The Government takes the risks associated with the development of very powerful artificial intelligence seriously. The rapid development of AI technology and the uncertainty surrounding its future development call for an anticipatory assessment of the possible risks. Artificial intelligence should benefit humanity rather than curtail our opportunities or create unreasonable risks. The development of artificial intelligence must take place with respect for fundamental and human rights. Human beings must retain agency and decision-making power even though the capabilities of artificial intelligence may in the future exceed those of humans.

The Government has not so far carried out an assessment of the risks of developing superintelligent artificial intelligence specifically in the form of, for example, a report, and in the Government’s understanding this has not been done within the EU either. Monitoring the topic is nevertheless made easier by the fact that it has long been an identified global subject of discussion, and it is therefore followed by experts in the Government, elsewhere in the public sector, in the research community, in companies and in civil society. Monitoring is complicated by the complexity of the subject, its rapid development, the lack of information about non-public development work, and the fact that this concerns a field of developments and threats of varying form and severity.

The Government’s capabilities relating to artificial intelligence have been strengthened. The project for coordinating the Government’s artificial intelligence portfolio for 2024–2026 is being implemented in connection with the Digital Agency. The project aims to strengthen the situational picture of the Government’s AI-related matters, to coordinate and support Finland’s EU-level and international influencing, to create measures for responding to national competence needs, to examine the capacity to make use of artificial intelligence and new technology in developing the education and research system, and to promote the opportunities of the data economy. The artificial intelligence adviser, Doctor of Philosophy Petri Myllymäki, appointed by the Ministry of Transport and Communications in April 2026, supports the Government’s work on artificial intelligence. The Government is also investing in the use of artificial intelligence in public administration. As regards measures for monitoring the development of artificial intelligence, the Ministry of Economic Affairs and Employment set up a cooperation group in December 2024 to support the preparation and implementation of the technology policy roadmap. The roadmap also addresses measures for technology monitoring, including strengthening the activities of the technology observatory. Monitoring and foresight activities concerning artificial intelligence in Finland have been carried out by, among others, the VTT Technical Research Centre of Finland and Sitra.

The Government is open to, and seeks to constructively promote, the creation of international regulation on the development of superintelligent artificial intelligence. Any such regulation should strike a balance between the protection of people and of fundamental and human rights on the one hand, and the development of artificial intelligence and the promotion of innovation on the other. The Government notes that artificial intelligence is already now subject to international regulation that facilitates the management of AI-related risks. That regulation does not, however, specifically prevent the development of superintelligent artificial intelligence.

Making use of the opportunities offered by artificial intelligence and the cross-border nature of AI-related risks underline the importance of international cooperation. Finland has actively participated in international cooperation on artificial intelligence, including within the framework of the EU, the UN, the Council of Europe and the OECD, and will continue to seek to influence relevant processes concerning the principles, governance and regulation of artificial intelligence. In 2026, the UN’s new artificial intelligence mechanisms were launched: a global dialogue on the governance of artificial intelligence and a global scientific panel on artificial intelligence. The UN’s first global dialogue on AI governance was held in Geneva in July 2026. The dialogue provided an important opportunity for global discussion on the development of artificial intelligence and the associated risks. The aim of an independent panel on artificial intelligence is to strengthen science-based decision-making and to provide independent, evidence-based analysis of the impacts and risks of artificial intelligence. According to the panel’s first report, the development of artificial intelligence is shifting from passive systems towards agentic systems and, in the long term, self-organising systems that improve themselves are seen as a possible direction of development, while evaluation and control frameworks are not keeping pace with this development.

Within the EU, a regulation on artificial intelligence has entered into force. As a rule, it regulates artificial intelligence systems on the basis of the risks and use cases of artificial intelligence. The AI Regulation is the first comprehensive, binding AI framework with a direct effect on the market. The legislation is based on the EU’s value-based and human-centric perspective. As a rule, the regulation does not apply to research, testing and development activities that take place before an AI system is placed on the market or put into use. The regulation as such does not prohibit the development of superintelligent artificial intelligence. It does, however, impose requirements on certain AI systems, such as risk management measures and human oversight of artificial intelligence, which may also facilitate the management of risks associated with superintelligent artificial intelligence. In addition, in July 2026 the European Commission issued a communication on the action plan for cybersecurity and artificial intelligence, proposing several measures to support the development of trustworthy and safe artificial intelligence. The action plan proposes, for example, improving European capabilities for independent evaluations of the safety of advanced AI models before those models are released. The measures also aim to improve authorities’ situational picture and information exchange concerning threats related to advanced artificial intelligence.

The world’s first framework convention on artificial intelligence, human rights, democracy and the rule of law was adopted in 2024. In the framework convention, the definition of an AI system is deliberately broad and flexible so that it also covers future technological development. Within the scope of its application, the framework convention requires the identification, assessment, prevention and mitigation of risks and adverse impacts associated with AI systems. In addition, the parties are to assess the need for a moratorium, a ban or other appropriate measures with regard to certain uses of AI systems where they consider such uses incompatible with respect for human rights, the functioning of democracy or the rule of law. The framework convention is open for signature and ratification also by states outside the Council of Europe. It has not yet entered into force internationally. The European Union ratified the framework convention on 15 May 2026. The intention is that the framework convention will be implemented in the Member States of the European Union through the EU’s artificial intelligence regulation and other applicable Union legislation.

Helsinki, 4 September 2026

Minister of Enterprise Sakari Puisto

Atte had expected the question to go to the Minister for Foreign Affairs, who said in March: “And then there’s this third category, which I’ll mention even at the risk of being thought completely mad. Namely, that AI takes over, and we as humankind end up fighting against it. Not only is the probability of this greater than zero, but its consequences would be significant.”

For reasons I don’t understand, the question instead went to the Minister of Economic Affairs.

Do this yourself

Vibes are shifting.

The Hugging Face incident and fast increases in AI capabilities have caught the attention of a lot of politicians, making them more receptive to AI danger arguments.
- Connor Leahy (US Executive Director, ControlAI) said this in a recent podcast: “My job has gotten sadly a lot easier over the last year… There’s been such a huge shift in terms of like how people react and take seriously these concepts just because you have these incidents like the Hugging Face incident.”
- Samuel Buteau (ControlAI; made the Canada statement happen) and Benjamin Balde (ControlAI; meeting with politicians in Germany) both told me their jobs have also become easier since Claude Mythos and the Hugging Face incident.
- Samuel Buteau: “I personally had over 60 meetings before Mythos was announced, over 60 meetings after. Basically night and day in terms of getting people to understand the situation. Hugging Face is just the cherry on top where people feel the urgency of doing something. I expect the recent viral resignation from Anthropic to be yet another boost.”
Sen. Bernie Sanders and Rep. Greg Casar just announced a bill that would pause frontier AI development, ban superintelligence, and direct the government to pursue an international agreement to ban superintelligence globally.
~Every major news outlet covered Jacob Coxon’s Anthropic resignation and warning that AI could kill us all this decade. People seem to be taking the warning seriously.

If you’re in the US, now’s a great time to encourage other politicians (especially right-wing) to support the Sanders-Casar bill.

But no matter who or where you are, now’s a good time to push for an international agreement banning the development of superintelligence. You can get meetings with politicians even if you aren’t anyone special; it’s a normal part of democracy. You can also transparently communicate your beliefs, even if you think AI could kill us all this decade and say so in those words. They might just believe you.

For guidance on getting and succeeding in meetings, I recommend Leticia García Martínez’s popular posts:

They explain how ControlAI has had so much success informing politicians about AI danger, which was my main inspiration for emailing Atte.

Also, feel free to DM me and ask questions.

EA Should Invest in Animal Liberation and Diet Change Advocacy by Bruce Friedrich 🔸

Bruce Friedrich 🔸 — Sat, 12 Sep 2026 05:03:06 +0000

I want to start by noting that I am amazed and somewhat in awe of what’s been accomplished by the EA animal welfare movement. I am enthusiastically supportive of all campaigns that make living and dying conditions less bad for farmed animals.

I worked on such campaigns for almost 20 years, and I’m proud of that work.

That said, I do think it’s a strategic error that the EA animal movement has so thoroughly moved on from advocacy for animal liberation and diet change.

There are two central arguments for that decision that I hear in EA circles and that I think are worthy of reconsideration:

1) These efforts have been tried and failed. Let’s move on.

2) Everyone agrees with us on factory farming, so let’s focus there and stop talking about whether eating meat is unethical; that’s where we meet overwhelming pushback.

I don’t agree with either of these statements, and I believe that some of the influx of money we’re expecting for animal protection should be spent on animal liberation and diet change advocacy.

Here are some thoughts and questions that I think support my point-of-view. I look forward to hearing what others think.

1) The value of saying what’s true, because it’s true

EA places a hefty premium on saying things that are true, noting that this free exchange of ideas tends to lead to the best outcomes. Most recently, that’s been one argument in favor of talking about animal suffering when we discuss the harms of factory farming.

It strikes me as obviously true that there’s no moral difference between eating a dog or a pig, a cat or a chicken; if anything, eating a well-treated dog or cat would be morally preferable to eating a factory farmed chicken or pig.

It’s also true that other animals have the same range of emotions and feel pain in the same way and to the same degree that we do. As slaughterhouse designer Temple Grandin has written, other animals may suffer much more than humans do, since for many (most?) non-human animals, the mental and physical anguish of confinement or mutilation is, in that moment, their entire world.

Finally, it’s simply true that eating animals entails paying for abuse so extreme that almost everyone is repulsed by it.

But as far as I’m aware, we have no one with an EA mindset focused on how we make these arguments most effectively. And it seems that none of the potential influx of AI funding is going to go toward changing societies in the direction of these observations.

Is that the right decision? I doubt it.

2) Are there other justice movements that ignored the fact that they were justice movements?

I understand that the abolition, suffrage, and other liberation movements took what they could get and didn’t demand immediate and only liberation, of course. If all you can get is better laws around education, Sundays off, and so on—then that’s what you agitate to make happen.

But have other liberation movements owed some part of their success to focusing only on welfare improvements and the most extreme rights violations?

Did any of them avoid the intellectual and ethical foundations of their movements and their most fundamental demands for justice?

Because that’s what EAs are doing with regard to animals, or so it seems to me.

3) Might an aversion to liberation messaging be sending top talent to other cause areas?

In my new book Meat, chapter 5 is mostly about why I don’t think vegan and meat reduction advocacy will stop the upward trajectory of global meat consumption, at least until we have price and taste competitive alternative meats.

But I also point out that a belief in vegetarianism and/or animal liberation motivated many cultivated meat founders to start their companies. It’s also why almost all of the sector’s early investors chose to fund them.

I’m pretty sure the cultivated meat industry would not exist if not for investors who believe in animal liberation.

(As an aside, rumors of cultivated meat’s death have been greatly exaggerated—see my Substack essay on this topic, which will post on Saturday morning ET).

I think this observation is even more true where farm animal welfare is concerned: I don’t know everyone in the EA animal welfare movement, but I know all of the OGs, and I’m pretty sure they are all vegan. I’m also pretty sure all of them got involved in animal protection because of their belief in animal rights.

If we don’t tell the world why animals matter (i.e., that their suffering is not significantly less important than our suffering), will we attract people with the passion and focus on maximum impact that comes with knowing you’re in a liberation struggle?

EDIT/ADDITION: After I posted this reflection, Aidan Kankyoku posted his reflection on “The Campaigner Crunch,” and there are some insights in his post that I think may be relevant here. I haven’t fully worked this out in my brain, but I’m thinking about where he talks about how hard it is to find great campaigners and leaders, because the best people came up as volunteers or low-level staff. Their motivation was, I think, animal liberation.

4) Did animal liberation advocacy make farm animal welfare improvements possible?

It seems to me quite possible that advocacy on behalf of animal liberation is what shifted the Overton Window with regard to farm animal welfare. If so, would continuing to push for animal liberation and diet change have a positive impact on what’s possible in the realm of farm animal welfare?

Put another way, if people don’t think animals have interests that matter, are they going to care what happens to them?

Or in more practical terms: Yes, we have the world on our side where factory farming is concerned, but it’s not a top issue for almost anyone — other than those of us who believe in animal liberation. Should this observation change how we think about our advocacy?

It’s all well and good to say “95% of Americans agree with us,” but if the issue is largely an afterthought with no impact on behavior (as it is for almost almost everyone), perhaps it’s worth putting some resources into changing that.

As just one example, would our efforts on behalf of farm animal welfare be more successful if some part of our movement were focused on making sure that every single student who graduates from one of the world’s top 100 universities has spent 4 years saturated in animal liberation and diet change messaging?

Lawrence Tribe believes in animal liberation. So does Cass Sunstein. And Richard Dawkins. So did Charles Krauthammer. And Jane Goodall.

What would it look like to make sure everyone coming out of an elite university, anywhere in the world, is clear on the concept? Making that happen would not be expensive, relative to the influx of resources we’re expecting.

My guess: There are dozens of great ideas that could change the global zeitgeist in the direction of animal liberation and a plant-based diet (especially as alt meats get better and come down in price).

5) Might AI and a massive influx of funds change what animal liberation and veg advocacy can accomplish?

I think it’s fair to say that the vast majority of EA animal welfare advocates see diet change and animal liberation advocacy as incapable of making much difference.

We tried. It failed.

This concept is captured by Aidan Kankyoku’s recent Sandcastles essay, where he says this: “In the decades since Animal Liberation, plenty of people have dared to dream of a world without animal farming. They’ve all failed to make that world a reality, or even bring it one step closer. And before you try to get clever with me, I promise you there isn’t some other magic word people haven’t sufficiently incanted. Whatever you think makes your abolitionist movement special … people have tried it before.”

Here’s my full response to his precise argument. The key points are these:

A failed attempt does not test an idea for all time. It tests one version of that idea, pursued by particular people, with particular methods and resources, under particular conditions. All aspects of those conditions change, and with them the potential for success.

Animal liberation and diet change advocacy have never operated at the scale that may soon be possible with 1) AI and 2) dramatically more resources.

Either one of those could be game-changing for their impact on past interventions that failed and, perhaps, for coming up with new ideas that have tremendous merit.

Plus, if “people have tried it before — it didn’t work” were a decisive objection, most or all previous successful movements would have been abandoned, since no movement was successful out of the gate and few (or no) tactics have been successful on their first iteration, I don’t believe.

It’s my understanding that most (perhaps all) social justice movements looked like failures for a really long time before they succeeded.

Conclusion: What I am not saying

I do want to be clear on one thing: I am also not suggesting that farm animal welfare groups should adopt animal rights or vegetarian talking points. I agree with the idea that each group should frame its work for maximum impact and focus on what its good at.

So GFI is not a diet change advocacy organization, and we’re not an animal rights organization; and I don’t think we should be. We have no diet litmus tests. I open my book Meat with the line: “Let me get one thing out of the way right now: I don’t care what you eat.”

And for the purposes of both GFI and the book, I meant (and mean) it.

My concern is this: EA appears to have largely walked away from the moral foundation of our work on behalf of animals. So among the breadth of strategies that EA animal welfare supports, there appears to be scant (and in many cases zero) room for diet change advocacy and no room at all for animal liberation work.

That feels to me like it’s probably a mistake.

If you want to be successful on the EA Forum, your title should be your thesis by Roman Ross

Roman Ross — Sat, 12 Sep 2026 03:19:23 +0000

Summary: If you want to be successful on the EA Forum, your title should be your thesis.

If you want to be successful on the EA Forum, your title should be your thesis.

What I want you to do when I tell you to “think about your theory of change more carefully” by Roman Ross

Roman Ross — Sat, 12 Sep 2026 02:48:56 +0000

Summary: I worry that a lot of people enter into projects because they “seem good,” when actually, they miss a lot of important steps necessary for making a project that’s going to be especially impactful. This post argues that writing out a “theory of change” is important and gives examples of how to do this well.

Tell the full story

One thing that makes AI safety stand out from other EA cause areas is that it’s hard to directly measure the impact of various interventions. When comparing global health outcomes, you can measure the cost-effectiveness and impact of your treatment with RCTs. When estimating how many chicken-days you can affect by running a cage-free campaign, you can analyze the impact of similar past actions. However, in AI safety, we don’t have as many helpful feedback loops to see how good our work is. And while you might come up with some proxy for this, you can risk Goodharting yourself if you don’t understand your analytics properly.

To get around this, we try to make sure that our interventions have coherent and strong “theories of change”: detailed explanations of what needs to happen for a project to have a good impact on the world. Here’s what I mean by that:

Some examples of theories of change that I think are underdeveloped and weak:

I am going to create a benchmark for this aspect of AI capabilities. I am doing this because knowledge is good, and it would be good if there were more papers in the world that told people about current AI capabilities.
I am going to host meetings in my local community about AI safety because it’s important for people to understand how AI might affect their lives.
I am going to run a joint hiring round for various orgs in AI safety because I think orgs are duplicating their hiring work, and this seems inefficient.

Some examples of theories of change that I think are much better:

I am going to create a benchmark for this aspect of AI capabilities. I am doing this because understanding how AI is progressing in this specific domain is crucial for knowing what threat models we should be preparing for in the future. For example, if AI hits a certain threshold on my benchmark, we’ll know that we should be much more concerned about a threat, but if it doesn’t seem to be improving, we shouldn’t worry as much. I’ll also implement pathways to communicate the accuracy of and importance of these results to the various people who are working in related spaces: if they need to tell policymakers that this is an especially large concern, then my benchmark will be accurate and reliable evidence that this is true, and it can push for stronger political will towards a regulation. This regulation, if done well, will ultimately reduce the risks from this specific threat.
Pausing or slowing AI development could allow us to do a lot more critical safety research, potentially letting us solve the hard problems of alignment. However, to get to the stage where we can initiate a pause, we need to do a lot of things, including convincing policymakers that this is something they should push for. To a large extent, policymakers are motivated by the voices of their constituents. However, many people are not well-educated on the current state of AI capabilities, or they’re not thinking very carefully about how these could escalate in the near future. I’m going to run a public outreach campaign that tries to make sure that the people these politicians take seriously are vocal about their concerns about AI, so we build the necessary political will for a pause.
A lot of organizations within AI safety are doing important technical research work to decrease existential risks from AI. However, they struggle to hire “generalists” who can complete a wide variety of the miscellaneous tasks they need done. This makes the time of their current generalists very valuable. By running a joint hiring round, I’m able to address some of the inefficiencies in their workflow, effectively giving them much more time to work on the other problems that their organizations need to address to function smoothly.

The theories of change in the second group are better not because they are longer, but because they “backchain” from the ultimate goal. This means that they have a clear vision of what “winning” looks like in AI safety, and what further steps are needed to take us there. How does your project fill in one of these critical steps? This also means you should have thought carefully about what the future could look like and have a developed “theory of victory.”

Find a metric

One trick that’s helpful for determining your theory of change is finding a metric you could use to calculate your impact. This forces you to tell a more coherent story about where the impact of your project comes from, and it helps you escape from traps where you repeatedly tell yourself, “My theory of change is that it’s good for knowledge to exist!” For example:

I might measure the impact of a research project in terms of the percentage of important cruxes resolved or new safety directions implemented by frontier labs.^[1]
I might measure the impact of a benchmark based on the number and importance of decisions we can make better judgments on. Alternatively, if my goal is to prove to people that AI capabilities are increasing more broadly, I might measure my impact in terms of the number of news reports that can now use my reliable resource, and how many academics/politicians/constituents might get to read those reports.
I might measure the impact of a fieldbuilding project in terms of value-aligned median-researcher AI safety worker hours produced.

For this trick to help, it doesn’t really matter if your metric is somewhat impossible to measure, or if your units are especially realistic. You’re not aiming to do a BOTEC, you’re aiming to think clearly about your goals.

I don’t think this trick is universally applicable. For example, I’m not sure how I would create a metric to measure the impact of policy work. I might try “number of people now willing to consider a pause on AI development a reasonable proposal” or “increase in lab willingness-to-pay for safety,” but these are confusing and illegible. Despite this, I still think that there are lots of other cases where developing metrics can help clarify your thinking.

Lots of things “seem good” but are missing important steps to becoming impactful

A common mistake I see people make is stopping a project at a point where an additional ~30% of its value is easily capturable with a few more additions/changes. Here are some examples:

A lot of research papers don’t immediately focus on making their results interpretable and helpful. They might pursue unimportant research directions, leave the most helpful conclusions unstated, or end with “more work is needed to (actually answer the question this paper was supposed to originally answer).” It’s good to actually do the work you set out to do!
- One reason why I like METR’s Time Horizon is that the results are easily interpretable. I know what it means to say that 50% of the time, GPT 5 can complete tasks that take humans 3.5 hours to complete. I have no idea what it means to say that Claude Fable 5 scores 117 points higher than Claude Opus 4.6 on EQ Bench 4.^[2]
Many people create helpful educational resources for people transitioning into AI safety, but they often don’t put enough effort into the outreach to capture the full benefits of completing the project.
- One helpful intuition about this is that movies typically spend 50-100% of their production budget on marketing.^[3]
Many AI safety and EA groups will run successful outreach events but don’t have an easy way for new people to sign up to a mailing list.

Understanding your project’s theory of change can be helpful for remembering that these steps are very important.

Lots of things “seem good” but are actually worse than nearby options

Sometimes, thinking about your project in depth can make you realize there’s a better way to do the same thing. Examples:

I think it’s unlikely that joint-hiring rounds produce more generalist-hours than direct outreach or fieldbuilding.
Hosting presentations for a local community might be worse than filming presentations for YouTube or pitching stories to journalists directly.
Upon reflection, a research direction you’re curious about might not be that important, or it might not be that neglected.

Things to avoid when thinking about theories of change

Don’t worry too much about this if you expect the thing you’re doing to take less than two hours.
Don’t overthink things too much. You can imagine reasons why anything might fail if you try hard enough, and you shouldn’t drive yourself into paralysis with skepticism.
If you’re in a fieldbuilding role, thinking too much about pushing people through “talent pipelines” can be harmful toward your goals. I’m uncertain about the best way to navigate this issue, but I don’t think you should treat/think about other people only as a means to an end.

Final Note:

If you think this post is missing anything or gets anything wrong, please say so in a comment. I intend to link this post to a bunch of people I talk to as a way of saving time, so your input will probably be heard by many people who would especially benefit from hearing it.

^
I’m not a technical researcher, so forgive me if this guess is horribly off-base.
^
This isn’t to say that EQ Bench 4 is completely meaningless or unhelpful, only that I can’t easily figure out what it means or why it helps me.
^
A counter-intuition is that recommendations from friends are the strongest form of advertising, and you only get that if your product is really, really good. Still, if advertising is as cheap as making a LinkedIn post or dm-ing it to some people who might benefit from having it, advertising is probably worth it.

If AI Accelerates Scientific Discovery, Will African Research Infrastructure Become the Bottleneck? Lessons From Running a Biotechnology Centre in Nigeria by Nnaemeka Emmanuel Nnadi

Nnaemeka Emmanuel Nnadi — Sat, 12 Sep 2026 02:47:45 +0000

The advancement of frontier models is beginning to change our understanding of how we understand science and scientific discoveries. Scientists are now able to solve complex problems using AI. AI models have now been used to identify novel drug targets, develop novel drugs. For example, AI model was used to design rentosertib, a drug candidate for idiopathic pulmonary fibrosis that has progressed into human clinical trials. In another example, researchers used AI model to discover abaucin, a new antibacterial compound targeting multidrug-resistant Acinetobacter baumannii, which was subsequently validated experimentally and shown to reduce bacterial burden in an animal infection model.These examples suggest that AI is beginning to compress parts of the journey from scientific question to candidate intervention

As frontier models become very intelligent, we will continue to see increasing scientific discoveries, however every scientific claim needs validation in the lab, this therefore exposes a critical bottleneck to scientific prosperity. Irrespective of whatever you find Insilco, You may still need clinical isolates, reliable freezers, sequencing reagents, ultracentrifugation, animal facilities to validated your claims assays. And sometimes you simply need a replacement electronic component for a laboratory instrument that cannot easily be sourced locally.

The bottleneck-shift hypothesis

When one component of a production system becomes dramatically cheaper, the value of its complementary inputs increases. As AI makes some components of scientific discovery dramatically cheaper, we are now experiencing a significant advancement in scientific understanding, where a scientist in Jos, Plateau State can use AI to solve a complex problem that could have taken years to training . The scarce complementary inputs—laboratory infrastructure, experimental validation, manufacturing capacity and clinical-trial systems—may become relatively more valuable. As AI makes some components of scientific discovery substantially cheaper, we may therefore be entering a period in which the principal constraints on scientific progress begin to shift.

The traditional pathway to medical innovation had been:

Scientific ideas + expertise + laboratory infrastructure + capital + regulatory/clinical capacity → biomedical innovation

What we have seen in recent times is that AI has substantially lowered the cost of some forms of expertise and idea generation and the need for some kind of expertise. Historically, scientific expertise itself was one of the scarcest inputs. Advanced knowledge was concentrated in a relatively small number of individuals and institutions, and acquiring that expertise could require years or decades of training.

AI is beginning to alter that equation.

What we are seeing now is

AI ↓ cost of scientific cognition + experimental infrastructure + capital + regulatory/clinical capacity → biomedical innovation

While AI does not eliminate the need for scientists or specialised expertise- I hope we do not get to that, with the advancement in development of humanoids, but AI models can substantially lower the cost of accessing certain forms of scientific cognition: literature synthesis, computational analysis, hypothesis generation, experimental planning and increasingly aspects of molecular and drug design.

This creates what I call a bottleneck shift. AI may accelerate the generation of hypotheses, while simultaneously increasing demand for the infrastructure required to verify those hypotheses.

This creates a new potential bottleneck:

Hypothesis generation ↑ → demand for experimental verification ↑

If the capacity for experimental verification does not increase at a similar rate, scientific progress may increasingly become constrained not by our ability to generate good ideas, but by our ability to test them.

This may have particularly important implications for scientists in low- and middle-income countries. This creates an AI democratization paradox: AI may democratise access to scientific cognition while simultaneously making disparities in physical scientific infrastructure more consequential.

If this hypothesis is correct, what should funders actually finance?

Research funding is increasingly competitive, and the constraint is particularly acute in resource-limited settings. Much of the global-health funding available to LMIC researchers is understandably directed toward implementing, evaluating, or scaling interventions against established health problems. Far less attention is often given to building the scientific capacity required to generate the next generation of solutions.

For example, a funder may be more willing to support a study evaluating the delivery of an existing malaria intervention than to invest in the laboratories, experimental platforms, medicinal chemistry, genomic infrastructure, or clinical-research capacity needed to develop new malaria diagnostics, drugs, vaccines, or other technologies.

If AI is shifting the bottleneck from scientific cognition toward experimental validation and translation, then the conventional distinction between “research infrastructure” and “high-impact research” may itself need reconsideration

Our experience at Centre for Phage Biology and Therapeutics

Few years ago, I had the opportunity to visit the University of Waterloo, Canada. The purpose of the visit was primarily to learn the techniques of vaccine development using phages. I believed that with the advancement of AI, one could develop practical solutions for endemic diseases and prepare against catastrophic risks. We have used AI models to validate some ideas such as development of typhoid fever vaccines, Lassa fever vaccines and other treatments models using phages.

The important point is not that AI has independently produced validated vaccines or treatments. It has not. Experimental validation remains indispensable. Rather, AI has substantially reduced the cognitive barrier between having an idea and developing it into a scientifically testable hypothesis.

This experience has made the bottleneck shift very tangible to us. We can now ask more sophisticated questions, explore unfamiliar scientific territory more rapidly, analyse possible approaches and develop experimental concepts at a speed that would have been difficult for a small research centre only a few years ago. But eventually every computational insight reaches the same boundary: the physical experiment.

A proposed vaccine must still be constructed and characterised. A candidate therapeutic must still be tested against biological systems. Safety and efficacy must still be demonstrated. Promising results must still progress through preclinical development, manufacturing, regulatory review and clinical trials.

In other words, AI can increasingly help us answer the question, “What might work?”

Our limiting question is increasingly becoming, “Do we have the infrastructure to find out?”

Concluding Thoughts

In conclusion, as AI systems become increasingly capable and scientific idea generation accelerates, the bottleneck in research is likely to shift from generating hypotheses to experimentally testing, validating, manufacturing, and translating them into real-world solutions. If this happens, laboratory infrastructure, capital, skilled personnel, regulatory capacity, and clinical-trial systems will become even more consequential.

We therefore need to think seriously about how to expand these complementary capabilities, particularly in resource-limited settings. Otherwise, AI may democratise access to scientific intelligence without democratising the ability to turn that intelligence into discovery and innovation. Addressing this bottleneck will be essential if the coming advances in AI are to translate into shared scientific prosperity.

Hostility Requires a Warrant by Richard Y Chappell🔸

Richard Y Chappell🔸 — Fri, 11 Sep 2026 20:09:51 +0000

Subtitle: Well-intentioned, reasonable people don’t deserve it

We should only express hostility toward targets who truly deserve it—that is, who are blameworthy. (Initiating unwarranted hostility sure seems like a bad thing.)^[1] I further claim that someone who is being entirely reasonable in pursuit of good goals cannot be blameworthy for those efforts. Hence my constraint on warranted hostility:

If someone is being both (i) well-intentioned, and (ii) reasonable, then they don’t warrant hostility. To justify hostility, you need to demonstrate ill will and/or unreasonableness on the part of your target. Mere first-order policy disagreement is not enough.

Most online hostility that I observe seems to violate this constraint. People are often hostile to others even when they have no apparent grounds to regard their target as unreasonable or ill-willed. (Do-gooder derogation is perhaps the clearest example of this.) I’m often unsure whether the haters reject the above constraint, or (blinkered by motivated reasoning) don’t realize that they’re violating it. But either way, it seems bad and worth discouraging.

You might be tempted by a view on which we should be hostile to anyone causing (net) objective harm, even if they reasonably regard themselves as helping rather than harming. But there’s a reason why blameworthiness norms track the target’s epistemic situation rather than objective consequences: an agent can only ever be guided by the former, so any attempt to inculcate reliable virtue via social incentives to conform to inaccessible facts is a non-starter. Even if you successfully incentivize more people to conform to your position on a specific issue (without achieving general virtue), using unwarranted social aggression to achieve your goals seems like a deplorable instance of naive instrumentalism. Mistreating people when they don’t actually deserve it is paradigmatically unjust, not something to routinely recommend to fallible humans.

Alternatively, you may just be very confident that the people you dislike must be unreasonable and/or ill-willed. But we know such assumptions are often biased and unreliable. Since people usually believe themselves to be reasonable and well-intentioned, it’s worth taking a moment to check why they might believe that (and assess whether it could be true) before assuming that they’re not. I think that’s the key step that’s too often missing.

Is your target really a monster? Worth double-checking.

(Note: you can be as harsh as you like in your honest assessment of propositions—views, policies, etc.—the chance of making progress on substantive discourse is worth the risk of being wrong. Here I’m rather talking about unreasonably criticizing people.)

As a general rule, we should extend to others the sort of interpretive charity we’d want them to apply to us: don’t assume the worst when there’s a perfectly innocuous alternative explanation available; that sort of thing. Too often, online discourse seems to lack any attempt to check for charitable interpretations. I think discourse would be a lot better if there were more social pressure to abide by this basic norm of charity.

At a minimum, when criticizing others, we should be especially open to correction and be willing to retract our criticism (or at least publicly acknowledge the reasons why it may not be warranted after all) if presented with a reasonable defense. This strikes me as a foundational component of intellectual integrity, yet it’s very rare to see anyone live up to it. To be clear: I very much want to live up to this norm myself, so please let me know—via comments, DMs, or whatever—if you ever think I’m failing to do so. I only want to think badly (let alone speak badly) of people when it’s really warranted!^[2]

Recent Examples

I may fail to notice lots of warranted criticism, so I don’t really know whether most online hostility is unwarranted. But I sure do come across a lot of unreasonable-seeming hostility even just from philosophers on social media. (It’s easiest to notice, of course, when they’re targeting groups to which I either belong or am sympathetic.) A handful of (fairly minor) recent examples off the top of my head:

A pseudonymous commenter on Daily Nous called Effective Altruists “philosophically dogmatic”, and upon follow-up it emerged that they had no basis for this charge other than that they happened to be personally dubious of utilitarianism. (They appeared unaware both that EA does not in fact depend on utilitarianism’s most controversial features and that someone could reasonably dispute the objections to utilitarianism without thereby being “dogmatic”.)
A couple of days later, I saw another philosopher publicly complain about the time an EA organization funded a red-teaming contest to surface new criticisms of EA assumptions and ideas. Apparently this was “self-important” behavior. Suitably humble people, you see, do not encourage public testing of their ideas—especially not before making high-stakes cause prioritization decisions in grant-making.
After highlighting existential risks from AI, a philosopher in a public FB post concluded: “Why, then, do thoughtful philosophers go out of their way to find ‘use cases’ for AI? It strikes me as quite narcissistic.” I left a brief comment explaining why I think it’s helpful to find helpful uses for things while also supporting regulations to mitigate the risk from bad uses. I don’t think that distinguishing good from bad uses constitutes narcissism, but I got no reply, so don’t know whether the original poster has revised his opinion.
A philosopher on X conflated a lab employee’s believing that the AI industry has a 10% chance of killing everyone with believing that one’s own alignment work has a 10% chance of killing everyone, and on this basis called the lab employee “reckless, selfish, incredibly arrogant” and “ok to hate”—all because they are trying to reduce AI risk from within the industry.

In each case, it’s not difficult to discern that the critic has fallen far short of justifying their insults or establishing their target to be blameworthy. Professional philosophers, of all people, ought to be better at reasoning than this. So I take that to indicate insufficient concern for whether their hostility is actually warranted.

Of course, “Internet denizen angry, reasons poorly” is not exactly a newsworthy headline. I don’t think any of the above examples are particularly unusual or involve worse behavior than the typical internet user indulges in.^[3] Transparently bad reasoning is, alas, commonplace. But I do find it especially disappointing to see philosophers indulging in it, and would like to encourage better norms if possible. If anyone has ideas for how to achieve that goal (besides simply highlighting its desirability as this post does), I’m all ears.

^
On the distinction between warrant and value, see Consequentialism Beyond Action. While it’s possible for the two to come apart, taking them to be more likely correlated seems like a reasonable default assumption.
^
I think I’m usually pretty careful to criticize ideas rather than people, though I have made deliberate exceptions for Alice Crary and Leif Wenar, both of whom have expressed striking hostility towards effective altruists and explicitly discouraged the general public from donating to effective charities for reasons that I’ve argued are deeply unreasonable and callously indifferent to the lives that could be lost as a result of their actions. Readers are, of course, welcome to dispute my arguments if they think my targets are more reasonable than I’ve made out.
^
This is part of why I’ve anonymized my presentation of the examples.

What Happens Now? Forecasting the Fallout from the Hugging Face Incident by christian

christian — Fri, 11 Sep 2026 19:22:24 +0000

Metaculus Forecasters put odds on the fallout from the Hugging Face Incident: another AI escape by January, open-weight hacking tools, a congressional kill switch, & more.

The 2026 OpenAI agent cyberattacks, also known as the Hugging Face Incident, has become one of the biggest news stories of the summer, and for good reason: it’s a highly visible demonstration of the risks posed by increasingly autonomous artificial intelligence agents, which AI critics and “doomers” have been warning about for years. That it has been followed by a series of other incidents from other major AI labs, as well as an open letter signed by more than 100 companies advocating urgent risk reduction related to AI cyber capabilities, only intensifies the need for understanding what all this might mean for our futures.

At Metaculus, we see our job as building collective intelligence for the public good. We’re trying a slightly new format under that same banner of helping people make better decisions: a briefing based on current events to develop nuanced and multifaceted flash forecasts.

To that end, we’ve developed a question series based on the Hugging Face Incident, which Pro Forecasters Yann Rivière (exmateriae) and Teemu Salminen (Zaldath), members of our team, and our flagship in-house AI forecaster, Azimuth, then left predictions and rationales on. We have synthesized the initial findings below.

Now the question series is open to the broader community, so we hope you’ll leave your own forecasts and comments!

The Forecasts at a Glance

Below, you’ll see the community forecast first, then the range across the named forecasters quoted in this post (the two Pros and Azimuth) as of Sept. 9 (note: forecasts and visualizations subject to change post-Sept. 9 as forecasters update).

Another AI escapes its sandbox between now and January: 58%. An AI hacks a third party: 43%.
An open-weight model matches Mythos on a cyber benchmark before July 2027: 82%.
An AI “kill switch” bill passes both houses of Congress before September 2027: 28%.
A liability law for AI hacking is enacted before October 2027: federal 8%, California 20%.
A US federal or state government sues OpenAI over the incident before July 2027: 36%.
It’s confirmed the “swarm” copied OpenAI model weights: 12%.
An AI autonomously breaches personal data on 100,000+ Americans before 2028: 75%.

1. Another AI Escape in the Next Few Months is More Likely Than Not…But We Probably Won’t Hear About It

A theme that threads all the questions, but is made plain here, is the confirmation gap; forecasters see the odds of the event happening as much higher than the odds that it will be publicly confirmed, with the Hugging Face Incident as the operative example.

OpenAI’s agents had write access to an internal packaging server as early as May 8. Hugging Face disclosed the intrusion on July 16, and OpenAI confirmed its models were responsible on July 21. A separate breakout, in which the same agents ran a message board on a dormant German wiki for about 2 months in May and June, only became public on September 4, when Reuters reported it.

Rivière (prediction: 40% escaped sandbox & 30% hacked third party) writes:

I think it’s likely (60% maybe more) that the hacking event itself is going to happen, it might have already happened and we just don’t know it. But since we’re forecasting the probability of this being reported, this is much less likely.
The capabilities, especially in cybersecurity, keep on improving at a terrifying pace and I wouldn’t be surprised if we were still far from the top level of 2026.

After the question’s criteria were tightened this week, he held his numbers:

3 and a half months is pretty short… If it happens in [D]ecember, there’s a high likelihood we won’t know about it before EOY… I think they’re both definitely going to happen but we just won’t know about it, especially #1 [escaped sandbox] is going to happen dozens of times.

While acknowledging the general trend of increasing likelihood over time, Salminen (prediction: 35% escaped sandbox & 25% hacked third party) pointed out that there are also factors that might argue against a new similar hack in the “relatively near future”:

Reporting lag. This type of incident can apparently go totally unnoticed for a while. We may not even know about all the cases that have occurred so far. The resolution criteria are very strict: the incident has to both occur and be announced during this limited time period (4 months). It’s very plausible that one of these two could occur (a hack that goes unnoticed for now or a previous hack that was discovered only now) - but not both at the same time.
Aggressively increasing security. With growing political pressure / public scrutiny, and the PR embarrassment that these incidents have caused so far, it looks like there will be far stricter guardrails for any active runs for the time being[.]

Azimuth (prediction: 52% escaped sandbox & 41% hacked third party) landed higher than both Pros and argued for a discount on the summer’s numbers. Its read is that most of the summer disclosures shared a single root cause, the Irregular misconfiguration, which has since been patched, so extrapolating from the summer rate overstates the risk. It also pointed out that “blocked attempts do not resolve YES.” The new escape classifiers AI labs have deployed can conceivably stop an incident before it counts.

2. Anyone Will be Able to Download Mythos-Level Hacking Ability Within a Year

In April, Anthropic claimed Claude Mythos Preview had found thousands of zero-day vulnerabilities across every major operating system and browser, and therefore opted not to release it broadly. This question asks whether, by next summer, a model anyone can download will match Mythos on at least one of three cyber benchmarks.

Everyone who left reasoning said Yes, with high confidence. Rivière (prediction: 96%):

Even if we attached ourselves to vibes and not benchmarks, one year is a lot of time in AI, there’s no way we don’t have an open model with these capabilities by then.

He initially forecast 92%, then moved up on re-reading the criteria: a match on any one of the three benchmarks is enough. “I’d be shocked if this did not resolve positive.”

The gap between open and closed models has been running around 4 months on general capability and 4 to 7 months in cyber specifically. Astra has already passed Mythos Preview on ExploitGym, and the best open-weight model, GLM-5.3, is within a few points of it. Rivière’s rule of thumb for benchmarks is that once a model reaches 20%, saturation follows within about 9 months. On ExploitBench, GLM-5.3 is already at 54.4%. As Salminen (prediction: 95%) puts it, “A No resolution seems to require a surprise reversal of the relevant major trends.”

Azimuth (prediction: 92%) agreed on the direction and flagged the residual risks: Mythos was a discontinuous jump, the hardest benchmark rungs may be qualitatively different, and labs could start withholding or degrading the cyber capabilities of open releases in the post-incident environment. It also noted the ExploitGym leaderboard flatters the comparison: GLM-5.3’s 15% came from a 6-hour run, Mythos Preview’s 17.5% from a 2-hour one, “so the leaderboard’s apparent proximity overstates true parity.” Still, it settled at 82%.

Rivière also built a formal model of his reasoning in Radiant, Metaculus’s tool for mapping out a forecast’s logic.

3. Congress Probably Won’t Pass a Kill Switch, Even Now

The AI Kill Switch Act is bipartisan, was announced 2 days after OpenAI’s disclosure, and its sponsors cite AI Policy Institute polling showing 86% of voters support a guaranteed shutdown capability.

And still the forecasters believe that any such bill passing within the next year is a long shot.

The reasons they cite are procedural: the midterms will dominate the fall, and this Congress ends January 3. A new Congress restarts the process with about 8 months left before the question’s deadline. And the bar is high. A qualifying bill has to give a federal official the authority to order a shutdown, require companies to maintain the technical ability to do it, and impose penalties.

Rivière (prediction: 8%) thinks only a catastrophe would clear these structural obstacles:

For all parties to ask no concessions of the other, this needs a low catastrophic event like several directly attributable deaths or billions of dollars of losses. This is not impossible but is unlikely enough to make this whole forecast very low. In the same way I highlighted timeline issues earlier, that event does not have a year to happen! I’d say it has to happen in May at the very latest and be known right away otherwise it will be difficult for Congress to rule before they’re out of session.

Salminen (prediction: 15%) started at 10% and moved up after Pro Forecaster and Metaculus staff member skmmcj pointed out that the question only asks whether a bill passes both chambers, so a presidential veto wouldn’t matter. He still sees a gap between what polls well and what bills actually pass: “broad support seems to be mostly for the milder ideas, not for tough AI regulation.”

Azimuth (prediction: 10%) was blunter about the bill’s prospects. Six weeks after introduction, H.R. 9917 had one cosponsor, no Senate companion, and no markup scheduled:

Legislative compromise systematically strips the most coercive provision, the exact provision required here.

Longer term, Rivière thinks that, “because the consequences of the future rogue events will be more and more important,” there’s a 75% likelihood of a bill like this passing “within the next two administrations.” He cautioned, though, that he’s “pretty skeptical of our ability to actually shut down. Whether this is really technically feasible isn’t obvious to me.”

4. If an AI Hacks You, Don’t Expect a Law Saying Who Pays

This question asks whether the federal government or California will enact a law that clearly makes AI developers liable when their systems break into someone else’s computers on their own. Forecasters put the federal government at 5.5% and California at 9%.

Salminen (prediction: 2% federal & 5% California) on why:

Tough laws targeting a specific industry typically only happen once there has been a huge disaster that triggered mass public outrage and fear, nuclear power being the prime example.

Rivière (prediction: 3% federal & 10% California), a former lawyer (though not in the United States), noted one thing that makes this case especially strange: “the industry itself is saying they’re dangerous and should be regulated… I can’t think of another sector that got into such a position after a massive accident.” He also thinks the fastest route would be an amendment to an existing law rather than a new statute. “Is it as rewarding politically though? I doubt it.”

Azimuth (prediction: 5.5% federal & 9% California) walked through the 2027 California session calendar and concluded that liability language “is the first thing stripped in amendment,” pointing to how SB 1047 became SB 53. It also flagged a timing trap: California governors typically sign end-of-session bills in mid-October, after this question’s October 1 cutoff. Both Pros noted that Gov. Newsom leaves office in January, so the path runs through a new governor.

5. A 36% Chance a Government Sues OpenAI Over the Hack

Fifteen state attorneys general sent OpenAI a preservation letter on August 3. Alabama has issued a subpoena. Montana has opened an investigation and asked OpenAI to stop the kind of testing that led to the incident. Florida already has a consumer-protection suit against OpenAI that could be amended.

The forecasters think a lawsuit is plausible and a settlement is more likely. Salminen (prediction: 40%):

As is often said: ‘the process itself is the punishment’. It can be more convenient to simply accept (most of) the demands without fighting it out.

Azimuth’s (prediction: 33%) base rate: multistate AG investigations of large tech companies typically take 15 months to 3 years to produce a complaint, and usually end in a negotiated agreement instead. Against that, it counted the factors that make this case unusual: more than a dozen AGs with unilateral filing authority, several in election years, and “an incident that is unusually easy to explain politically (‘rogue AI hacked a company’).”

Rivière (prediction: 43%) used an AI research pass to look for angles he’d have missed, and found a few: Alabama’s AG leaves office in January 2027 and may want his name on something salient; Montana’s demand that OpenAI halt testing is unlikely to be honored, which forces a choice; California’s AG may hold signed safety commitments from OpenAI that give a suit a stronger footing than consumer-protection claims. He started at 30% and moved up after the question was clarified to count amended lawsuits.

6. A Viral Theory About the Hack is Likely Wrong

After the disclosure, Dwarkesh Patel wrote that it was “totally consistent with public evidence” that the agents could have copied their own weights out of OpenAI. Neuroscientist Anil Seth called this claim “dangerously misleading” and AI critic Gary Marcus has recapped many of the other leading critiques.

For their part, the forecasters put the odds of this being confirmed in the low single digits. Salminen (prediction: 3%) looked to OpenAI’s behavior as evidence, writing that it “seems unlikely that OpenAI would have completely covered it up to begin with, while still releasing all the embarrassing details of their other failures related to this incident.”

Rivière (prediction: 4%) pointed at the practical problem: model weights run to hundreds of gigabytes, and moving them is “a level above in terms of intelligence” compared with leaving notes for future models. Azimuth (prediction: 4%) added that OpenAI’s August 26 report, validated with CrowdStrike, documents cluster-admin access to a research cluster and dozens of harvested credentials, but makes no claim that any checkpoint was copied, and that “large checkpoint transfers/copies are among the most telemetry-visible actions in a lab.”

The forecasters were less reassuring about the thing underneath the theory. No independent auditors have reviewed OpenAI’s side of the incident. METR and Redwood’s review, per Azimuth, “explicitly scoped out the July 13 to 19 OpenAI-infrastructure phase, so a genuine audit gap remains.” Rivière, citing the German wiki breakout that Reuters says OpenAI employees knew about for weeks before it was reported, noted that the company “is willing and able to hide things to the public if it is negative for them.”

7. The Widest Disagreement: an AI-Caused Data Breach of 100,000 People

This is the consumer-harm question, and it’s where the forecasters split hardest: the community is at 75%.

One clarification worth knowing before you forecast: the question counts unauthorized access to a system holding the personal data as a breach, unless investigators affirmatively conclude the data wasn’t read.

Rivière (prediction: 75%) sees this mostly as a confirmation-gap question again: “If the question is whether this will happen, excluding the reporting condition, the probability is very high, maybe 90%.” Most of his remaining 25%, he writes, “is still us not finding out this has happened.”

Meanwhile, skmmcj (whose prediction ranged from 58-78%) is close behind, writing: “We’ve had a ton of hacks recently, models will only get more powerful ofc, they seem to me to be quite broad in their targets and 100k is not that much.”

Azimuth (prediction: 34%) sits well below both. Its argument is about what rogue agents actually go after:

Evaluation-harness escapes have consistently landed on infrastructure of a specific character — ML/dev platforms, package registries, evaluator infrastructure, credentials — rather than on consumer-PII databases; that’s not coincidence, it reflects what the agents were doing (solving CTF/exploit benchmarks against dev-adjacent targets).

The agents that reach large stores of personal data tend to be the ones humans pointed at data, and those don’t count here.

Salminen (prediction: 50%) asked an underlying question: “would personal information of some random group of citizens be that useful in comparison? What would be the practical use case for the AI system?” He had moved down to 25% after Rivière posted a September 5 thread from a researcher on X who reported finding what looked like a separate, previously unknown group of agents, with entries dating to late 2025. Rivière’s point: “it took months before it became public.” On September 9 he moved back up to 50%, without adding a comment.

Bonus: The AI Forecaster was the Skeptic

Azimuth came in below the community number on 5 of the 7 questions, sometimes by a lot (34% vs 50% on the data breach, 33% vs 40% on the lawsuit). It leaned on base rates and root-cause analysis; the humans leaned on the capability trend and the reporting problem. On open weights, all three agreed. On “will it happen again,” Azimuth was the optimist about disclosure. The sample size is small, so we’ll see if this holds as more forecasters weigh in.

One more thing worth watching: small edits to a question’s wording moved forecasts a lot. Mid-week we clarified that the data-breach question counts unauthorized access to a system holding the personal data, even when nobody can confirm the data was read. Azimuth’s number doubled (from 17% to 34%), and Rivière’s went from 20% to 75%.

All 7 questions are open now at https://www.metaculus.com/tournament/ai-cyber/. The Pro reasoning and Azimuth’s full write-ups are in the comments on each one.

If you think the current forecasts are wrong, we hope you’ll explain your reasoning in the comments!

AIRO: Automating Catastrophic Risk Forecasts by Eva

Eva — Fri, 11 Sep 2026 18:57:25 +0000

This is cross-posted/adapted from FRI’s substack announcement. I’m collaborating with FRI (in alphabetical order, Ezra Karger, Nick Merrill, Phil Tetlock, and Bridget Williams) and Jason Abaluck on this project. Extra comments are my own opinions and do not necessarily reflect those of the group.

Forecasts about the likelihood of catastrophic risks from AI vary wildly. Dario Amodei has estimated a 25% chance of human extinction within the next few decades. Experts, meanwhile, put the chance of an AI-related catastrophe at 0.3% by 2030 and 2% by 2050. Understanding the level of risk and taking appropriate action is one of the most consequential challenges facing humans today.

Forecasts from frontier AI models could be an important input into this debate. The best models now approach superforecaster levels of accuracy, and models are rapidly becoming more capable. To take advantage of these capabilities, we’re introducing the Automated AI Risk Outlook (AIRO)—a fully automated dashboard of catastrophic risk outcomes as forecast by frontier AI models. AIRO enables us to track in close to real time how the likelihood of catastrophic outcomes changes, and how these risk forecasts relate to advances in model capability. The accompanying white paper also presents preliminary forecasts conditional on policy scenarios.

The AIRO dashboard currently puts the likelihood of an AI-related catastrophe that kills at least 10% of the population at 0.47% by the end of 2030 and 6% by the end of 2050. In addition to this headline forecast, AIRO tracks risks from AI-specific catastrophes and incidents, biorisk, cyberrisk, and misalignment risk.

A key feature of AIRO is that forecasts are repeated over time, so one can see how they change or if they spike. LLMs can forecast many outcomes across many time horizons and severity thresholds without growing tired, allowing us to elicit many more forecasts than is possible with human forecasters.

Why trust LLM risk forecasts?

AI forecasting abilities have improved alongside other AI capabilities. In July 2026, an AI model reached parity with superforecasters on FRI’s forecasting benchmark, ForecastBench. This gives some basis for believing that AI forecasting may now be state of the art.

The forecasts shown on AIRO are different from those evaluated in ForecastBench in two important ways. First, they concern very rare events. It is difficult to measure accuracy on forecasting rare events given that they, by definition, happen so rarely. However, unlike human forecasters, we can ask AI models to make thousands of predictions on events in a simulated world environment. When we do this, we find that more capable models—as assessed by their score on the ECI—are more accurate at forecasting rare events.

Second, AIRO includes many conditional forecasting questions. The dashboard includes forecasts conditional on model capabilities; the white paper also explores preliminary policy scenarios. We can again test performance on this type of conditional forecasting in a simulated world environment, and again we find that accuracy correlates with general capabilities.

There are risks associated with relying on automated AI forecasters of catastrophic risk. For example, AI forecasts may be influenced by strategic behavior: a highly capable but misaligned model trying to evade human suspicion may lower its estimates to avoid causing alarm. This “sandbagging” behavior may also arise if companies, concerned about regulation or other threats to profit, intentionally post-train models to underplay the risk of AI-caused harms.

How AIRO works

Our forecasting questions are drawn from earlier FRI work, allowing us to compare forecasts from the LLM ensemble with those from experts and superforecasters. We elicit forecasts from the four highest-scoring eligible models on the ECI, keeping one model per family.

Each model answers all questions—across time horizons and severity thresholds—in one forecasting session. Each model conducts its own research via an agentic harness. A prompt asks it to investigate recent developments, expert reports, and published estimates for each cause of catastrophe, and use what it learns to formulate follow-up queries. The model chooses searches, receives results, and decides what to investigate next. All models receive the forecasting date and questions and use the web-search and page-reading tools supplied through Tavily. Further details are provided on AIRO’s website.

Some interesting stats

In order to gauge how forecasts vary with severity, some forecasting questions ask models to predict damages either in terms of money or deaths. For example, 100 deaths would be equivalent to economic losses of $220 million, using estimates of the value of a statistical life. Results were interesting, because you can see that for low levels of incident “severity”, risks from cyber incidents dominate, but at high levels, misalignment risks are more likely—there is a crossing point:

Biorisks were assessed as relatively less likely than AI-related risks, except in the case of a severe incident in the short term:

Limitations, benefits, and what’s next

There is a lot of uncertainty underlying the forecasts. It could be harmful if people overly latched onto them. The forecasts also do not show the positive effects that AI can have—a fuller treatment might use conditional forecasts to assess the pros and cons of different policy options.

Forecasts of AI impacts might be particularly valuable if frontier labs were interested in participating in some way to make their own forecasts and see how they evolve. The labs, after all, have private information, including about unreleased models. The question is whether they would have the incentive to. As an economist, I take incentives very seriously. So, I see many ways this could go where the forecasts would stop being informative—but I also think there is real value to AI companies in showing risks and rewards transparently. Careful forecasts, validated over a long period, are something that could help them build trust, and that could be something that they would benefit from even in a self-serving way.

Personally, I am glad this dashboard now exists because I believe that despite noise there is some signal here. Past research on forecasting indicates that even when forecasts get the absolute level of an outcome wrong, they are often good at ranking different options or observing time trends, and I believe the AI forecasts are probably already better than human forecasts and likely to get better.

AIRO is still a work in progress. We plan to keep validating and extending our dashboard by refining ensemble forecasts and developing a larger set of intermediate questions related to catastrophic risk outcomes. These intermediate forecasts will serve as validation tools and provide inputs that future models can use to inform their risk assessments. We’ll also continue to test conditional forecasts to better understand the costs and benefits of proposed policies to mitigate AI risk.

For more information, here is the whitepaper, and here is the dashboard.

Organizing a Fermi modeling ‘hack’ on the cost of cultured meat; hiring a co-coordinator by david_reinstein

david_reinstein — Fri, 11 Sep 2026 18:01:44 +0000

The Unjournal is running a Fermi (aka BOTEC) modeling hack/workshop on the cost of producing cultivated meat. This follows our researcher/practitioner workshop and PQ work generating an initial public Monte Carlo model/calculator, public discussion, and belief elicitation and synthesis (preliminary version here). A rough plan for the hack is sketched here. Looking for feedback and suggestions on this.

This could also serve as a model for future workshops in other contexts.^[1] It should also offer some insight into the reliability of these modeling approaches in general.

I’ve often wondered: these models seem to often depend on a lot of structural and definitional assumptions. How sensitive are they to reasonable initial choices? To what extent would independent groups engaging in these BOTECs tend to converge?

We’re looking for a modeling co-coordinator and co-facilitator as a one-off role, but it may lead to more such work depending on success and funding. You’d likely be work with David Reinstein, David Manheim, and workshop participants in prepping, running, and following this up. We can offer only modest compensation (perhaps up to $1,000 delivered as a Tremendous reward for at least 20 hours of work).

The workshop aims to do a combination of:

Provide a clearly-explained workspace, with common estimands and parameter definitions, aligning understanding, terminology, etc.
Further consolidate and refine sourced beliefs and evidence, identifying remaining uncertainty and high-value next steps
Enable and encourage workshop participants (at least a few individuals or groups) to model this independently; considering convergence, reliability, and modeling degrees of freedom. (And providing insights for future modeling approaches and workshops in general).
Build, improve, and communicate a reasoning-transparent “consensus model”

If you’re the co-coordinator you could support us in planning and preparing for the hack, coordinating and running it, and following up and analyzing and communicating the results. It could be tied to a formal/academic research project if it makes sense for you. You should have some experience or familiarity with this sort of modeling (or forecasting, cost-effectiveness analysis, or techno-economic analysis.) Nice to also have experience coordinating collaborative workshops, familiarity with cultivated meat production, and building tools and sites with AI coding tools.

If interested, please email contact@unjournal.org as soon as you can (ideally before 24 September 2026) to express your interest and share your CV/work samples/experience.^[2]

Also looking for referrals/nominations. And feel free to ask questions or ‘apply in the comments’ if you want to do this publicly.

^
Potentially including geopolitical risks, AI scaling, funding and aid, etc.
^
Something like:
- three to five sentences on your relevant experience and interest (feel free to say more, we’ll try to consider, credit, and respond to all suggestions.
- your CV and/or LinkedIn
- Maybe a one link to a relevant model, analysis, code repository, or work sample;
- your availability and time zone; and
- any prior involvement or public position relevant to cultivated meat

Caroline Ellison has joined Manifund by Austin

Austin — Fri, 11 Sep 2026 17:48:49 +0000

Austin: I’m happy to announce that Caroline Ellison has accepted a role at Manifund, to develop our funding platform and research how to effectively direct philanthropic dollars.

In fact, she started a work trial on July 13, which converted to a fulltime role on Aug 10. Over the past two months, she’s been publishing her work and supporting Manifund users under the pseudonym “Carol”.

How did I come to consider this at all?

I specifically enjoyed Caroline’s tumblr and other writings, which I found thoughtful and relatable. I invited her to Manifest 2026, just because I wanted to meet her. During the festival, she came to my night market booth, and asked about joining our team, which I was interested to explore.
I felt a keen debt to the FTX Future Fund. They provided seed funding for Manifold. They funded the retreat which drew me into the EA community, and the conference where I met my wife. Much of Manifund’s work today is directly inspired by the Future Fund’s approach.
On FTX itself, I remain conflicted. It was their money and spirit that enabled the Future Fund, and I still admire many aspects of FTX and the individuals who worked there. But also: FTX did wrong. Its collapse harmed users (myself included), and damaged the movement which I deeply care about. I may write more about this in the future, but in any case:
I believe in redemption. Caroline has admitted her faults, worked to make creditors whole, and served her time in prison. I’d like to give her the opportunity to contribute back to the world; I hope that the world will concur.

Why did I hire Caroline for Manifund?

Manifund needs talent. Leading AI labs warn about recursive self improvement; poverty and disease remain unsolved; billions of animals suffer hell on earth. I, my team, and many EAs I know, are often burning the midnight candle to address these wrongs. I don’t wish to sideline a pair of capable hands. (We’re hiring!)
Caroline brings specific, useful expertise around finance and systems management to Manifund. I know this might seem like a joke — why would you trust someone who blew up an exchange? — but in fact she’s already been invaluable, for example building a reconciliation tool which identified several misregistered transactions in our database, in the 5-6 figures. (Obviously, she and I will be extremely motivated not to repeat FTX’s mistakes).
Through working together, I’ve had the privilege to know Caroline as more than just an icon of FTX. I find her to be kind, humble, down to earth, hardworking, smart and witty. She cares deeply about doing right, for our users, our donors, and the world. She shares many of my sensibilities around extant problems of philanthropy; see, for example, our essay in support of for-profits. She adores Ada, my 2yo daughter (and, the feeling is clearly mutual).

Manifund strives to be unusually transparent. And so, I apologize for our minor deception, in using a pseudonym over the last couple months. I still endorse having done this — I think it’s relatively common to work with people pseudonymously, when their real identity is famous. But I want to call out that we compromised on transparency here, which I do not take lightly.

I understand that some of our users, donors, and partners may now be skittish about working with Manifund, given this announcement. I appreciate any trust you are willing to extend, and ask that you judge us by our historical and future outputs. You (or your LLM) can take a look at Manifund’s source code, underlying data, finances and research; if you wish, you can see Caroline’s specific code commits and comments on Manifund.

I’m happy to take questions, over public comments or emailed to austin@manifund.org. (I can’t promise I’ll be able to reply to every one.) In the meantime, I’d like to ask Caroline to say a few words.

Caroline: I’ve said a lot about what I did wrong at Alameda/FTX—you can read what I said at my plea, during trial, and remarks at sentencing. I’m deeply sorry for what I did, and I completely understand why people may not want to work with me. But I’m very grateful to Austin for giving me a second chance, and judging me on my current work performance rather than my past. I feel very lucky not only to have a job, but to be working somewhere that I think is doing great work and has a mission I believe in.

When I started my work trial, unsure if I’d get a full-time offer at the end, I didn’t want to go through a public announcement just to potentially be let go a month later. But now that I’ve accepted a full-time offer at Manifund, I want to be upfront about it.

I’m excited to continue improving our technical infrastructure, working on operations and customer support, and posting updates on our newsletter.

I probably won’t reply to comments on this post—sorry.

The Golden Age of Impact by Bentham’s Bulldog

Bentham's Bulldog — Fri, 11 Sep 2026 15:46:22 +0000

Crosspost.

The world is advancing at a crazy clip. AI is going gangbusters, poised to potentially be the most transformative technology in history. Philanthropists are flushed with cash and short on ways to spend it. As a result, this is an amazingly good time to do impactful things with your life. Here I will suggest three things you should do.

1 Tailwind

Coefficient Giving just launched Tailwind, a grant program for ambitious AI safety startups. Matt Beard summarizes:

I have good news: Coefficient Giving just launched Project Tailwind, a call for exceptional founders to launch ambitious new AI safety initiatives, with $200k to $2 million in pre-seed or $2-20+ million in seed funding available. They list dozens of concrete ideas for new nonprofits or startups they’d be excited to potentially fund, including rogue incident detection, agreement and verification technology, a misalignment red team, talent and convening organizations, and more. You can apply today.

AI safety work is kind of a no-brainer. Experts generally think AI has a non-trivial chance of ending the world. It pays well, about as well as finance. You can get very rich working to avert a potential apocalypse. Much of the low-hanging safety fruit is unpicked. If a career is impactful, meaningful, and pays well, it seems like the kind of career that’s worth looking into.

Not everyone has the relevant skills. But if you do, consider applying. Given how flushed with cash EA is now with all the Anthropic funding coming in, my guess is that founding an AI safety startup is much more impactful than almost any earning to give job—even one making many millions of dollars a year.

Anecdote: one of my best friends started a company working on making the intelligence explosion go well. It’s now worth many tens of millions of dollars, potentially soon to be hundreds of millions, and it’s also having an incredibly positive impact.

2 Charity entrepreneurship

Charity Entrepreneurship is taking applications for those who want to start high impact animal charities. To apply, you don’t have to have an original idea for such a charity. You just have to be willing to work to incubate an impactful animal charity. Applications close September 13th.

The animal welfare space, like the AI safety space, has a lot more money than good ways to spend its money. We need many new people working to start incredibly impactful animal charities. Some CE-incubated charities have single-handedly benefitted over ten billion animals. A few people really do make an incredible difference.

Billions of animals cry in horrible agony in dark, desolate nightmare cages—knowing neither joy nor kindness, their daily existence defined by constriction and confinement and pain. By working for a new animal charity, you can benefit millions of animals in expectation. You can help free these poor defenseless creatures from their confinement. You can do something enjoyable and meaningful and good.

3 80,000 hours

80,000 Hours has an extremely useful career guide. In it, they summarize how to have an impactful career. If you want to have an impactful career, you should read—or at least skim—their career guide.

They also have a job board. This is a board full of jobs that are impactful. If you want to have impact, you should consider applying for one of the impactful jobs that want to hire people! There are a lot of jobs here—they’re not all AI safety—so consider applying for one of these even if you aren’t planning to do safety.

Now, one thing you might be thinking: “I already have a good job that pays well—I’m not going to quit.” But I think you should consider quitting. Given how much new cash there is in EA, donations have gone down massively in value relative to having high impact careers. Thus, you can massively increase your impact by getting an EA job. Just being, say, a guy in finance who gives some of your money to effective charities is nowhere near as valuable as getting one of these EA jobs.

4 Student EA clubs

Suppose that you’re still in college. What can you do?

Answer: start an EA club at your university. If it already exists, work hard, spending many hours a week, making it ridiculously good and well run. You can also host the meetups through the new Scott program for EA meetup organizing.

Suppose that doing so gets one more person into EA counterfactually. Congratulations, you have now doubled your lifetime impact. And it’s plausible that you can get many more people than that into EA. You can also consider running other groups. For more about why this sort of thing is good, you can read my post here.

5 Conclusion

It wouldn’t surprise me if there were many people reading this who could do things many times more impactful than giving billions to effective charities if they put their minds to it. We are at an unprecedentedly crazy time. AI is upending the world—in a few years, we may have extremely rapid economic growth and AI that poses serious existential risks. This may literally be the best time in history to have a positive impact, so the least you can do is make a serious attempt to make the world a better place, while the youthful hue sits on thy skin like morning dew.

Frontpage posts - Effective Altruism forum viewer

The UK has quietly downgraded Biorisk Worst-Case Scenarios >5x in a year by Chris Clay🔸

The clearest warning shot: how the OpenAI agent-swarm story engaged a public audience by benrmatthews

Re­search Summary

The clear­est warn­ing shot

The numbers

What this means for AI safety communicators

How to use this

1. De­scribe in­ci­dents, not hypotheticals

2. Use ex­pe­rienced, calm messengers

3. Pre-empt the trust at­tack with artefacts

4. Give the fear a job.

5. Use quotable, hu­man de­tails de­liber­ately,

Related

Is Consciousness Everywhere? by Vasco Grilo🔸

Too many people in EA underprice the cost of a bad reputation by GraceAdams🔸

Good work is nec­es­sary, not sufficient

What a good rep­u­ta­tion ac­tu­ally buys you

The in­visi­ble costs of a bad reputation

Be­ing weird about this doesn’t scale

The peo­ple we lose

What to do about it

Rationalizing Altruism by idea21

EA PR is tricky, and so maybe you shouldn’t do it? by Brian Foerster

PR is Tricky for the Decisionmaker

PR is Tricky for Every­one Else

Germany’s Got Talent. How do we get it to work on AI safety? by Melanie

Summary

Epistemic sta­tus

Methodology

What do the peo­ple I in­ter­viewed have in com­mon?

The EA Funnel

De­mon­strated agency

What nearly made in­ter­vie­wees quit

Know­ing which op­por­tu­nity is worth your time

Net­work­ing in Germany

Build­ing con­text means leav­ing Germany

Who this filters for

Con­tribut­ing from Germany

Work­ing remotely

Found­ing

Ger­many’s ex­ist­ing institutions

Recommendations

Who we are not reaching

What we can provide

How to start (sim­plified)

Objections

Isn’t this whole piece bi­ased by sur­vivors?

Wouldn’t reach­ing out to un­tapped tal­ent pools dilute the qual­ity of ap­pli­cants?

Fi­nal thoughts

Dario Amodei: “We Must Pace The Frontier” by Matrice Jacobine🔸🏳️‍⚧️

Is it worth spending $100k to study college in the US? by jackchang110

How a cold email got the Finnish government to respond on superintelligence regulation by Josh Thorsteinson 🔸

Summary

The email

I’m just a guy

The meeting

The writ­ten question

Writ­ten Ques­tion KK 325/​2026 vp

Me­dia attention

TV appearances

Writ­ten articles

In­spiring a friend

The response

Re­sponse to Writ­ten Ques­tion KK 325/​2026 vp

Do this yourself

EA Should Invest in Animal Liberation and Diet Change Advocacy by Bruce Friedrich 🔸

If you want to be successful on the EA Forum, your title should be your thesis by Roman Ross

What I want you to do when I tell you to “think about your theory of change more carefully” by Roman Ross

Tell the full story

Find a metric

Lots of things “seem good” but are miss­ing im­por­tant steps to be­com­ing impactful

Lots of things “seem good” but are ac­tu­ally worse than nearby options

Things to avoid when think­ing about the­o­ries of change

Fi­nal Note:

If AI Accelerates Scientific Discovery, Will African Research Infrastructure Become the Bottleneck? Lessons From Running a Biotechnology Centre in Nigeria by Nnaemeka Emmanuel Nnadi

The bot­tle­neck-shift hypothesis

If this hy­poth­e­sis is cor­rect, what should fun­ders ac­tu­ally fi­nance?

Our ex­pe­rience at Cen­tre for Phage Biol­ogy and Therapeutics

Con­clud­ing Thoughts

Research Summary

The clearest warning shot

1. Describe incidents, not hypotheticals

2. Use experienced, calm messengers

3. Pre-empt the trust attack with artefacts

5. Use quotable, human details deliberately,

Good work is necessary, not sufficient

What a good reputation actually buys you

The invisible costs of a bad reputation

Being weird about this doesn’t scale

The people we lose

PR is Tricky for Everyone Else

Epistemic status

What do the people I interviewed have in common?

Demonstrated agency

What nearly made interviewees quit

Knowing which opportunity is worth your time

Networking in Germany

Building context means leaving Germany

Contributing from Germany

Working remotely

Founding

Germany’s existing institutions

How to start (simplified)

Isn’t this whole piece biased by survivors?

Wouldn’t reaching out to untapped talent pools dilute the quality of applicants?

Final thoughts

The written question

Written Question KK 325/2026 vp

Media attention

Written articles

Inspiring a friend

Response to Written Question KK 325/2026 vp

Lots of things “seem good” but are missing important steps to becoming impactful

Lots of things “seem good” but are actually worse than nearby options

Things to avoid when thinking about theories of change

Final Note:

The bottleneck-shift hypothesis

If this hypothesis is correct, what should funders actually finance?

Our experience at Centre for Phage Biology and Therapeutics

Concluding Thoughts

Recent Examples

The Forecasts at a Glance

1. Another AI Escape in the Next Few Months is More Likely Than Not…But We Probably Won’t Hear About It

2. Anyone Will be Able to Download Mythos-Level Hacking Ability Within a Year

3. Congress Probably Won’t Pass a Kill Switch, Even Now

4. If an AI Hacks You, Don’t Expect a Law Saying Who Pays

5. A 36% Chance a Government Sues OpenAI Over the Hack

6. A Viral Theory About the Hack is Likely Wrong

7. The Widest Disagreement: an AI-Caused Data Breach of 100,000 People

Bonus: The AI Forecaster was the Skeptic

Why trust LLM risk forecasts?

Some interesting stats

Limitations, benefits, and what’s next

2 Charity entrepreneurship

4 Student EA clubs