Top OpenAI Catastrophic Risk Official Steps Down Abruptly

By Garrison @ 2025-04-16T16:04 (+29)

This is a linkpost to https://garrisonlovely.substack.com/p/breaking-top-openai-catastrophic

This is the full text of a post from Obsolete, a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to Build Machine Superintelligence. Consider subscribing to stay up to date with my work.

OpenAI's top safety staffer responsible for mitigating catastrophic risks quietly stepped down from the role weeks ago, according to a LinkedIn announcement posted yesterday.

Joaquin Quiñonero Candela, who took over OpenAI's Preparedness team in July, announced on LinkedIn that he has taken on a new role at the company:

I'm an intern! After 11 years since my last commit, I'm back to building. I first transitioned to management in 2009, and got more and more disconnected from code and hands-on work. Three weeks ago, I turned it all upside down, and became an intern in one of our awesome teams that's focused on healthcare applications of AI.

Candela's LinkedIn bio now describes him as the "Former Head of Preparedness at OpenAI."

An OpenAI spokesperson told Obsolete that Candela "was really closely involved in preparing the successor to the preparedness framework" and "will probably be involved in preparedness in some capacity" but is currently "focusing on different areas within the company that he's really excited about."

The spokesperson added that the company recently restructured its safety organization, consolidating "all governance under the Safety Advisory Group" (SAG) — a committee chaired by five-year OpenAI veteran Sandhini Agarwal. The SAG uses a rotational leadership structure with one-year terms, designed, they said, to balance "continuity of knowledge and expertise" with "fresh and timely perspectives."

Meanwhile, OpenAI's preparedness work is now distributed across multiple teams, focused on things like capabilities, evaluations, and safety mitigations, the spokesperson said.

Candela's departure from the team comes amidst OpenAI's contentious attempt to shed the last vestiges of nonprofit control and follows a string of scandals and high profile exits in the last year.

It also marks the second major shakeup in the Preparedness team’s short history. In July, OpenAI removed Aleksander Mądry from his role as head of Preparedness — also without a public announcement. The Information reported that the MIT professor was reassigned to work on AI reasoning just days before US senators sent a letter to CEO Sam Altman regarding "emerging safety concerns" at the company.

Following Mądry's reassignment, Candela took over, and Tejal Patwardhan, a 2020 Harvard graduate, began managing day-to-day operations, according to The Information story.

Mądry's quiet move reflects a pattern of leadership changes to OpenAI's safety teams that continues with Candela's departure.

The Preparedness team was established in December 2023 to track and mitigate "catastrophic risks related to frontier AI models," according to the company's Preparedness Framework, which was introduced as "a living document describing OpenAI’s processes to track, evaluate, forecast, and protect against catastrophic risks posed by increasingly powerful models."

The Framework focuses on risks related to cybersecurity, persuasion, model autonomy, and chemical, biological, radiological, and nuclear weapons.

OpenAI published the second version of its Preparedness Framework around noon, local time today, shortly after Obsolete contacted the company for comment.

The crumbling of safety leadership

OpenAI has seen an exodus of leadership and safety staff in the last year.

Company cofounder and AI alignment lead, John Schulman, left in August for Anthropic, a rival firm started by an earlier wave of departing OpenAI safety staff.

Lilian Weng, OpenAI's safety lead, left in November and subsequently joined Thinking Machines Labs — a startup launched earlier this year by Mira Murati, who served as OpenAI's CTO from 2022 until her abrupt departure amidst the company's October fundraising round. Schulman joined Murati's company in February.

OpenAI's Superalignment team, tasked with figuring out how to build smarter-than-human AI safely, was disbanded in May. The team leads, OpenAI cofounder Ilya Sutskever and longtime safety researcher Jan Leike, both left the same month. During his departure, Leike publicly stated that "safety culture and processes have taken a backseat to shiny products" at OpenAI. Fortune reported that the Superalignment team never got the computing power it was promised.

In October, Miles Brundage, OpenAI's Senior Advisor for AGI readiness, resigned after more than six years at the company; his team was disbanded and absorbed into other departments.

Brundage was one of the last remaining members of OpenAI's early safety-focused staff and had been increasingly vocal about his concerns. In his departure announcement, he wrote that "neither OpenAI nor any other frontier lab is ready" for artificial general intelligence (AGI) — the very technology the company is explicitly trying to build. He cited publishing constraints as one reason for leaving, suggesting the company was restricting what he could say publicly about AI risks.

Brundage also broke with Altman by advocating for cooperation with China on AI safety rather than competition, warning that a "zero-sum mentality increases the likelihood of corner-cutting on safety."

A former senior OpenAI employee told Obsolete that Mądry's reassignment was particularly alarming. "At a certain point, he was the only person in there with a safety-focused role who was empowered at all," the former employee said.

Who is leading safety at OpenAI?

With most safety-focused leaders gone or reassigned, OpenAI's formal governance structure has become increasingly important — but also increasingly opaque.

In May, OpenAI announced the creation of its Safety and Security Committee (SSC), tasked with making recommendations to the full board on "critical safety and security decisions for OpenAI projects and operations." Its original members included a subset of its nonprofit board, including Altman, along with Madry, Weng, Schulman, Matt Knight, the head of security, and Jakub Pachocki, the chief scientist.

Of these original members, only Knight and Pachocki remain in these or similar roles at OpenAI.

OpenAI announced in September that board member and Carnegie Mellon professor Zico Kolter would join the SSC as its chair and that Altman was no longer on the committee. When asked about Altman's departure, the OpenAI spokesperson declined to comment.

The updated version of the Preparedness Framework published today goes into more detail on the roles, responsibilities, and decision-making processes of the SSC and introduces the "Safety Advisory Group (SAG)" — a related committee made up of OpenAI staff.

However, the updated document does not identify the members of the SAG. According to the OpenAI spokesperson, the SAG has been working under Agarwal's leadership for two months. They described her as "functionally heading up all of the governance work," including "all of the evaluation calls about what [risk] mitigations are necessary."

The lack of transparency around safety leadership extends beyond public announcements. “Even while working at OpenAI, details about safety procedures were very siloed. I could never really tell what we had promised, if we had done it, or who was working on it,” a former employee wrote to Obsolete.

Growing concerns

These leadership changes come amid mounting questions about OpenAI's commitment to safety.

The Financial Times reported last week that "OpenAI slash[ed] AI model safety testing time" from months to days. When asked about this story, the company spokesperson directed Obsolete back to the updated Preparedness Framework — saying that "our safety practices continue to be really rigorous" and suggesting that characterizations of reduced testing were not "very fair."

And just yesterday, the company released GPT-4.1 without publishing a corresponding safety report. OpenAI's announcement touts the model's significant improvements over its flagship multimodal model, GPT-4o, in areas like coding and instruction following.

Conducting pre-release safety evaluations on frontier AI models and publishing the results alongside the model launch has become a common practice for the industry — one that OpenAI committed to at the 2024 Seoul AI Summit.

When questioned by TechCrunch, OpenAI claimed that "GPT-4.1 is not a frontier model, so there won't be a separate system card released for it."

However, the company released DeepResearch, a powerful web-searching tool, weeks before publishing a safety report, which refers to the product as a frontier model.

Following the release of the updated Framework, former OpenAI safety researcher Steven Adler tweeted that he's "overall happy to see the Preparedness Framework updated." But he also called out the company for "quietly reducing its safety commitments," pointing to OpenAI's abandonment of an earlier promise to conduct safety testing on models finetuned to perform better in certain risky domains, like bioengineering.

Safety reports have been a primary tool for transparency in the AI industry, providing details on testing conducted to evaluate a model's risks. After conducting safety evaluations, OpenAI and Anthropic each found that their most advanced models are close to being able to meaningfully assist non-experts in the creation of bioweapons. And OpenAI had previously called system cards "a key part" of its approach to accountability ahead of the 2023 UK AI Safety Summit.

In the United States, frontier AI developers are governed by voluntary commitments, which they can violate without real consequence. Many of these companies, including OpenAI and Google, lobbied hard last year against California AI safety bill SB 1047, the most significant effort to codify some of these commitments.

As AI models get more capable and autonomous, companies appear to be increasingly cutting corners on safety.

Google's Gemini 2.5 Pro model is considered by many to be the most capable on the market, but the company still hasn't released a safety report, which Fortune reported last week violated voluntary commitments the company made to the White House and at the Seoul summit.

The competitive pressure to release faster and with fewer safeguards will likely increase from here, raising alarming questions about whether meaningful guardrails will be in place when they're needed most.

If you enjoyed this post, please subscribe to Obsolete.