Joining the Carnegie Endowment for International Peace

By Holden Karnofsky @ 2024-04-29T15:45 (+228)

Effective today, I’ve left Open Philanthropy and joined the Carnegie Endowment for International Peace^[1] as a Visiting Scholar. At Carnegie, I will analyze and write about topics relevant to AI risk reduction. In the short term, I will focus on (a) what AI capabilities might increase the risk of a global catastrophe; (b) how we can catch early warning signs of these capabilities; and (c) what protective measures (for example, strong information security) are important for safely handling such capabilities. This is a continuation of the work I’ve been doing over the last ~year.

I want to be explicit about why I’m leaving Open Philanthropy. It’s because my work no longer involves significant involvement in grantmaking, and given that I’ve overseen grantmaking historically, it’s a significant problem for there to be confusion on this point. Philanthropy comes with particular power dynamics that I’d like to move away from, and I also think Open Philanthropy would benefit from less ambiguity about my role in its funding decisions (especially given the fact that I’m married to the President of a major AI company). I’m proud of my role in helping build Open Philanthropy, I love the team and organization, and I’m confident in the leadership it’s now under; I think it does the best philanthropy in the world, and will continue to do so after I move on. I will continue to serve on its board of directors (at least for the time being).

While I’ll miss the Open Philanthropy team, I am excited about joining Carnegie. 

• Tino Cuellar, Carnegie’s  President, has been an advocate for taking (what I see as) the biggest risks from AI seriously. Carnegie is looking to increase its attention to AI risk, and has a number of other scholars working on it, including Matt Sheehan, who specializes in China’s AI ecosystem (an especially crucial topic in my view).
• Carnegie’s leadership has shown enthusiasm for the work I’ve been doing and plan to continue. I expect that I’ll have support and freedom, in addition to an expanded platform and network, in continuing my work there.
• I’m generally interested in engaging more on AI risk with people outside my existing networks. I think it will be important to build an increasingly big tent over time, and I’ve tried to work on approaches to risk reduction (such as responsible scaling) that have particularly strong potential to resonate outside of existing AI-risk-focused communities. The Carnegie network is appealing because it’s well outside my usual network, while having many people with (a) genuine interest in risks from AI that could rise to the level of international security issues; (b) knowledge of international affairs.
• I resonate with Carnegie’s mission of “helping countries and institutions take on the most difficult global problems and advance peace,” and what I’ve read of its work has generally had a sober, nuanced, peace-oriented style that I like.

I’m looking forward to working at Carnegie, despite the bittersweetness of leaving Open Phil. To a significant extent, though, the TL;DR of this post is that I am continuing the work I’ve been doing for over a year: helping to design and advocate for a framework that seeks to get early warning signs of key risks from AI, accompanied by precommitments to have sufficient protections in place by the time they come (or to pause AI development and deployment until these protections get to where they need to be).

1. ^{^}

    I will be at the California office and won’t be relocating.