Should you start a for-profit AI safety org?

By Kat Woods 🔶 ⏸️ @ 2025-08-15T13:52 (+9)

This is one of the most common questions I get when people use my AI safety career advice service. 

My short answer is: yes, you can start a for-profit AI safety org, but you should try non-profit first. 

It mostly comes down between a trade-off between distorted incentives and amounts of funding. 

Non-profits slightly distort your incentives, trying to do things that please funders rather than directly doing what most improves the world. 

But usually the funders also want to improve the world, so it’s mostly just “distorting” it to their judgement of what’s impactful, rather than moving away from impact entirely. 

For-profits massively distort your incentives. 

Your customers usually don’t care about their purchases making the world better. They care about making their lives immediately better. 

Yes, for-profits you have a higher potential upside of raising far more funding. 

However, broadly speaking, there are two important inputs into how much impact you have: 

  1. Direction
  2. Speed 

Going fast in the wrong direction doesn’t matter. This would be having a lot of money to do things, but the things don’t actually help with AI safety. 

Especially given how relatively easy it is in this field to think you’re helping but actually making things worse, this should be a very strong consideration.

Of course, conversely, knowing the right direction but not moving anywhere is pointless. And doing AI safety research can be cheap (if it’s mostly theoretical work and you can find good people who are willing to work for little), but often it’s very expensive. So expensive that only a handful of funders can realistically fund your work. So if you’ve tried applying to them and they’ve taken a disliking to you for whatever reason, often going for-profit is the only real option left. 

In that case, it can be worth trying for-profit. Just spend a lot of time and energy figuring out ways to make sure that it doesn’t distort your end goal and end up making you a slave to Moloch. 

This is a cross-post from my Substack