There Should Be More Alignment-Driven Startups

I work with a lot of startups as a software developer and investor.  I'd love to lend a helping hand.

Executive summary: The authors argue that founding more AI safety-focused startups would increase the probability of solving the AI alignment problem by developing human and organizational capital, unlocking access to financial capital, fostering productive incentive structures, and enabling neglected approaches to be explored.

Key points:

1. Startups can access funding sources unavailable to non-profits and unlock productive incentive structures suited to tackling complex technical challenges like AI alignment.
2. More startups could explore a wider range of neglected approaches to alignment, rather than concentrating efforts on a small set of prevailing ideas.
3. Alignment may require significant resources that existing organizations are too hesitant to invest, which a speculative investment model could enable.
4. Building an ecosystem of alignment-focused investors, founders, and employees now prepares for expected increases in alignment funding and technical needs.
5. Customer feedback and market dynamics could help evaluate alignment approaches' scalability better than purely theoretical arguments.
6. The authors call for more ambitious alignment startups, skunkworks-style experimentation, consulting to develop skills, and outreach to interested investors and founders.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

(comment crossposted from LW)

While the coauthors broadly agree about points listed in the post, I wanted to stick my neck out a bit more and assign some numbers to one of the core points. I think on present margins, voluntary restraint slows down capabilities progress by at most 5% while probably halving safety progress, and this doesn't seem like a good trade. [The numbers seem like they were different in the past, but the counterfactuals here are hard to estimate.] I think if you measure by the number of people involved, the effect of restraint is substantially lower; here I'm assuming that people who are most interested in AI safety are probably most focused on the sorts of research directions that I think could be transformative, and so have an outsized impact.