AI Safety & Entrepreneurship v1.0
By Chris Leong @ 2025-04-26T14:37 (+27)
Why did I create both a post and a wiki article?
Posts are best for making sure people see the initial version of the post, whilst Wiki articles are best for long-term maintenance. Posting an article that links to a Wiki page provides the best of both worlds.
For the most up-to-date version, see the Wiki page.
Articles:
There should be more AI safety organisations
Why does the AI Safety Community need help founding projects?
Alignment can be the ‘clean energy’ of AI
AI Tools for Existential Security
Incubation Programs:
Def/acc at Entrepreneur First - this is a new program, which focuses on "defensive" tech, which will produce a wide variety of startups, some of which may help with reducing x-risk, but others may include AI applications like autonomous weapons or diagnostic medical software, which, while they generally won't increase x-risk, may not reduce it either.
Catalyze - AI Safety Incubation
Seldon Accelerator: “We take on AGI risk with optimism”
AE Studio has expressed interest in helping seed neglected approaches, incl. ideas that take the form of business ideas
AIS Friendly General Program:
Founding to Give: General EA incubator
Entrepreneur First - They run def/acc, so this could be a decent alternative
Halycon Futures: “We’re an entrepreneurial nonprofit and VC fund dedicated to making AI safe, secure and good for humanity.” They do new project incubation, grants, investments, ect.
Communities:
Entrepreneurship Channel EA Anyway
AI Safety Founders: Website, Discord, LinkedIn
VC:
AI Safety Seed Funding Network
Lionheart Ventures - “We invest in the wise development of transformative technologies.”
Juniper Ventures - Invests in companies “working to make AI safe, secure and beneficial for humanity”. Published the AI Assurance Tech Report
Polaris Ventures - “We support projects and people aiming to build a future guided by wisdom and compassion for all”
Mythos Ventures - “an early-stage venture capital firm investing in prosocial technologies for the transformative AI era.”
Metaplanet - “We support novel and evidence-based innovation that could produce an outsized return for the benefit of humankind.”
Anthology Fund - Backed by Anthropic. One of the five key areas is “trust and safety tooling”
Menlo Ventures - Lead the GoodFire round (article)
Organisational Support:
Ashgro - fiscal sponsorship
Rethink Priorities Special Projects - provides fiscal sponsorship or incubation
Other:
Hackathon for AI Safety Startups - Ran once by Apart Research, may run again
Constellation Residency - Year-long position
Nonlinear - Free coaching for people who are running an AI safety startup or who are considering starting one