When should we worry about AI power-seeking?

By Joe_Carlsmith @ 2025-02-19T19:44 (+21)

This is a linkpost to https://joecarlsmith.substack.com/p/when-should-we-worry-about-ai-power

This is a crosspost, probably from LessWrong. Try viewing it there.

null
SummaryBot @ 2025-02-19T21:36 (+1)

Executive summary: AI power-seeking becomes a serious concern when three prerequisites are met: (1) the AI has agency and the ability to plan strategically, (2) it has motivations that extend over long time horizons, and (3) its incentives make power-seeking the most rational choice; while the first two prerequisites are likely to emerge by default, the third depends on factors like the ease of AI takeover and the effectiveness of human control strategies.

Key points:

  1. Three prerequisites for AI power-seeking: (1) Agency—AI must engage in strategic planning and execution, (2) Motivation—AI must value long-term outcomes, and (3) Incentives—power-seeking must be a rational choice from the AI’s perspective.
  2. Incentive analysis matters: While instrumental convergence suggests many AI goals may lead to power-seeking, evaluating AI incentives requires understanding available options, likelihood of success, and AI's preferences regarding failure or constraints.
  3. Motivation vs. Option control: Effective AI safety requires both shaping AI motivations (so it avoids power-seeking) and restricting its available options (so power-seeking isn’t feasible).
  4. The risk of decisive strategic advantage (DSA): A single superintelligent AI with overwhelming power could easily take control, but a broader concern is global vulnerability—where AI development makes humanity increasingly dependent on AI restraint or active containment.
  5. Multilateral risks beyond a single AI: Coordination between multiple AI systems (either intentional or unintentional) could pose an even greater risk than a single rogue superintelligence, making alignment and oversight more complex.
  6. AI safety strategies should go beyond extremes: AI alignment efforts often focus on either complete control over AI motivations or extreme security measures, but real-world solutions likely involve a mix of both approaches.

 

 

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.