The benefits and risks of optimism (about AI safety)

By Karl von Wendt @ 2023-12-03T12:45 (+3)

This is a crosspost, probably from LessWrong. Try viewing it there.

null
titotal @ 2023-12-03T14:03 (+7)

I define “optimism” as the tendency to weigh positive outcomes higher than their expected value and negative outcomes lower

I don't think this is the definition that Pope & Belrose are using. I think they are using it in the same sense as "I'm optimistic about my relationship": A genuine belief that something will go well. 

I think they claim to be optimistic because they genuinely believe that the development of AI will have good effects and that significant harms are unlikely, and they want policies such as open sourcing to reflect that. 

Karl von Wendt @ 2023-12-03T15:21 (+1)

I don't think that there's a huge difference. As long as there aren't very strong fact-based arguments for exactly how likely it is that we will be able to control AGI, my definition of "optimists" will end up with a significantly higher probability of things going well. From what I read, I believe that Belrose and Pope have this basic bias towards "AGI is beneficial" and weigh the upside potential higher than the downside risks. They then present arguments in favor of that position. This is of course just an impression, I can't prove it. In any case, even if they genuinely believe that everything they say is correct, they still should put in massive caveats and point out where exactly their arguments are weak or could be questioned. But that is not what a self-declared "optimist" does.  So, instead they just present their beliefs. That's okay, but it is clearly a sign of optimism the way I define it.

titotal @ 2023-12-05T17:26 (+4)

This seems like a selective demand. I believe that doomers have a bias towards "AGI is destructive". Will you comment on doomer posts, demanding they add in massive caveats and point out exactly where their arguments are weak? 

If you don't agree with Pope and Belrose, argue with them on the facts. Don't argue with disingenuous semantic games, and pretend that the word "optimist" doesn't have more than one definition in the dictionary. 

Karl von Wendt @ 2023-12-06T15:24 (+1)

I agree that some "doomers" (you may count me as one) are "pessimistic", being biased towards a negative outcome. I can't rule out that I'm "overly cautious". However, I'd argue that this is net positive for AI safety on the same grounds that I think optimism as I defined it is net positive under different circumstances, as described.

I agree that the word "optimism" can be used in different ways, that's why I gave a definition of the way I usually use it. My post was a reaction to Pope and Belrose, but as I stated, not about their arguments but generally about being "optimistic" in the way I defined it. Nora Belrose said in a comment on LessWrong that my way of defining optimism is not how they meant it, and as long as I don't analyze their texts, I have to accept that. But I think my definition of optimism fits in the range of common uses of the word (see Wikipedia, for example). All I did was trying to point out that this kind of "positive outcome bias" may be beneficial under certain circumstances, but not for thinking about AI safety.

I believe that if Pope and Belrose try to have a truly rational and unbiased stance, the term "AI Optimism" is at least misleading, as it can be understood in the way I have understood it. I hope this post is at least helpful in the sense that I have pointed that possible misunderstanding out.

SummaryBot @ 2023-12-04T14:06 (+1)

Executive summary: The author argues that optimism can drive progress but becomes dangerous when downside risks are high, as with optimistic assumptions about AI safety.

Key points:

  1. Optimism motivates exploration and fuels adaptation, though often inefficiently. It likely evolved to promote growth despite negative expected outcomes.
  2. However, optimism has downsides like wasted effort and becomes clearly problematic when potential harms are severe or irreversible.
  3. AI safety intrinsically risks catastrophic and existential harms if control fails, so optimism seems unwarranted and dangerous absent strong arguments it is achievable.
  4. More limited optimism about trying particular approaches to AI safety may still be warranted based on upside, even if success chances are low.
  5. But general optimism about controlling any AGI system developed risks trivializing crucial safety issues and diminishing chances of averting disaster.

 

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.