Don't treat probabilities less than 0.5 as if they're 0

By MichaelDickens @ 2025-02-26T05:14 (+36)

Example: "[wishy-washy argument that AI isn't risky], therefore we shouldn't work on AI safety." How confident are you about that? From your perspective, there's a non-trivial possibility that you're wrong. And I don't even mean 1%, I mean like 30%. Almost everyone working on AI safety think it has less than a 50% chance of killing everyone, but it's still a good expected value to work on it.

Example: "Shrimp are not moral patients so we shouldn't try to help them." Again, how confident are you about that? There's no way you can be confident enough for this argument to change your prioritization. The margin of error on the cost-effectiveness of some intervention is way higher than the difference in subjective probability on "shrimp are sentient" between someone who does, and someone who does not, care about shrimp welfare.

EAs are better at avoiding this fallacy than pretty much any other group, but still broadly bad at it.

Edited to add more examples:

  1. "I think there will be a slow takeoff, therefore we will spend ~zero effort planning for what to do in case of an intelligence explosion."
  2. "I think we can solve AI alignment by getting AI to do our homework. Therefore it's fine for our alignment plans to critically depend on this working, and we don't need to bother with a backup plan."
  3. "I don't foresee any discontinuous leaps in AI capabilities, so it's safe to keep building iteratively more powerful models, and we can calibrate risk as we go."

(I'm sure there are plenty of examples outside of AI safety, but that's what I've been thinking about lately.)

Davidmanheim @ 2025-02-26T07:39 (+10)

Not to answer the question, but to add a couple links that I know you're aware of but didn't explicitly mention, there are two reasons that EA does better than most groups. First, the fact that EA is adjacent to and overlaps with the lesswrong-style rationality community, and the multiple years of texts on better probabilistic reasoning and why and how to reason more explicitly had a huge impact. And second, the similarly adjacent forecasting community, which was kickstarted in a real sense by people affiliated with FHI (Matheny and IARPA, Robin Hanson, and Tetlock's later involvement.)

Both of these communities have spent time thinking about better probabilistic reasoning, and have lots of things to say about not just the issue of thinking probabilistically in general instead of implicitly asserting certainty based on which side of 50% things are. And many in EA, including myself, have long-advocated the ideas being even more centrally embraced in EA discussions. (Especially because I will claim that the concerns of the rationality community keep being relevant to EA's failures, or being prescient of later-embraced EA concerns and ideas.)

Non-zero-sum James @ 2025-02-27T05:43 (+1)

Some make a similar mistake with commonalities between human values, taking what is probably a 90% commonality (in our experience of injury, social rejection, sickness) and dismissing it under the a blanket "everything is subjective, completely unique to each individual", and concluding therefore that we can't make any generalisations about shared human values, and therefore are arrogant to believe we can say anything with authority about human wellbeing in general. I think this is a major hurdle to consensus—in fact, a form of consensus denial.