Who's track record of AI predictions would you like to see evaluated?

By Jonny Spicer 🔸 @ 2025-01-29T11:57 (+10)

As uncertainty grows around how AI development will affect culture and society, it becomes more valuable to compare track records of predictions about technological progress. 

I've recently been working on automating parts of the methodology from Arb's Scoring The Big 3's Predictive Performance report[1], and have had some promising preliminary results. I hope to try to automate most of the steps in the original report, making it feasible to analyse many more track records and publish the results.

I am particularly interested in the following questions:

  1. Which track record(s) would you find valuable to have evaluated in a similar way to Asimov, Clarke and Heinlein’s, as in the Arb report?
  2. What would you want to see from an LLM-based evaluation that would give you confidence that the results are meaningful and accurate?

 

  1. ^

    See also original Cold Takes post explaining why such evaluations are valuable 

adnhw🔸 @ 2025-01-29T20:18 (+7)

I'd prefer to see you pick 'people who have made AI predictions who are not famous for those predictions' in some random-ish way. I could just say 'Gary Marcus' and be done, but I'd only be saying that because he disagrees with me on AI progress and I think he'd look bad if his track record was examined. 

You're probably not trying to be super scientific, but I definitely wouldn't cite anything to a policymaker that was cherry-picked. I also wouldn't cite your tool if you only found an effect because you mainly compared people who are famous for scaling predictions, like Gary Marcus and Gwern.

Thanks for all the work you guys do! And love the new website.

Nathan Young @ 2025-01-30T12:28 (+2)

Oppenheimer

Nathan Young @ 2025-01-30T09:37 (+2)

Alexander Hamilton

Nathan Young @ 2025-01-30T09:37 (+2)

I would want to see a big database of the exact wording of the statement that I could look up or at least some large random sample.

Nathan Young @ 2025-01-30T09:36 (+2)

Paul Christiano

Nathan Young @ 2025-01-30T09:36 (+2)

Nixon

Nathan Young @ 2025-01-30T09:36 (+2)

Napoleon

Nathan Young @ 2025-01-30T09:36 (+2)

Kissinger

Nathan Young @ 2025-01-30T09:35 (+2)

Einstein

Nathan Young @ 2025-01-30T09:35 (+2)

Von Neumann

Nathan Young @ 2025-01-30T09:35 (+2)

Eliezer Yudkowsky