Announcing the Winners of the 2023 Open Philanthropy AI Worldviews Contest

Very curious to hear how Open Philanthropy has updated as a result of this competition.

Looking at the winning entries would seem to suggest that Open Philanthropy is likely to now be less worried about these risks, but it would be interesting to know by how much.

Congratulations to the winners.  

My question is, now that we have winners, how do we make the best use of this opportunity? What further actions would help people think better about such questions, either those at OP or otherwise? 

This is complicated by the caveats. We don't know which of the points made are the ones the judges found to be interesting or useful, or which ones merely crystalized disagreements, and which ones were mostly rejected. 

As is, my expectation is that the authors (of both the winners and other entries) put a lot of effort into this, and the judges a lot of effort into evaluations, and almost no one will know what to do with all of that and the opportunity will by default be wasted. 

So if there's opportunity here, what is it? For me, or for others?

As another commentor notes, at the time I offered a rebuttal to the interest rates post, that I would still stand by almost verbatim, and I'm confused why this post was still judged so highly - or what we should be paying attention to there, beyond the one line 'the market's interest rates do not reflect the possibility of transformative AI.' 

I will refrain from commenting on the others since I haven't given them a proper reading yet (or if I did, I don't remember it). 

As a potential experiment/aid here, I created Manifold markets (1,2,3,4,5,6) on whether my review of these six would be retrospectively considered by me a good use of time.

Congrats to the winners, readers, and writers!
 

Two big surprises for me:

(1) It seems like 5/6 of the essays are about AI risk, and not TAGI by 2043. I thought there were going to be 3 winners on each topic, but perhaps that was never stated in the rules. Rereading, it just says there would be two 1st places, two 2nd places, and two 3rd places. Seems the judges were more interested in (or persuaded by) arguments on AI safety & alignment, rather than TAGI within 20 years. A bit disappointing for everyone who wrote on the second topic. If the judges were more interested in safety & alignment forecasting, that would have been nice to know ahead of time.

(2) I'm also surprised that the Dissolving AI Risk paper was chosen. (No disrespect intended; it was clearly a thoughtful piece.)

To me, it makes perfect sense to dissolve the Fermi paradox by pointing out that the expected # of alien civilizations is a very different quantity than the probability of 0 alien civilizations. It's logically possible to have both a high expectation and a high probability of 0.

But it makes almost no sense to me to dissolve probabilities by factoring them into probabilities of probabilities, and then take the geometric mean of that distribution. Taking the geometric mean of subprobabilities feels like a sleight of hand to end up with a lower number than what you started with, with zero new information added in the process. I feel like I must have missed the main point, so I'll reread the paper.

Edit: After re-reading, it makes more sense to me. The paper takes the geometric means of odds ratios in order to aggregate survey entries. It doesn't take the geometric mean of probabilities, and it doesn't slice up probabilities arbitrarily (as they are the distribution over surveyed forecasters).

Edit2: As Jaime says below, the greater error is assuming independence of each stage. The original discussion got quite nerd-sniped by the geometric averaging, which is a bit of a shame, as there's a lot more to the piece to discuss and debate.

Congratulations to the winners.

I haven't engaged deeply with any of the winning posts like the judges have, but I engaged shallowly with 3–4 when they were written. I thought they were methodologically doomed (‘Dissolving’ AI Risk) or constituted very weak evidence even if they were basically right (AGI and the EMH and especially Reference Class-Based Priors). (I apologize for this criticism-without-justification, but explaining details is not worth it and probably the comments on those posts do a fine job.)

Normally I wouldn't say this. But OP is high-status and I worry people will defer to its judgment about Good Posts and then defer to these posts (despite the disclaimer that the judges disagreed with some) or add them to reading lists or something, which I think would be mostly bad.

(To avoid being purely negative: I think excellent quantitative work on AI forecasting that directly addresses the big question—when will powerful AI appear—includes bioanchors, the Davidson takeoff report, and Fun with +12 OOMs of Compute.)

Congrats to the winners! It's interesting to see how surprised people are. Of these six, I think only David Wheaton on deceptive alignment was really on my radar. Some other highlights that didn't get much discussion:

• Marius Hobbhahn's Disagreements with Bio Anchors that lead to shorter timelines (1 comment)
  • In the author's words: 'I, therefore, think of the following post as “if bio anchors influence your timelines, then you should really consider these arguments and, as a consequence, put more weight on short timelines if you agree with them”. I think there are important considerations that are hard to model with bio anchors and therefore also added my personal timelines in the table below for reference.'
  • Even though Bio Anchors doesn't particularly influence my timelines, I find Hobbhahn's thoughtful and systematic engagement worthwhile.
• Kiel Brennan-Marquez's Cooperation, Avoidance, and Indifference: Alternate Futures for Misaligned AGI (1 comment)
  • Maybe this question is seen as elementary or settled by "you are made of atoms", but even so, I think other equilibria could be better explored. This essay is clear and concise, has some novel (to me) points, and could serve as a signpost for further exploration.
• Matt Beard's AI Doom and David Hume: A Defence of Empiricism in AI Safety (6 comments)
  • On reasoning described by Tom Davidson: "This is significantly closer to Descartes’ meditative contemplation than Hume’s empiricist critique of the limits of reason. Davidson literally describes someone thinking in isolation based on limited data. The assumption is that knowledge of future AI capabilities can be usefully derived through reason, which I think we should challenge."
  • Beard doesn't mean to pick on Davidson, but I really think his methods deserve more skepticism. Even before specific critiques, I'm generally pessimistic about how informative models like Davidson's can be. I was also very surprised by some of his comments on the 80,000 Hours podcast (including those highlighted by Beard). Otherwise, Beard's recommendations are pretty vague but agreeable.

Jason - thanks for the news about the winning essays.

If appropriate, I would appreciate any reactions the judges had to my essay about a moral backlash against the AI industry slowing progress towards AGI. I'm working on refining the argument, so any feedback would be useful (even if only communicated privately, e.g. by email).  

We’re honored to have received a prize for our work on the Existential Risk Persuasion Tournament (XPT)!

We’re in the process of extending this work and better connecting forecasts to policy decisionmaking. If you’re interested in working on this and our other projects, we wanted to mention that we’re currently hiring for full-time Research Analysts and Data Analysts. We’re also hiring for part-time Research Assistants.

In some cases, the judges liked that an entry crisply argued for a conclusion the judges did not agree with—the clear articulation of an argument makes it easier for others to engage. One does not need to find a piece wholly persuasive to believe that it usefully contributes to the collective debate about AI timelines or the threat that advanced AI systems might pose.

Facilitating useful engagement seems like a fine judging criterion, but was there any engagement or rebuttal to the winning pieces that the judges found particularly compelling? It seems worth mentioning such commentary if so.

Neither of the two winning pieces significantly updated my own views, and (to my eye) look sufficiently rebutted that observers taking a more outside view might similarly be hesitant to update about any AI x-risk claims without taking the commentary into account.

On the EMH piece, I think Zvi's post is a good rebuttal on its own and a good summary of some other rebuttals.

On the Evolution piece, lots of the top LW comments raise good points. My own view is that the piece is a decent argument that AI systems produced by current training methods are unlikely to undergo a SLT. But the actual SLT argument applies to systems in the human-level regime and above; current training methods do not result in systems anywhere near human-level in the relevant sense. So even if true, the claim that current methods are dis-analogous to evolution isn't directly relevant to the x-risk question, unless you already accept that current methods and trends related to below-human level AI will scale to human-level AI and beyond in predictable ways. But that's exactly what the actual SLT argument is intended to argue against!

On advice of Manifold I read Quintin's entry and wrote up a response on LessWrong. I would be curious to hear from OpenPhil and others if and how they found this useful at all. Unless I get unexpectedly good feedback on that, I do not anticipate engaging with the other four.