How to do conceptual research: Case study interview with Caspar Oesterheld

By Chi @ 2024-05-14T15:09 (+26)

Caspar Oesterheld came up with two of the most important concepts in my field of work: Evidential Cooperation in Large Worlds and Safe Pareto Improvements. He also came up with a potential implementation of evidential decision theory in boundedly rational agents called decision auctions, wrote a comprehensive review of anthropics and how it interacts with decision theory which most of my anthropics discussions built on, and independently decided to work on AI some time late 2009 or early 2010.


 

Needless to say, I have a lot of respect for Caspar’s work. I’ve often felt very confused about what to do in my attempts at conceptual research, so I decided to ask Caspar how he did his research. Below is my writeup from the resulting conversation.

How Caspar came up with surrogate goals

The process

Caspar’s reflections on what was important during the process

How Caspar came up with ECL

The process

Caspar’s reflections on what was important during the process


 

How Caspar came up with decision auctions

The process

[editor’s note: I find it notable that all the linked papers are in CS venues rather than economics. That said, while Yiling Chen is a CS professor, she studied economics and has an economics PhD.]

How Caspar decided to work on superhuman AI in late 2009 or early 2010

My impression is that a few people in AI safety independently decided that AI was the most important lever over the future and then discovered LessWrong, Eliezer Yudkowsky, and the AI safety community. Caspar is one of those people. While this didn’t turn out to be unique or counterfactually impactful, I am including his story for deciding to work on superhuman AI. The story is from notes Caspar left in writing after the interview. I mostly copied them verbatim with some light editing for clarity and left it in first person.

The process

“Much of this happened when I was very young, so there's some naivete throughout:


 

Caspar’s reflections on what was important during the process


 

General notes on his approach to research

What does research concretely look like in his case?

Thinks he might do when he does research, in no particular order:


 

Research immersion


 

Goal orientation vs. curiosity orientation


SummaryBot @ 2024-05-15T13:50 (+1)

Executive summary: Caspar Oesterheld shares insights into his conceptual research process, highlighting the importance of immersion, goal-orientation balanced with curiosity, and iterative high-level thinking alongside narrow projects.

Key points:

  1. Caspar developed key concepts like surrogate goals, evidential cooperation in large worlds (ECL), and decision auctions through a combination of deep immersion in research areas, high-level thinking, and building on existing ideas.
  2. Spending significant time (e.g. 6 months FTE) in a research area helps build useful heuristics and intuitions.
  3. Balancing narrow technical projects with regular high-level thinking about the overall problem space is important. Getting stuck in familiar reasoning loops can sometimes lead to breakthroughs.
  4. Having the right combination of ideas salient in mind can spark insights, as with realizing the importance of AI after learning about automated theorem proving.
  5. Research involves diverse activities like reading, writing, discussion, solo thinking, and background rumination. Occasional obsessive immersion alternating with background immersion can be helpful.
  6. Goal-orientation is important, especially with shorter timelines, but curiosity and exploration also play a key role. Academia introduces additional publishing incentives.

 

 

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.