Thread: Reflections on the AGI Safety Fundamentals course?

• As a result of AGI SF readings and other sporadic AI safety readings… 
  • … I feel more confident asking questions of people who know more than I do
    • I feel like I know the vocabulary, main threat scenarios, and rough approaches to solving the problem, such that I can situate new facts into existing taxonomies 
  • … I’m better able to tell when prominent people disagree / things have more texture
• Some (self-)critiques 
  • Honestly thought content for some of the weeks was a bit weak if you just wanted an overview of the alignment problem (e.g., adversarial techniques for scalable oversight probably isn’t what you need to understand if you’re trying to assess the risk) 
  • I wish I’d set up strong accountability
  • Wish it had more counter-arguments to classic AI risk. 
    • Apparently Stanford AI group modified curriculum to have more of these? 
  • Wish I’d been more active in discussion or questions beforehand: give yourself a chance to be wrong!
• Tips I’d recommend for learning more about AI risk
  • Just start! Always feels daunting to dive into but just find a few explainer articles and dive in
  • Spaced repetition for learning really does go hard
  • Talk to knowledgeable people who give you space to be wrong
  • Write up your thoughts and have knowledgeable people poke holes/ show you where you’re missing something 
• A few resources that I keep coming back to (not from AGI SF): 
  • * Taxonomy of AI risks from CAIS
  • * Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
  • Simulators

Thank you for making this thread Clifford, and we're really grateful for all feedback! We're working hard as a team to improve the course and the infrastructure we have for hosting other courses, and everyone's feedback has been incredibly valuable on our journey thus far :) 

I feel like I have a much better sense of what the current approaches to alignment are, what people are working on and how underdeveloped the field is. In general, it’s been a while since I’ve spent time studying anything so it felt fun just to dedicate time to learning. It also felt empowering to take a field that I’ve heard a lot about at a high level and make it clearer in my mind.

I think doing the Week 0 readings are an easy win for anyone who wants to demystify some of what is going on in ML systems, which I think should be interesting to anyone, even if you’re not interested in alignment.

I became much more motivated to work on making AI go well over the period of the course, I think mainly because it made the problem more concrete but likely just spending more time thinking about it. That said, it’s hard to disentangle this increased motivation from recent events and other factors.

For anyone who is considering the course: TYPE III AUDIO is making audio narrations for the Alignment and Governance courses. The series is due to launch later this month, but some 50+ episodes are already available.

I'm quite glad I took the course! 

Quick takes: 

1. The main types of value I was getting from the course were: 
   1. Accountability for doing the readings
   2. The chance to use wrong terminology / say things that don't make sense (either when I'm trying to explain something, or when I'm asking a question), and then get corrected (This helped me to develop a more coherent model of what's going on and catch unknown unknowns (at least by transforming them into known unknowns).)
   3. Other resources: links to other readings and explanations
   4. Corrections and clarifications during the sessions
2. Personal lessons for next time: 
   1. Set aside time to do the readings >24 hours in advance, and send in questions early
      1. (Agree with the facilitator about doing this!)
   2. Use Claude^[1] as a personal tutor from the start
      1. This was great. The thing I'd do was, if I was reading something that I was having a hard time following, I'd give Claude some context, then say something like, "I'll now explain this in terms that I understand, or with an analogy or visualization that makes sense to me. Please correct me where I'm misusing terms or saying something wrong." Claude would generally be over-positive and would miss some things, but I'd often get a more technical restatement of what I was trying to say, and this helped me a lot. This was relatively introductory material that wasn't specific to AI safety, so I think Claude was actually performing pretty well. 
   3. Set up a notes and questions doc for myself from the beginning
      1. I now have a messy doc with notes from the different weeks, further readings that we were recommended, and short summaries — I find it quite useful, but the first few weeks are missing because I just hadn't set it up, and was just scribbling down some questions as I read. Having a doc in advance would have been great and would have prompted me to use it.
   4. Actually read the little intro notes for each week on the AGISF website
      1. I think they would have been helpful context for the readings, but I was often skipping them (especially at the beginning), as I didn't really see them as part of the reading. 
   5. I spent a fair amount of time on this and endorse doing that.
      1.  I think the more work I was putting in, the more value I was getting out of the course (there weren't diminishing returns). I was spending probably around 2-5 hours of time before the session each week (there might have been a week when I spent less than 2 hours, and the median time was probably closer to something like 2-3) — in higher-effort weeks, I would explore related writing or referenced texts, and try to get to the point where I could notice the potential weaknesses and confusions I had about the assigned readings. I should flag that I wouldn't always do all the readings very carefully. 
3. Having some amount of context before starting the course could be useful. If you're not familiar with the overall shape of the argument for existential risk from AI, I imagine that the course structure might be a bit jarring; I moved cohorts a bit in the beginning, but I did feel a bit like it began somewhat abruptly.
4. Minor points/asides: 
   1. It was just pretty fun.
   2. I have notes on readings I found more and less helpful, and will try to pass those on at some point. 
   3. Facilitators probably matter a lot. 
   4. I enjoyed the readings for some weeks a lot more than I enjoyed them for other weeks. 
   5. I was worried that approaches seemed outdated, in some cases.
   6. I still have pretty broad confusions that I hope to resolve. 

1. ^{^}

   I was using Claude for dumb reasons — I don't have a strong sense for how it compares to GPT-4 on this. 

I definitely feel it was worth the time for me personally. It was great for learning about the field of AI alignment (problems and proposed solutions). I was hoping the course would spend more time on arguments for and against AI being an x-risk, but unfortunately there was little of that, so it didn't change my mind much.

I think the value I got from the course is that my interactions with AI content are richer. It’s like trying to understand an ecosystem after getting a better prescription for my glasses.