LLMs won’t lead to AGI - Francois Chollet

Thanks for sharing toby, I had just finished listening to the podcast and was about to share it here but it turns out you beat me to it! I think I'll do a post going into the interview (Zvi-style)^[1] and bringing up the most interesting points and cruxes, and why the ARC Challenge matters. To quickly give my thoughts on some of the things you bring up:

• The ARC Challenge is the best benchmark out their imo, and it's telling that labs don't release their scores on it. Chollet says in the interview that they test it but because they score badly, the don't release them.
• On timelines, Chollet says that OpenAI's success led the field to 1) stop sharing Frontier research and 2) make the field focus on LLMs alone, thereby setting back timelines to AGI. I'd also suggest that the 'AGI in 2/3 years' claims don't make much sense to me unless you take an LLMs+scaling maximalist perspective.

And to respond to some other comments here:

• To huw, I think the AI Safety field is mixed. The original perspective was that ASI would be like an AIXI model, but the success of transformers have changed that. Existing models and dependents could be economically damaging, but taking away the existential risk undermines the astronomical value of AI Safety from an EA perspective.
• To OCB, I think we just disagree about how far LLMs are away from this. i think less that ARC is 'neat' and more that it shows a critical failure model in the LLM paradigm. In the interview Chollet argues that the 'scaffolding' is actually the hard part of reasoning, and I agree with him.
• To Mo, I guess Chollet's perspective would be that you need 'open-endedness' to be able to automate many/most work? A big crux I think here is whether 'PASTA' is possible at all, or at least whether it can be used as a way to bootstrap everything else. I'm more of the perspective that science is probably the last thing that can possibly be automated, but that might depend on your definition of science. 
  • I'm quite sceptical of Davidson's work, and probably Karnofsky's, but I'll need to revisit them in detail to treat them fairly. 
  • The Metaculus AGI markets are, to me, crazy low. In both cases the resolution criteria are some LLM unfriendly, it seems that people are more going off 'vibes' and not reading the fine print. Right now, for instance, any OpenAI model will be easily discovered in a proper imitation game by asking it to do something that violates the terms of service.

I'll go into more depth on my follow-up post, and I'll edit this bit of my comment wiht a link once I'm done.

1. ^{^}

   In style only, I make no claims as to quality

I do think this set of benchmarks is neat.

My best guess, however, is that we're within spitting distance^[1] of scaffolded LLMs being able to solve these. (Unscaffolded LLMs would I think be way off.)

1. ^{^}

   What I really mean by this is something like "gee it really seems like you should be able to do this already with good enough scaffolding". Then my actual timeline for that to turn into a real system that someone's built which has done it is uncertain, and plausible values range from "it's already happened" to "it takes two or three more years".

The ARC Prize website takes this definitional stance on AGI:

Consensus but wrong:

AGI is a system that can automate the majority of economically valuable work.

Correct:

AGI is a system that can efficiently acquire new skills and solve open-ended problems.

Something like the former definition, central to reports like Tom Davidson's CCF-based takeoff speeds for Open Phil, basically drops out of (the first half of the reasoning behind) the big-picture view summarized in Holden Karnofsky's most important century series: to quote him, the long-run future would be radically unfamiliar and could come much faster than we think, simply because standard economic growth models imply that any technology that could fully automate innovation would cause an "economic singularity"; one such technology could be what Holden calls PASTA ("Process for Automating Scientific and Technological Advancement"). In What kind of AI? he elaborates (emphasis mine)

I mean PASTA to refer to either a single system or a collection of systems that can collectively do this sort of automation. ...

By talking about PASTA, I'm partly trying to get rid of some unnecessary baggage in the debate over "artificial general intelligence." I don't think we need artificial general intelligence in order for this century to be the most important in history. Something narrower - as PASTA might be - would be plenty for that. ...

I don't particularly expect all of this to happen as part of a single, deliberate development process. Over time, I expect different AI systems to be used for different and increasingly broad tasks, including and especially tasks that help complement human activities on scientific and technological advancement. There could be many different types of AI systems, each with its own revenue model and feedback loop, and their collective abilities could grow to the point where at some point, some set of them is able to do everything (with respect to scientific and technological advancement) that formerly required a human.

This is why I think it's basically justified to care about economy-growing automation of innovation as "the right working definition" from the x-risk reduction perspective for a funder like Open Phil in particular, which isn't what an AI researcher like Francois Chollet cares about. Which is fine, different folks care about different things. But calling the first definition "wrong" feels like the sort of mistake you make when you haven't at least good-faith effort tried to do what Scott suggested here with the first definition: 

... if you're looking into something controversial, you might have to just read the biased sources on both sides, then try to reconcile them.

Success often feels like realizing that a topic you thought would have one clear answer actually has a million different answers depending on how you ask the question. You start with something like "did the economy do better or worse this year?", you find that it's actually a thousand different questions like "did unemployment get better or worse this year?" vs. "did the stock market get better or worse this year?" and end up with things even more complicated like "did employment as measured in percentage of job-seekers finding a job within six months get better" vs. "did employment as measured in total percent of workforce working get better?". Then finally once you've disentangled all that and realized that the people saying "employment is getting better" or "employment is getting worse" are using statistics about subtly different things and talking past each other, you use all of the specific things you've discovered to reconstruct a picture of whether, in the ways important to you, the economy really is getting better or worse.

Note also that PASTA is a lot looser definitionally than the AGI defined in Metaculus' When will the first general AI system be devised, tested, and publicly announced? (2031 as of time of writing), which requires the sort of properties Chollet would probably approve (single unified software system, not a cobbled-together set of task-specialized subsystems), yet if the PASTA collective functionally completes the innovation -> resources -> PASTA -> innovation -> ... economic growth loop, that would already be x-risk relevant. The argument would then need to be "something like the Chollet's / Metaculus' definition is necessary to complete the growth loop", which would be a testable hypothesis.

I was under the impression that most people in AI safety felt this way—that transformers (or diffusion models) weren't going to be the major underpinning of AGI. As has been noted a lot, they're really good at achieving human-level performance in most tasks, particularly with more data & training, but that they can't generalise well and are hence unlikely to be the 'G' in AGI. Rather:

1. Existing models will be economically devastating for large sections of the economy anyway
2. The rate of progress across multiple domains of AI is concerning, and that the increased funding to AI more generally will flow back to new development domains
3. Even if neither of these things are true, we still want to advocate for increased controls around the development of future architectures

But please forgive me if I had the wrong impression here.

The tasks thematically seem like Raven's Progressive Matrices (a non-verbal test of intelligence where a pattern is established and you have to apply it to a new exemplar), link here

Thanks for the post! My quick thoughts:

> I'm not hugely worried about today's LLMs causing x-risk.
> I do think they could cause catastrophic harm in the hands of bioterrorists, but that's about it 
> I am going to basically shit my pants when an AI agent can,
1/ take a brief from me for a brand new tv,
2/ have it be delivered to my home on time, on spec and on budget,
3/ have also organised installation by a technician
4/ all while I'm out of the loop after step 1

I've played/designed a lot of induction puzzles, and I think that the thing Chollet dismissively calls "memorization" might actually be all the human brain is doing when we develop the capacity to solve them. If so, there's a some possibility that the first real world transformative AGI will be ineligible for the prize.

There's a big issue with semantics.

Most people would agree that an AI that automates ~99% of remote jobs is AGI, but Chollet disagrees because the AI might be "just memorizing".

The key thing is whether scaling will be enough to automate AI research, but unfortunately, they don't discuss this in the podcast.