A concern about the “evolutionary anchor” of Ajeya Cotra’s report on AI timelines.

Ajeya's report addresses this in the "What if training data or environments will be a bottleneck?" section, in particular in the "Having computation to run training environments" subsection:

An implicit assumption made by all the biological anchor hypotheses is that the overwhelming majority of the computational cost of training will come from running the model that is being trained, rather than from running its training environment. 

This is clearly the case for a transformative model which only operates on text, code, images, audio, and video since in that case the “environment” (the strings of tokens or pixels being processed) requires a negligible amount of computation and memory compared to what is required for a large model. Additionally, as I mentioned above, it seems possible that some highly abstract mathematical environments which are very cheap to run could nonetheless be very rich and support extremely intelligent agents. I think this is likely to be sufficient for training a transformative model, although I am not confident.  

If reinforcement learning in a rich simulated world (e.g. complex physics or other creatures) is required to train a transformative model, it is less clear whether model computation will dominate the computation of the environment. Nonetheless, I still believe this is likely. My understanding is that the computation used to run video game-playing agents is currently in the same ballpark as the computation used to run the game engine. Given these models are far from perfect play, there is likely still substantial room to improve on those same environments with a larger model. It doesn’t seem likely that the computational cost of environments will need to grow faster than the computational cost of agents going forward.  (If several intelligent agents must interact with one another in the environment, it seems likely that all agents can be copies of the same model.) 

In the main report, I assume that the computation required to train a transformative model under this path can be well-approximated by FHKP, where F is the model’s FLOP / subj sec, H is the model’s horizon length in subjective seconds, P is the parameter count of the model, and and K describe scaling behavior. I do not add an additional term for the computational cost of running the environment.

I did this analysis a while back, but it's worth doing again, let's see what happens:

If you are spending 1e25 FLOP per simulated second simulating the neurons of the creatures, you can afford to spend 4e24 FLOP per simulated second simulating the environment & it will just be a rounding error on your overall calculation so it won't change the bottom line. So the question is, can we make a sufficiently detailed environment for 4e24 FLOP per second?

There are 5e14 square meters on the surface of the earth according to wolfram alpha.

So that's about 1e10 FLOP per second per square meter available. So, you could divide the world into 10x10 meter squares and then have a 1e12 FLOP computer assigned to each square to handle the physics and graphics. If I'm reading this chart right, that's about what a fancy high-end graphics card can do.  (Depends on if you want double or single-precision I think?). That feels like probably enough to me; certainly you could have a very detailed physics simulation at least. Remember also that you can e.g. use a planet 1 OOM smaller than Earth but with more habitable regions, and also dynamically allocate compute so that you have more of it where your creatures are and don't waste as much simulating empty areas. Also, if you think this is still maybe not enough, IIRC Ajeya has error bars of like +/- SIX orders of magnitude on her estimate, so you can just add 3 OOMs no problem without really changing the bottom line that much.

It would be harder if you wanted to assign a graphics card to each nematode worm, instead of each chunk of territory. There are a lot of nematode worms or similar tiny creatures--Ajeya says 1e21 alive at any given point of time. So that would only leave you with 10,000 flops per second per worm to do the physics and graphics!  If you instead wanted a proper graphics card for each worm you'd probably have to add 7 OOMs to that, getting you up to a 100 GFLOP card. This would be a bit higher than Ajeya estimated; it would be +25 OOMs more than GPT-3 cost instead of +18.

Personally I don't think the worms matter that much, so I think the true answer is more likely to be along the lines of "a graphics card per human-sized creature" which would be something like 10 billion graphics cards which would let you have 5e14 FLOP per card which would let you create some near-photorealistic real time graphics for each human-sized creature. 

Then there's also all the various ways in which we could optimize the evolutionary simulation e.g. as described here. I wouldn't be surprised if this shaves off 6 OOMs of cost.



 

I'm entirely unconvinced that this is a relevant concern - if training data is the equivalent of complex environments, we kind-of get it for free, and even where we don't, we can simulate natural environments and other agents much more cheaply than nature.

Another consideration that I think Ajeya thought about but didn't go into the report is that anthropic considerations might mean that our planet's evolutionary path is unusually short compared to arbitrarily selected evolutionary paths. Which means using our specific evolutionary path as an anchor underestimates timelines.

(Of course there are many factors that suggest the evolutionary anchor is an overestimate as well). 

(Originally sent as feedback privately regarding an earlier version, then Nuño requested publishing as a comment instead)

Overall I think it's a fair point and would be curious what the response would be from Ajeya. I think you downplay the importance others have placed in the evolution anchor more than needed and you also could engage a bit more with Ajeya's existing arguments in the report.

You say that you think the evolution anchor isn't viewed as as important by others, but I'm not so sure: see e.g. Holden's blog post here which emphasizes it as an upper bound. I'd also look through this shortform. In particular it notes that Ajeya mentions in the report "I expect that some ML researchers would want to argue that we would need substantially more computation than was performed in the brains of all animals over evolutionary history; while I disagree with this, it seems that the Evolution Anchor hypothesis should place substantial weight on this possibility."

There's also this section of the report of which some parts somewhat close to directly address your concern, e.g.: "There are also some specific ways it seems that we could improve upon the “simulate natural selection” baseline, prima facie. For example, population sizes are a consequence of the carrying capacity of an ecological niche rather than being tuned to minimize the amount of computation used to evolve intelligent animals; it seems likely that they were far too large from a computational-efficiency standpoint. Additionally, the genetic fitness signal in the natural environment was often highly noisy, whereas we could plausibly design artificial environments which provide much cleaner tests of precisely the behaviors we are looking to select for."

Thanks for writing this post, have added it to my collection "AI timelines via bioanchors: the debate in one place".

A fairly minor point. You write:

After reading this, I think that the reader should:

//

3. Slightly increase credence that something like The Patient Philanthropy Project, which pretty much conditions on no AGI soon, is a good idea.

It seems to me that to the extent that one is persuaded by your argument, one should update in the direction of "patient longtermism". But that's different from patient philanthropy, as Owen Cotton-Barratt points out:

[T]here is no direct implication that "patient longtermists" should be less willing to spend money now than "urgent longtermists". Rather I think it's an open question which will depend on a lot of messy empirics (about giving opportunities) which position should be more in favour of saving money now. My current guess is to recommend spending rather than saving money at current margins to both patient and urgent longtermists.

This doesn't mean that what you wrote is strictly wrong, but it seems worth noticing.

Not that important, but...  in terms of what intuitions people have, the split of the computation into neurons/environment  is not a reasonable model of how life works. Simple organisms do a ton of non-neuron-based computations distributed across many cells, and are able to solve pretty complex optimization problems. The neurons/environment  split pushes this into the environment , and this means the environment was sort of complex in a way for which people don't have good ituitions (e.g. instead of mostly thinking about costs of physics simulation, they should include stuff like immunse system simulations).

Great post! It does seem prima facie infeasible that recapitulating evolution would even be computable. Another thing to consider is that trying to simulate evolution may not yield general intelligence if just run once and it may need to be simulated many times in order to stumble on general intelligence which adds to the amount of computation that may be needed if turns out that coming up with general intelligence is very unlikely.

Even if simulating the environment is costly, we can be more efficient than evolution, so the environment implies simulated-evolution is harder but other factors Ajeya doesn't consider implies simulated-evolution is much easier.

Really interesting. I think there are connections to the extended mind thesis - where mental processes are in part constituted by the external body and world, such that cognition can't be neatly circumscribed to the brain.  This seems a deeper system than is modelled by 'computation done by neurons + training data'. 

Another complication is the relationship between ecological or environmental complexity and the evolution of intelligence. Peter Godfrey-Smith's 'Environmental Complexity and
the Evolution of Cognition' is a good read on this. Other comments on this post point to video game worlds and getting interaction by copying the evolving agent - but I think this may drastically understate the complexity of co-evolving sets of organisms in the real world.  

I think it's unlikely that developing artificial intelligence requires these wrinkles of mind extension or environmental complexity. But I interpret the evolutionary anchor argument as a generous upper bound based on what we know evolution did at least once. For that purpose, our model should probably defer to evolution's wrinkles rather than assume they're irrelevant.