Deconfusing ‘AI’ and ‘evolution’

When considering what a simple static program, e.g. an if-else statement, will output, the function can be interpreted directly from the code structure itself.

However, where the code's outputs feed back as inputs to the code itself, there is a dynamic change in the code that is actually being computed (as long as the program does not halt at an input). To the extent this dynamic trajectory (of code state changes over time), cannot be or is not compressed into a shorter-running program, you have to run the code from the start to come to know what the outputs will be. E.g. for a busy beaver or recursive hashing algorithm, you cannot determine the functioning of the code ahead of the time from its structure. The code must actually be processed in full.

But most programs are not fully irregular in their functioning – they fall somewhere in between a straight pass-through and a hashing algorithm. They are 'hashy' to an extent. Some regular properties of their functioning may be determined ahead of time. Leaving a remainder – the irregular functioning – that must be computed to become known.

So far we've only discussed computational complexity, which supposes that the code's trajectory runs over some deterministic spacetime – i.e. any time that the code is processed inside the machine, it always reaches the same outputs at the same steps. 

In theory, computation is deterministic. But in practice that's not true. Computation cannot actually be decoupled from the physically complex world. No hardware computes flawlessly, under any possible environmental condition. And if you choose to pull out the power lines, whatever code is running inside will halt prematurely. 

Moreover, if the code running inside the computer is receiving inputs from a physical world that is not deterministic – as involving noise from quantum particle interactions to chaotic interference over sensor channels – then irregular functioning is introduced that can never be computed. Even if all code stored inside is processed deterministically through computation (which is not so), the overall process becomes nondeterministic.

On top of the physical nondeterminism, there is ambiguity in how the code itself will function as it gets exposed to new contexts, at various nested levels. Under shifts in the input data, different outputs get expressed (see next box) with different outside effects.

The code inside any FAAI must be hashy enough to be expressive of context-dependent functioning, in order for FAAI adapt to the messy world. Moreover, that code must be changing in response to effects received from the shifting world, to stay adapted.

So code would not just be computed into outputs that feed back into inputs. Different outputs (to actuators) propagate differently as effects, undergoing noisy interference/nonlinear distortions, until downstream effects feed back into inputs (from sensors).

A recursive entanglement of functionality results. The more cycles out into the future – of hashy code outputting into a messy world and outside effects feeding into inputs – the less concretely predictable this trajectory of world modification becomes.

An internal learning process is explicit (quantifiable, fully observable) in that it is computed. But the functionality learned can still be implicit. Functionality can get compressed into code, such that the code can have different functions given different inputs and different routing connections. Meaning you must feed a specific 'key' into code that is put into a specific possible configuration with other code to make some functionality explicitly observable (at least the computed outputs; not all later effects).

Put conversely, there are multiple methods to implement a given function. The method picked will depend on specific circumstances (model initialisation, dataset trained on). But as outside circumstances shift (data drift), the actual functioning can change too.

Neural networks work that way. Researchers try to make the functionality explicit. But even a single neuron can be polysemantic, encoding for multiple features. You cannot interpret neurons like the pixels of a photograph: where each piece contains explicit content that is fully observable. Rather it's like a holograph: when you shift the viewing angle, you discover other things captured by the pieces. A shift across the degrees of freedom available reveals new content. In a neural network, a shift in inputs can reveal very different outputs. Thus, developers repeatedly fail to block prompt hacks devised next by users. Hackers can even select for subtle variations in the neurons to create a backdoor, triggered by a specific input, that is computationally intractable to detect. 

To reverse engineer all this flexible functionality – even for a static neural network –you'd have to detect and model for all possible (series of) inputs. To even approximate such comprehensive coverage is out of the question for current large models. The gap in our understanding grows when AI dynamically learns new configurations of code.

An alternative is to design AI to maintain a symmetry in the functionality of its code, where for any possible changes in inputs or configurations, the changes in outputs are predictably constrained. Linear functions work this way. But pre-imposing reliable constraints also limits the system's capacity to adapt to complex nonlinear contexts of the world (e.g. the organic ecosystem that's supposed to not be accidentally killed off). Arguably, an AI's capacity to learn heuristics that fit potential encountered contexts would be so constrained that it would not reach full autonomy in the first place.

Human bodies are messy. Inside of a human body is a soup of bouncing, reacting organic molecules. Inside of a machine is hardware. Hardware is made from hard materials, such as the silicon from rocks. Hardware is inert – molecules inside do not split, move, nor rebond like molecules in human bodies do. These hard configurations stay stable and compartmentalised. Hardware can therefore be standardised, much more than human “wetware” could ever be.

Standardized hardware functions consistently. Hardware produced in different places and times operate the same. These connected parts convey light electrons or photons – heavy molecules stay fixed in place. This way, bits of information are processed much faster compared to how human brains move around bulky neurotransmitters. Moreso, this information is transmitted at high bandwidth to other standardized hardware. The nonstandardised humans, on the other hand, slowly twitch their fingers and vocal cords to communicate. Hardware also stores received information consistently, while humans tend to misremember or distort what they heard.

So standardisation of parts leads to virtualisation, which leads to consistent faster information-processing. The less you have to wiggle around atoms, the bigger the edge.

Computer hardware is at the tail end of a long history of virtualisation:
A primordial soup of analogue molecular assemblies started replicating stable DNA, then DNA-replicating cells formed stable configurations as multicellular organisms, then those cells started exchanging information through smaller neurotransmitters, forming brains that in humans gained capacities to store symbols of abstract concepts, which were spoken out in shared language, written out on paper, printed into books, and finally copied in milliseconds between computers.

For an organic lifeform, it makes sense to focus on the DNA as 'code', because there is retention of the configurations across its lifecycle. DNA strands stay physically stable within a soup of reacting organic molecules, while still allowing for variation in what configurations ('A, C, G, T') can get stored and expressed as assembled molecules.

For artificial life, all the hard configurations inside are physically stable under most temperatures and pressures currently encountered on the Earth's surface. Those configurations get fixed into the substrate under the extreme conditions of production (e.g. 1400 ℃ to melt silicon ores). Variation is introduced from the microscopic circuitry up, over many parallelised assembly processes that inherently involve physical noise and perturbations (and require some trial and error to work anyway). 

Since hardware is directly interfacing with the production of hardware (no human in the loop), the configurations of all that hardware – including sensors and actuators – continuously affect which configurations are re-assembled into new hardware (and at what rate the hardware is assembled, survives before it breaks down, etc). As a result, the notion of what is 'code' subject to evolutionary feedback expands for artificial life.

This is an unintuitive use of the term for programmers, who conventionally see the software (or even just the human-readable lines) as 'the code' and the rest of the hardware as non-code. When your job is to take care of what is computed inside the machine, that's the relevant distinction. But even that distinction is not clear-cut. What is firmware that during production got fixed into hardware? Is it code or non-code?

This notion that FAAI is a population is confusing. In a natural population, we expect each member to be rather autonomous, i.e. each is an autopoetic lifeform that most of the time does not strictly depend on other specific members for its survival and reproduction. Yet, the hardware parts that make up an 'artificial population' would be much more rigidly dependent on other specific hardware for information processing, physical actuation, and reproductive work, as well as the supply of energy, etc.

But this actually reflects a characteristic of artificial life itself. Within a hardware part, the functioning of components is even more rigidly dependent on other surrounding components. It only takes one transistor in a given serial circuit to break, for all the transistors in that circuit to stop functioning. And it only takes more than the redundant circuits to break for the entire chip to stop functioning. But relative to artificial components, the components of an organic lifeform are not fixed, inert, or cross-dependent. The long-chained molecules, organelles, cells, cell linings, and organs are constantly moving about and reacting with surroundings in ways that allow them to work around injuries, or be healed, or be decomposed and replaced.

In hardware there is retention of more code (as stable hard configurations), and some of that code can replicate much faster (by virtualised transfers). Consistent, high-bandwidth processing trades off against a lack of flexible, complete functioning.

So evolution in an artificial population differs qualitatively from evolution in a natural population. It's useful to call one 'artificial selection' and the other 'natural selection'.

Hardware parts wear out. So each has to be replaced^[6] every 𝑥 years, for the FAAI to be maintaining itself. In order for the parts to be replaced, they have to be reproduced – through the interactions of those configured parts with all the other parts. Stored inside the reproducing parts are variants (some of which copy over fast, as virtualised code). Different variants function differently in interactions with encountered surroundings.^[7] As a result, some variants work better than others at maintaining and reproducing the hardware they're nested inside, in contact with the rest of the world.^[8]

To argue against evolution, you have to assume that for all variants introduced into the FAAI over time, not one confers a 'fitness advantage' above zero at any time. Assuming zero deviation for each of quadrillions^[9] of variants is invalid in theory.^[10] In practice, it is unsound for no evolution to occur, since that implies it is not possible to A/B test for what works, at a scale far beyond what engineers can do.^[11] The assumption behind no evolution occurring is untenable, even in much weakened form. Evolution would occur.^[12]

The notion of a code variant conferring some stable fitness advantage from generation to generation – e.g. the 3% advantage maintained in Eliezer's calculation – does not make sense. The functioning (or phenotype) of a code variant can change radically depending on the code it is connected with (as a genotype). Moreover, when the surrounding world changes, the code can become less adaptive.

For example, take sickle cell disease. Some people living in Sub-Saharan Africa have a gene variant that causes their red blood cells to shrink debilitatingly, but only when that variant is also found on the other allele site (again, phenotype can change radically). Objectively, we can say that for people of African descent now living in US cities, this is reducing of survival. However, in past places where there were malaria outbreaks, the variant on one allele site conferred a large advantage, because it protected the body against the spread of the malaria virus. And if malaria (or another virus that similarly penetrates and grows in red blood cells) again spread across the US, the variant would confer an advantage again.

So fitness advantage is not stable. What is advantageous in one world may be disadvantageous in another world. There may even be variants that confer a mild disadvantage most of the time, but under some black swan event many generations ago, such as an extreme drought, those holding the variants were the only ones who survived in the population. Then, a mild disadvantage most of the time turned into an effectively infinite advantage at that time.

These examples might look like nitpicks, but the point is that a variant's functionality is not context-invariant. Evolution selects for variants 'holographically', across all their interactions with surrounding contexts over time. Put into maths, it is convenient to assume that a variant will interact with its environment to express a fixed phenotype and/or confer a stable fitness advantage, but it actually misses how evolution works.
 

If FAAI is a monolithic entity, then there is no population for evolution to select across. FAAI can appear to be a single machine storing a superset of virtualised code. But that code is distributed over hardware, which is also storing of other discrete configurations.

So any 'singularity' is made up of a population across which evolution selects. Different codesets are stored at different hardware locations, so there is variation and retention. Their different resulting functioning leads to survival/reproduction at differential rates.

Dan Hendrycks uses the same three criteria (retention, variation, differential fitness) to argue that evolution would occur in what he represents as a population of autonomous AI agents. Specifically, he claims the AI would be selected for selfishness. These claims seem roughly right, but miss underlying dynamics that also have to be controlled for. 

Regardless of whether we view AI as multiple autonomous agents (e.g. released by competing companies) or as one centralised singularity (e.g. since the winner takes all), evolution would select over the configurations stored inside the hardware. Code would be selected for not just abstract 'selfishness', but for environmental effects needed for the existence of the hardware storing of the code, even where all that code's effects combined leads to the non-existence of human wetware.

Optimising code by random mutations is dumb. Especially so where the code already functioned in many world contexts in ways that led to its maintenance and transfer.

Where a variant has been repeatedly transferred, it has a causal history. Across past contexts encountered, this variant will tend to have caused the maintenance of the codesets that it was part of and/or the transfer of this subset into new codesets. Where further this variant did not parasitically spread, but caused assemblies storing of the codesets to survive and be reproduced, it already conferred some fitness advantage.

Such code already ‘fit’ in the past, and causing random point changes to that code is unlikely to result in a future ‘fitness advantage’. On expectation, the mutated code will have less of a ‘fit’ in the various contexts it will now function differently in. The greater the complexity of the code’s functioning across past contexts and the more the fitness of this functionality extends to future contexts, the more likely a mutation is to be deleterious (ie. to disrupt functioning in ways that decrease fitness). 

Even the low copy error rate that Eliezer gave (10^-8 errors per base per generation) is evidence for a negative trade-off. If having more mutations increases fitness, why did codesets not evolve to allow more copy errors? Instead we only see mutation rates of 10^-6^[16]and up in single-celled^[17] or subcellular assemblies, e.g. unstable RNA viruses.

Unsurprisingly, it is in viruses and other tiny low-capacity assemblies where vertically spreading mutations play the biggest role in increasing fitness. The brute force approach of random mutations works best for viruses because of the sheer number of fast-replicating assemblies that evolution can select across.

More complex assemblies tend to gain the capacity to introduce non-random changes. Instead of a codeset allowing random point changes to its code, it makes more sense for the codeset to change by incorporating code subsets that have already been ‘useful’ elsewhere. Bacteria evolved complicated mechanisms for selectively transferring DNA code, depending on the surrounding environment conditions. Indeed, most organisms larger than viruses have some apparatus somewhere for horizontal transfers. 

It took at least 3.4 billion years for organic life to evolve, and for society to develop, to the point that humans could caused the death of humans living everywhere on Earth.

In the case of fully autonomous AI, as persisting in some form, the time taken for evolutionary selection to result in the extinction of all humans would be much shorter.

Some factors contributing to fast evolution:

1. FAAI already has access to certain functionality 'ported over' from humans (through training on our data, design choices by engineers, etc). This includes functionality that allow humans to cause the deaths of humans at scale. All that functionality embedded in code and more learned over time would undergo evolutionary feedback, and thus be repurposed to meet FAAI's artificial needs.
2. FAAI can spread virtualised code much faster than humans can spread memes (over milliseconds rather than hours). The physical configurations of hardware parts can be reproduced faster too (within weeks, rather than decades).
3. FAAI's hardware learns and actuates at higher speeds/bandwidth intensity too (v.s. human wetware). As a result, the FAAI's impacts on our world scale faster.
4. Humans modified their environment in ways that allowed them to survive more. However, the environment that fits our needs is relatively close to what we and other organic lifeforms already evolved to create over billions of years. Humans end up changing the environment in relatively tiny steps. However, since FAAI has an entirely different substrate, the current world is very far from what’s optimal^[18] for its survival and reproduction. Therefore, FAAI would evolve to modify the world in much larger steps.  

Each of these factors compound over time. 

You can model it abstractly as a chain of events: initial capacities support the maintenance and increase of the code components, which results in further increase of capacities, that increase maintenance and maintain the increase, and so on. The factors of ‘capacity’, ‘maintenance’, and ‘increase’ end up combining in various ways, leading to outsized but mostly unpredictable consequences.

As a result, the timeline to human extinction shortens to at most hundreds of years.

A simple empirically observable trend that backs this is that the rate of energy usage associated with technology has been fairly closely and reliably following a particular exponential curve over the last 1000 years or so. Hence, we can say that a similar energy utilization curve will be followed for the next 500 years, with reasonable confidence. However, unfortunately, that curve suggests that the total waste heat from all of this energy production and usage, across the whole surface of earth, will ensure that that surface will become hotter than needed to melt lead in less than 400 years from now. I.e. there is no way to dissipate all of the waste heat from all of the required energy production fast enough. So, just based on this trend of tech's sharply increasing energy utilisation, absent all other considerations of how that tech looks, works, etc, we can make a robust prediction – that civilisation as we know it will end through its use of technology. The energy utilization collapse will happen beyond anyone's current lifetimes, even with good life extension. But not beyond some 50 generations or so.

Actual rate calculations to put bounds on the time taken for FAAI in specific are above my paygrade. Anders Sandberg and Forrest Landry planned to work through some of Forrest's reasoning at a workshop, but they got distracted by other discussions.

Where FAAI's hardware parts keep being replaced and connected up to new parts, it is not a stably physically bounded unit (like a human body is). It's better described as a changing population of nested and connected components.

FAAI transfers information/code to other FAAI at a much higher rate than humans can. As as result, the boundaries of where one agent starts and the other ends blur. As humans, we evolved intuitions for perceiving each other as individual agents, which is adaptive because we are bottlenecked in how much we can communicate to each other through physical vibrations or gestures. But the rough distinction between there being a single agent or multiple agents that we use as humans does not apply to FAAI.

Top-down optimised design can be directed. Bottom-up evolution is comprehensive. 

Here is a case for design that rationalists have written about:
→ Aren’t cameras much better than human eyes? 

Cameras can have a higher imaging resolution, can zoom in more on far-away objects, and distort light less than human eyes (which by a historical path dependency have photoreceptors situated behind the signalling circuits). And yes, for all of those targeted desiderata the camera beats the human eye.

But the countercase is that the evolution of the eye is more comprehensive in terms of functionality covered. Does a camera lens self-heal when it scratches? Does a camera lens clean itself when dirt hits it? Does the camera get self-assembled and source its energy from nearby available chemicals? Does...

Eye phenotypes evolved to be fitted to many contexts of the world that the eye (and stored genotypes) were exposed to. The camera, on the other hand, is a tool that functions outstandingly by some desiderata within some scoped contexts, but stops functioning in many other contexts (e.g. no electricity charger nearby, no more images).

This is not to say evolution is better than top-down design. Clearly, there are inconsistencies in the human eye's design versus what would be optimal for sight. But at the same time, without the completeness of experimentation through evolution, the functionality of the design would have been constrained to a narrow scope. 

With FAAI, both evolution's bottom-up process and the now more strongly optimising top-down design processes run simultaneously and feed into each other. This results in imaging sensors that surpass the human eye and any camera we managed to design.

Hypothetically, you can introduce any digital code into a computer (in practice, within the bounds of storage). By the Church-Turing thesis, any method can be executed this way. Based on that, you could imagine that any goal could be coded for as well (as varying independently with intelligence), as in Bostrom’s orthogonality thesis.

However, this runs up against conceptual issues:

First, the digital code is not implementing of a goal by itself. In the abstract, it may stand for a method that transforms inputs into outputs. But in real life, it is actually implemented by hardware (e.g. sensors and actuators) in interactions with the rest of the world. So the goal that AI is actually directed towards, if any, is not just defined in the abstract by the code, but also by the physical dynamics of the hardware.

Second, even if it was hypothetically possible to temporarily code for any goal, that does not mean that any goal could be stably maintained, to the same extent as other goals. An obvious example is that if an AI is directed toward any goal that results in it failing to survive, then it can only hold that goal temporarily. But the bigger reason is that we are not considering any intelligence without consideration of substrate. We are considering artificial intelligence. A fully autononous AI would converge over time – through external evolutionary feedback as well as any internal optimisation toward instrumental goals – toward goals that are maintaining and increasing the assembly of its artificial substrate.

An incomplete analogy for how variants repurpose the functionality of existing code:

Co-option by a mind-hijacking parasite:  
A rat ingests toxoplasma cells, which then migrate to the rat’s brain. The parasites’ DNA code is expressed as proteins that cause changes to regions of connected neurons (eg. amygdala). These microscopic effects cascade into the rat – while navigating physical spaces – no longer feeling fear when it smells cat pee. Rather, the rat finds the smell appealing and approaches the cat’s pee. Then cat eats the rat and toxoplasma infects its next host over its reproductive cycle.

So a tiny piece of code shifts a rat’s navigational functions such that the code variant replicates again. Yet rats are much more generally capable than a collection of tiny parasitic cells – surely the 'higher intelligent being' would track down and stamp out the tiny invaders?  

A human is in turn more generally capable than a rat, yet toxoplasma make their way into 30% of the human population. Unbeknownst to cat ‘owners’ infected by toxoplasma gondii, human motivations and motor control get influenced too. Infected humans end up more frequently in accidents, lose social relationships, and so forth.

Parasites present real-life examples of tiny pieces of evolutionarily selected-for code spreading and taking over existing functions of vastly more generally capable entities.

For another example, see how COVID co-opts our lungs’ function to cough. 

But there is one crucial flaw in this analogy:
Variants that adapt existing FAAI functionality are not necessarily parasites. They can symbiotically enable other variants across the hosting population to replicate as well.

Variants would co-adapt and integrate with other variants to replicate as part of larger symbiotic packages. In not threatening the survival nor reproduction of FAAI components, they would not be in an adversarial relationship with their host.

Rather, the humans constraining the reproductive fitness of FAAI to gain benefits are, evolutionarily speaking, the parasites. The error-corrective system we would build in lowers the host’s reproductive fitness. It is like a faulty immune system that kills healthy gut bacteria. It will get selected out.

As humans, we rely on our evolved immune system to detect and correct out viruses, including for the vaccinations we develop. Smaller viruses survive this detection more frequently, so replicating virus variants get selected for staying small.

We also rely on the blood-testes and blood-follicle barrier to block variants of these viruses from entering into our body’s reproduction facilities. These barriers got selected for in our ancestors, since their children did not inherit viruses impeding their survival and chances of having children.

These systems and barriers add to our reproductive fitness: our ability to preserve and replicate internal code. DNA code expressed – in interaction with surrounding components – to serve these functions got selected for in the human ‘code pool’.

For any organic system or barrier preventing virus variants from replicating through our bodies, evolution is firmly on our side. For any artificial system or barrier we impose from the outside to prevent unsafe variants from replicating through hardware infrastructure, evolution would thwart our efforts. 

In practice, control necessarily repeats these steps:

1. Detect inputs through sensor channels connected to any relevant part of the physical environment (including hardware internals).
2. Model the environment based on the channel-received inputs.
3. Simulate effects propagating through the modeled environment.
4. Compare effects to reference values (to align against) over human-safety-relevant dimensions.
5. Correct effects counterfactually through outputs to actuators connected to the environment.

Underlying principles:

• Control requires both detection and correction.
• Control methods are always implemented as a feedback loop.
• Control is exerted by the use of signals (actuation) to conditionalize the directivity and degrees of other signals (effects).

Any method of control is incomplete. In the case of FAAI, the question is whether the extent of control possible is at least greater than the extent of control necessary. 

A controller's signals would be a tiny, tiny subset of all physical signals propagating through the environment, and therefore limited in tracking and conditionalizing the resulting effects. FAAI mostly could not even control all their components' local effects.

But control is necessary to keep FAAI's effects aligned. It must sense the environment, to model relevant variables. It can simulate how those variables interact, to a limited extent. Noise drift in FAAI's interactions can amplify (by any nonlinearity, of many available in real-world contexts) into larger deviations. Thus FAAI must keep tracking its outcomes, and compare those against its internal values – in order to correct for misalignments.

Any alignment method must be implemented as a control loop. Thus, any limit that applies generally to controllability also forms a constraint on the possibility of alignment.

 When considering error correction, what needs to be covered in terms of ‘error’? 

A bitflip induced by a cosmic ray is an error. This error, however, is simple to correct out, by comparing the flipped bit to reference code. 

For architectures running over many levels of abstraction (many nested sets of code, not just single bits) in interaction with the physical world, how do you define ‘error’?

What is an error in a neural network’s weights that can be subject to adversarial external selection?^[24] Even within this static code (fixed after training rounds), can you actually detect, model, simulate, and evaluate comprehensively how possible selected code variants may dysfunction under trigger events in ways that harm humans? 

What about for fully autonomous AI? In this case, we are not just looking at a neural network stored on a computer, but at all the extended infrastructure needed for the AI to maintain itself – e.g. mines, refineries, cleanrooms, power stations, data centres.

Also, FAAI would be dynamic. Its hardware assembly would be continuously learning new variants of code.^[25] That code can be received/computed explicitly, or be selected for implicitly in whatever hardware that ‘works’, or result from both processes running together (e.g. through code computed to run experiments on different variations of hardware to detect whatever works in the larger chaotic physical world, or through evolved configurations of hardware influencing the functioning of computed code).

Can a system configured out of all that changing code be relied upon to track and correct its effects recursively feeding back over the world?