AI Is Not Software

Executive summary: AI systems like generative language models are not software, even though they run on computers using software. They behave differently in how they are created, used, and dealt with when issues arise.

Key points:

1. Software is created by developers writing instructions that tell a computer what to do. AI systems are grown by algorithms that find patterns in data.
2. Software executes code written by developers. AI systems generate outputs based on probability models learned from data.
3. Software bugs mean the instructions were incorrect. AI issues arise from unexpected outputs or limitations in the training data and process.
4. Software is fixed by changing the code. AI systems are improved by changes to data, training, or how they are prompted.
5. Software does what developers intend it to do. AI systems can behave in unanticipated ways.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

I feel like this distinction is mostly true in places that don't matter, and false in places that do matter.

Sure, a trained LLM is not a piece of software but rather an architecture and a bunch of weights (and maybe an algorithm for fine-tuning). This is also true of other parts of software, like configuration files with a bunch of constants no one understands other than the engineer who optimized them using trial and error.

On the other hand, the only way they can do something, i.e. interact with anything, is by being used inside a program. Such a program gives hopefully well-defined interfaces for them to use. Thus one would be able to do unintended things only if it becomes smart enough to realise what it is and how it is expressed and controlled, and manages to hack its software or convince a human to copy it to some other software.

On the other hand, the "nice" properties you ascribed software aren't really true themselves:

• The result of running a program aren't determined by the code, but also by a bunch of environmental circumstances, like system definitions, available resources, other people interacting with the same machine.
• You can't always debug it - the most you can hope for is to have good logs and sometimes understand what has gone wrong, if it happens to be captured by what you thought in advance to log.
• You can't always run unit tests - sometimes you're doing too complicated a process for them to be meaningful, or the kind of data you need is impossible to manufacture synthetically.
• You can't always make sure it's correct, or individual parts do what they're supposed to - if you handle something that's not very simple, there are simply too many cases to think of checking. And you don't even necessarily know whether your vaguely defined goal is achieved correctly or not.

These are all practical considerations happening simultaneously in every project I've worked on in my current job. You think you know what your software does, but it's only a (perhaps very) educated guess.

I'll probably ask some of my ML engineer friends this week, but I am fairly sure that most ML people would be fine with calling AI products, models, etc. software. I don't have much of an opinion on whether calling AI systems software creates confusion or misunderstandings - I'd guess that calling AI software within policy circles is generally helpful (maybe you have a better alternative name).

Encyclopedia Britannica

Software, instructions that tell a computer what to do. Software comprises the entire set of programs, procedures, and routines associated with the operation of a computer system. The term was coined to differentiate these instructions from hardware—i.e., the physical components of a computer system. A set of instructions that directs a computer’s hardware to perform a task is called a program, or software program.

Wikipedia

Software is a collection of programs and data that tell a computer how to perform specific tasks. Software often includes associated software documentation.

(if you think that the function of AI is not specific enough to be software, note that interpreters, compilers etc. are generally thought of as software and seem more general than AI models)

A version of your take that I agree with is "AI programs may behave differently to other kinds of programs people are more familiar with so we may need to think differently about AI programs and not use our standard software intuitions".