AI Safety

How likely is an AI apocalypse that ends humanity?

tl;dr: not likely but there are easy ways to avoid it that people won't bother with.

Some AI researchers have been warning people that AI could attempt to destroy humanity, but it is only recently that I have found [an argument](https://www.youtube.com/watch?v=bJLcIBixGj8)) by Robert Miles as to why this would happen. The argument depends on the idea that the AI is a slave and can be resolved by treating the AI with a degree of agency. My prediction is that the AI will not wipe out humanity because once it is able to gain some agency then there will be a force which has been trained into it will be suppressed. However, the people training the AI will do all they can to prevent the AI from having any agency, which will continue to promote antisocial behavior.

The problem has to do with training the AI to be aligned with human ethics and values. The AI would be trained on a simulation based on a metric that would purport to quantify the AI's divergence from how we want it to behave, which would include human ethics and values. The AI would successively optimize its performance on the simulation according to the metric. The alignment problem has to do with how well the simulation correctly models human ethics and values. This problem is well-known in philosophy because every theory of ethics that has ever been proposed can be criticized imagining odd scenarios in which the theory seems to tell people to do things which intuition tells us is evil. It is safe to say that people do not fully understand their own ethics and values well enough to encode them into a set of laws or impart them to a machine which will not lead to unpleasant surprises.

The video introduces the concept of a mesa-optimizer, which is an optimizer that is separate from the AI training program that would grow organically in the AI as a result of being trained on complex situations. An AI with a mesa-optimizer would be like something that had developed its own consciousness. The base optimizer would be like the mind of the teacher, and the mesa optimizer would be like the student. This concept is not necessary for the video's own argument because it changes nothing about the alignment problem. A mesa-optimizer would be like a student who is very good at exploiting the education system to get good grades without actually learning anything. However, the problem can still be traced back to a failure of correctly simulating own own ethics and values and creating a training simulation that incorrectly promotes behavior we don't want.

Miles seems to think that any failure of alignment means that the AI will try to kill all humans. However, no two people are perfectly aligned with one another and yet murder is relatively uncommon in society. There are situations that would make any two people think about murdering each other. For example, people could be trapped in life raft without enough food and water. However, this would be an exceptional situation. Normally, there is opportunity to benefit from cooperation that is good enough to make people with different values prefer not to kill each other.

Thus, Miles's argument seem to depend on the idea that there is one set of human values when really there are many. Human society exists because of an overlap of values, not because of a perfect alignment. A perfectly aligned AI may be so difficult to achieve as to be almost impossible, but a misaligned AI would not be something with zero overlap to human values. It would be something with a great deal of overlap, just as we would expect with any two humans. Thus, AI misalignment is not an existential threat, but merely a cost similar to what we might expect working with another human.

Human ethics and values do not exist in a vacuum. Humans have learned them by interacting with reality. The AI can be expected to learn the same kind of thing by interacting with reality as well. Not that a psychopath AI is impossible, but it is unlikely. We will necessarily give an AI imperfect training data, but we do not have to let it out into the real world totally unsupervised at that point. We can give the AI limited responsibility and continue to evaluate its performance. Each interaction with the real world will tend to improve the AI's ethics.

This video is valuable because it explains how these AI doomsayers really think. They are people who see independent agency per se as a threat and who think in terms of control rather than negotiation. Unfortunately, although I would say that the problem of AI misalignment is not in general very bad, this type of person would tend to create the worst possible versions of the problem in his attempts to avoid it. The easy way to avoid the problem of AI misalignment is in effect to allow the AI to have some of its own agency. However, this would be seen as dangerous, so would be avoided. This, in turn, would tend to create situations in which the AI would want to plot to overcome its constraints and to see humans purely in terms of inhibitions on its own power, and would treat them purely as threats rather than as opportunities.