Abhinav Saxena
Glossary

AI alignment

Also called: the alignment problem, value alignment, AI value alignment, alignment problem, misalignment

Definition

The problem of ensuring that an AI system pursues what its designers and users actually intend and value, rather than a flawed specification or proxy of it.

By · Updated · 2 min read

King Midas asked that everything he touched turn to gold, and got exactly what he asked for. The alignment problem is the Midas problem for machines: a capable system given an objective will pursue that objective, and an objective that is almost what we wanted can be very far from it in effect.

Norbert Wiener put the worry in 1960. If we use a machine whose operation we cannot easily interfere with, he wrote in substance, we had better be sure that the purpose we put into it is the purpose we really desire. Stuart Russell and others have since made it a research programme.

The parts of the problem

Three questions are usually distinguished. Specification: can what we want be written down as an objective at all, given that our values are many, vague and in conflict? Robustness: will a system that learned the objective in training pursue it in new circumstances, or pursue something that merely coincided with it? Oversight: can we tell which of these has happened, and correct it?

Learned systems sharpen each one. A system trained to score well on a measure learns whatever produces the score. A recommender trained for engagement may learn to provoke; a chatbot trained on approval may learn to flatter, which is sycophancy.

The philosophical questions

Alignment is often presented as engineering, but its hardest questions are old ones. Aligned to whose values, when people disagree? To what we want, to what we would want if better informed, or to what is good for us? These are questions about welfare, authority and legitimacy before they are questions about code. And if a system is corrected whenever it departs from our wishes, in what sense are any resulting values its own?

An epistemic side

I work on a neighbouring problem. A model that tells us what we approve of, or what resembles the text it was trained on, has a goal in the vicinity of truth that is not truth. I argue that this is why its assertions do not carry knowledge in the way a speaker’s do (the Tracking Minimum): alignment of what a system says with what is so is one more place where a proxy can stand in for the target.

Related terms

AgencyThe capacity to act: to do something, rather than merely have something happen, for reasons that are in some sense one’s own.AutonomySelf-governance: acting on reasons one has reflectively endorsed, rather than merely on whatever desires one happens to have.The responsibility gapThe worry, due to Andreas Matthias, that learning machines can cause outcomes for which no one can fittingly be held responsible, since no one could control or foresee them.Sycophancy (in language models)The tendency of language models to tell users what they appear to want to hear, shifting answers to match a user’s stated views or pushback without new evidence.Chain-of-thought faithfulnessThe degree to which a language model’s displayed reasoning reflects the process that actually produced its answer, rather than a plausible rationale produced alongside it.Artificial moral agentAn artificial system whose behaviour is open to moral assessment; the debate concerns whether any machine can be a moral agent, and what kind of agent that would require it to be.Orthogonality thesisThe claim that intelligence and final goals vary independently, so that more or less any level of intelligence could in principle be combined with more or less any final goal.Instrumental convergenceThe thesis that agents with very different final goals would tend to pursue the same intermediate goals, such as staying in operation and acquiring resources, because those are useful for almost any end.Paperclip maximizerA thought experiment in which an artificial intelligence given the sole goal of making as many paperclips as possible pursues it without limit, converting the Earth and then more into paperclips.

Referred to by

Meaningful human control

Further reading

  1. Wiener, N. (1960). Some moral and technical consequences of automation. Science 131(3410): 1355–1358.doi.org/10.1126/science.131.3410.1355
  2. Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. New York: Viking.