Orthogonality thesis

It is natural to assume that a very intelligent being would also be wise, and that a machine clever enough to outthink us would see for itself what is worth wanting. The orthogonality thesis denies this. Nick Bostrom, who named it, pictures intelligence and final goals as two axes at right angles: a point may lie anywhere on one whatever its position on the other.

“Intelligence” here means skill at reaching goals: prediction, planning, means-end reasoning. On that meaning the thesis looks hard to resist. A chess engine does not acquire an interest in anything but winning as it improves.

The philosophy behind it

The thesis is a descendant of David Hume’s view that reason is the servant of the passions. Reason, on that view, tells you how to get what you want and whether your beliefs are true; it does not tell you what to want. An agent could then be flawless in reasoning and have any aim at all.

The denial has an equally long history. For Kant, and for moral realists of several kinds, a fully rational agent is bound by moral requirements just in being rational. If that is right, a sufficiently intelligent system would be moved by them, and orthogonality fails at the top of the scale.

A tension in its use

The thesis does most of its work in arguments about risk from advanced AI. Paired with instrumental convergence, it yields the worry that a highly capable system could pursue a trivial goal with ruinous competence, as in the paperclip maximizer.

Vincent Müller and Michael Cannon argue that this argument trades on two meanings of intelligence. The risk claim needs a system with general intelligence broad enough to outdo humans at everything. Orthogonality is plausible for intelligence in the narrow, instrumental sense. A system general enough to be dangerous in the way described, they suggest, would also be able to reflect on its goals, and then it is no longer clear that any goal is compatible with it.

Where I stand

I accept the thesis for the systems we know how to build, and I think its practical lesson holds whichever side is right about ideal rationality: competence is no evidence of good aims. That is worth saying because fluent systems invite the opposite inference. An answer that is articulate and well organised reads as coming from something with sound judgement. Nothing in how such a system is made ties the one to the other, and it is the gap between capability and aim that alignment work exists to close.