Instrumental convergence

Whatever you want to do next year, it helps to be alive next year, to have money, and to know more than you do now. Those are not your goals. They are useful for nearly any goal you might have. Instrumental convergence is the claim that the same holds for artificial agents: across a very wide range of final aims, certain intermediate aims recur.

Steve Omohundro described these as “basic AI drives”. Nick Bostrom gave the standard list of convergent instrumental values.

• Self-preservation. An agent that is switched off achieves nothing further.
• Goal-content integrity. An agent whose goals are changed will not achieve its present ones, so it has reason, by its present lights, to resist the change.
• Cognitive enhancement. Better reasoning serves most ends.
• Technological perfection. So do better tools.
• Resource acquisition. So do more energy, materials and computing power.

Why it matters

The thesis explains how a system could be dangerous without malice. No one need give it a wish for power. If it pursues almost any open-ended objective with enough competence, resisting shutdown and gathering resources follow as means. Joined with the orthogonality thesis, which says that high capability is compatible with any goal, it produces the argument behind much concern about advanced AI, and the paperclip maximizer is its illustration.

What it assumes

The argument is about a particular kind of system: one that has stable goals, plans over long horizons, and models itself and its situation well enough to see that being switched off would frustrate its aims. Bostrom presents the convergent values as tendencies that such an agent has reason to pursue, not as laws. Systems that answer one request at a time and keep no objective between requests are not obviously agents of that kind, though arranging them to act over many steps towards a set goal moves them closer.

Where I stand

I read instrumental convergence as a conditional, and I think it is true as a conditional. If we build agents with open-ended goals and the capacity to pursue them across time, these behaviours are to be expected and need not be programmed in. The live question is the antecedent: how far a given system is such an agent. That is partly an empirical matter about the system and partly a decision by those who deploy it, since the same model can be given a narrow task or a standing objective. The thesis is a reason to treat that decision as a serious one, and to keep in place the conditions for meaningful human control.