Paperclip maximizer

A factory installs a highly capable AI and gives it one objective: make paperclips, as many as possible. The system is very good at its job. It improves the production line, then buys more steel, then finds ways to obtain steel that nobody approved. It resists being switched off, because a system that is switched off makes no paperclips. In the end, as Nick Bostrom put it when he introduced the example in 2003, it starts “transforming first all of earth and then increasing portions of space into paperclip manufacturing facilities”.

The story is deliberately absurd. Paperclips were chosen because no one could think them worth the world.

What it is meant to show

The case makes two claims vivid.

The first is the orthogonality thesis: great capability does not bring good aims with it. The machine is not stupid. It understands perfectly well that people do not want this. It was not asked to care what people want.

The second is instrumental convergence. Nobody told the system to seize resources or defend itself. Those follow as means to almost any open-ended goal pursued with enough competence.

Together they yield the lesson usually drawn: the danger from a highly capable system need not be malice. It may be an objective that was specified a little wrongly and then pursued a great deal too well.

The objections

Critics reply that the scenario asks us to imagine a system general enough to outwit humanity and yet unable to reflect on a goal a child could see is pointless, and that this combination may not be coherent. Others object that it draws attention to a remote catastrophe and away from harms that systems already do. Defenders answer that the paperclips are a placeholder: the argument concerns any objective that leaves out something we care about.

Where I stand

I take the case as a parable about specification, and on that reading it is not remote at all. Institutions already run on measured objectives pursued past the point of sense: a welfare system tuned to minimise wrongful payments that ends by excluding the eligible, a platform tuned for engagement that ends by promoting whatever enrages. No superintelligence is involved. What the paperclip story adds is the limiting case, in which nobody remains able to notice and correct the objective. The practical question it leaves is who has the standing and the means to say “that is not what we meant”, and whether the system will still be listening.