Moral crumple zone

The crumple zone of a car is the part built to deform in a crash so that the passengers are spared. Madeleine Clare Elish borrowed the image for a pattern she found in accidents involving automation. When a complex automated system fails, the blame collects on the nearest human being, and the system itself, with the organisation behind it, is spared. The human is the part designed, in effect, to take the impact.

Her cases come from aviation and nuclear power, including the partial meltdown at Three Mile Island and the loss of Air France flight 447. In each, operators had been moved from doing a task to supervising a machine that did it, were handed control back at the worst moment, and were then judged as though they had been in command throughout. “Human error” was the finding. The design that made the error likely was not on trial.

Why it happens

Automation tends to be introduced with a person left “in the loop” as a safeguard. But monitoring a system that almost never fails is a task people do badly, as the research on automation bias shows, and skills that are not practised decay. The supervisor therefore has responsibility without the means to discharge it. Law and public opinion still look for a person, and find the one who was present.

How it relates to the responsibility gap

The responsibility gap is the worry that for some harms done by learning systems no one can fairly be held responsible. The moral crumple zone is what often happens in practice instead: someone is held responsible, but not fairly. The gap is filled with the wrong person.

Where I stand

I take Elish’s point to be a warning about a popular fix. “Keep a human in the loop” sounds like accountability, and a named person on the form does satisfy the wish to know who answers. But answerability should follow control and knowledge. Where an institution arranges matters so that a clerk, a driver or a moderator signs off on what they cannot meaningfully check, the signature protects the institution, not the public. The honest test of a human-in-the-loop design is whether the human could have done otherwise, and would have known to.