Abhinav Saxena
Glossary

Meaningful human control

Also called: MHC, human in the loop, human oversight of AI, human on the loop, human control of autonomous systems

Definition

The requirement that people, and not automated systems, remain in control of and morally responsible for consequential decisions, in a sense stronger than merely having a person present who could in principle intervene.

By · Updated · 2 min read

The phrase entered public debate around 2013 in arguments about autonomous weapons, where campaigners held that decisions to use lethal force must remain under meaningful human control. It has since spread to self-driving cars, medical systems and public administration. The word doing the work is “meaningful”. Nearly every automated system has a human somewhere. The question is what kind of involvement counts.

The tracking and tracing account

The most developed philosophical answer is Filippo Santoni de Sio and Jeroen van den Hoven’s. Drawing on John Martin Fischer and Mark Ravizza’s theory of responsibility as guidance control, they give two conditions.

  • Tracking. The system’s behaviour must respond to the relevant moral reasons of the relevant people, and to the relevant facts of the situation. A system that would act the same whatever its designers and users had reason to want is not under their control, however many buttons they have.
  • Tracing. The system’s behaviour must be traceable to at least one person in the chain of design and use who understands what the system can do and understands that they are responsible for its effects.

On this account control does not require a hand on the switch at the moment of action. A well-designed system operating alone may be under meaningful control. A system with an operator who is present but cannot follow what it is doing is not.

Why the weaker notion fails

“Human in the loop” asks only for presence. Presence without understanding or time to act yields the moral crumple zone: a person who takes the blame for what they could not have prevented.

Where I stand

I think tracking and tracing are the right conditions, and I would add a third party to the picture. The account is written from the side of those who design and operate a system. For systems that decide about people, such as eligibility for a benefit, control is meaningful only if the person decided about can also reach a human being who is able to understand the decision and empowered to change it. Control that runs only upward, to the institution, secures responsibility on paper. Whether anyone is in fact answerable is tested when someone has been wronged and asks who will put it right.

Related terms

The responsibility gapThe worry, due to Andreas Matthias, that learning machines can cause outcomes for which no one can fittingly be held responsible, since no one could control or foresee them.Moral crumple zoneThe position of a human operator in a highly automated system who absorbs the moral and legal blame when the system fails, although they had little real control over what it did.ResponsibilityThe relation between an agent and an outcome in virtue of which the agent can rightly be praised, blamed or asked to answer for it.AutonomySelf-governance: acting on reasons one has reflectively endorsed, rather than merely on whatever desires one happens to have.AI alignmentThe problem of ensuring that an AI system pursues what its designers and users actually intend and value, rather than a flawed specification or proxy of it.Instrumental convergenceThe thesis that agents with very different final goals would tend to pursue the same intermediate goals, such as staying in operation and acquiring resources, because those are useful for almost any end.

Questions that turn on it

QuestionShould AI make decisions about people?

Referred to by

The problem of many handsRight to explanationPaperclip maximizerValue-sensitive design

Further reading

  1. Santoni de Sio, F. and van den Hoven, J. (2018). Meaningful human control over autonomous systems: A philosophical account. Frontiers in Robotics and AI 5: 15.doi.org/10.3389/frobt.2018.00015
  2. Fischer, J. M. and Ravizza, M. (1998). Responsibility and Control. Cambridge: Cambridge University Press.