Should AI make decisions about people?
Only under conditions that are usually not met. For decisions that determine a person’s access to welfare, credit, work or liberty, an automated system is acceptable only if the person can see the criteria applied to them, is not left with nothing when the system fails, and can reach someone with the power to correct it. Where those conditions cannot be met, the decision should not be automated, however accurate the system.
Automated systems already decide, or shape the decision, on who receives welfare payments, who is offered credit, whose job application is read and whose tax return is examined. The question is no longer whether this will happen but what would make it acceptable.
What has gone wrong
Three cases recur in the literature because they share a pattern.
In Australia, an automated scheme known as Robodebt raised debts against welfare recipients by averaging their annual income across fortnights. The method was unlawful, hundreds of thousands of people were pursued for money many did not owe, and a royal commission later condemned the scheme.
In the Netherlands, a system for detecting fraud in childcare benefits wrongly labelled thousands of families as fraudsters and demanded repayments that ruined many of them. The government resigned over it in 2021.
In India, linking food rations to biometric authentication under Aadhaar meant that when a fingerprint failed to match or a connection dropped, people entitled to grain went without it, as Jean Drèze and colleagues documented in Jharkhand.
In none of these was the main failure a subtle statistical one. People could not find out why the system had decided as it did, had no working route to challenge it, and bore the whole cost of its errors.
Why accuracy is not enough
A system that is right ninety-nine times in a hundred is wrong about a great many people when it processes millions. Those people are not consoled by the average. They were owed a correct decision, or failing that a way to get one, and the question of what the institution owes them does not go away because the error rate is low.
Why consent is not enough
The usual ethical test for a technology is whether the people affected agreed to it, and that test assumes they could have refused. For a welfare system, refusing is not an option: the system is how one comes to count as entitled at all. I call this constitutive dependence. Where it holds, a signature on a consent form shows only that the person needed what was on the other side.
Where I stand
I argue that for systems people depend on in this way, legitimacy rests on three obligations that the institution cannot ask anyone to waive.
- Legibility. The criteria by which the system recognises a person must fit their real circumstances and be visible to them. A fingerprint reader that cannot read the worn hands of a labourer fails this before any decision is made.
- Non-abandonment. A failure of the system must not leave the person without the thing they are entitled to. There must be another way through.
- Remediation. When the system is wrong, someone must have the power and the duty to put it right, promptly, without the burden of proof resting on the person harmed.
These are tests of the institution rather than of the algorithm: a simple rule-based system can fail all three, and a sophisticated one could pass them. I think the debate should accordingly move away from whether machines may decide and towards whether the body deploying them remains answerable to the people decided about.
Questions people also ask
Is it legal for AI to make decisions about people?
Whether it is legal depends on the place and the decision. European Union data protection law gives a right not to be subject to solely automated decisions with legal or similarly significant effects, with exceptions, and the EU’s AI Act places duties on systems used in areas such as employment, credit, welfare and policing. Many other jurisdictions have no general rule.
Are AI decisions more accurate than human ones?
For some narrow prediction tasks, statistical methods have long matched or beaten human judgement. But accuracy on average is not the whole question. The errors fall on particular people, a system repeats its mistake on everyone like them, and a person has standing to object to how they were treated even when the average is good.
Does having a human in the loop solve the problem?
A human in the loop solves it only if the human can understand the system’s recommendation, has time and authority to depart from it, and in practice does. A person who approves what they cannot check adds a signature and absorbs the blame without adding a safeguard.
Can you refuse to have a decision made about you by AI?
Where the law gives such a right you can invoke it. In many settings, though, refusing means going without the benefit, the loan or the job, which is why I argue that consent is the wrong test for systems people depend on.
Read next
Sources and further reading
- Saxena, A. (2026). Beyond consent: Algorithmic welfare, constitutive dependence, and the limits of liberal AI ethics. AI and Ethics 6: 414.doi.org/10.1007/s43681-026-01270-5
- Eubanks, V. (2018). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. New York: St. Martin’s Press.
- Santoni de Sio, F. and van den Hoven, J. (2018). Meaningful human control over autonomous systems: A philosophical account. Frontiers in Robotics and AI 5: 15.doi.org/10.3389/frobt.2018.00015
- Wachter, S., Mittelstadt, B. and Floridi, L. (2017). Why a right to explanation of automated decision-making does not exist in the General Data Protection Regulation. International Data Privacy Law 7(2): 76–99.doi.org/10.1093/idpl/ipx005
- Drèze, J., Khalid, N., Khera, R. and Somanchi, A. (2017). Aadhaar and food security in Jharkhand: Pain without gain? Economic and Political Weekly 52(50): 50–59.
Cite this page
Saxena, A. (2026). Should AI make decisions about people? abhinavsaxena.in. https://abhinavsaxena.in/questions/should-ai-make-decisions-about-people/