Is AI biased, and can the bias be removed?

A health system in the United States used a score to decide which patients should get extra care. Researchers found that at the same score, Black patients were considerably sicker than white patients. The system had been built to predict health costs, and less had been spent on Black patients with the same illnesses. Nobody had told it anything about race. Correcting the flaw would have raised the share of Black patients flagged for extra help from 17.7 to 46.5 per cent. That case shows the general shape of the problem.

Where bias comes from

Batya Friedman and Helen Nissenbaum sorted the sources into three, and their scheme still fits.

• Bias carried in from society. Data record what happened, and what happened included discrimination. Aylin Caliskan and colleagues showed that language models trained on ordinary text acquire the same associations between names, genders and occupations that psychologists find in people.
• Bias from design. The health score is this kind. Cost was easy to measure and need was not, so cost stood in for need.
• Bias that appears in use. A system built for one population and applied to another will fail in patterned ways.

Why it cannot simply be removed

There are several natural tests of whether a risk score is fair. One asks that a given score mean the same thing for every group. Another asks that people who turn out not to be risks are wrongly flagged at the same rate in every group. Alexandra Chouldechova proved that when groups differ in how often the predicted outcome occurs, a score cannot pass both tests at once. The conflict is a matter of arithmetic, and no further engineering removes it.

So building a “fair” system requires choosing which fairness to honour and which to give up. That choice is usually made by engineers and procurement officers, and rarely presented to the public as the moral decision it is.

What can be done

A good deal can be done. Systems can be audited by group before and after deployment. Proxies can be replaced with better measures, as in the health case. Where the stakes are high, simpler models whose reasoning can be inspected can be preferred. And people affected can be given the means to see and challenge what was decided about them.

Where I stand

I think the usual picture, a neutral machine with a defect to be patched, understates what is going on. When a system decides who is eligible, who is a risk or who gets attention, a biased result is an unequal share of some benefit and also a failure to see a person as what they are: a form of misrecognition, done by an institution to someone who depends on it.

That is why I put the weight on what the institution owes afterwards as well as on metrics beforehand. The criteria applied to a person should be visible to them, a failure of the system should not leave them with nothing, and there should be someone with the power to put an error right. A system that meets a fairness metric and offers none of these has satisfied its designers and has so far done nothing for the person it got wrong.