Abhinav Saxena
Glossary

Algorithmic bias

Also called: AI bias, machine bias, bias in machine learning, algorithmic discrimination, algorithmic fairness

Definition

Systematic and unfair difference in how a computer system treats individuals or groups, arising from the data it learned from, the way the problem was specified, or the setting in which it is used.

By · Updated · 2 min read

A hospital system uses a score to decide which patients get extra care. The score predicts future health costs, on the reasonable-sounding assumption that sicker people cost more. But less had been spent on Black patients than on equally sick white patients, so at the same score Black patients were sicker. Obermeyer and colleagues, who found this in a widely used commercial tool, estimated that correcting it would raise the share of Black patients flagged for extra help from 17.7 to 46.5 per cent. No one had written race into the system.

That is the pattern the term names: a difference in treatment that is systematic, not random, and unfair, not merely unequal.

Where it comes from

Batya Friedman and Helen Nissenbaum, writing before modern machine learning, sorted the sources into three, and the sorting still holds.

  • Pre-existing bias is carried in from society: data that record past discrimination teach a system to continue it.
  • Technical bias comes from design choices: what is measured, what stands in for what cannot be measured, what is optimised. The hospital score is this kind; cost was a proxy for need.
  • Emergent bias appears in use, when a system meets a population or a purpose it was not built for.

Why “remove the bias” is not simple

Several reasonable statistical definitions of fairness cannot all be met at once when groups differ in base rates, so choosing one is a moral decision made in mathematical dress. And deleting the protected attribute does not help when other variables encode it.

Where I stand

I think the usual framing, a neutral system plus a flaw, understates the case. Where a system decides who counts as eligible, creditworthy or at risk, a biased classification is not only an unequal distribution of some good. It is a failure to see a person as what they are, which is why I treat it alongside misrecognition and epistemic injustice. The remedy then includes more than better metrics: the person classified must be able to find out how they were classified and to contest it.

Related terms

Epistemic InjusticeA wrong done to someone specifically in their capacity as a knower, as when a speaker’s word is given less credibility than it deserves because of who she is.MisrecognitionFailure to accord someone the standing, regard or status due to them, treated in recognition theory as a distinct moral and political wrong.Algorithmic welfareThe administration of welfare entitlements through systems that combine automated decisioning, criterion-enforcement without discretion at the point of application, and integration across databases.Epistemic opacityThe condition of a process whose epistemically relevant elements an agent cannot all know, as with a complex simulation or a trained neural network.Legibility (of the state)James C. Scott’s term for a state’s capacity to make its population and territory readable for administration, through standard names, maps, registers and measures.Value-sensitive designAn approach to designing technology that accounts for human values in a principled and systematic way throughout the design process, treating them as requirements alongside function and cost.Right to explanationThe claim that a person subject to an automated decision is entitled to be told why it was made, and the legal provisions, chiefly in European Union law, that are said to grant or to fall short of granting that entitlement.

Questions that turn on it

QuestionIs AI biased, and can the bias be removed?

Further reading

  1. Friedman, B. and Nissenbaum, H. (1996). Bias in computer systems. ACM Transactions on Information Systems 14(3): 330–347.doi.org/10.1145/230538.230561
  2. Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science 366(6464): 447–453.doi.org/10.1126/science.aax2342