Explainable AI (XAI)

A bank’s model refuses a loan. The applicant asks why. If the model is a deep network with millions of weights, nobody at the bank can answer by pointing to the step where the decision was made. Explainable AI is the name for the attempts to give an answer anyway.

There are two families of method. The first builds models that are interpretable from the start: short decision lists, sparse scoring systems, models small enough that a person can follow them. The second keeps the opaque model and adds an explanation afterwards, by showing which features moved the output most or by fitting a simple model that imitates the complex one near a single case. Feature-attribution tools such as LIME and SHAP belong to this second, post hoc family.

The two standing objections

Tim Miller’s survey argued that the field had been explaining to itself. Research on how people explain shows that a good explanation is contrastive (why this and not that), selective (a few causes, not all of them), and social (fitted to what the asker already believes). A ranked list of feature weights is none of these.

Cynthia Rudin’s objection goes deeper. A post hoc explanation is a second model, and it cannot be perfectly faithful to the first: if it were, it would be the first. Where it differs, the explanation is false, and nothing in it says where. For decisions about liberty, health and money she argues for interpretable models instead, and disputes that they must be less accurate.

Where I stand

I think the faithfulness problem is the central one, and it is the same problem that arises when a language model explains its own reasoning. An explanation is worth having only if what it says is connected to why the output was produced. A persuasive account with no such connection gives the feeling of understanding and removes the motive to check, which is worse than an honest “we cannot say”. Explanation is also not the only route to warranted reliance: a well-tested instrument can be relied on without being understood. But that is a claim about the user’s knowledge, not about what an institution owes the person it has refused.