Does ChatGPT lie, or is a 'hallucination' something else?

When a chatbot invents a court case or a citation, it is tempting to say that it lied. It is equally tempting, and more common in the industry, to call the error a hallucination, as if the system ordinarily perceived the facts and occasionally saw things that were not there. Both descriptions get the structure wrong, in opposite directions.

Why it is not lying

To lie is to assert what you believe to be false, usually in order to be believed. On the best current evidence, the production of a language model’s open-domain answer is not governed by anything that plays the role of a belief which the answer could contradict. Researchers have found internal structure that correlates with truth, and it can be used to steer outputs; whether it governs what the model says when nobody steers is not established. Without that, there is nothing for a false answer to be insincere about.

Why ‘hallucination’ misleads

The word implies that true answers come from a faculty working properly and false ones from a faculty misfiring, and that is not the picture. On the standard description, true and false answers come out of the same process: the selection of continuations that are likely given what the model learned, adjusted by training against human approval. Where likely text and true text coincide, which in well-covered domains is very often, the answer is right. Where they come apart, it is wrong. Nothing in the production of the particular answer distinguishes the two cases.

A ‘hallucination’ is therefore not a malfunction of a truth-tracking process but what an accuracy-without-tracking process does when its proxy for truth, the regularities of the text it learned from, fails to fit the case. That is why models reproduce common misconceptions that are widely written down, and why a request to explain an answer produces a fluent rationale that can be disconnected from what actually drove the answer.

What follows

The practical lesson is the one I draw in Should you trust what ChatGPT tells you?: fluency and confidence are not evidence, because they are outputs of the same process as the answer. The theoretical lesson is that the line that matters is not between honest and dishonest outputs, or between seeing and hallucinating, but between production that is governed by the truth of what is said and production that is not. That line is the Tracking Minimum, and current systems’ open-domain answers fall on the wrong side of it.