Should you trust what ChatGPT tells you?

‘Trust’ covers two different relations, and the answer depends on which one is meant. Faulkner distinguishes predictive trust, the expectation that something will behave as it has, from affective trust, the reliance we place in a person who invites it and becomes answerable for what she says. You can have the first towards a thermometer, a bridge or a chatbot. The second is what makes testimony work, and a chatbot is not the kind of thing towards which it is warranted.

What you are actually doing when you rely on it

When a friend who is a doctor tells you a dose, you can take her word for it: the warrant for your belief runs through her, and she is answerable for it. When a chatbot gives you the same dose, nothing runs through it in that way. If you come to know the dose, it is because you have reason to think the system is reliable on questions like this, in conditions like yours, and nothing in the answer gives you reason to doubt it. The work is yours. That is what I mean by saying that relying on a language model is reading an instrument, not receiving testimony.

How to read the instrument well

Four habits follow from the structure rather than from any particular system’s error rate.

1. Check in proportion to the stakes. A wrong film date costs little; a wrong dose, a wrong legal deadline or a wrong figure in a submission costs a great deal. The higher the stakes, the more the answer should be treated as a lead rather than a result.
2. Check against something independent. A follow-up question to the same system is not a second opinion. A source that does not depend on the system is: the label on the box, the statute itself, the paper the claim came from, a calculation you run.
3. Ask whether your question is typical. A system’s record was made on some distribution of questions. The further yours is from the ordinary (rare names, recent events, local facts, a precise number), the less that record tells you about this answer.
4. Prefer answers that can verify themselves. Code that runs, arithmetic you can redo and quotations you can open shift the burden from trust to checking, which is where it belongs.

Why fluency is not evidence

Human speakers who are unsure tend to sound unsure. A language model’s expressions of confidence are produced by the same process as its answers, so they are not a separate report on how reliable the answer is. A system that is right most of the time will generate false answers with the same fluency as true ones, because nothing in the production of an open-domain answer is keyed to the truth of that particular answer. Its fluency shows that it is working and is silent on whether it is right.