Chain-of-thought faithfulness

Chain-of-thought faithfulness is the degree to which the reasoning a language model displays reflects the process that actually produced its answer, rather than a plausible rationale produced alongside it. Prompting models to reason step by step improves their performance on many tasks, and newer systems generate extended reasoning traces before answering. Whether those traces can be read as the model’s reasons is a separate question. Jacovi and Goldberg separated faithfulness, accuracy to the process, from plausibility, persuasiveness to a human reader.

The evidence is mixed in an instructive way. Turpin and colleagues showed that answers can be driven by features of the prompt, such as a suggested answer or a reordering of options, that the displayed reasoning never mentions. Lanham and colleagues measured faithfulness directly, by truncating traces, inserting mistakes and paraphrasing, and found that how far the answer depends on the trace varies with model and task.

This matters to my argument because it marks the one place where Burge’s route to knowledge from a computational source could open. Burge allows that a recipient can come to know from such a source by thinking its reasoning through, incorporating it and appreciating its force from the inside. A post-hoc explanation, produced on request after the answer, cannot serve, since it is a further output of the same process rather than a record of the process that produced the answer. A trace generated before the answer is different: it is causally upstream of the answer, and its role can be measured. That is a real concession.

It does not yet reach what Burge requires. Measured faithfulness is partial and varies, and causal upstreamness is not contact with grounds. What is needed is that the recipient, by thinking the reasoning through, comes into contact with the grounds on which the assertion rests. A trace that shapes an answer without being governed by the evidential bearing of what it cites can be followed without thereby appreciating any justification. Faithfulness research is therefore among the most important evidence on whether these systems satisfy the Tracking Minimum, and my verdict should move with it.