Frequency reliability

Frequency reliability is reliability understood as a high rate of true outputs over a fixed distribution of cases, whatever explains that rate. It is what a benchmark measures. I distinguish it from tracking reliability, the counterfactual sensitivity of a source’s production to the truth, and argue that testimony requires the second.

The two come apart in a familiar case. A broken thermometer in a room held at constant temperature by a thermostat is right every time it is read. Its accuracy is stable, grounded in the thermostat, and explicable, not a fluke. Yet nothing about the temperature explains what it displays, and no one thinks that posting its readings transmits knowledge. Grounded frequency reliability is non-accidental relative to the mechanism that produces it while remaining accidental relative to the truth of the particular content reported.

That does not make a track record worthless. It can enter a user’s epistemic position, but it enters as inductive evidence about the source, and warrant so structured is the warrant of instrument reading: establish a record under conditions of use and project it to the case at hand.

Language models are often highly frequency-reliable, and successive systems have become more so. But the rate anyone can certify is measured under validation conditions, and it projects to a particular output only on the assumption that the case is typical of what was sampled, an assumption the system neither polices nor signals the failure of. A higher rate does not change what the rate is a rate of. Where frequency and truth diverge, a merely frequency-reliable practice generates true belief, not knowledge, and false belief with the same fluency.

Process reliabilism, a frequency notion attaching to types, meets the generality problem here. Type the thermometer’s process narrowly enough, as reading-this-thermometer-in-this-room, and it comes out reliable. Something must constrain the typing, and the constraint Goldberg accepts, that the reliability belong to the source’s own epistemic operation, is the one I call endogeneity.