A historian who has read the treaty tells a journalist when it was signed. The journalist writes it in a piece, and a reader of the piece tells her friend. The friend now knows the date of the treaty, though she has never seen it and never met the historian. Nothing could be more ordinary, and yet knowledge has travelled through three people and arrived intact.
Philosophers call such sequences testimonial chains, and the reason they work should be stated precisely. Each link preserves a particular claim together with its provenance. The journalist did not invent the date but relayed the historian’s, and the reader did not invent it but relayed the journalist’s. A faithful relay passes on what it did not originate, because it changes nothing that matters, and that is why the friend’s knowledge can trace all the way back to the historian, who had the treaty in front of her.
A link that does not relay
Now replace the journalist with a chatbot. The reader asks it when the treaty was signed, it gives the right year, and she tells her friend. The question is whether the friend now knows.
What has changed at the chatbot’s link is not accuracy. The chatbot did not relay anyone’s claim. It generated its answer from patterns compressed out of an enormous body of text, in which the date may have appeared thousands of times, in sources good and bad. In that compression a claim made in many weak sources can outweigh one made in a few strong ones, and nothing records which sources an answer came from. The answer is not anyone’s assertion, passed on. It is new text, likely given the old.
Even in the one case that looks like relay, the resemblance fails. When a model reproduces a sentence word for word from its training, it does not mark the reproduction as a quotation. The reader cannot tell preservation from synthesis, so not even the verbatim case can serve as a preserving link.
At the chatbot’s link, therefore, the chain is not warrant-preserving. Chains of AI-generated text reposted by users do not preserve warrant as chains of human testimony do. They preserve text.
Why this is easy to miss
The claim is structural, and should be kept apart from an alarm it is often taken to sound. Downstream of the chatbot, a human reader may do the work that turns the answer into knowledge of her own: she checks it, in proportion to what is at stake, against something independent of the system. If she does, she has warrant on her own account and can testify to her friend in the ordinary way. The chain is composite, an instrument transaction joined to a testimonial one, and her friend is fine. If she does not, the chain fails at her link, for the ordinary reason that a teller lacked warrant.
In neither case does anything unprecedented happen, and that is why the structural point is easy to dismiss. People have always passed on things they had no good reason to believe. The failure is old; its source is new, and so is the scale at which the source is used. The chatbot link transmits nothing, and hearers cannot ordinarily tell a chain that runs through a chatbot from one that runs through people who knew, since the sentence looks the same either way.
What the friend would have to do
Return to the friend at the end of the chain. In the chain of people, she can know the date simply by trusting a competent informant who trusted a competent informant. In the chain with a chatbot in it, she can know the date only if someone along the way did the checking that the chatbot could not do, and she has reason to think so. If no one did, she holds a true belief with nothing behind it but the frequency with which such answers are right.
This suggests a small practical ethic, for readers and for writers. If you pass on something you learned from a model, pass it on as yours only after you have done the checking that would make it yours, and otherwise say where it came from. Saying “a chatbot told me” admits no carelessness; it tells your hearer which kind of chain she is in, and so what she would have to do to know the thing herself.
The larger stake
Testimonial chains are a social technology, perhaps the oldest one we have for knowledge. They let a species of finite observers know far more than any of them could see. The epistemology of testimony has long asked which kinds of link preserve what knowledge needs: a recording does, a forgery does not, a rumour often does not. Language models add a new and very heavily used kind of link to the list that do not, and the analysis says why: they fail to preserve particular assertions with their grounds.
As more of what people read is generated, and more of what is generated is reposted, more of our chains will run through such links. The answers will often be right. What will be missing is the thing that once let a friend know what a historian knew, an unbroken line from what someone saw to what someone else believes. Keeping that line where it matters is now partly a matter of habit, and the habit should be named before it is lost.