What happens to knowledge when people pass on AI answers?

Testimonial chains are one of the oldest social technologies for knowledge. A historian tells a journalist, the journalist tells her readers, a reader tells a friend, and the friend can end up knowing what the historian knew. The chain works because each link preserves a particular claim together with its provenance. A recording or a faithful relay can pass on what it did not originate, because it changes nothing that matters.

Where an AI link breaks the chain

When one of the links is a language model, the chain is not warrant-preserving at that point. The model does not relay a particular assertion with its grounds. It generates new text from patterns compressed out of an enormous body of writing, in which a claim repeated in many weak sources can outweigh one made in a few strong ones, and in which nothing attaches to a given output the address of the ground it came from. Even when a model reproduces a sentence word for word, it does not mark the reproduction as a quotation, so the reader cannot tell preservation from synthesis.

Chains of AI-generated text reposted by users therefore do not preserve warrant as chains of human testimony do. They preserve text.

A structural claim, not an alarm

The claim should be kept apart from a sociological claim people often hear in it. Downstream of the AI link, a human reader may do the work of reading the instrument well: checking the answer, in proportion to the stakes, against something independent. If she does, she acquires warrant of her own and can testify on her own account, and her hearers are fine. If she does not, the chain fails at her link, for the ordinary reason that a teller lacked warrant.

Nothing unprecedented happens in either case, and the temptation is to conclude that nothing has been shown. What has been shown is what the AI link itself transmits, which is nothing. That is a fact about the structure of the chain, not about how often the structure occurs. It matters practically because hearers cannot ordinarily tell a chain that runs through a model from one that runs through people who knew.

What follows for readers and writers

If you pass on something you learned from a model, the honest course is to pass it on as yours only after you have done the checking that would make it yours, and otherwise to say where it came from. Saying ‘an AI told me’ is then no confession of laziness: it tells your hearer which kind of chain she is in, and so what she would have to do to know it herself.