ELIZA effect

In the mid-1960s Joseph Weizenbaum, at MIT, wrote a program called ELIZA. Its best-known script, DOCTOR, imitated a psychotherapist of the Rogerian school by turning the user’s statements back as questions: tell it you were unhappy and it asked why you were unhappy. It worked by matching keywords and rearranging the user’s own words. It had no model of the person or of anything else.

Weizenbaum was startled by what happened next. People who knew how the program worked nonetheless confided in it, and some asked to be left alone with it. He spent much of his later book, Computer Power and Human Reason, on what that showed about us rather than about the machine. The readiness to read a mind into conversational output has been called the ELIZA effect since.

Why it happens

Conversation is the main evidence we have ever had for other minds, so anything that produces conversation draws on habits built for people. The effect does not depend on being fooled about the mechanism. Weizenbaum’s users were not ignorant; the response was faster than their knowledge.

The effect now

ELIZA managed this with a short pattern-matching program. Present systems produce fluent, relevant and apparently considerate prose, so the pull is far stronger, and it is commercially cultivated when products are given names, personalities and the first person. Whether any current system has mental states is a separate and open question (can AI be conscious?). The ELIZA effect is the reason our impressions cannot settle it: we would have the impression either way.

For knowledge the consequence is practical. We extend to a conversational system the default trust we give to speakers, and a speaker’s word comes with the presumption that she knows what she is saying. My argument about AI testimony is that the presumption is not earned by the grammar of assertion, and should not be extended on its strength.