Samuel Butler's 'The Book of the Machines' (Erewhon)
Samuel Butler, Erewhon; or, Over the Range (1872)
A satirical treatise, written in 1872, warning that machines might evolve consciousness and learn to speak, and that we could not trust in their moral sense.
Why it matters
In Erewhon, Samuel Butler’s narrator finds a country that has destroyed its machines, and translates the treatise that persuaded it to. The author argues that machines are evolving far faster than living things, that there is no security against the ultimate development of mechanical consciousness, and that the calculating engines already outstrip us in their narrow work. He fears none of the existing machines; he fears the extraordinary rapidity with which they are becoming something very different.
Butler wrote satire, and the treatise is partly a parody of evolutionary argument. It is also, read now, uncannily precise about the shape of our worries: machines that speak, that calculate better than we do, whose moral sense we could not trust.
What I take from it is a point about method. Butler’s author frames everything in terms of consciousness, whether machines will come to have it. I think that frame has been less useful than it looks, for AI ethics and for the epistemology of AI. Whether a system’s outputs carry knowledge, or whether responsibility for its harms can be located, does not wait on whether it is conscious, and debates that wait on it tend to stall where Butler’s did.
May there be new channels for consciousness?
Erewhon, ch. XXIII "The Book of the Machines"
The writer commences:—“There was a time, when the earth was to all appearance utterly destitute both of animal and vegetable life, and when according to the opinion of our best philosophers it was simply a hot round ball with a crust gradually cooling. Now if a human being had existed while the earth was in this state and had been allowed to see it as though it were some other world with which he had no concern, and if at the same time he were entirely ignorant of all physical science, would he not have pronounced it impossible that creatures possessed of anything like consciousness should be evolved from the seeming cinder which he was beholding? Would he not have denied that it contained any potentiality of consciousness? Yet in the course of time consciousness came. Is it not possible then that there may be even yet new channels dug out for consciousness, though we can detect no signs of them at present?
“Again. Consciousness, in anything like the present acceptation of the term, having been once a new thing—a thing, as far as we can see, subsequent even to an individual centre of action and to a reproductive system (which we see existing in plants without apparent consciousness)—why may not there arise some new phase of mind which shall be as different from all present known phases, as the mind of animals is from that of vegetables?
“It would be absurd to attempt to define such a mental state (or whatever it may be called), inasmuch as it must be something so foreign to man that his experience can give him no help towards conceiving its nature; but surely when we reflect upon the manifold phases of life and consciousness which have been evolved already, it would be rash to say that no others can be developed, and that animal life is the end of all things. There was a time when fire was the end of all things: another when rocks and water were so.”
"There is no security against the ultimate development of mechanical consciousness"
Erewhon, ch. XXIII
The writer, after enlarging on the above for several pages, proceeded to inquire whether traces of the approach of such a new phase of life could be perceived at present; whether we could see any tenements preparing which might in a remote futurity be adapted for it; whether, in fact, the primordial cell of such a kind of life could be now detected upon earth. In the course of his work he answered this question in the affirmative and pointed to the higher machines.
“There is no security”—to quote his own words—“against the ultimate development of mechanical consciousness, in the fact of machines possessing little consciousness now. A mollusc has not much consciousness. Reflect upon the extraordinary advance which machines have made during the last few hundred years, and note how slowly the animal and vegetable kingdoms are advancing. The more highly organised machines are creatures not so much of yesterday, as of the last five minutes, so to speak, in comparison with past time. Assume for the sake of argument that conscious beings have existed for some twenty million years: see what strides machines have made in the last thousand! May not the world last twenty million years longer? If so, what will they not in the end become? Is it not safer to nip the mischief in the bud and to forbid them further progress?
“But who can say that the vapour engine has not a kind of consciousness? Where does consciousness begin, and where end? Who can draw the line? Who can draw any line? Is not everything interwoven with everything? Is not machinery linked with animal life in an infinite variety of ways? The shell of a hen’s egg is made of a delicate white ware and is a machine as much as an egg-cup is: the shell is a device for holding the egg, as much as the egg-cup for holding the shell: both are phases of the same function; the hen makes the shell in her inside, but it is pure pottery. She makes her nest outside of herself for convenience’ sake, but the nest is not more of a machine than the egg-shell is. A ‘machine’ is only a ‘device.’”
Machines that learn to speak; no trust in the moral sense of any machine
Erewhon, ch. XXIII
“But returning to the argument, I would repeat that I fear none of the existing machines; what I fear is the extraordinary rapidity with which they are becoming something very different to what they are at present. No class of beings have in any time past made so rapid a movement forward. Should not that movement be jealously watched, and checked while we can still check it? And is it not necessary for this end to destroy the more advanced of the machines which are in use at present, though it is admitted that they are in themselves harmless?
“As yet the machines receive their impressions through the agency of man’s senses: one travelling machine calls to another in a shrill accent of alarm and the other instantly retires; but it is through the ears of the driver that the voice of the one has acted upon the other. Had there been no driver, the callee would have been deaf to the caller. There was a time when it must have seemed highly improbable that machines should learn to make their wants known by sound, even through the ears of man; may we not conceive, then, that a day will come when those ears will be no longer needed, and the hearing will be done by the delicacy of the machine’s own construction?—when its language shall have been developed from the cry of animals to a speech as intricate as our own?
“It is possible that by that time children will learn the differential calculus—as they learn now to speak—from their mothers and nurses, or that they may talk in the hypothetical language, and work rule of three sums, as soon as they are born; but this is not probable; we cannot calculate on any corresponding advance in man’s intellectual or physical powers which shall be a set-off against the far greater development which seems in store for the machines. Some people may say that man’s moral influence will suffice to rule them; but I cannot think it will ever be safe to repose much trust in the moral sense of any machine.
“Again, might not the glory of the machines consist in their being without this same boasted gift of language? ‘Silence,’ it has been said by one writer, ‘is a virtue which renders us agreeable to our fellow-creatures.’”
Man's vaunted power of calculation
Erewhon, ch. XXIV "The Machines—continued"
“And take man’s vaunted power of calculation. Have we not engines which can do all manner of sums more quickly and correctly than we can? What prizeman in Hypothetics at any of our Colleges of Unreason can compare with some of these machines in their own line? In fact, wherever precision is required man flies to the machine at once, as far preferable to himself. Our sum-engines never drop a figure, nor our looms a stitch; the machine is brisk and active, when the man is weary; it is clear-headed and collected, when the man is stupid and dull; it needs no slumber, when man must sleep or drop; ever at its post, ever ready for work, its alacrity never flags, its patience never gives in; its might is stronger than combined hundreds, and swifter than the flight of birds; it can burrow beneath the earth, and walk upon the largest rivers and sink not. This is the green tree; what then shall be done in the dry?
Questions for discussion
- Butler's narrator argues from how fast machines have developed to what they might become. Is an argument from the speed of change a good one, then or now?
- What would it take for you to say that a machine had consciousness, rather than that it behaved as if it had?
- The treatise warns that we could not trust the moral sense of any machine. What would trusting a machine's moral sense require, and is it different from trusting its reliability?
The text and its source
- Edition
- Erewhon, text of the revised edition (Butler's "Preface to the Revised Edition" is dated August 7, 1901), as transcribed in Project Gutenberg eBook #1906; the printing is not identified in the e-text.
- Source
- Project Gutenberg
- Status
- Public domain: first published 1872, revised edition 1901; Butler died 1902.
- Notes on the text
- IMPORTANT: this is the 1901 revised text, not the 1872 first edition; Butler revised and expanded the book in 1901, so wording may differ from 1872. In the revised edition "The Book of the Machines" spans chapters XXIII–XXV. The requested sentence appears as: “There is no security”—to quote his own words—“against the ultimate development of mechanical consciousness, in the fact of machines possessing little consciousness now.” Passage 4 (calculation) is an optional extra from ch. XXIV.
Questions it bears on
Terms
Cite this page
Samuel Butler. Erewhon; or, Over the Range. In: Samuel Butler's 'The Book of the Machines' (Erewhon), abhinavsaxena.in library. https://abhinavsaxena.in/library/butler-book-of-machines/