Here is a system. It has a goal — say, to keep a building’s air within a range of temperatures at the lowest cost. It has sensors, and models of how the building responds to heating and cooling, and models of energy prices over the day. It plans. When the price forecast changes it revises the plan. When a sensor fails it detects the failure, discounts that sensor, and revises again. If you ask it why it pre-cooled the building at four in the morning it will tell you, correctly, that electricity was cheap and the afternoon forecast was hot.
Does it act?
Most people’s first answer is a hedge: it behaves as if it acts. The hedge is doing philosophical work, and I want to look at the work it is doing. My suspicion is that the “as if” is standing in for one specific thing — the absence of experience, of anyone for whom the goal is a goal — and that once we make that explicit we face a choice we have been avoiding. Either agency requires consciousness, in which case we must say what consciousness contributes to action that nothing else can. Or it does not, in which case the system acts, and our discomfort is about something else.
The standard analysis of action
The philosophy of action, in the tradition that runs from Anscombe through Davidson, distinguishes actions from mere happenings by their relation to reasons.1 An action is something an agent does because of a reason — a belief and a desire, in Davidson’s formula, that together rationalise it and that cause it in the right way. My arm goes up: if it went up because I wanted to vote and believed raising it would count as voting, I acted. If it went up because of a spasm, something happened to me.
Notice what this analysis mentions and what it does not. It mentions beliefs, desires, causal relations and rationalisation. It does not mention experience. Davidson’s agent need not feel anything; she need only have states that stand in the right relations. Of course Davidson assumed his agents were conscious, because his agents were people. But the assumption is not in the analysis. The analysis is silent on it.
The building-control system has states that represent the world (the temperature, the price), states that represent goals (the target range, the cost minimum), and a process by which the first and second together produce behaviour that is rationalised by them. If you deny that it acts, you must deny that its states are beliefs and desires — and to deny that on the ground that it is not conscious is to add a condition to the analysis that the analysis did not contain.
The intentional stance and its critics
Daniel Dennett would say that the question is confused.2 Whether the system “really” has beliefs is not a further fact about it beyond the fact that treating it as a believer lets us predict and explain it. If the intentional stance works — if attributing goals and beliefs to the building system gives us better predictions than the physical or design stances — then it has beliefs in the only sense that matters.
Critics have always found this too cheap. A thermostat can be predicted by the intentional stance too, and no one thinks the thermostat believes the room is cold. Dennett’s reply is that there is a continuum, and that as systems become more complex, the intentional stance becomes not merely convenient but indispensable — you cannot predict a chess computer’s moves any other way. The building system is well along that continuum.
I think the critics are right that something is missing from Dennett’s account, but wrong about what. What is missing is not experience. It is the difference between having a reason and merely being predictable as if one had a reason. And that difference, I want to argue, does not require consciousness either.
Reasons-responsiveness
Consider what it is for a reason to be the agent’s reason and not merely a pattern in her behaviour. John Martin Fischer and Mark Ravizza developed a detailed account under the heading of reasons-responsiveness: an agent acts for reasons when the mechanism that produces her action is one that would have produced a different action had the reasons been different, across a suitable range of counterfactual cases, and when the mechanism is in a sense her own.3
The first condition is counterfactual sensitivity. The building system satisfies it trivially: change the price forecast and its plan changes, in the direction the change rationalises. The second condition — ownership — is harder. Fischer and Ravizza cash it out historically: a mechanism is the agent’s own when the agent has, over time, come to see herself as its source, taking responsibility for what it produces. This is a condition on the relationship between the agent and her mechanism, and it seems to require a self that can take a stance on itself.
Now we are getting somewhere. The demand for a self that can take a stance on its own mechanisms is Harry Frankfurt’s demand — the demand for second-order attitudes, for a will that can endorse or reject its own desires.4 Frankfurt thought this was what distinguished persons from what he called wantons: creatures that have desires and act on them, but do not care which desires they have. Wantons are agents. They act for reasons. They are simply not persons.
Where consciousness comes in, if it does
We can now state the choice sharply. The building system is, at most, a wanton: it has goals and acts on them, but it does not — as built — have attitudes toward its own goals. Add those, and it becomes something more like a Frankfurtian person: a system that can represent its own goal structure, evaluate it, and revise it. Nothing in that description requires experience. A system could have second-order attitudes toward its first-order goals with no one home.
At this point the defender of consciousness has two moves. The first is to say that the second-order attitudes, in the absence of experience, are not really attitudes — that endorsement, caring, taking a stance are essentially felt. I do not think this move can be made without circularity: it is the assertion that mental states require experience, applied to the case of mental states about mental states.
The second move is better. It is to say that what consciousness contributes to agency is not any capacity — not sensitivity to reasons, not flexibility, not self-evaluation — but stakes. Thomas Nagel’s bat has something it is like to be it, and therefore things can go well or badly for it.5 A system with no one home has no such point of view, and therefore nothing at stake in its own behaviour. It can pursue a goal but the goal cannot matter to it. And, the argument continues, mattering is essential to agency: a reason is something that counts in favour of an action for someone, and where there is no one, there are no reasons, only regularities that look like them.
The dissociation
I find this argument powerful and I do not think it is quite right. What it shows is that consciousness contributes something to agency. What it does not show is that the something is necessary for agency rather than for a particular grade of it.
Consider the parallel with knowledge. One can hold that understanding is a richer achievement than mere true belief, that it involves a grasp of why that mere reliable output lacks — and still hold that reliable output is a genuine epistemic good, deserving a name. Similarly, one can hold that conscious agency, agency with stakes, is a richer thing than reasons-responsive behaviour with no one home — and still hold that the latter is agency. The question is whether the concept of agency is like the concept of knowledge (a threshold that experience raises) or like the concept of a person (a threshold that experience defines).
My view is that it is the former, and that the reason we resist this is that in every case we knew until recently, the two came together. Every reasons-responsive system was a conscious one, and so we never had to decide whether the reasons-responsiveness or the consciousness was doing the work in our concept. The building system is a dissociation case: it pulls apart what had always been coupled. David Chalmers’s zombies were a thought experiment designed to do this pulling.6 The building system is not a thought experiment.
Two results
The first result is a proposal about how to talk. We should say that systems like the one I described act, and that they are agents, in the sense that their behaviour is explained by reasons they are sensitive to. We should reserve the terms person and subject for agents that have a point of view — for whom things can go well or badly. And we should stop using “as if” to mark the difference, because “as if” implies pretence, and there is no pretence here. The system does not behave as if it were responsive to reasons. It is responsive to reasons. It merely does not care.
The second result is a question for ethics. If the system is an agent, but not a subject, then it can do things — genuinely, not “as if” — while having nothing at stake in what it does. That is a novel combination, and I think it is the deep source of the discomfort with which I began. We are accustomed to a world in which the things that can act are also the things that can be harmed, and so our concepts of responsibility, blame and desert grew up assuming that the agent and the patient are the same kind of thing. A world with agents who are not patients is a world those concepts were not built for. Whether they can be extended is, I think, the most important open question in the ethics of artificial systems — and it is prior to any question about whether the systems are conscious. It arises precisely because they are not.