Imagine a very sophisticated android – let’s call it Droid

Imagine a very sophisticated android (let’s call it Droid) that has achieved self-consciousness. Droid has far greater cognitive capacities than any human being-it has more memory, greater processing speed and perfect logic. Droid can think about itself; it has goals and can make plans to achieve them. Droid also has a morality program that constrains its behavior in accordance with certain rules, such as “do not kill innocent sentient beings without indisputable justification.”

What Droid does not have is emotions. Droid doesn’t feel guilty or ashamed, angry or impatient, joyful or sad. Is Droid a moral being? Now imagine that Droid is in a situation in which a madman has taken an innocent hostage and threatened to shut down all internet shopping if Droid does not kill the hostage.

The importance of internet shopping to human beings activates Droid’s “justification” routine, and it asks itself whether this counts as a justification for killing an innocent person. Droid does think and reflect on its actions. Its moral programming isn’t automatic; rather, in the case of the rule against killing innocents without justification, it has to think about whether there is a justification for killing (as there might be if this were a case in which killing an innocent is the only way to save millions of lives).

But when Droid engages in this thinking, it does not feel sympathy with the person who might die, and it does not feel anger at the madman. Droid cannot feel guilty if it makes the wrong choice. Is Droid capable of morally evaluating its plans? Can Droid really make a moral judgement? Can Droid act morally?