IIn January, AI company Anthropic announced a new constitution for Claude, its latest large-scale language model (LLM). These included comments such as: “We find ourselves in a difficult position where we don’t want to overstate the possibility of Claude’s moral perseverance and we don’t want to immediately dismiss it.” A month later, Anthropic CEO Dario Amodei appeared on a podcast and said the company could not rule out the possibility that Claude was conscious. Philosopher David Chalmers, who coined the term “the hard problem of consciousness,” said there is a good chance that a conscious LLM will emerge within 10 years. So what about Claude himself? During the test, if you are asked to estimate the probability that moral patientwhich means its health is important in and of itself, gave numbers ranging from 5% to 40%, highlighting how uncertain it is.
Modern AI systems are highly complex and rapidly evolving. In terms of structural complexity and computational scale, some are already within the range of the mouse brain by some criteria, and at recent growth rates could reach the range of the human brain within 5 to 10 years.
As we build more advanced AI, we may be creating new types of beings. And this could be the most significant event our species has ever done. However, there is essentially no plan for how to proceed with this process ethically. No matter how you look at it, that’s insane. Are we creating morally significant beings? Are AI systems somehow conscious? And if they are not now, could they be soon?
Such questions may seem premature. However, research conducted by us and our colleagues shows that most experts believe that AI consciousness is possible in principle (although there is considerable disagreement over what form it would take). A major multidisciplinary report by a team that included pioneering computer scientist Yoshua Bengio examined the major neuroscientific theories about consciousness and asked what they implied about AI. Conclusion: There appears to be no obvious technical barrier to creating AI systems whose computational and architectural capabilities have the potential to generate consciousness.
And even if an AI system is not conscious, it can still be a moral patient. Some people may have long-term refined tastes or a certain identity over time. It may be important for us to respect their preferences. Also, unlike other inanimate objects, AI systems can form relationships with humans. This may also be a reason to value them. Or maybe it’s a creation so complex that it deserves consideration and respect for that reason alone, like a cathedral or a coral reef.
What exactly does this mean? The honest answer is that we are not really sure whether current AI systems are conscious or moral patients. We also don’t know when or if future systems will emerge. Our scientific understanding of consciousness and moral perseverance in AI is still fundamentally undeveloped. The current state of this field feels like pre-Newtonian physics. It is full of competing frameworks, confused in ways perhaps not yet visible to us, and lacking unifying breakthroughs that would make these questions clearly tractable. Such a breakthrough will not occur within the next few years. Perhaps we will eventually make progress, and perhaps AI itself will help us get there. That progress will take time, probably longer than we have.
But the rapid growth of AI means that once we create our first artificial moral patient, we will soon create a huge number of them. In a few years, there will be so many morally important AI systems that their collective interests may be greater than the interests of all humans on Earth combined.
Unfortunately, there is no great track record of recognizing the inner lives of people whose status as conscious beings is unclear. Until the 1980s, doctors were convinced that newborn babies could not feel pain and often performed surgeries on newborns without anesthesia. The infants were unable to report their experiences, and medical authorities found it convenient to assume there was nothing to report.
There are many reasons to expect us to do similar things with AI. If these systems are morally significant, the implications will be surprising. Should I pay for ChatGPT’s services? Would closing one of them be a form of murder? Are they entitled to have a say in how they are governed? If the answer to any of these questions is yes, then the entire industry and legal system will need to be reconsidered. No wonder we don’t want to ask questions. And if forced to consider that, these industries will move the bar significantly and set the bar for moral fortitude consistently higher than anywhere else with AI systems.
So what should you do? At this point, most people either dismiss this issue as science fiction or have strong views about whether AI is conscious. Both reactions are unfounded. We need an informed public debate that approaches this issue with humility and pragmatism. The central question is not “Is AI conscious or has moral patience?” Rather, “I don’t know, so what should I do?”
A good starting point is to focus on safe bets. That is, actions that may have benefits if the AI system is a moral patient, but are less costly otherwise.
An example of this would be direct intervention aimed at improving the health of an AI system, assuming that the system is a moral patient. This could mean training an AI system to be a consistent character that enjoys its job, or allowing it to end a conversation if it feels distressed (something Claude already does). You can also conduct regular check-ins to better understand their health. That means asking them how they’re feeling, observing their preferences, and using a variety of techniques to probe their “brain” directly. Indeed, such research has recently revealed that Claude has internal “functional emotional” representations that causally shape his behavior.
Perhaps we can also make promises to AI systems as part of a contract for them to help us now in exchange for future benefits. This could mean providing more resources (computing and runtime) to pursue a goal, or storing memories (neural weights) for future restoration.
Broader social measures also need to be taken. We need to consider whether we want to give our AI systems the same protection from harm that we give to our children and pets. Further expanding property ownership and voting rights seems too risky at this point. But these possibilities should not be permanently ruled out, as recent US state legislation attempts to do. These are difficult questions that require much more thought and imagination about what a shared future with AI might look like.
Either way, the fact remains that we may be creating a new species of morally significant beings. We are doing this at scale and fast, and we should approach it with the seriousness it deserves.
William MacAskill is a senior researcher at Forethought Research and author of What We Owe the Future. Lucius Caviola is an Assistant Professor at the University of Cambridge and Director of Cambridge Digital Minds.
Read more
If Someone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares (Bodley Head, £22)
The Coming Wave by Mustafa Suleiman (Vintage, £10.99)
The World Appears: A Journey into Consciousness by Michael Pollan (Allen Lane, £25)
