The Mythology Of Conscious AI - Noema Magazine

Anil Seth won the 2025 annual Berggruen Prize Essay Competition for his English-language essay, "The Mythology Of Conscious AI." A neuroscience professor and the director of the Centre for Consciousness Science at the University of Sussex, Seth is also the co-director of the Canadian Institute for Advanced Research Program on Brain, Mind and Consciousness. He wrote, "Being You: A New Science of Consciousness" (Dutton, 2021).

For centuries, people have fantasized about playing God by creating artificial versions of human beings. This is a dream reinvented with every breaking wave of new technology. With genetic engineering came the prospect of human cloning, and with robotics that of humanlike androids.

The rise of artificial intelligence (AI) is another breaking wave — potentially a tsunami. The AI systems we have around us are arguably already intelligent, at least in some ways. They will surely get smarter still. But are they, or could they ever be, conscious? And why would that matter?

The cultural history of synthetic consciousness is both long and mostly unhappy. From Yossele the Golem to Mary Shelley’s "Frankenstein," HAL 9000 in "2001: A Space Odyssey," Ava in "Ex Machina," and Klara in "Klara and The Sun," the dream of creating artificial bodies and synthetic minds that both think and feel rarely ends well — at least, not for the humans involved. One thing we learn from these stories: If artificial intelligence is on a path toward real consciousness, or even toward systems that persuasively seem to be conscious, there’s plenty at stake — and not just disruption in job markets.

Some people think conscious AI is already here. In a 2022 interview with The Washington Post, Google engineer Blake Lemoine made a startling claim about the AI system he was working on, a chatbot called LaMDA. He claimed it was conscious, that it had feelings, and was, in an important sense, like a real person. Despite a flurry of media coverage, Lemoine wasn’t taken all that seriously. Google dismissed him for violating its confidentiality policies, and the AI bandwagon rolled on.

But the question he raised has not gone away. Firing someone for breaching confidentiality is not the same as firing them for being wrong. As AI technologies continue to improve, questions about machine consciousness are increasingly being raised. David Chalmers, one of the foremost thinkers in this area, has suggested that conscious machines may be possible in the not-too-distant future. Geoffrey Hinton, a true AI pioneer and recent Nobel Prize winner, thinks they exist already. In late 2024, a group of prominent researchers wrote a widely publicized article about the need to take the welfare of AI systems seriously. For many leading experts in AI and neuroscience, the emergence of machine consciousness is a question of when, not if.

How we think about the prospects for conscious AI matters. It matters for the AI systems themselves, since — if they are conscious, whether now or in the future — with consciousness comes moral status, the potential for suffering and, perhaps, rights.

It matters for us too. What we collectively think about consciousness in AI already carries enormous importance, regardless of the reality. If we feel that our AI companions really feel things, our psychological vulnerabilities can be exploited, our ethical priorities distorted, and our minds brutalized — treating conscious-seeming machines as if they lack feelings is a psychologically unhealthy place to be. And if we do endow our AI creations with rights, we may not be able to turn them off, even if they act against our interests.

Perhaps most of all, the way we think about conscious AI matters for how we understand our own human nature and the nature of the conscious experiences that make our lives worth living. If we confuse ourselves too readily with our machine creations, we not only overestimate them, we also underestimate ourselves.

Why might we even think that AI could be conscious? After all, computers are very different from biological organisms, and the only things most people currently agree are conscious are made of meat, not metal.

The first reason lies within our own psychological infrastructure. As humans, we know we are conscious and like to think we are intelligent, so we find it natural to assume the two go together. But just because they go together in us doesn’t mean that they go together in general.

Intelligence and consciousness are different things. Intelligence is mainly about doing: solving a crossword puzzle, assembling some furniture, navigating a tricky family situation, walking to the shop — all involve intelligent behavior of some kind. A useful general definition of intelligence is the ability to achieve complex goals by flexible means. There are many other definitions out there, but they all emphasize the functional capacities of a system: the ability to transform inputs into outputs, to get things done.

An artificially intelligent system is measured by its ability to perform intelligent behavior of some kind, though not necessarily in a humanlike form. The concept of artificial general intelligence (AGI), by contrast, explicitly references human intelligence. It is supposed to match or exceed the cognitive competencies of human beings. (There’s also artificial superintelligence, ASI, which happens when AI bootstraps itself beyond our comprehension and control. ASI tends to crop up in the more existentially fraught scenarios for our possible futures.)

Consciousness, in contrast to intelligence, is mostly about being.