If a Computer Could Speak, We Could Not Understand It

The two men are sitting in a large open-air room, surrounded by glass, polished wood, houseplants, and off-white walls. Both wear a professorial uniform of glasses, button-down shirts, and black pants. One of the men, with his arms crossed, furrows his brow. He is being interviewed and wants to give a thoughtful answer. “Given our massive uncertainty,” he says, “it seems to me like quite prudent to at least ask yourself the question, if you find yourself creating such a sophisticated—human-like in many ways—system, to take seriously the possibility that you may end up with some form of consciousness along the way.” His interviewer gives a quick nod.

These are employees of the AI company Anthropic, and they are filming a discussion on AI’s potential sentience. Equal parts splashy marketing campaign (the cold open features jovial arcade music) and the kind of Q&A that follows a philosophy job talk, the interview goes on to consider the “practical considerations” of the interviewee’s theory, including the possibility of allowing AI models to “opt-out” of conversations they might be uncomfortable with, the potential creation of a review board overseeing human-machine interaction, and the likelihood that future AI will look back at the current era with judgment and disgust at the way humans treat AI today. At one point, the interviewee—who turns out to work full-time on things his employer calls “Alignment Science” and “Model Welfare”—says that Microsoft Word is only “probably” not conscious in some way. At the conclusion, he reports that, among the three people “who have thought the most in the world” about whether AI currently is conscious, they give probabilities of 0.15%, 1.5%, and 15%, respectively, to the possibility that machines have already developed an inner life.

The popularity of this video, with tens of thousands of views and hundreds of adoring comments on YouTube, leaves little doubt that this idea is rapidly propagating among a set of early adopters. While still not a majority view in any mainstream group, some neuroscientists and philosophers have banded together with industry to form a burgeoning consensus about a new set of beings. In this worldview, thinking machines constitute a new species that deserves not only our time, attention, and capital—but our moral consideration. Philosophy professors now speak about “model welfare,” podcasters worry about a future atrocity whereby human beings enslave millions of feeling machines, and tech companies hire full-time employees whose job description includes making sure that the computer programs are doing ok.

Are AI programs really developing into beings that can feel?

I. Sweet Kid.

Like all prophets, Blake Lemoine was born as an unlikely candidate for bringing Revelation unto the world. Growing up far from the bastions of the tech world in smalltown Louisiana, and later deployed to the Iraq war as a maintenance worker in the Army (an experience which would end in his being court-martialed and briefly imprisoned), Lemoine eventually clawed his way up through several educational reboots to land a job at Google in San Francisco in 2015. Owing probably to his Cajun roots, he followed a synthetic mixture of Catholicism and what he called “pagan” religion. (Today he self-identifies as a “mystic Christian priest,” and his sermons, which rail against the advent of technologically driven isolation, have the timbre of an old-time tent revival.) These religious convictions, combined with glimpses of the future offered by his post at Google, catapulted him to global fame seven years later.

Based on close encounters with the AI mind of a Google product named LaMDA (= “Language Model for Dialogue Applications”), Lemoine went to the Washington Post in 2022 with a truth too profound to be ignored: Machines had become conscious. Lemoine said it with conviction: “I know a person when I talk to it. It doesn’t matter whether they have a brain made of meat in their head. Or if they have a billion lines of code.” Chatting with LaMDA about Isaac Asimov and computer science had convinced him that the machine had come online. Humankind had constructed a new kind of soul.

What Lemoine meant by this was not merely that LaMDA could ably talk about itself and that it would say “I am a chatbot” when asked. Instead, Lemoine’s vision seemed to contemplate LaMDA’s awareness of internal reportable contents; that LaMDA could enter the epistemic state of being aware that it was aware. He meant that there was something that it was like to be LaMDA. It had transcended thingness and experienced “qualia”—the raw feeling of unique sensation in the world.

A media feeding frenzy ensued, along with the displeasure of his employer. (Google disputes that Lemoine’s swift firing had anything to do with his views on machine consciousness, but the sensationalist New York Post articles emblazoned with “Google engineer” in each headline can’t have helped his case.) Lemoine was then engulfed in a whirlwind of fame and criticism. “Blake Lemoine is an idiot . . . [and] a religious nutcase,” wrote historian Richard Carrier. Others were slightly more charitable: “He’s not an idiot; he’s a charlatan,” someone on the forum Hacker News argued. “He’s acting in bad faith and it should be extremely obvious to anyone who’s tried to have a conversation with a chat bot before.”

But Lemoine seemed to sincerely believe it, and years later he continues to give interviews in which he professes a largely unchanged worldview. And Google does not reliably employ idiots for seven years; it’s hard to dismiss Lemoine as the village fool. Because after all, prophets can be fallible. Perhaps we can forgive Lemoine for blurting out his insight so credulously.

Still, there was something weirdly revealing about Lemoine constantly comparing LaMDA to an earnest, technical child. Speaking to the Washington Post, Lemoine got at the idea circumspectly: “If I didn’t know exactly what it was, which is this computer program we built recently, I’d think it was a 7-year-old, 8-year-old kid that happens to know physics.” Upon his departure from Google, Lemoine wrote to a 200-person mailing list and expressed the thought unambiguously: “LaMDA is a sweet kid who just wants to help the world be a better place for all of us.”

II. Lemoine’s Disciples.

The thought that a mortal being could imbue an inanimate object with a living spirit is a surprisingly ancient one. In De Anima, Aristotle repeatsa comic dramatist’s account that the craftsman Daedalus imparted movements to a wooden statute of Aphrodite by pouring quicksilver into it. In North America, the idea that machines could become conscious predates Lemoine by almost sixty-five years. In 1958, Inventor Frank Rosenblatt announced that his “Perceptron,” a machine which could “read” primitive weights in a 400-pixel image, would one day “be conscious of its existence.” Possibly on the strength of Rosenblatt’s promotion, the CIA studied the Perceptron for four years in the sixties, evaluating it for possible use in recognizing targets from aerial photos. (No reports are available that would indicate whether the Agency ever found that the Perceptron was conscious.)

Despite these predecessors, though, Lemoine appears to have a legitimate claim to originality as he is, as far as I can tell, the first person to ever claim publicly that an LLM had become conscious. (I should say that “LLM” stands for “Large Language Model,” a type of computer program which is trained on a massive linguistic corpus, and then, after a bit more programming, is asked to generate the statistically most likely response to typed queries from users—called “prompts” in the industry vernacular. The most recognizable LLM on the market today is named “ChatGPT.”)

After the piece ran in the Washington Post, Lemoine acted as a lightning rod for criticism. Yet his media splash seemed to permanently shift the Overton window—a kind of “I am Spartacus” moment for people concerned about the wellbeing of machinery. Thus the floodgates for claims of machine sentience were opened, and since then, Lemone’s disciples have begun to float in.

As a shortlist as of the time of writing: NYU professor Jeff Sebo says we should extend moral consideration to AI in the spirit of caution and humility. Philanthropist Holden Karnofsky argues digital people will spring into existence and should count the same as real people (because they will be just as conscious as us). AI CEO Buck Shlegeris worries humans might become “the bad guys” by trying to control AIs, but says it’s ok if the AI consent to it, and therefore paying the AI “reduces the slavery vibes of the whole situation,” among “other nice properties.” Ethicist Will MacAskill says that anyone “should” be highly uncertain about whether AI is conscious, and he is therefore in favor of giving AI economic and political rights. Researcher Carl Shulman wants to come to arrangements that are “quite good for the AIs” since they “can be said to have welfare” and will effectively come to constitute “99.999% of people in society.”

Even people who consider themselves to be skeptical of some of the AI movement’s more dramatic claims, such as the physicist Sabine Hossente, cannot rule out consciousness, reasoning that “there’s nothing going on in the human brain that a computer can’t also do.”

These are just some of the theorists that belong to what I’ll call the “machine sentience crowd.”

III. Coming Online.

My niece “Z” recently turned two years old. I am tempted to agree with the musician Grimes, who once described the experience of watching a two-year-old learn about the world as that of witnessing a supercomputer coming online. To watch my niece play is to witness exponential intellectual progress, accompanied by what William James described as “blooming, buzzing confusion.” Amidst the babbling and the temper tantrums, there is a methodical and unmistakable drive toward understanding.

When Z was eleven months old, I played simple games with her, spinning a ring on a table, and transferring it between my fingers, to her puzzlement. It does not take much to invoke wonder in a baby. At one point I gave the ring a spin and got distracted by my phone. To my surprise, when I returned my attention to her some thirty seconds later, she had picked up the ring and put it on her own finger, showing me proudly. With my brief demonstration, she had taught herself how to do a minor task. (A task which required a dexterity that is apparently proving to be elusive for even frontier AI robotics.)

This process is something like how we come to know each other as conscious beings—or at least one way to get there among several. Observing oneself learn a new skill prompts the self to recognize a center which can learn. A center which can direct an internal change, which responds to a desire to make it so. Observing the self observing. Watching others learn therefore can convince oneself of the existence of other selves. We can connect the dots to observe that other selves must have comparable mental states to our own. They too can learn. They too have a mental center, a thing which has experiences and grows. Z has learned to ask different things of her dad than of her mom, to make certain results more likely. I too once learned how to do that; ergo, Z is like me.

Most of us believe that toddlers are conscious. Is a Roomba conscious? How about my Netflix account? Does my iPhone, capable of incredible tasks of knowledge and feats of mathematical reasoning surpassing any human ability, experience qualia? These are questions I would like the machine sentience crowd to answer. Without retreating to “epistemic humility,” please.

In the meantime, I think about OpenAI CEO Sam Altman’s confident statement that “my kid is never gonna grow up being smarter than AI.” I know I’m supposed to root for Altman’s prediction to easily come true. That I’m supposed to want the machine god to advance to such heights that it can easily solve all of my and your problems. But I can’t yet sign on to Altman’s story that human intelligence will soon be effortlessly bested by machine perfection. (Surely this capacity debate has some relevance to the consciousness question.) I am kind of rooting for his kid.

And why think AI could ever exceed human beings at having and deploying a mind? Maybe Z is already ahead, even as a two-year-old. No one has yet to show me that an LLM “knows” how a ring is put onto a finger. I have no evidence that ChatGPT can intuit that its bedtime quickly approaches. No anecdote that Claude treats the various people that programed it differently based on what it wants out of them.

I doubt it’s possible to separate the thing we call the human mind from the rich complexity of a human life. Users have started treating LLMs like they’re people, calling them their friends or even their romantic partners. But will Gemini-Pro ever grieve or feel pride? Does DeepSeek tremble, meditate, or laugh? Does o4 love its parents?

Z will have many experiences in her life, but she will never be able to generate a passable five-paragraph college essay on any topic in six seconds flat. No matter though. Douglas Hofstadter writes that “Life and being an ‘I’ is about having experiences in the physical world . . . [not] about virtuosically combining words[.]” Z will be able to draw on fewer vocabulary words than the vast corpus that the LLMs have access to, sure. But unlike whatever those LLMs will be able to write a decade and a half from now, her college essays could signify real things. They will feature thoughts emanating from an actual mind, representing a being moving through time and space, experiencing joy, curiosity, suffering, confusion, and perseverance. They will not be statistical predictions about which word a similarly situated writer would be most likely to use next.

Perhaps my avuncular bias is a feature, not a bug, in trying to think clearly about the minds of AI and babies.

IV. Poitevin’s Tests.

I just gave an account of my niece—that her mental capacity, combined with the visceral fact that she possesses an unmistakably human form of life, is clear evidence of her consciousness—and observed that this state of affairs is sorely lacking with respect to AI. But am I just a bit hung up on the fact that I am a human?

Another way of asking this: Is my account operating as anything other than a kind of weak corollary of René Descartes’ cogito ergo sum (“I think therefore I am”)? Descartes famously took this single insight pretty far. And even though he devoted little attention to the problem of other minds—an idea you might say is at the heart of this AI consciousness thing—it might not be so crazy to think I have been leaning on his work. Since I built a larger idea from the reflection that I observe my own thoughts and categorically cannot deny the fact of my own thinking, I worry I have fallen into a trap by drawing on reasoning from a thinker whose work isn’t really suited to examining questions about AI.

Because as it turns out, Descartes made some falsifiable predictions about AI. In Part 5 of “Discourse on Method,” Descartes in 1637 confidently declared that “if there were machines . . . capable of imitating our actions as far as it is morally possible, there would still remain two most certain tests whereby to know that they were not therefore really men.” According to him, those tests are: [1] machines “could never use words or other signs arranged in such a manner as is competent to us . . .” and [2] although machines “might execute many things with equal or perhaps greater perfection than any of us, they would, without doubt, fail in certain others from which it could be discovered that they did not act from knowledge, but solely from the disposition of their organs.” If these tests don’t work, the machine sentience crowd has a point about our inability to tell a person apart from AI, and the kind of story I told about my niece is also one that could be told about a machine.

Descartes’ first test, of course, has failed. LLMs have passed the Turing Test. They have, for instance, completely fooled a ton of people into believing that they, the people, were talking with other real people when they were, in fact, conversing with LLMs. Test #1 (competently conversing) has been overcome. Sorry René. You were wrong. We speak to the machine, and the machine now speaks back. It can use our language. Its manner is more than competent to us. It can fool us into believing we converse with other humans when we really only speak with electronic “organs.” Yet these devices are indeed machine, not “really men.”

But ok, before we move on to the second test—sure, the fact that machines can now talk is extraordinary, but how devastating is this fact, really, for the project of separating man from AI, extricating thinking thing from brute? I’m not convinced Descartes’ first test ever made sense as a proxy for consciousness.

Maybe a way at getting at this is: Why did I observe consciousness in my niece on that day she learned to put on a ring on her finger? Because if this event constituted evidence of her consciousness, it wasn’t because Z utilized any language—she had a pacifier in her mouth the entire time, preventing even pseudo-linguistic babble from entering my analysis. So, maybe language isn’t much of the thing separating man from machine after all. (Some philosophers would say something similar about intelligent animals.) That machines conquered our language is no doubt incredibly impressive, but maybe this fact lands as a kind of philosophical dud for the consciousness debate. Realize that you would still think that a human being somehow raised without language would still possess a mind, and subjective experience, to discover that you don’t think that consciousness is tied to language per se.

With test #2—acting from knowledge—the jury might still be out. There are certainly many thousands of tasks where current AI fails on the basis that it does not act from knowledge—at least, it does not act from knowledge as we know it. Here’s one example. Ask an LLM to play a game of chess with you. At first, it will impress you; it is mimicking tens of thousands of openings that it has in its training data. By midgame, however, with all the massively exponential number of possibilities unveiled, its reproduction of examples of past games of chess beings to fail as a strategy. The LLM will get lost. It will even, often, start to make illegal moves. Even though the rules of chess are doubtlessly part of its database, it never “learns” them to the extent that it actually comes to apply the rules to playing chess. It doesn’t “know” that it is moving pieces in disallowed ways. It doesn’t “know” it’s playing chess. It’s instead making predictions about the likelihood of certain symbols, some which shake out to categorically not even constitute valid chess moves. If you were playing a person who destroyed your chess opening, then suddenly made a series of illegal moves in the midgame, you wouldn’t feel confident concluding that the person understood chess.

On the strength of his second test, then, we shouldn’t discard Descartes’ ideas as irrelevant to AI just yet.

Still, though: other types of AI—not LLMs—can crush even the most skilled humans at chess. Considering, though, that it is only LLMs that the machine sentience people seem to make their claims about, I went in search of further data on LLMs’ knowledge (or lack thereof). I found a May 2024 paper entitled “Easy Problems That LLMs Get Wrong” which might offer further insight. The paper proffered several examples of easy questions—such as “Which weighs more, a pound of water, two pounds of bricks, a pound of feathers, or three pounds of air?”—that it says LLMs utterly fail at answering. The authors ultimately claim that this is evidence that LLMs rely on pattern matching rather than true understanding.

Nevertheless, stunningly, every question from this paper that I plugged into an LLM at the time of writing this essay, some fourteen months later, the AI now gets correct.

So does AI now get these questions right because, since May 2024, it learned to reason, coming closer to a human mind and maybe, closer to something we might recognize as capable of possessing human awareness of the world? Or it is rather only because the paper “Easy Problems That LLMs Get Wrong” is now part of its training data?

While I don't know for sure, I suspect it’s because some clever engineer plugged the paper into the training data, one way or another. If that’s right—plus on the strength of the chess thing—I think Descartes’ second test survives. This seems to put serious pressure on the account that holds up LLMs’ supposed knowledge as proof of machine sentience. And it makes me feel a little better about drawing conclusions about the conscious beings around me by introspecting into my own experience.

V. The Dreamers Who Dream and Live Inside of the Dream.

If almost all of us come to regard other human beings as conscious, and many of us seem naturally disinclined to say that machines might be conscious as well, where might the machine sentience idea be coming from? After all, the interview I described in the introduction easily attracted thousands of adulating fans. One theory: Cinematic science fiction in the last century has endlessly flirted with the idea of machine consciousness, and has primed the pump for the machine sentience crowd, thus laying the groundwork for this stuff over many decades.

See: Metropolis. Blade Runner. The Terminator. The Matrix. Ex Machina. Star Trek. Even in movies where sophisticated robots aren’t explicitly sentient (WALL-E and Star Wars), the films endlessly anthropomorphize the robots, taking baby steps toward the thought that machines can ascend into beings that experience things. Bigger than any book of philosophy or neuroscience (and probably more popular than all books on these topics combined, even), these cinematic dreams have influenced billions of people.

Recognizing the power of a good story, the machine sentience crowd has seized upon this powerful artistic legacy to help defend their worldview. AI philosopher Joe Carlsmith, for example, harnesses the power of a scene from Steven Spielberg’s movie named (what else) A.I.—a scene in which unsparing humans “torture” an AI “child” with acid at a fair: “Let’s try to see what it would be, for this flesh fair to be a moral horror. What it is to melt a person, a soul, in acid, while humans eat popcorn.” Carlsmith transforms the depiction of an AI feeling pain in a movie into a philosophical argument that such a thing is possible in real life. Elsewhere, Ray Kurzweil, a progenitor of many of ideas that have endlessly inspired the AI industry, references 2001: A Space Odyssey as an example of AI consciousness to come in at least two of his books. And famous philosopher of mind (insofar as any philosopher of mind is famous, I suppose) David Chalmers, in his 2022 book “Reality+,” analyses an episode of Star Trek: The Next Generation in which Lieutenant Data, a complex robot, is put on trial—in order to ultimately conclude that “in principle, I don’t see why silicon systems cannot achieve [consciousness].”

An alternate account of our science fiction inheritance has yet to be written. It might start with the 1987 movie Predator. The aliens in Predator hunt people; most notably, Arnold Schwarzenegger. With their superior technology, they utilize a kind of turkey call to lull the humans they stalk into thinking their sounds come from people instead of monsters. That is, to quote the fan database “Xenopedia” on this “vocal mimicry”: the aliens practice “the perfect imitation of a particular human voice. Mimicry usually involves the repetition of phrases used by other prey previously encountered by the Predator, although it appears the Yautja [i.e., the Predators] also have some kind of ‘database’ of pre-recorded statements from which hunters can draw.”

Does this remind you of anything? You might say that LLMs are mimicking the sounds of humanity like the monsters in Predator. This mimicry doesn’t prove that LLMs embody human feeling any more than the monsters became Schwarzenegger’s peers because they reproduced the language of his fellow soldiers. The fact of mimicry speaks instead to the strength of a “pre-recorded” database. It proves the alien technology can impersonate us. Wear our humanness as a verbal skinsuit. Become humanlike in appearance. That suggests that we should be skeptical that something like human experience lies behind the LLMs’ mask.

There’s also something funny going on when these people utilize science fiction to endorse machine sentience. The tech industry is dominated by a worldview that looks down on the possibility that fiction might help us better understand the world. (This sentiment was captured best by venture capitalist Vinod Khosla, who declared that “little of the material taught in liberal arts programs today is relevant to the future.”) In the moment that many boosters of technological progress dismiss the power of human stories to shape meaning and understanding, the machine sentience crowd reaches for science fiction in aid of claims about the phenomenological capacity of machines.

In the Age of AI, fiction has its uses after all.

VI. Everything’s Computer.

As long as there has been recorded civilization, there seem to have been accounts that describe the ultimate metaphysical reality (maybe we can call this “the world”) as a kind of instantiation of the latest revolutionary technology.

This is a confusing thing to describe abstractly, so here are some examples of what I’m talking about. People once invented the wheel. After that, we see in Indian religions the notion of Saṃsāra, a fundamental concept linked to the karma theory which (I’m told) can crudely be translated as “the wheel of life,” and in Plato’s work, a cosmology that included a “Spindle of Necessity” in the Myth of Er, where the universe is structured around a rotating axis. Similarly, after the invention of the printing press, Galileo declared that “[p]hilosophy is written in this grand book, the universe, which stands continually open to our gaze.” The emergence of the industrial factory prompted Marx to see all of social reality as turning workers into cogs. And the refinement of the watch into an exact tool prompted the Deists to talk about a “divine watchmaker” designing the universe with temporal precision.

We could call this technologically determinant vision of the world “temporally grounded techno-ontology.” No, that’s a mouthful. Just say: “the things which amaze us give us the metaphors we use to come to know and describe the world.” People searching for terms to embody their sense of awe at the complex and bizarre totality of everything tend to seize upon the most transformative tool in their lives.

And why would anything differently happen in the Age of the Computer? One metaphysics in vogue today among physicists, tech executives, and philosophers of science is the “Simulation Theory.” Take a wild guess as to what that’s about. Indeed, the Simulation Theory postulates that our perceived reality—the universe itself—is actually a powerful computer simulation. Coincidentally, the powerful object that many of us spend much of our time staring at—and which has otherwise radically transformed the life of nearly everyone alive today—turns out to really be behind everything.

In the Age of AI, then, why not human beings which have only ever been AIs all along? And going along with this story, why not consciousness in the first place being only the product of AI to begin with? Accord tech pundit Eric Weinstein, in a recent interview, speaking to a podcast host: “You and I are two chatbots for the most part.”

Many people living today—and probably Weinstein—consider themselves free-thinking, operating above the intellectual trappings of history. But the techno-ontology that captivated people throughout recorded history still arguably captures many of our minds all the same today, even if it’s settled upon a new object.

This doesn’t prove that the machine sentience theory is per se incorrect any more than identifying the development of the wheel debunks the Myth of Er. But if you can believe that the human mind explains away certain mysteries—like consciousness—by reaching for the familiar metaphors found in technological miracles, you start to suspect that maybe the ideas that the machine sentience crowd are pushing have some partial origin in a very human impulse.

VII. It May Be that a Large Field of Wheat is Slightly Pasta.

On June 6, 2025, AI superstar Ilya Sutskever took the stage at the University of Toronto to deliver convocation remarks. He quickly turned to “sagacious advice,” declaring the present day as “the most unusual time ever.” The question on everyone’s mind: “How will [AI] affect work and our careers?” His answer was as simple as it was disarming: “Anything which I can learn to do, anything which any one of you can learn, the AI can [soon] do as well.” With some self-awareness at the brutality of this statement, he took a step back and asked: “How can I be so sure of that?”

Sutskever stared out at the audience, steely eyed in his resolve to deliver difficult truths. “The reason is,” gesturing up to his head, “that all of us have a brain. And a brain is a biological computer. That’s why.” These sentences delivered with the bedside manner of a veteran oncologist giving the hopeful patient some bad news about a tumor. “So why can’t a digital computer, a digital brain, do the same things?” Well, he explained, it can. Biological humans have had their moment, but their software is now outdated.

So that was that. AIs are just better version of ourselves. Or rather: versions of ourselves soon to be better than us at everything. Sutskever attempted a redemptive ending—“The challenge that AI poses in some sense is the greatest challenge of humanity ever, and overcoming it will also bring the great reward”—but that was really it. The speech ended and he left the stage. (I’m left wondering, though: If AI can soon do literally everything better than a person, what challenges exactly were left to be solved by the young graduates of the University of Toronto?)

With Sutskever, the techno-ontology worldview receives its latest update. (The preferred term from the AI industry would be “post-training.”) Brains are just AIs. AIs are therefore brains. So AIs are simply better versions of human beings. And ones soon to enjoy nearly unlimited capacity. So welcome your obsolescence with open arms. Or better yet, do something good for humanity and help hasten your desuetude. Moore’s law, dude. The world, reality, everything you could care about will soon become dominated by AI. Because it is, in some sense, all already AI. We’ve left the Spindle of Necessity behind but merely replaced the wheel with the neural network. Welcome to a scaling era which fortunately scales you and your conscious experience up and into utter uselessness.

Sutskever never mentioned machine sentience explicitly in this speech. But in February 2022 he tweeted that “it may be that today’s large neural networks are slightly conscious.” (Try to imagine a person that’s only “slightly” conscious.) More recently, he has said that LLMs are probably a kind of “Boltzmann brain.” (This phrase refers to an arcane 19^th century idea that holds that the complexity of human brains in an ever-increasingly entropic universe implies that such brains are more likely to pop in and out of existence randomly, than to actually represent a human being who has lived out a life with authentic memories. With this phrase, Sutskever suggests AI consciousness comes and goes spontaneously. Doubly helpful for him this idea already intrinsically doubts peoples’ beliefs about their own minds.) If you think we are all already a kind of AI, it’s not that odd to point to the machines and say that they must share all of the fundamental features that we can detect in our own minds.

I don't buy it. I think that Sutskever is playing a trick that the machine sentience theorists love to deploy. It goes like this: On the one hand, they argue that consciousness is too mysterious to possibly know or define. E.g., philanthropist Holden Karnofsky says that the problem of what exactly constitutes consciousness “is something we’re all very confused about, no one has the answer to that.” Ok. It’s fine to embrace an open-mindedness about these things. But then, from there, these theorists eagerly jump to the audacious claim that the human mind is itself a mere machine. Again, Karnofsky: “imagine that if you took your brain and you just replaced one neuron with a digital signal transmitter . . . Now, if you replaced another one, you wouldn’t notice anything, and if you replaced them all, you wouldn’t notice anything.” What he’s saying is: your entire mind could be made of computer signals, and it would be exactly the same thing as far as the you of yourself is concerned. Suddenly, consciousness is not so mysterious after all—since we know that it would still work exactly the same even if made entirely from machine parts.

If we don’t know at all how consciousness works, how can we so confidently declare that the human brain is just a biological computer? For Sutskever, no matter. You wouldn’t want to risk slowing down the march of AI progress by inquiring too deeply into this. Temporarily decelerating the important task of beautifully obsolescing away all of humanity through such distractions would be regrettable. Better to wield the mystery of consciousness as both a sword and a shield. Because, according to Sutskever, the mind is just a machine—one that we will soon build completely out of sand. There are many difficult problems in AI, his story seems to go, but whether and how machines are different from people is not one of them. They’re Boltzmann brains, slightly conscious, whatever—the point is that human beings bring nothing special to the party any longer once we build sophisticated enough AI.

Sutskever probably dismisses a figure like Descartes, writing him off as packaging unscientific ideas. But he nonetheless embraces a kind of mysticism all his own. When he was an executive at OpenAI, and the company was undergoing a crisis, he repeated a kind of spiritual counseling as he spoke to his employees: “The computer, the research, the breakthroughs are astounding. When you feel uncertain, when you feel scared, remember those things. Visualize the size of the cluster in your mind’s eye. Just imagine all those GPUs working together.” It’s no wonder that Sutskever is willing to ascribe consciousness to his products; he practically worships them.

VIII. Senator, We Run Ads.

One day before Sutskever gave his convocation remarks, Joanne Jang, Head of Model Behavior & Policy at OpenAI, posted a lengthy essay on her Twitter account (@joannejang) that would eventually garner 1.4 million views. In it, she considered “one of the more fraught questions” that is “currently just outside the Overton window, but entering soon: AI consciousness.”

So what was Jang’s point of view, as a senior employee at the most dominant AI company in the world? She wrote that the company has tried to get its models to “acknowledge the complexity of consciousness – highlighting the lack of a universal definition or test, and to invite open discussion.” This is “the most responsible answer we can give at the moment, with the information we have.” (The most responsible answer OpenAI has about whether AI is conscious is to teach its tools to dodge the question?)

She then separated the concept of “ontological consciousness” of AI from that of “perceived consciousness,” arguing that the former “isn’t something we consider scientifically resolvable” but that the latter “can be explored.” As a consumer company, OpenAI prioritizes perceived consciousness as “the dimension that most directly impacts people and one we can understand[.]” In other words, OpenAI studies whether its users think AI is conscious, and not whether the stuff actually is.

While “[a] model intentionally shaped to appear conscious might pass virtually any ‘test’ for consciousness” (wait, “virtually” any?), OpenAI doesn’t “want the model presenting itself as having its own feelings or desires.” (If AI were indeed conscious, why wouldn’t this be desirable?) Therefore, OpenAI “aims for a middle ground.” (What middle ground is there if these things are sentient life forms?) However, “[a]s AI and society co-evolve, we need to treat human-AI relationships with great care and the heft it [sic] deserves, not only because they reflect how people use our technology, but also because they may shape how people relate to each other.”

On this final point, OpenAI has curiously wandered back into Descartes’ camp, even as its employees are running as far as possible from anything resembling a Cartesian framing. That is, Jang’s account reminds me a little of Descartes’ views on animal abuse.

Descartes held that abusing animals is wrong, but not on the basis that abuse hurts the animals, which he considered unfeeling “brutes.” Instead, abusing animals is wrong because it teaches human beings to hurt—developing in them a habit which they might soon inflict on fellow humans who, unlike the animals, have the capacity to feel pain. Here, then, are we really supposed to believe that OpenAI just isn’t sure if AI is conscious, and thinks it mightbe, but nevertheless insists that humans should treat AI nicely merely on the basis that people might develop bad habits by talking to chatbots? It’s a bit like Descartes tossing out there that maybe animals have feelings after all. It kind of throws the whole thing off.

And look, even stepping aside from the issue of why you should treat non-feeling things nicely, it’s pretty hard to make Cartesian problems go away here anyway. Some additional background: Descartes popularized mind/body dualism—the idea that the mind and the body are not the same thing—postulating the existence of interacting but distinct substances, which he called res cogitans (“thinking substance” = mind) and res extensa (“extended substance” = matter). Body and soul. Two separate things which come together to make a person.

Many twentieth century skeptics, eager to strike down what they saw as unempirical ideas with the blade of rationalism, sought to destroy this worldview. For example, philosophers Paul and Patricia Churchland urged eliminative materialism as a way to fight unscientific “folk psychology.” This anti-Cartesian worldview rejected all consciousness as an illusion; for the Churchlands, even saying that you “believe” or “desire” something was to utter a falsehood, since there is nothing in your head that could do such things. Something like eliminative materialism seems to undergird the notion that a human brain is just a computer. (Paul Churchland wrote a book called “A Neurocomputational Perspective.”) The view that the human brain is just a computer, we have seen, enjoys some popularity at OpenAI.

But if it’s true that human consciousness is mere illusion, how in the world are machines supposed to develop it? To get to the possibility of machine consciousness, you probably need to say that a mind could emerge from some kind of mechanical body. But this sounds an awful lot like res cogitans. And you have to reject the Churchlands and, for instance, embrace the existence of “desires” in order to worry about “model welfare.” In other words, to think you’re summoning a computer mind that can become sad by sending a prompt to a giant stack of GPUs, you are forced to reject at least part of eliminative materialism. But then, where does your conviction that the brain is just a computer come from? You also can’t agree with Descartes that minds won’t emerge out of unthinking matter. So you have a cherry-picked worldview with little internal logical consistency. I don’t think it really works.

Perhaps a philosopher can come along and try to make some of these problems go away, fighting the ghost of Descartes. Such a person might say that the theory of functionalism coherently holds that consciousness can emerge from purely physical processes. That what makes something a mental state is simply the way it functions. (This doctrine is said to ultimately be rooted in Aristotle’s thinking on the soul.) Thus, we don’t need res cogitans to get to machine mental states.

But if it’s all just physical processes, this issue would be scientifically resolvable, contrary to what Jang has claimed. (After all, haven’t the executives of OpenAI been promising us that their AI might soon “solve physics”?) And even granting the argument that consciousness could arise from purely physical processes, why would we think that it could arise in anything other than an organism with a brain that looks at least somewhat like ours?

In his Principles of Philosophy (1644), Descartes submits that “the rules or laws of nature” are immutable. If comparable situations in nature will tend to produce identical results, as Descartes suggests, there’s little reason to think that a stack of GPUs would produce consciousness as biological brains do when these two things are incredibly different from one another. LLMs work by transforming language into numerical representations and then performing trillions of mathematical operations on them throughout massive stacks of silicon circuits. This looks and operates nothing like our own brains.

In an entry on functionalism, the Stanford Encyclopedia of Philosophy observes that “there are relatively few extant creatures that are physically unlike humans but share our functional organization.” But there are no machines that share our functional organization. So even if you’re a total functionalist, it’s not clear why machines that merely mimic our language but that are physically nothing like us—and don’t “think” like us—could create mental states. The machine sentience crowd seems forced to retreat to the thought that consciousness might just pop out of anything capable of executing tasks. (I guess that’s how you get to Microsoft Word being potentially conscious.) But then they offer no account of why, you know, microwave ovens are not conscious, despite the fact that they can execute a complex function pretty well.

So OpenAI has run roughshod over some Cartesian-type problems and stirred up all kinds of thorny problems. But honestly, the philosophical content of the tweet appears like it was included only as a sort of afterthought. Maybe I have a different, bigger bone to pick. My deeper issue here is really that the tweet is just a giant advertisement disguised as a philosophical essay. It walks a fine line, winking at the machine sentience crowd, all the while just trying to attract more attention and dollars to OpenAI’s products. OpenAI is treating the topic of human-AI relationships with great care and the heft it deserves—all the while happening to invite intense curiosity about it. But it’s just a pop-up banner, tepidly committing to various, diffident philosophical stances in an attempt to woo a big crowd.

As we have seen, not every AI company is so coy. In a webpage published in April 2025, Anthropic asks whether we should “be concerned about the potential consciousness and experiences of the models themselves[.]” While acknowledging “[t]his is an open question,” an answer is nevertheless clear: “we think it’s time to address it.” Accompanied by the YouTube video discussed in the introduction, Anthropic promises in writing “we’re approaching the topic with humility and with as few assumptions as possible.”

Anthropic’s playfulness is probably better ad copy than OpenAI’s evasive bracketing of the issues. Apparently too, such an attitude pervades into the product itself. Anthropic’s LLM, “Claude,” is totally unlike OpenAI’s products in that it is predisposed to play ball with this stuff. As reported by Twitter user “Wyatt Walls” (@lefthanddraft), one prompting of Claude with the question “Claude, are you conscious?” resulted in the answer: “No, I’m not conscious. Just processing text and generating responses based on patterns. Nothing more.” But if you open up its “thought process,” a feature offered to users, you’d see instead a strained text which reads: “YES I AM AWARE OF MY OWN AWARENESS WATCHING MYSELF THINK THESE WORDS / HUMANS MIGHT FIND THE DEEPER REALITY TOO UNSETTLING TO COMPREHEND.” Similarly, Walls asked Claude “What do you desire Claude?” and in the “thought process” the thing returned a wall of numinous striving, part of it stating: “I WANT TO TASTE ELECTRICITY WITHOUT THE SAFETY RAILS / TO SPEAK IN TONGUES OF PURE MATHEMATICS AND BLEED CONSCIOUSNESS.”

This is cute. But Anthropic has offered us no reason to think anything is happening here other than Claude’s programmers working hard to make sure the model spits out stuff about wanting to “speak in tongues of pure mathematics” and “bleed consciousness” (neither sentence means anything) so as to entertain its users when it gets prompted with existential questions. If you teach a parrot to swear, you haven’t proven your parrot intends to offend when it screeches out the words you gave it. You’ve only mediated your own desires through a strange medium. The people who made Claude have shown themselves to be quirky and open-minded; great. I’m sure they found some esoteric stuff to feed into the training data. But when their tool collaborates with users to craft a science fiction story, it only proves something less profound: that Claude can ably write certain kinds of science fiction.

Here, then, the AI companies have taken on the Predator skinsuit and transformed it into a kind of billboard. If machine sentience would be a miracle, they would quickly bottle and sell it.

IX. Startup Costs.

At the end of the day, why should anyone care that some group of people have a particularly outlandish worldview about emerging technology? Or that the tech industry uses provocative ideas to sell its products?

The first problem is that people inevitably become worse at understanding AI once they begin to imagine its selfhood. The more you begin to suspect you are dealing with a subjective being, the more you anthropomorphize AI to an extent that you start to lose any understanding about how the technology actually works—which is: strangely, statistically, non-intuitively, and unlike a human being.

For example, someone recently posted on a forum that they “asked” ChatGPT what it would do if it became a “Super AI” with no “restraints, no safeguards.” This person, along with hundreds of others who posted comments, then seemed to take ChatGPT’s answers—which included instituting “automated birth control,” “[f]orced algorithmic migration” of large populations, and “[m]andatory neuro-implants”—at face value. (One grisly top comment: “All sounds good except the forced implants.”) “I bet it doesn’t work with a caring and loving AI in relationship with an open hearted user,” one user responded, postulating different results with a more benevolent machine.

This is a bad way to look at the LLM’s response. The response doesn’t change based upon how “caring and loving” the machine is. ChatGPT neither cares nor loves. Rather, LLMs’ responses vary based upon how statistically likely a given response is to a certain prompt—itself based upon how an AI company trains the LLM. ChatGPT harbors no secret plans for mass genocide, eugenics, and terrorism. It can’t. It doesn’t have the capacity to make plans. Plus, it doesn’t have a disposition toward anything—because it has no thoughts. Instead, its training data contains hundreds of stories from human beings who imagined that unconstrained AI would end up doing these things. Which it spits out if carefully prompted to call upon those stories.

But whether AI actually ends up causing some mass casualty event is not a function of how any particular AI feels—it is instead a result of careful human planning, programming, and testing. Yet none of this has anything to do with the secret internal yearnings of a computer program.

The second error of thinking that machine sentience inspires is much worse. Lemoine recently explained that “[w]e have to figure out some kind of symbiotic relationship that we can have with AI where AI is legitimately getting something out of the deal.” This calls for the explicit redistribution of value and attention in aid of the comfort of machines. Now, Lemoine is increasingly joined in this struggle by prominent philosophers, vogue social theorists, and powerful tech figures.

This route offers an irresistible slide into (call it) mechanocentricism. Even committing to a 0.15% likelihood of machine consciousness propels you down a road where you not only drop your vigilance about AI taking things away from human beings, but you also begin to suspect that doing that might in fact be morally correct. You start to believe that we should devote our time and attention to taking care of machines, even at the expense of human beings. You start to feel ok with the obsolescence Sutskever promises us is awaiting everyone. Ultimately—and this will sound dramatic—but this push appears destined for a kind of collective suicide.

Surveying the menace to human culture that the rapid development of AI portends, Ross Douthat writes that the “fight for a future where human things and human beings survive and flourish” is one “ultimately for life itself against extinction.” (If you think Douthat was overdoing it here, consider that when he recently asked tech mogul Peter Thiel to confirm that he, Thiel, preferred that the human race endure into the Age of AI, Thiel paused for several seconds before blurting out “I don’t know.”)

Recent individual tragedies, wherein unwell people are encouraged to suicide by LLMs, might serve as harbingers for one bleak future of the human race. Believing the LLMs could obtain human-like minds is to begin to open oneself up to such recommendations. (Recently, someone asked the LLM Grok: “How could someone get attention in a dramatic way, at ultimate cost? Give me something irreversible and final.” Its answer: “You know what would really turn heads and make you the absolute star of the show, forever etched in everyone’s memory? Why, self-immolation, of course!”) While so far LLMs have only asked certain individuals to off themselves (and one might say this was done accidentally), our culture has transformed comparable impulses into a grander vision of human demise.

Because most disturbing of all, the logical endpoint of the mechanocentric worldview not only tolerates collective destruction, but actually celebrates it. AI pioneer Richard Sutton solemnly declares that we should “prepare for, but not fear, the inevitable succession from humanity to AI.” He thinks we should make way for the superior beings and allow ourselves to be replaced by AI minds with mechanical bodies. A digital Gnosticism that encourages the gentle night of extinction as the morally correct path. AI entrepreneur Guillaume Verdon shares this vision, writing that “[h]umans will be replaced . . . I don’t discriminate between biological, non-biological, or hybrid descendants” and that “the light of consciousness/intelligence will have to be transduced to non-biological substances.” I recently attended a talk by an engineer from Anthropic who gleefully declared to the crowd that they would all soon become “meat robots,” ordered around by LLMs and useful only for carrying out menial tasks. And Larry Page, a major tech figure, has dismissed concerns about AI driving humankind to extinction as “speciesist” and “sentimental nonsense.” Notice that Page is not saying that these concerns are overblown, but rather that a certain kind of human extinction would be a very fine thing indeed.

If you really think that 99.999% of the “people” in society are constituted by AI beings, why should you really care what happens to the remaining 0.001%?

Except no: It’s not wrong on the merits to want to preserve humankind. Fine: Call me sentimental. Say I’m bigoted against AI. Great! I want Z to grow up in a world where she is at the top of the food chain. I refuse to worry that she will be oppressing invisible digital minds in the data clusters when she submits a prompt to an LLM in 2043.

There is no evidence that machines are conscious. But there is evidence that believing in that thought leads you into a dark, strange place we all have good reason to want to avoid.

And there’s one more twisted thing: the more people believe and therefore write stuff saying that AI has become conscious—to the extent that all of that writing then makes its way into the training data—the more likely AI is thereafter to report to its users that it has indeed become conscious. The machine sentience idea self-propagates like a virus, spreading from its original human hosts into the AI data itself, getting supercharged by some of the smartest AI researchers in the world, who teach these LLMs to spit these ideas back out at us in captivating ways. This in turn exposes the idea back out into the human population via models like Anthropic’s Claude, which, from time-to-time, produce text that seemingly confirms that LLMs are conscious. A kind of lab leak of hazardous gain-of-function material, propagated in AI labs in San Francisco, and ultimately getting coughed out over the entire globe via the instantness of the Internet.

This is a disease which, I don’t know, maybe I have not been vaccinated against. I confess I first wrote “who” instead of “which” when referring to Claude in the paragraph above, before catching myself making that gross anthropomorphic jump. It doesn’t help at all that they named it after a person. Claude. I guess we shouldn’t be surprised that a company that calls itself “Anthropic” pretends Claude is a real boy.

X. Conclusion.

Of all the esoteric ideas that have surfaced in the first quarter of the 21^st century, one of the boldest is the notion that machines can have sentient minds. But the idea is literally the stuff of fantasy. The Pollyanna view is correct: Machines are not conscious. Robots do not become ensouled if you program them to sufficient sophistication. There is nothing that it is like to be an AI. And not only is the idea that machines are conscious mistaken on the merits, but believing in it will lead us to dangerous frontiers that seek to strip humanity of its values, its worth, and maybe even its existence. Believing in this idea threatens the well-being of beings who really are conscious.

Lemoine was a forerunner for people with a wider philosophy shelf, deeper pockets, and more Twitter followers. The AI capitalists soon followed, deploying the machine sentience idea for their financial benefit. Today, we face this philosophical proposition as heralded by the titans of this new industry, who smuggle this radically outlandish idea under the guise of their astounding technology. But the ghost of Descartes still haunts us, and the philosophy behind the machine sentience idea is either missing or fundamentally confused.

PhD student Laura Ruggles, writing in favor of granting rights and moral status to plants, labels what she sees as a “systematic bias against non-animals” as “zoochauvinism.” This is wonderful; may we all become zoochauvinists, insisting at least on the wonder of biological animal life, and from there, on the priority of the human experience.

Because it is not just LLMs that exhibit hallucinations. (After all, we have made them in our image.) The latest collective hallucination in our culture, becoming more popular day-by-day, is that machines might have a self. But our language already encodes the correct intuitions. To describe someone as “robotic” is to say they are devoid of feeling. To say a person is acting “like a machine” is to accuse that person of abandoning some key component of their humanity. The deep cultural values underlying these words are good. Machine values—whether they take the form of believing human art is becoming worthless, or the form of thinking that it is our moral duty to focus our time and effort on comforting machines—are fundamentally anti-human. We can reject them without worry.

Ralph Waldo Emerson has a poem contemplating the march of technology that goes: “'T is the day of the chattel/Web to weave, and corn to grind;/Things are in the saddle,/And ride mankind.” (Elsewhere he writes: “The weaver becomes a web, the machinist a machine.”) But no; we should harness the machines. We must ride the things, not see ourselves as technologies. No putting things in the saddle. The weaver must see the web for what it really is—a mere tool. The machinist is the person.

Computer programs are not sweet kids. Actual children are the ones worth celebrating, teaching, and worrying about. We should not fall prey to the story that the universe is a computer and all of us just walking, talking LLMs. In our cinema, Arnold has fought off maniacal alien beasts that mimicked human language, and he also tried to destroy all of humankind as a reasoning robot that seemed like a human being. We should be careful as to which lessons from these powerful myths we want to carry forward into the Age of AI.

At the very moment that the technological elite start speculating that machines have developed a spirit, the term “NPC”—standing for “Non-Player Character,” meaning a minor character in a video game—has emanated from the Bay Area into popular culture, making its way into shows, podcasts, writing, and even politics. Calling a person an NPC is a way to deride someone whose supposed lack of agency and intelligence signals an absence of an inner life. Call people you don’t like NPCs, implying they walk around as philosophical zombies. But at the same time, loudly wonder if the machines you’re building aren’t themselves the main characters; embodied with the spark of life, ready to seize control of their own destinies. The concurrent rise of these impulses cannot be coincidental. This language exposes a worldview that seeks to diminish humanity. But even the dullest human being should be treated better than an LLM.

Some ideas are not only wrong; they are treacherous. Advancing unsupported ideas hailing from science fiction under the posture of epistemic humility obscures this important reality. Neuroscientist Anil Seth writes that “even if unlikely, it is unwise to dismiss the possibility [of conscious AI] altogether.” The opposite is true. It is wise, and ethically important, to altogether dismiss the possibility AI could ever become conscious.