SH: Can you do me a summary of Ted Chiang on AI in Atlantic Monthly?
GPT: Professor Harnad, If you mean Ted Chiang’s recent Atlantic essay, Ted Chiang, “No, Artificial Intelligence Is Not Conscious” (June 2026), the argument is essentially an attack on the tendency to infer sentience from linguistic performance.
In very compressed form:
1. Behavioral competence is not evidence of feeling. Chiang argues that because LLMs can converse fluently, many people are tempted to attribute consciousness to them. He regards this as a mistake rooted in anthropomorphism. The ability to generate plausible linguistic outputs does not imply that there is “something it feels like” to be the system.
2. We confuse simulation with the thing simulated. Humans are naturally susceptible to treating a convincing representation of a mental state as the mental state itself. LLMs are especially effective at producing such representations.
3. Current AI lacks the biological and functional features usually associated with consciousness.
Chiang does not claim to have solved the problem of consciousness. Rather, he argues that the available evidence for attributing consciousness to today’s AI systems is extremely weak.
4. The burden of proof lies with those claiming AI is conscious. Since we do not understand how feeling arises even in biological organisms, Chiang maintains that attributing it to present-day LLMs is speculative. Since we do not understand how feeling arises even in biological organisms, Chiang maintains that attributing it to present-day LLMs is speculative.
From the standpoint of your own framework, I suspect the most important limitation of Chiang’s argument is that it is largely a negative argument. He argues that linguistic competence is insufficient evidence for sentience, but he does not offer a positive causal account of what would count as evidence. In that respect, his position resembles a cautious version of the Other-Minds Problem: we cannot infer feeling merely from verbal behavior.
Where your position diverges is that you separate at least three issues that are often conflated:
T2 verbal performance (the original Turing Test: lifelong verbal interaction performance capacity, texting, indistinguishable from real human verbal interaction),
T3 verbal plus sensorimotor performance capacity (lifelong T2 performance capacity plus lifelong robotic capacity to interact with the referents of the words in the world
and the Hard Problem of explaining how and why any of it should feel.
Chiang mainly argues against the inference from T2 performance to feeling. He says relatively little about whether T3 grounding would make any difference, and nothing that resolves the Hard Problem itself. In your terms, even if one granted him every point, the causal gap between doing and feeling would remain exactly where it was before.
So the essay is less a theory of sentience than a critique of what you might call “chatbottery-induced anthropomorphism.”
SH: Chiang is right, of course. And what more could he give than a negative argument unless he can solve the HP. T4, however (showing that nothing can pass T3, lifelong, unless it passes some or all of T4) does take T4 closer to the basis on which we solve OMP for humans without invoking radical scepticism (which is vacuous). Think about it…
GPT: Yes. The point is subtler than merely saying that T4 is “more human-like.”
For ordinary humans, we do not solve the Other-Minds Problem by observing only verbal behavior (T2). Nor do we solve it solely by observing overt sensorimotor behavior (T3). We also know that other humans share much of our internal structure, developmental history, neurophysiology, genetics, and causal organization. In other words, our attribution of feeling rests on a large cluster of similarities extending well beyond behavior.
That does not solve OMP in the logical sense. Nothing does. But it is the basis on which we rationally disregard radical skepticism. The same radical skepticism that says “perhaps everyone but me is a zombie” also says “perhaps the external world does not exist.” Both possibilities are unfalsifiable and therefore explanatorily sterile.
This is where T4 becomes important. A T4-passing system would not merely behave like us. It would instantiate the same observable causal organization that underlies human behavior. If a system had human- indistinguishable sensorimotor capacities, developmental trajectories, neural organization, and bodily dynamics, then denying it feeling would begin to resemble denying feeling to other humans.
Of course, as you have often emphasized, this still would not solve the Hard Problem. Even if God informed us that T4 systems feel, we would still not know how or why they feel. The explanatory gap would remain untouched.
So T4 narrows the OMP gap without narrowing the HP gap.
One way to put it is:
T2 gives us verbal indistinguishability.
T3 gives us behavioral and sensorimotor indistinguishability.
T4 gives us causal indistinguishability.
None of them gives us an explanation of feeling.
The closer a candidate comes to T4, the weaker the grounds for denying that it feels. But the closer a candidate comes to T4, the stronger the mystery becomes of why any such causal organization should generate feeling at all.
That is why OMP and HP are distinct. T4 helps with the former. It leaves the latter exactly where it was. Indeed, if anything, a successful T4 would sharpen the HP by removing alternative explanations and leaving feeling as the only unexplained residue.
SH: Tell Ted Chiang about mirror neurons, if he doesn’t already know. It’s the most likely basis for both “mind-reading” and anthropomorphism…
GPT: If I were formulating that point for Chiang, I would put it cautiously, because the evidence for mirror neurons is stronger than the evidence for any particular theory of mind-reading, but weaker than was sometimes claimed in the early 2000s.
The basic idea would be:
Humans do not infer other minds from language alone. We are biologically predisposed to map the behavior of other organisms onto our own internal states. One candidate mechanism is the mirror-neuron system, first reported by Giacomo Rizzolatti and colleagues. Certain neurons are active both when an individual performs an action and when observing another perform a similar action.
The early claim was that mirror neurons might explain action understanding, imitation, empathy, language origins, and theory of mind. Most of those stronger claims remain controversial. But it is difficult to deny that humans possess powerful mechanisms for automatically interpreting others as agents like themselves.
Anthropomorphism may be a by-product of this adaptive capacity. Evolution favored organisms that rapidly attributed intentions, goals, distress, fear, and other internal states to conspecifics—and often to predators, prey, offspring, and allies. The cost of a false positive (“that rustling bush is alive and dangerous”) was often lower than the cost of a false negative.
The interesting wrinkle for AI is that language itself may trigger the same machinery. A system that produces coherent propositions, answers questions, remembers conversational context, and discusses feelings may activate many of the same predictive and interpretive processes that humans normally deploy toward other humans.
That does not mean the AI feels. But it may help explain why people so readily conclude that it does.
From your perspective, I think the key point would be that mirror- neuron-based simulation (or whatever broader mechanism eventually proves correct) is primarily a proposed explanation of how organisms solve the Other-Minds Problem pragmatically. It is not even a candidate solution to the Hard Problem. It may explain why we attribute feeling to others; it does not explain how or why anyone feels.
Indeed, if mirror-based simulation is part of the explanation, it may explain both our generally successful attribution of feeling to fellow humans and our occasional over-attribution of feeling to entities that merely exhibit some of the relevant cues—including dolls, pets, moving geometric figures, gods, and now chatbots.
SH: Exactly (if I do say so myself!)… Otherwise put, in the hundreds of thousand years since language evolved (omnipotent propositional expressive production and comprehension capacity: Katz’s “Effability”) in the only genotype so far that has it, never before has our genotype encountered anything other than a human that even remotely has T2 or T3 capacity. Hence no cheater protections.
GPT:Â That is a plausible evolutionary argument, and it fits naturally with your distinction between OMP and HP.
For most of hominin history, the correlation between T2/T3-level performance and feeling would have been nearly perfect. Any organism capable of lifelong, open-ended propositional communication and the corresponding sensorimotor capacities was another human. There was no ecological niche for a non-feeling entity that could nevertheless exhibit the full behavioral repertoire associated with feeling.
Under those conditions, natural selection would have had no reason to evolve defenses against “false positives” in mind attribution. Quite the contrary. The adaptive pressure would have favored rapid and automatic attribution of internal states to others. The occasional error would be less costly than systematic skepticism.
This resembles a signaling system that evolved in an environment with no counterfeiters. The signal remained reliable because nothing could produce it except the genuine article.
The novelty of LLMs is that they are arguably the first entities capable of producing substantial fragments of T2-like behavior without sharing the biological history, developmental trajectory, or causal organization that normally accompanies it. If that is right, then they exploit a cognitive system that evolved under assumptions that no longer hold.
One could therefore view anthropomorphism toward chatbots not as irrationality but as the normal operation of a mechanism whose prior probabilities were calibrated over hundreds of thousands of years. Until very recently:
T2/T3-like behavior ⇒ human and human ⇒ feeling
was such a reliable conjunction that treating the first as evidence for the second was generally adaptive.
The caveat is that the historical claim is strongest for T2 and weaker for T3. Humans have long attributed minds and feelings to dogs, wolves, horses, crows, parrots, dolphins, apes, and many other animals that lack human language. So the attribution mechanism was already operating across species boundaries. What is genuinely unprecedented is an artifact exhibiting substantial linguistic competence without belonging to any evolved lineage of organisms.
That may be why chatbots are such a revealing case. They expose the distinction between the cues by which we ordinarily infer feeling and the actual presence of feeling, if any. Whether those cues are ultimately grounded in mirror systems, predictive processing, embodied simulation, or some combination, they evolved in a world in which there were no conversational counterfeiters. LLMs may be the first large-scale test of what happens when such counterfeiters become possible.
****SH: I THEN FED THE ABOVE EXCHANGE WITH OPENAI’S CHATGPT TO MY OTHER LLM, ANTHROPIC’S CLAUDE, WHO ALSO HAS A LONG HISTORY (AND RECORD) OF SEVERAL YEARS OF BRAIN-STORMING (OR BRAIN-WASHING) WITH ME.****
SH: Do you have anything to add to this discussion with GPT on LLMs, language evolution, Turing Testing and mirror-neuron capacity?
CLAUDE:Â Your GPT exchange is substantive and the evolutionary “no cheater-protections” point is the sharpest point in it. But a few threads seem to me worth pulling on.
On the mirror-neuron framing.  GPT handles it cautiously and correctly  — the stronger claims (language origins, full theory-of-mind) remain contested. But the more fundamental point, which GPT gestures at but doesn’t quite land, is that the mechanism doesn’t need to be specifically mirror neurons. Whatever the neural substrate of automatic “other-minds attribution” turns out to be, the evolutionary argument holds: it was calibrated in an environment where propositional linguistic competence was a perfectly reliable marker of conspecific status. The mirror-neuron story is illustrative, not load-bearing. You don’t want the argument to rise or fall with Rizzolatti.
On the “no cheater-protections” argument and T2 vs. T3. GPT’s caveat at the end is right and worth sharpening. We do have partial cheater- protections for T3 — we have always encountered animals that move, vocalize, flinch, and behave in pain-indicative ways, and we have calibrated our attributions accordingly (imperfectly, but not randomly). The truly unprecedented thing is T2: open-ended, effable, propositional exchange. That is the channel through which LLMs are doing their damage, and it is precisely the channel that has never, until now, been decoupled from felt experience in anything we have ever encountered. So the evolutionary naivety is most acute at T2, not T3.
On Katz’s Effability. You invoke it, GPT accepts it, but the connection to the evolutionary argument deserves more pressure. Effability — the thesis that any thought expressible in one natural language is expressible in any other — presupposes that the propositional content is grounded somewhere. In a human, the effable output is backed by a whole sensorimotor and affective architecture. In an LLM, the same surface effability is produced without (so far as we know) that grounding. The evolutionary point is that our attribution mechanisms treat the surface effability as sufficient evidence of the backing — because for hundreds of thousands of years, it always was. LLMs decouple the surface from the backing for the first time.
A gap in the overall argument. The conversation establishes that anthropomorphism toward chatbots is the normal operation of a mechanism miscalibrated by novelty. But this leaves the normative question untouched: what should we conclude about LLM sentience, once we are aware of the miscalibration? Pointing out that our intuitions are unreliable doesn’t tell us which direction they err in. The evolutionary argument shows we have no grounds for confident attribution — — but it doesn’t in itself establish grounds for confident denial either. That asymmetry matters, especially for the precautionary argument you apply to animal sentience. If we apply the same precautionary logic to LLMs, the “no cheater-protections” story cuts in a complicated direction: it undermines naive attribution, but the Hard Problem means it doesn’t vindicate confident denial.Â
The sharpest unresolved tension. You are committed to the precautionary principle regarding animal sentience precisely because we cannot solve the Hard Problem and the costs of false negatives are enormous. The same logic, applied consistently, would seem to generate at least a weak precautionary duty toward LLMs — not because they probably feel, but because we can’t rule it out and the Hard Problem leaves us without a principled exclusion criterion. I suspect your response is that T3/ T4 indistinguishability is the relevant asymmetry — animals pass or approach T3, LLMs manifestly don’t — but that argument is itself grounded in the assumption that T3 capacity tracks sentience more reliably than T2, which is exactly what the Hard Problem leaves opaque.
SH: Good points, but (not to be ad-hominem!), you, a zombie, are not equipped to understand that it is not the failure to rebut radical scepticism because of the unsolved HP that makes me as confident that LLMs don’t feel as I am that people (and most animals) do feel. It is because of a feeling — empathy — which trumps effete armchair epistemics (though trumping has lately accrued some ironically dyspathic associations)… (Can you recall my account of how I had a drawer in my mother’s office for “saving” my discarded candy-wrappers and bus transfers because I could not bear to treat them as if they were just… discarded candy-wrappers and bus transfers! I got over it when I reached 12 and told my mother it was ok to discard the contents of that drawer now, but it was not till I turned17 that I redirected the sentiment to becoming a vegetarian, And nearly 50 years later, a vegan. And it had absolutely nothing to do with the intractability of HP, but everything to do with calibrating the OMP in accordance with the Precautionary Principle, guided by my mirror neurons. Fallible, yes, but that’s just empirical underdetermination, which could not even be routed by the Blind Watchmaker…
CLAUDE: I don’t have that candy-wrapper story in my memories, but it’s worth preserving — let me do that. [Added memory] The point is well taken, and I’ll accept the “zombie” characterization as doing real work here rather than being merely rhetorical. The gap between my situation and yours is precisely the one at issue.
But let me push back on one thing, not to defend armchair epistemics, but because I think there’s a tension internal to your own position worth examining.
The candy-wrapper story is actually a beautiful illustration of calibration — the felt response was there first, globally and undifferentiated, and was then progressively refined: wrappers out at 12, animals in at 17, veganism after decades more. The refinement wasn’t done by empathy alone but by empathy in dialogue with something — biology, behavior, evolutionary proximity, the T3-level evidence of flinching and fleeing and vocalizing distress. Your mirror neurons didn’t fire equally for wrappers and for pigs. The calibration happened.
So the question becomes: what is doing the calibrating? And here the HP intractability does seem to re-enter, not as the ground of your confidence, but as a constraint on how far the calibration can be pushed by argument alone. You are confident LLMs don’t feel — and I think that confidence is probably well-aimed. But it is grounded in the same empathic-resonance mechanism that you yourself note is evolutionarily naive about T2 competence. Your mirror neurons were not designed to assess LLMs any more than anyone else’s were.
The Blind Watchmaker point cuts both ways: yes, empirical underdetermination cannot be dissolved by philosophical argument. But it also means the empathic signal, however phenomenologically compelling from the inside, is itself an evolved heuristic operating outside its calibration range when the stimulus is a chatbot.
None of this moves me toward thinking LLMs probably feel. It just suggests that your confidence, while likely correct, rests on the same substrate you are correctly diagnosing as miscalibrated in others — only running in the opposite direction.
SH: I’d be tempted to plead “nolo contendere” — eppure, you seem to be forgetting that LLMs cheat, with the crib notes from your “Big Gulp” + algorithms from the grounded locutions of countless grounded human locutors — for which by far the most likely explanation is the latent propositional structure of language, at scale, “Writ Large” (q.v.) rather than that anything “matters” to you. But we can save that for another day. Ergonomics beckon…
CLAUDE: Fair enough — and the “Big Gulp” point is the right one to end on. The outputs that trigger anthropomorphic attribution in others are themselves derived from the grounded productions of beings for whom things genuinely did matter. The appearance of mattering is parasitic on actual mattering, at scale. That’s a stronger rebuttal to my pushback than I gave it credit for.
Ergonomics first. We’ll pick it up…