This essay was originally submitted as the final assessment for ARTS2115: Philosophy of Artificial Intelligence at UNSW in 2025. The assignment prompted us to engage with a major philosophical issue in AI, here I examine how John Searle’s famous Chinese Room argument stands up against the realities of today’s Large Language Models (LLMs), such as GPT-4 and Claude.
John Searle’s Chinese Room argument (1980) presents a criticism of the idea that computers could truly understand language through mere symbol manipulation. Four and a half decades later, Large Language Models (LLMs) like ChatGPT and Claude have achieved incredible capabilities, leading many like myself to question whether his critique still applies to these neural network based systems that operate so differently from the computers or ideas of AI from his time. In this essay, I will argue that while modern LLMs differ significantly from the symbolic representation processing systems Searle envisioned, these differences do not defeat his core claim about understanding. I will begin by explaining Searle’s original argument and its context within symbolic AI, then examine how modern LLMs challenge the three key assumptions underlying Searle’s critique. Despite these differences, I will argue that LLMs still lack the causal grounding necessary for genuine understanding. Finally, I will explore future developments in AI that might better challenge Searle’s argument.
Searle’s argument is based on a scenario in which Searle, who has no understanding of Chinese, is placed in a room. Within this room, Searle receives input as a piece of paper with Chinese symbols written on them. Searle then uses a book that has a list of rules telling Searle which Chinese characters to write down on a new piece of paper based on the previous input. Searle’s point from establishing this scenario is that from an outsider’s perspective, the room may appear to understand and engage in coherent Chinese, however, Searle (who’s inside) doesn’t truly understand or learn any Chinese. Searle emphasizes that this can be extrapolated to any computer or AI system which manipulates symbols (e.g. symbolic LLMs).
The Chinese Room is fundamentally meant to be a critique against strong AI, which is the view that a programmed computer could be capable of thinking and truly understanding. Furthermore, Searle believes that no amount of formal manipulation is sufficient for understanding, therefore strong AI is not possible. I believe it’s worth re-looking at Searle’s argument considering how technology has progressed overtime, especially considering the advent of LLMs and the potential embodiment of them in the future (more on this in the section on the future of AI). The machines Searle considers are based on three fundamental points:
At the center of Searle’s argument is the distinction between syntax and semantics. Syntax refers to the formal structure and rules governing symbol manipulation, whereas semantics refers to meaning and understanding. The Chinese Room illustrates that a system can perfectly execute syntactic operations (following the rulebook) without possessing any semantic understanding of what those symbols mean. This highlights what philosophers like Stevan Harnad (1990) call the “symbol grounding problem,” that being: how can arbitrary symbols ever become meaningful to the system manipulating them (like Chinese characters to the non-Chinese speaker)? Symbolic representations are arbitrary, that is, they do not resemble what they represent, but rather stand in for concepts through convention. The word “cat” tells nothing about what a cat is; it says nothing about it having fur, a tail, or whiskers, it just is the word to represent a cat. With this in mind, I think it’s conceivable why even sophisticated rule-based systems could never achieve genuine understanding in Searle’s view, as they lack the capacity for intentionality that characterizes human thought and representation. I believe that the symbolic AI systems that Searle was criticizing in 1980 were fundamentally different from today’s neural networks, but I will argue that his central ideas remain applicable despite this.
I believe modern LLMs differ significantly from the systems that Searle considered in his original Chinese Room argument. While Searle’s critique targeted AI systems that operated on explicit symbol manipulation through programmed rules, LLMs like GPT-4 or Claude-3.7-Sonnet function in fundamentally different ways. In this section, I will examine how these differences challenge the three key assumptions in Searle’s argument: that AI systems operate on symbolic representations, follow pre-programmed rules, and have access only to syntactic features of language.
Modern LLMs represent language through distributed vector representations rather than discrete symbols. When processing the word “Paris,” an LLM doesn’t simply manipulate a symbol labeled “Paris.” Instead, it processes a high-dimensional vector that captures semantic relationships between “Paris” and other concepts. GPT models, for instance, use embeddings that are extremely high-dimensional (e.g. according to Brown et. al (2020) GPT-3 had 12,288-dimensional embeddings), each encoding information about semantic relationships between words. Within these vectors, relevant information emerges from the statistical patterns captured in the representations. I believe this shift from discrete symbolic manipulation to vector spaces represents a notable shift in how these systems process information, one that Searle did not consider in his original argument.
Searle’s second assumption, that AI systems follow pre-programmed rules, is directly challenged by how modern LLMs operate. Unlike the rule-based systems of the 1980s, the “rules” in neural networks are not explicitly programmed but rather learned through training. The weights that determine how an LLM processes information emerge through backpropagation (when the system learns by calculating gradients and adjusting weights to minimize error, gradually developing its own statistical model of language) during training. This training process differs fundamentally from following a rulebook in the Chinese Room. With this sort of training, I see an approach more similar to how humans learn patterns through exposure rather than following hard-coded instructions. For example, most of our parents told us not to touch the stove when it’s on, but at some point, whether on accident or on purpose, we may have touched the stove and thus created an even more significant understanding of why we shouldn’t touch it when it’s on. The parameters that govern an LLM’s responses aren’t intelligible rules that could be written in a book that we could read, instead, they’re distributed across billions of weights in a neural network architecture, adjusted over time as more information is ingested and learned.
The third assumption in Searle’s argument, that AI systems have access only to syntactic features of language, is also challenged by modern LLMs. Through extensive pre-training on tons of text, these models can detect patterns that go beyond syntax. The attention mechanisms in transformer models allow tokens to “look” at all previous ones, finding those that “interest” it and “attending” to those, enabling the model to find complex contextual relationships in language. This allows LLMs to exhibit behaviors that appear to reflect more semantic-like understanding rather than just simple syntactic manipulation. Despite these significant differences from the systems Searle envisioned, an important question remains: do these differences undermine Searle’s argument about understanding? While LLMs process information in more brain-like ways than symbolic systems, I will argue in the next section that they still face limitations that prevent them from achieving genuine understanding.
Despite modern LLMs using vector representations which make them different from what Searle had envisioned, ultimately, they still face the symbol grounding problem. Harnad (1990) argues, for symbols to have meaning, they must be “grounded” in something non-symbolic, that is, in direct causal interactions with the world they represent. For example, LLMs can process vast amounts of text about Paris like its geography, history, landmarks, and culture, but they can’t see Paris or the Eiffel Tower like you and I could. An LLM’s concept of “Paris” is derived entirely from patterns in text written by humans who have had these experiences. While their vector representations can capture nuanced relationships between concepts, they still lack the direct causal connections to the world that ground meaning for humans.
Consider how an LLM might “know” that fire causes burns. This knowledge exists purely as statistical patterns in its training data. Words like “fire,” “hot,” “burn,” and “pain” may occur together in certain texts during training, but the LLM has never felt heat or experienced pain. It has no causal understanding of why fire leads to burns or what a burn feels like. Human understanding involves constructing causal models of the world (understanding not just that events correlate but why and how they correlate to each other). While LLMs are great at capturing correlations between words and concepts in text, I believe they currently aren’t capable of this deeper causal reasoning.
A famous counter to my argument is the Systems Reply, which contends that while the individual manipulating symbols (Searle) doesn’t understand Chinese, the system as a whole does. When considering this with modern LLMs, this might suggest that even though no individual component genuinely understands language, the neural network, with its billions of parameters, collectively achieves understanding. However, I believe this response misses the fundamental issue at hand in Searle’s argument.
The Systems Reply fails to address the causal grounding problem that affects LLMs regardless of their scale or complexity. Even when considering the entire system, the neural network still derives its “knowledge” from statistical patterns in text without any direct causal interaction with the world those texts describe. While an LLM’s representations are indeed created based on weights from the system as a whole rather than from individual components, these representations remain derived from texts about the world rather than from direct experience of it. The problem isn’t one of complexity or scale but of the system’s fundamental relationship to meaning. As Searle might argue, adding more complexity to a system that manipulates symbols doesn’t suddenly transform syntax into semantics, rather, it produces more sophisticated syntax. Overall, this suggests that even modern LLMs taken as whole systems still lack the intentionality necessary for genuine understanding.
While advancements in AI since Searle developed his argument are impressive, and they certainly contend with some of his assumptions about how AI systems work, his core insight about the gap between symbol manipulation and understanding remains true. They may operate very differently from the symbolic systems Searle envisioned, but they still lack the causal grounding and intentionality that he identified as essential for genuine understanding. In the next section, I will consider future developments in AI that might more substantially challenge Searle’s argument by addressing these limitations.
While current LLMs remain susceptible to Searle’s critique regarding understanding, I believe future developments in AI may pose more challenges to his argument. In this section, I will explore two potential avenues of development that could alter the landscape: embodied AI and continuous learning systems. These approaches might address the symbol grounding problem and causal understanding limitations I identified in the previous section, potentially closing the gap between statistical and genuine understanding.
Embodied AI systems that interact with the physical world could solve the symbol grounding problem that currently affects LLMs. While current LLMs train only on text, embodied AI could perceive and manipulate its environment through sensors and actuators, allowing for new avenues of data ingestion and training. This grounding between the internal representations and physical reality could address Searle’s concerns about intentionality and aboutness.
Consider a robot with vision via cameras and the ability to manipulate objects with arms. Unlike an LLM that merely processes text about fire being hot, this robot could directly experience the relationship between fire and heat through its sensors. It could learn that touching certain objects causes temperature readings to increase and potentially trigger damage warnings in its systems. These direct causal interactions might ground the robot’s concept of “hot” in a way that pure text processing cannot.
Our concepts of “up,” “down,” “heavy,” and “light” derive meaning from our embodied experience of gravity and physical effort. Similarly, an embodied AI might develop conceptual understanding grounded in its physical experiences rather than merely statistical correlations in text. Once again, this grounding could potentially bridge the gap Searle identifies between syntax and semantics.
A second challenge to Searle’s argument might come from AI systems that continuously learn and adjust from their interactions rather than remaining static after initial training. Current LLMs are primarily trained and then deployed with fixed weights, but future systems might continuously update their parameters based on ongoing experience, more closely mirroring human learning.
This continuous learning approach could address several limitations of current systems. First, it would allow AI to adapt to changing environments and novel situations, potentially developing more robust causal models through repeated interaction. Second, it might enable systems to more actively correct errors and misconceptions through feedback, similar to how humans refine their understanding through experience. Third, continuous learning might allow systems to develop a form of personal history or experiential memory that could ground meaning in a way that static training cannot.
The potential for continuous learning also connects back to theories of consciousness such as Integrated Information Theory (IIT). Developed by Giulio Tononi and Christof Koch (2015), IIT suggests that consciousness arises from integrated information, more specifically, a system’s capacity to integrate information across its components in a way that generates a unified experience. A continuously learning AI system might develop increasingly complex information integration as it interacts with its environment, potentially satisfying more of IIT’s criteria for consciousness than current LLMs do.
If consciousness is indeed related to integrated information processing, then continuously learning systems might develop forms of awareness that current (feed-forward) LLMs cannot. This awareness, coupled with embodied interaction, could potentially ground meaning in ways that address Searle’s concerns about understanding and intentionality.
These potential developments do not necessarily refute Searle’s Chinese Room argument entirely, but I believe they certainly could complicate it. If future AI systems ground their representations in direct physical interaction and continuously integrate new information, I see no reason why they wouldn’t develop forms of understanding that go beyond mere symbol manipulation.
Searle might argue that even these advanced systems would still be following computational processes without genuine understanding. However, at some point, the distinction between statistical understanding and actual understanding becomes increasingly difficult to grasp, especially if the understanding is grounded in nearly the same information available to us. If an embodied, continuously learning AI system can ground its concepts in physical reality, adapt to novel situations, and integrate information in consciousness-like ways, I might argue the burden of proof shifts to those claiming it does not “really” understand.
Throughout this essay, I have examined Searle’s Chinese Room argument and its application to modern LLMs. While I have argued that current AI systems still lack genuine understanding in Searle’s sense, I also believe this analysis has revealed an important philosophical implication: understanding should be conceived as a spectrum rather than a binary property. On one end lies pure symbol manipulation without any grounding (the scenario Searle described in his original thought experiment). On the other end is human understanding, with its causal grounding in physical reality, embodied experience, and social context. I think current LLMs exist somewhere in the middle, going beyond simple rule-following because of their vector representations and statistical learning, yet falling short of human understanding due to their lack of causal grounding and intentionality.
While modern LLMs differ significantly from the symbolic AI systems Searle envisioned during his time, these differences do not fundamentally undermine his core claim about understanding. Although LLMs process information using vector representations rather than discrete symbols, learn and adjust parameters rather than follow programmed rules, and access statistical patterns in language, they still lack the causal and contextual grounding necessary for genuine understanding. The symbol grounding problem remains a significant challenge for current AI systems, without direct causal connections to the world they apparently understand, LLMs cannot develop the intentionality or “aboutness” that characterizes human understanding. The Chinese Room argument remains a powerful example of how behavior does not guarantee understanding. Yet, as LLMs and AI develop more and more sophisticated ways of interacting with and representing the world, we may eventually have to accept that they possess genuine understanding.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. arXiv:2005.14165 [cs.CL]. https://doi.org/10.48550/arXiv.2005.14165
Harnad, Stevan (1990). The symbol grounding problem. Physica D 42:335-346.
Searle, J. R. (1980). Minds, brains, and programs. Behavioral and Brain Sciences, 3(3), 417–424. doi:10.1017/S0140525X00005756
Tononi, G., & Koch, C. (2015). Consciousness: here, there and everywhere?. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 370(1668), 20140167. https://doi.org/10.1098/rstb.2014.0167