Modern artificial intelligence chatbots can write essays and hold conversations that closely mimic human communication, raising the question of whether they process language like the human brain. A new neuroimaging study published in eLife suggests that when sentence meaning depends on word order rather than vocabulary alone, large language models organize meaning quite differently from the brain. Instead, human brain activity aligns more closely with computational models that explicitly track grammatical roles, such as who is performing an action and who is receiving it.
When people read a sentence, they do not simply add up the definitions of individual words. The brain also tracks syntax and semantic roles, which are the structural relationships specifying who did what to whom, allowing a reader to distinguish “the dog bit the man” from “the man bit the dog” despite their identical vocabulary.
Neuroscientists have found that specific brain regions help keep these roles straight. In a 2015 brain imaging study, researchers showed that neighboring subregions of the left temporal cortex, an area on the side of the brain involved in language comprehension, separately track the agent performing an action and the recipient of that action.
Computer scientists have approached language differently by building large language models, the artificial intelligence systems behind tools like ChatGPT, Claude, and Llama. Built on a neural network design called a transformer, these models learn by predicting missing words across massive text collections and represent word and sentence meanings as long lists of numbers called vectors or embeddings.
Because these vectors can be compared to brain activity, researchers began testing whether artificial networks and brains process meaning the same way. For instance, a brain-decoding study of people listening to stories paired functional magnetic resonance imaging (fMRI) with a language model to reconstruct the gist of narratives from brain scans, and a 2021 modeling and neuroimaging study found that transformer vectors predicted nearly all of the brain activity in response to sentences that could, in principle, be predicted.
Skeptics, however, have questioned whether high alignment scores reflect genuine grammatical understanding. Using datasets from the 2021 study, the authors of a 2024 reanalysis (posted as a preprint on arXiv that has not been peer reviewed) argued that much of this apparent similarity came from how data was split during testing and from basic features like the tendency of each brain measurement to resemble the one just before it.
A major challenge is that most neuroimaging studies use natural stories where different meanings use different words, allowing a model to match brain activity simply by recognizing vocabulary or resolving words with multiple meanings. To isolate sentence structure from vocabulary, James Fodor conducted the new study at the University of Melbourne with co-authors Carsten Murawski and Hitotsubashi University’s Shinsuke Suzuki. Fodor is now a postdoctoral research associate in the Department of Machine Learning at Carnegie Mellon University.
“Large Language Models (LLMs) such as Claude and ChatGPT are now able to produce lengthy grammatically correct natural language responses to all sorts of queries,” Fodor told PsyPost. “But does that mean these models truly understand language in a similar way to humans? Or are they more like sophisticated artificial parrots, combining memorized phrases in a way that mimics human language but without comprehending the underlying meaning? Our goal was to make progress in answering this fundamental question.”
The researchers handcrafted 108 single-clause English sentences divided into six subsets of 18 sentences each. Each subset started with a base event, such as “the cameraman brought the equipment to the director,” and altered it by adding adjectives, time or location modifiers, or substituting a noun or verb.
Central to the design were role-swapped sentences where nouns traded grammatical positions while keeping the words nearly identical, such as “the director brought the cameraman to the equipment.” Combining all 108 sentences yielded 5,778 unique pairs, including 918 pairs within the same subsets that shared much of their vocabulary, so that telling them apart depended more on structure.
The team evaluated four main modeling approaches, preregistering their primary predictions and posting their data and code publicly. Two approaches used numerical vectors: a baseline word-averaging model that ignored word order, and a transformer category averaging five models (ERNIE 2.0, AMRBart, SentBERT, DefSent, and OpenAI embeddings).
The other two approaches explicitly encoded structure. A graph-based model (Abstract Meaning Representation) mapped each sentence as a web of concepts and compared shared three-part branches, while a hybrid model, developed by two of the same researchers in a 2024 study, used GPT-4 to tag words by eight semantic roles (such as verb, agent, patient, time, and location), built separate word vectors for each role, and applied fixed role weights carried over from that earlier work.
For the neuroimaging experiment, 39 native English speakers (ages 18 to 40, average age 22) were recruited at the University of Melbourne. After excluding nine people for head motion or for scoring below 70 percent on attention checks, a threshold lowered from the preregistered 75 percent that added one participant, 30 participants remained. Inside a 7-Tesla fMRI scanner, which measures blood flow changes across tiny three-dimensional cubes of brain tissue called voxels, participants read all 108 sentences four times (2 to 7 seconds per sentence) and answered 40 comprehension questions.
The researchers analyzed brain activity from the middle three seconds of each trial rather than the preregistered final three seconds; an analysis of the final window gave similar but weaker results. They focused on voxels that responded consistently within the brain’s language network and across the cortex outside early visual areas. Using representational similarity analysis, they calculated how similar the brain’s activity patterns were for every sentence pair and tested whether each model ranked those pairs in the same order, controlling for sentence length differences.
Across all 5,778 sentence pairs, where most pairs used different vocabulary, all four model types positively aligned with language network activity. In individual-level analyses across all pairs, the difference between the word-averaging and transformer models was not statistically significant, while the hybrid model held a slight edge and the graph-based model aligned best.
On the 918 sentence pairs within the same vocabulary subsets, the results shifted sharply and matched the researchers’ preregistered order of performance. The word-averaging model showed a strong negative relationship with language network activity because it rated role-swapped sentences as nearly identical while the brain treated them as distinct. Although the transformer models beat the word-averaging baseline, they still showed an overall negative relationship with brain activity.
By contrast, both the hybrid and graph-based models achieved a modest positive alignment with brain activity on these same-vocabulary pairs (the difference between the two was not statistically significant). Similarity plots showed that transformers treated role-swapped sentences as roughly as similar as sentences that merely added a time or location modifier, whereas the hybrid and graph models assigned lower similarity to role-swapped pairs, matching the brain. Because the hybrid model was built to be highly sensitive to role swaps, the authors emphasize that their main takeaway is not that this specific model is superior overall, but that word-averaging and transformer representations are not sensitive enough to who-did-what roles.
“We found evidence that LLMs process language differently to humans,” Fodor explained. “This suggests that they may not ‘understand’ language in a fully human-like way.”
In absolute terms these correlations were small, around -0.05 for the transformer average and 0.07 for the hybrid model on the same-vocabulary pairs, as is typical of noisy fMRI data, but the pattern held consistently across participants.
“We found quite large effects that were consistent across models and participants, showing that LLMs process language in systematically different ways to humans,” he said. “However, whether this has practical implications for human use of LLMs in real-world tasks was beyond the scope of our study.”
Examining each layer of the Llama 3 transformer showed a slight positive relationship with brain activity only in layers 2 and 3, and negative relationships from layer 4 onward. Controlling for visual cortex activity caused even those early-layer correlations to drop sharply, which the authors read as a sign that they reflected superficial visual features like text length.
Within the language network, alignment with the structured models was strongest in temporal regions, though every subregion, including frontal ones, responded to both vocabulary and structure. Alignment also extended beyond the language network into areas such as the default mode network. On the same-vocabulary pairs, the graph model’s strongest alignment was in the visual cortex, which a peer reviewer flagged as a possible confound with visual features of the text.
The scans also showed that pairs of relatively long sentences produced similar brain activity across multiple areas regardless of meaning, which the authors suggest may reflect heavier memory load, deeper processing, or shared grammatical roles. Even after controlling for this minimum-length effect and visual cortex activity, the hybrid model maintained a positive alignment with brain activity on the same-vocabulary pairs, while the transformer and word-averaging models remained negatively aligned.
In a complementary online study on Prolific, 502 native English speakers from Australia and the United States (average age 30) rated subsets of the sentence pairs on a 1-to-7 similarity scale, with 486 participants retained after attention checks. Across all 5,778 pairs, the transformer and word-averaging models aligned best with human ratings, but on the 918 same-vocabulary pairs, the word-averaging model fell to last place while the hybrid model aligned most closely with human judgments, followed by the transformers and the graph model. Even so, human raters judged the role-swapped pairs as more similar than the hybrid model did, falling between the transformer and hybrid models.
In an exploratory analysis that was not preregistered, the researchers prompted GPT-4 directly via text to score each pair’s similarity from 0 to 1, finding that its direct ratings matched human judgments better than any model’s internal embeddings did. According to the authors, this indicates that while large language models can use grammar to answer questions about meaning, their internal vector representations do not organize syntax and semantics in a brain-like way.
“Our research provides one example of an important point in our age of AI,” Fodor said. “Just because a computer model can perform a task to a human level, it doesn’t follow that the model is doing so in a human-like way. This can lead to unexpected behavior or failure modes when the model is deployed outside of the context in which it was trained.”
These findings stand in tension with an earlier intracranial recording study of people engaged in unscripted conversation, which found that frontal and temporal neural activity aligned more strongly with higher-level semantic layers of language models than with lower-level word representations, though that study examined spontaneous dialogue rather than controlled sentences isolating word order.
At the same time, the results align with an earlier neuroimaging study of listeners hearing word sequences, which found that the brain tracked hierarchical levels of words, phrases, and sentences without statistical cues. Although the new study compared fMRI patterns during silent reading against computational models rather than tracking spoken phrases in real time, both indicate that grammatical structure is central to human language processing.
As with all research, there are some caveats to consider, and while eLife‘s editorial assessment rated the evidence as solid, it noted that the stimuli were highly controlled. The experiment used 108 single-clause sentences with repetitive structures rather than natural dialogue. Across all 5,778 pairs, where vocabulary varied naturally, transformers aligned best with human ratings, and some individual transformers matched brain activity better than the hybrid model. On brain data, however, every transformer fell below both structured models once visual cortex activity was controlled for.
The transformers tested were mostly systems built to produce sentence embeddings, plus Llama 3; the internal workings of chatbots such as GPT-4 or Claude were not analyzed. In addition, the online human ratings showed almost no correlation with the fMRI patterns on the same-vocabulary subsets, which the authors suggest may reflect differences between actively comparing sentence pairs side by side and passively reading single sentences.
“Our results do not show that LLMs cannot produce relevant or meaningful language, only that they likely do not understand such language in the way humans do,” Fodor noted. “This arises from differences in the way LLMs are trained and the tasks they are designed to perform. Whether such differences lead to practically important limitations for LLMs and their capabilities is a subject for further research.”
The fMRI analysis also averaged brain activity over a single three-second window per sentence, simplifying the dynamic, word-by-word process through which the brain builds meaning over time. Comparing vector and graph models required different mathematical similarity metrics that can affect model fit, and the authors note that future studies will need to link model representations with the real-time cascade of neural activity as sentences unfold.
“This research dealt with single sentences with relatively simple structure,” Fodor added. “In future work we would like to consider how LLMs represent more complex sentences and narrative structures. This is considerably more difficult to study and so is still relatively unexplored in existing research.”
The study, “Large language models poorly capture how word order shapes representations of sentence meaning in the human brain,” was authored by James Fodor, Carsten Murawski, and Shinsuke Suzuki.