Traditional psycholinguistic models explain human language using interpretable models that combine symbolic elements with rule-based operations. However, the recent emergence of autoregressive deep language models (DLMs), trained on vast amounts of real-world text data, has challenged this paradigm. These DLMs, exemplified by GPT-2, excel at next-word prediction, generating contextually appropriate and well-formed text without explicit knowledge of grammatical rules or parts of speech. Instead, they learn to encode word sequences into numerical vectors (contextual embeddings) and decode the next word based on these embeddings. While DLMs are highly effective, their relationship to human brain language processing remained unclear. Prior research used language models to extract semantic representations from brain activity but did not consider DLMs as cognitive models. This study directly investigates the hypothesis that the human brain and autoregressive DLMs share core computational principles in processing natural language.

Past research has leveraged language models and machine learning techniques to explore semantic representations in the brain. Studies have examined the neural correlates of semantic meaning and how these representations relate to computational models. However, these studies largely did not treat autoregressive DLMs as cognitive models for the brain's language processing mechanisms. The current theoretical perspectives argue that there are significant underlying connections between the computational mechanisms of DLMs and the human brain's language processing. This study builds upon and extends this theoretical work, providing empirical evidence to support the proposed relationship.

This study employed a multi-faceted methodology combining behavioral experiments and electrocorticography (ECOG) recordings. In the behavioral experiment, 50 participants were tasked with predicting the next word in a 30-minute podcast transcript using a sliding window paradigm. This provided a direct measure of human next-word prediction accuracy in a natural context. The participants' predictions were then compared with predictions from GPT-2, a state-of-the-art autoregressive DLM. For the ECOG experiment, nine epilepsy patients with implanted electrodes listened to the same podcast while their brain activity was recorded. High-frequency broadband power (70-200 Hz) was extracted from the ECOG data. Linear encoding models were used to predict brain activity from word embeddings (GloVe, word2vec, and GPT-2 contextual embeddings) at various lags relative to word onset, allowing the researchers to assess whether neural activity reflected next-word predictions before the word was even heard. A deep convolutional neural network was used for decoding, estimating the ability to reconstruct word embeddings from neural activity. GPT-2's internal confidence (entropy) and surprise (cross-entropy) measures were employed to model the relationship between pre-onset predictions and post-onset surprise signals in the brain.

The study's key findings strongly support the hypothesis that the human brain and autoregressive DLMs share fundamental computational principles. First, both humans and the DLM exhibit continuous context-dependent next-word prediction before word onset. Behavioral results showed that participants could predict upcoming words with considerably more accuracy than chance, and this human prediction accuracy closely matched GPT-2's predictions. Neurophysiologically, significant correlations were observed between neural activity and upcoming words up to 800 ms before word onset, providing direct evidence for this pre-onset prediction. Second, both the brain and DLM utilize pre-onset predictions to calculate post-onset surprise (prediction error). Increased neural activity for unpredictable words, 400 ms after word onset, matched the DLM's surprise calculations. Third, both the brain and the DLM represent words using contextual embeddings. Contextual embeddings from GPT-2 were shown to be superior to static embeddings (GloVe, word2vec) in predicting neural responses, demonstrating the importance of context in both systems. Finally, decoding analyses using a deep convolutional neural network showed that contextual embeddings significantly improved the classification accuracy of word identity from neural activity, both before and after word onset. This suggests that the brain uses contextual embeddings to represent both the context and upcoming words.

The findings provide strong empirical support for the idea that autoregressive DLMs provide a biologically feasible computational framework for understanding language neural processing. The shared computational principles suggest that the brain's language processing system may fundamentally operate through continuous prediction and error minimization, similar to how these DLMs are trained. The superior performance of contextual embeddings in both encoding and decoding analyses highlights the crucial role of context in representing word meaning in the brain. This challenges traditional linguistic models that often focus on isolated words and syntactic rules. The study directly demonstrates the brain's capacity for continuous, spontaneous next-word prediction in a natural context, a capacity previously supported primarily by indirect evidence. The results also highlight the power of utilizing DLMs as cognitive models to bridge the gap between computational and neural models of language. The alignment of pre-onset predictions with subsequent surprise signals provides a novel way to understand how the brain integrates prediction and error signals during language comprehension.

This study offers compelling evidence for three shared core computational principles between autoregressive DLMs and the human brain in processing natural language: continuous next-word prediction, error-driven surprise calculation, and contextual representation using embeddings. While DLMs and the brain may differ in their underlying mechanisms, these shared principles suggest a novel and biologically plausible computational framework for studying the neural basis of language. Future studies could explore the role of these predictive signals in language acquisition and examine other potential objectives the brain uses to learn language.

The study's reliance on ECOG data from a small number of epilepsy patients limits the generalizability of the findings to the wider population. The invasive nature of ECOG may also influence brain activity, potentially impacting the results. Additionally, the use of GPT-2, while a state-of-the-art model, represents only one architecture among many DLMs. The study's findings may not generalize equally well to other architectures. Further investigation is needed to clarify the extent to which other language models can provide similar insights into neural language processing.