The close relationship between language and thought has led to the misconception that strong language skills imply strong cognitive abilities, a fallacy amplified by recent advancements in Large Language Models (LLMs). LLMs' ability to generate fluent text has fueled claims of "sparks of artificial general intelligence." However, this paper challenges the conflation of language and thought by distinguishing between two types of linguistic competence: formal and functional. Formal competence encompasses the knowledge of linguistic rules and patterns, while functional competence involves using language strategically in real-world situations. This distinction is rooted in human neuroscience, where separate neural mechanisms support these two types of competence. The authors argue that a comprehensive evaluation of LLMs must consider both formal and functional competence, and that achieving human-like language proficiency necessitates mastery of both.

The paper reviews existing literature on LLMs, highlighting both their successes and limitations. It references previous work showcasing LLMs' surprising capabilities in tasks requiring complex syntactic understanding and their notable shortcomings in tasks demanding commonsense reasoning, world knowledge, and social cognition. The review underscores the need to move beyond evaluating LLMs solely on their ability to predict the next word in a sequence and instead focus on their broader cognitive capacities.

The paper's methodology centers on a conceptual framework that differentiates formal and functional linguistic competence. It leverages findings from cognitive neuroscience, particularly research on the human brain's language network and its distinctness from other cognitive networks involved in reasoning, world knowledge, and social understanding. The authors use this framework to systematically analyze the capabilities of contemporary LLMs, examining their performance across various tasks that necessitate different types of linguistic competence. The analysis considers specific domains critical for functional competence, including formal reasoning, world knowledge, situation modeling, and social cognition. The paper evaluates LLMs' performance on established benchmarks, considering both their successes and failures across different domains of linguistic competence. It also takes into account factors such as task-specific fine-tuning and the challenges of evaluating complex, closed models. This systematic evaluation allows the authors to gauge the gap between LLMs' formal and functional competence and to identify areas where their performance deviates from human capabilities.

The authors' key finding is a significant disparity between the formal and functional linguistic competence of current LLMs. They find that LLMs have reached near-human levels in formal linguistic competence, demonstrating a mastery of complex grammatical structures, hierarchical relationships, and linguistic abstractions. This success is evident in their performance on various benchmarks that assess proficiency in syntax, morphology, and semantics. However, their functional competence remains inconsistent and often lags behind human capabilities. The authors highlight LLMs' limitations in several key areas: 1. **Formal Reasoning:** LLMs struggle with complex mathematical and logical reasoning tasks, often failing to generalize beyond simple patterns present in their training data. 2. **World Knowledge:** LLMs frequently generate false statements ("hallucinations") and exhibit inconsistencies in their responses. Their commonsense knowledge is limited, particularly when superficial statistical patterns are controlled for. 3. **Situation Modeling:** LLMs struggle to maintain and update information about objects, agents, and events across extended narratives, sometimes referring to non-existent discourse entities. 4. **Social Reasoning:** LLMs' performance on pragmatic tasks, particularly those requiring Theory of Mind, is highly variable. While some fine-tuned models show near-human performance in specific sub-domains, overall results remain uneven and context-dependent. The authors note that even some forms of pragmatic inference that seem easy for LLMs may not be those supported by corresponding human brain networks. The authors' analysis suggests that achieving human-like language use requires not only mastering formal linguistic rules but also integrating diverse cognitive skills and mechanisms supporting functional competence. This finding is strongly supported by neuroscientific evidence that shows a dissociation between the brain networks responsible for language processing and other cognitive processes such as reasoning and social understanding.

The findings of this paper directly address the central research question of how well LLMs model human language and thought. By establishing a clear distinction between formal and functional linguistic competence, the authors demonstrate that current LLMs excel in formal language but significantly lag in real-world language use. This discrepancy highlights the limitations of solely relying on next-word prediction as the primary training objective for LLMs. The results underscore the need for a more comprehensive approach to LLM development, one that considers not just the syntax and semantics of language but also the broader cognitive functions needed for practical language understanding and communication. This integrated approach, reminiscent of the human brain's modular architecture, offers a path toward building more human-like and capable language models.

This paper significantly contributes to the field by highlighting the crucial distinction between formal and functional linguistic competence in LLMs and emphasizing the limitations of current models in achieving genuine human-like language use. The authors' findings suggest that future research should focus on developing more modular architectures that integrate language processing with other cognitive functions. This could involve either explicitly incorporating separate modules for different cognitive abilities or creating conditions that facilitate the emergence of such modularity during the training process. Future research also needs to address challenges in creating benchmarks that cleanly separate formal and functional linguistic competence. In essence, the paper advocates for a shift in LLM development, moving beyond simple scaling up to more nuanced and cognitively informed approaches.

While the paper presents a compelling framework, a limitation lies in the challenges of evaluating LLMs comprehensively. The closed nature of many state-of-the-art LLMs hinders rigorous analysis of their internal mechanisms. Furthermore, the study primarily focuses on English, and future research should explore the generalizability of its findings across different languages. Additionally, the reliance on existing benchmarks, some of which might be susceptible to ‘hacking’ by LLMs using flawed heuristics, warrants consideration. Finally, the paper does not definitively resolve the question of whether functional competence can be fully bootstrapped from language data alone, raising a critical area for future investigation.