Emergence of syntax and word prediction in an artificial neural circuit of the cerebellum

Language comprehension, a complex cognitive function, relies on the interaction between the neocortex and the cerebellum. While the neocortex's role is well-studied, the cerebellum's contribution at the circuit level remains largely unexplored. Current AI models, such as those based on the transformer algorithm, effectively process language but lack biological plausibility. These models often treat sentences as single units rather than sequences of words, unlike the brain's processing mechanism. Therefore, constructing biologically constrained artificial neural networks (ANNs) is crucial to understand the brain's language processing mechanisms. Evidence suggests that the right lateral cerebellum plays a significant role in language, particularly in next-word prediction and grammatical processing. Childhood damage to this region results in more severe and permanent language deficits compared to adult damage, indicating a crucial role in language acquisition and supporting subsequent neocortical acquisition. This study utilizes a biologically-constrained ANN, specifically focusing on the cerebellar circuit, to investigate how these language functions emerge.

Two key language functions linked to the right lateral cerebellum are next-word prediction and grammatical processing (specifically syntactic recognition). Next-word prediction aids faster and more accurate comprehension, particularly in noisy environments, while grammatical processing involves combining word meanings according to grammar. These cerebellar functions reflect broader cerebellar roles: prediction of external events, and rule extraction from sequences of events. Next-word prediction is a language-specific instance of the general prediction function, and syntactic recognition is a language-specific instance of rule extraction. While these functions support various cerebellar cognitive functions, the mechanisms for their realization within the cerebellum’s uniform cytoarchitecture remain unclear. Artificial neural network (ANN) modeling is essential to understand the network dynamics and investigate whether a single cerebellar circuit can support both prediction and rule extraction.

The researchers developed a biologically constrained cerebellar ANN (CANN) model. The CANN comprises three layers: an input layer (granule cells), a middle layer (Purkinje cells), and an output layer (nuclei neurons). It incorporates both a conventional feedforward pathway and a recently identified recurrent pathway, crucial for prediction in the cerebellum. The climbing fiber pathway delivers prediction errors for learning. Words are represented by sparse coding in the input cells, devoid of semantic or grammatical information. The correct answer signal, likewise, contains no syntactic information. The number of output cells mirrors the dimensionality of the input and correct answer signals. The CANN was trained using sentences from classic novels, with synaptic weights updated to minimize prediction error. The model's performance was evaluated based on correct prediction rates and the ability to predict words based on context (including predicting nouns after verbs). To analyze information processing, the researchers examined Purkinje cell activity using principal component analysis (PCA). To assess the role of the recurrent pathway, a version of the CANN with this pathway blocked was also tested. Additionally, three CANN variants were created with additional biological constraints: convergent recurrent pathways, inhibitory-restricted Purkinje-output connections, and excitatory/inhibitory-limited input-Purkinje projections, to examine the robustness of the findings. Finally, a "convergent CANN" was developed with fewer output cells and a modular structure, reflecting the cerebellum's anatomical features. This model utilizes population coding in its output, requiring a compressed word representation.

The CANN successfully learned next-word prediction, achieving significantly higher accuracy than chance. Importantly, the model also spontaneously acquired syntactic recognition in its middle layer, specifically S-V-O (subject-verb-object) information. This syntactic information emerged despite the absence of syntactic information in the input signals or the correct answer. The recurrent pathway proved essential for both prediction and syntactic processing. Furthermore, the three CANN variants with additional biological constraints maintained equivalent language functions. Purkinje cell activity patterns distinctly reflected S, V, and O words, demonstrating the extraction of S-V-O syntactic information. This syntactic information was predominantly found in the Purkinje cells, suggesting this layer is responsible for extracting syntactic information. The analysis showed that the information needed to distinguish between subject, verb and object words were largely aligned between PCA-derived components and SVM-derived components. The convergent CANN model, with its modular structure and population coding, also successfully learned next-word prediction and syntactic information. This version had a slightly lower prediction accuracy overall but performed better in predicting certain word types compared to the non-convergent model. Overall, these results suggest that a single cerebellar circuit architecture can support both next-word prediction and syntactic processing, with the recurrent pathway playing a vital role in both functions.

The CANN model’s success in replicating cerebellar language functions supports the hypothesis that a single circuit underlies both prediction and rule extraction. The model’s consistency with known cerebellar anatomy and physiology, including the recently identified recurrent pathway, further strengthens this conclusion. The recurrent pathway's essential role emphasizes its importance in long-time-step dependent predictions and syntactic processing. The emergence of syntactic processing upstream of predictive output neurons suggests that the cerebellar internal model not only predicts future events but also extracts structural features from past event sequences, potentially a previously overlooked function. This suggests a generalized cerebellar computation underlying various motor and cognitive functions, including sequence processing essential in tasks like tool use. The clinical findings of more severe and permanent language deficits in children with cerebellar damage compared to adults highlight the cerebellum’s critical role in developing neocortical language functions. The CANN’s capacity for independent syntactic information extraction suggests that during development, the cerebellum might provide this information to the neocortex, assisting in the maturation of language functions.

This study presents a biologically constrained ANN model of the cerebellum that successfully replicates next-word prediction and syntactic processing. The findings highlight the potential for a unified computational basis for these functions within a single cerebellar circuit, emphasizing the role of the recurrent pathway. The model provides insights into the distinct yet cooperative roles of the neocortex and cerebellum in language processing, especially during development. Future research should explore the CANN's capacity for other grammatical processes and investigate the potential of the convergent CANN’s novel circuit design for brain-inspired AI applications. The results also suggest that training in word prediction could improve syntactic comprehension, with potential implications for language therapy.

The study relies on an artificial neural network model, which, while biologically constrained, is still a simplification of the complex biological system. The model's reliance on sparse coding for word representation may not fully capture the richness of semantic and grammatical information present in natural language processing. The training data used might influence the results, and the generalizability to other languages or syntactic structures needs further investigation. The model's prediction accuracy, while exceeding chance level, is still relatively low compared to state-of-the-art language models, likely due to the simplified word representations and the inherent limitations of using a simplified biological model.