Human cognition exhibits a remarkable ability to form abstract concepts from limited experiences, a capacity that has long intrigued cognitive scientists and philosophers. This ability to transcend concrete details and grasp general principles has been a central topic of debate between two dominant modeling approaches: symbolic and connectionist models. Symbolic models, rooted in the idea of manipulating abstract symbols, excel at representing and reasoning with abstract concepts. However, they often struggle with the learning process, requiring complex search methods to discover the symbolic representations from raw data. Connectionist models, based on neural networks, effectively learn from data through training. However, these models typically require massive datasets to achieve human-level performance in abstract reasoning tasks, failing to capture the data efficiency of human learning. This paper introduces the concept of a "relational bottleneck", a novel reconciliation between these two approaches. This principle suggests that an inductive bias—a constraint on information processing—that focuses neural networks on relationships between inputs, rather than individual input attributes, enables data-efficient abstraction. By emphasizing relations, the relational bottleneck encourages the emergence of abstract, symbol-like mechanisms within neural networks. The authors propose that this principle offers a significant advancement in understanding human cognition and the creation of more powerful artificial learning systems. The review will examine information-theoretic formulations and analyze neural network architectures that embody the relational bottleneck, demonstrating their efficiency and capacity for systematic generalization.

The paper extensively reviews existing literature on abstract concept acquisition, highlighting the limitations of both symbolic and connectionist approaches. Symbolic approaches, while capable of representing abstract concepts through program induction, often face computational challenges in learning such programs from data. The authors mention several prominent examples of program induction models and their limitations in capturing the complexity and richness of human natural concepts. Connectionist models, particularly large language models, demonstrate impressive abilities in certain abstract tasks, but their dependence on massive training datasets stands in stark contrast to human learning efficiency. The review discusses various neuro-symbolic approaches that attempt to bridge the gap between symbolic and connectionist models, including binding-by-synchrony, tensor product variable-binding, and BoltzCONS, along with more recent vector symbolic architectures and deep learning combined with symbolic programs. These existing hybrid systems either rely on pre-specified symbolic primitives or necessitate separate symbolic processing stages. The paper positions the relational bottleneck as a different approach, integrating key aspects of symbolic computing (variable-binding and relational representations) into fully differentiable neural systems that can be trained end-to-end.

The core of the paper's methodology is a conceptual and computational exploration of the "relational bottleneck." This is defined as a mechanism restricting information flow to only relational aspects of inputs, discarding object-level features. The authors formalize this concept using information bottleneck theory, outlining the trade-off between compression and retaining relevant information. The information bottleneck objective function is presented to mathematically capture this trade-off. They then present three neural network architectures that implement this relational bottleneck through architectural inductive biases: 1) Emergent Symbol Binding Network (ESBN): This architecture separates perceptual inputs (fillers) from abstract representations (roles), mediating their interaction solely through similarity comparisons. The network learns to represent roles and fillers separately, facilitating abstract representation learning and rapid generalization to out-of-distribution inputs. 2) Compositional Relation Network (CoRelNet): This model directly computes a relation matrix from object embeddings, encapsulating relational information and passing it to a decoder network. The use of inner products ensures that only relational information is processed, allowing for parallel computation and rapid learning. 3) Abstractor: This architecture utilizes a novel relational cross-attention mechanism within a transformer framework, further emphasizing the relational bottleneck by using separate embeddings for keys, queries, and values. The learned values are independent of perceptual attributes, serving as learned symbols. The paper extensively compares these models to related architectures lacking a relational bottleneck, such as the Relation Net and standard transformers, demonstrating that the relational bottleneck is crucial for data efficiency and generalization. The authors also discuss how these architectures can be extended to model more complex relations, including asymmetric and higher-order relations, by employing recursive application of the bottleneck or using separate key and query projections.

The paper presents several key findings, all supporting the efficacy of the relational bottleneck as an inductive bias for efficient abstraction learning. First, the proposed neural network architectures—ESBN, CoRelNet, and Abstractor—consistently outperform related models that lack the relational bottleneck in various relational tasks. These architectures demonstrate significantly improved data efficiency, learning relational patterns from far fewer examples than traditional neural networks. This superior data efficiency is attributed to the constraint imposed by the relational bottleneck, focusing the network on relational patterns and preventing overfitting to irrelevant perceptual details. Second, the models show enhanced out-of-distribution generalization, successfully applying learned relational patterns to novel objects or situations. This systematic generalization underscores the abstract nature of the learned representations. Third, the ESBN is shown to model the development of counting in children, exhibiting a human-like inductive transition in learning to count. This aligns with the theoretical claim that the relational bottleneck facilitates a human-like developmental trajectory for abstract concept learning. Fourth, the paper explores how the relational bottleneck contributes to understanding cognitive capacity limits. The authors argue that the compositional nature of representations, often considered a strength of human cognition, can lead to capacity limitations due to interference between shared representations. This model offers a novel explanation for phenomena like working memory capacity limits. Finally, the paper explores potential neural mechanisms underlying the relational bottleneck, suggesting that the separation of abstract and perceptual processing might be implemented in the brain through distinct neocortical systems (parietal and temporal cortices) and their interaction via the episodic memory system (hippocampus). However, they acknowledge that other brain areas, such as the cerebellum and prefrontal cortex, might also contribute.

The findings of this paper strongly support the hypothesis that the relational bottleneck serves as a crucial inductive bias for efficient abstraction learning in both artificial and biological systems. The consistent outperformance of models incorporating this bias demonstrates its efficacy in promoting data efficiency and generalization. The ability of the ESBN to replicate the developmental trajectory of human counting abilities adds further credence to this hypothesis, suggesting that the relational bottleneck may be a fundamental mechanism underlying human cognitive development. The link between compositionality and capacity limitations presents a novel theoretical perspective on a long-standing problem in cognitive science. The proposed neural mechanisms provide a potential framework for understanding the biological implementation of this principle, though further research is needed to validate this claim. The results have implications for both cognitive science and artificial intelligence, suggesting that incorporating relational bottleneck principles could lead to the development of more efficient and robust artificial learning systems.

This paper introduces the relational bottleneck as a novel principle for explaining data-efficient abstraction in human cognition and artificial intelligence. Three distinct neural architectures implementing this principle demonstrate superior performance in relational learning tasks compared to related models. The implications of this work extend to cognitive development, capacity limits, and the potential neural substrates of abstract thought. Future research directions include exploring graded versions of the relational bottleneck, applying the principle to symbolic models, integrating other cognitive processes, and further investigating the neural implementation of the relational bottleneck.

While the paper presents compelling evidence for the relational bottleneck principle, certain limitations should be noted. The proposed neural architectures are primarily tested on relatively simple relational tasks. Further research is needed to assess their performance on more complex and naturalistic tasks. The proposed neural mechanisms are speculative, necessitating further neuroscientific investigation to validate the suggested brain regions and processes involved. The paper primarily focuses on relational aspects of cognition, potentially neglecting other important factors influencing human abstraction.