The Relational Bottleneck as an Inductive Bias for Efficient Abstraction

The paper addresses how humans acquire abstract, relational concepts from sparse data and seeks to reconcile symbolic and connectionist accounts of cognition. Symbolic approaches provide explicit, compositional, variable-binding mechanisms that enable systematic generalization but struggle with tractable program discovery. Connectionist models can learn from data at scale but typically require far more experience than humans and often fail to systematically generalize relations. The authors propose the relational bottleneck as a core inductive bias: constrain processing to relational information between objects while discarding object-specific attributes. This hypothesis is advanced as a way to attain symbolic-like abstraction and data efficiency within end-to-end trainable neural networks, and to provide a candidate principle for human cognitive development and brain function.

The review situates the relational bottleneck within decades of work on symbolic, connectionist, and neuro-symbolic systems. Early neural-symbolic methods targeted variable binding (e.g., binding-by-synchrony, tensor product representations, BoltzCONS) and vector symbolic architectures, generally relying on pre-specified symbolic primitives. More recent hybrids combine deep perception with symbolic program execution, or use deep learning to assemble predefined symbolic primitives in program induction. Other lines integrate symbolic features within differentiable systems: neural production systems, graph neural networks, discrete-valued networks, sparse causal graphs, and tensor-product-enriched transformers. The relational bottleneck differs by making relational representation and variable-binding emerge through architectural constraints (e.g., inner-product-based relation computation) without pre-specified symbols. The paper also contrasts architectures that compute relations via generic neural modules (e.g., Relation Networks) which may overfit to content, versus inner-product-based mechanisms that enforce genuinely relational representations, supporting better out-of-distribution generalization.

The work is a conceptual and architectural review supported by an information-theoretic formalization and empirical demonstrations from prior studies. Using information bottleneck theory, the authors formalize relational tasks as those where a minimal-sufficient compressed representation is purely relational R = {r(x_i, x_j)} over inputs X = (x_1,…,x_N). They advocate architectural inductive biases that enforce such a bottleneck, especially via inner products between learned encoders’ outputs (using separate key/query encoders to allow asymmetry). Three neural architectures implement this principle:

• Emergent Symbol Binding Network (ESBN): Separates perceptual (fillers) and abstract control (roles) pathways with an external memory that binds them. Retrieval uses similarity (inner products) between current perceptual query and stored perceptual keys to fetch abstract values (roles), while preventing perceptual content from entering the control pathway. This enforces abstraction and enables symbol-like role representations.
• Compositional Relation Network (CoRelNet): Encodes objects and computes a full relation matrix via pairwise inner products among object embeddings. Only this relation matrix is passed to a decoder, forming a feedforward relational bottleneck that avoids vanishing gradients associated with recurrent processing.
• Abstractor with relational cross-attention: Forms keys and queries from object embeddings but attends over learned value vectors that reference objects independently of their attributes. The relation matrix (query–key inner products) gates access to these abstract values, enabling asymmetric and multi-dimensional relations via separate projections and multi-head attention. The framework also extends to higher-order relations via hierarchical application (e.g., relational convolutional networks).

• Enforcing a relational bottleneck yields data-efficient learning and systematic generalization of relational patterns across novel objects and contexts.
• ESBN: Rapidly learns identity-rule relations and generalizes to out-of-distribution objects, requiring as few as 4 examples in some tasks; when 95 of 100 objects are withheld during training, ESBN generalizes while multiple baselines (Transformer, NTM, MNM, LSTM, PrediNet, RN) fail. ESBN uses identical role representations across fillers, demonstrating symbol-like abstraction.
• Abstractor: On asymmetric relation tasks (e.g., greater-than/less-than object sorting), learns substantially faster than a Transformer and an ablation replacing relational with standard cross-attention, and generalizes better out of distribution.
• Counting development (give-N): ESBN exhibits a human-like inductive transition around N>5, learning numbers 1–4 gradually and then rapidly generalizing to higher N, outperforming LSTM and Transformer which show linear or exponentially increasing learning curves. This arises from abstract procedures in the control pathway bound to perceptual content via memory.
• Capacity limits: A relational-bottleneck, compositional architecture explains severe capacity limitations (e.g., working memory) as interference from shared compositional codes—the curse of compositionality.
• Neuroscience alignment: The principle aligns with evidence for segregated abstract (parietal/prefrontal) vs perceptual (temporal) systems and suggests variable-binding via episodic memory/hippocampus or alternative rapid-plasticity mechanisms; consistent with neuroimaging and lesion findings on abstract reasoning.
• Formal foundation: Inner-product-based relation functions (with distinct key/query encoders) serve as universal approximators for relations, providing a general yet constrained inductive bias toward purely relational compressed representations.

The relational bottleneck directly targets the central question of how to acquire abstract, symbolic-like structure from limited data within neural networks. By restricting downstream processing to information contained in relations (e.g., similarity via inner products) and isolating abstract values from perceptual content, the reviewed architectures learn procedures and representations that transfer across instances and domains, reconciling symbolic systematicity with connectionist learning. This principle explains human-like rapid relational learning and inductive transitions (e.g., in counting), supports robust out-of-distribution generalization, and offers a mechanistic account of cognitive capacity limits as an inherent trade-off of compositional codes. The approach is consistent with known brain organization and suggests computational roles for memory systems in variable binding. Overall, the framework narrows the gap between symbolic and neural accounts, proposing a unifying inductive bias that fosters efficient abstraction in both artificial and biological cognition.

The paper proposes and formalizes the relational bottleneck as a functional principle and architectural inductive bias that promotes efficient learning of abstract, symbol-like relational structure in neural networks. Through ESBN, CoRelNet, and Abstractor, the framework demonstrates rapid acquisition and generalization of relations, human-like developmental trajectories in counting, and a normative account of capacity limits from compositional interference. The authors suggest future directions: developing graded bottlenecks to capture content effects; integrating attention, memory, and semantic systems; linking to analogical mapping algorithms and group-theoretic generalization; extending to higher-order and multi-dimensional relations; and exploring interactions with education and cultural curricula. The principle may guide both cognitive theory and the design of neuro-symbolic AI systems.

• Scope of empirical demonstrations: Many tasks emphasize same/different relations; broader evaluations across diverse asymmetric and higher-order relations are ongoing, though initial hierarchical relational architectures are promising.
• Architectural constraints: ESBN and CoRelNet (as presented) primarily capture single-dimensional relations; richer multi-dimensional and asymmetric relations require extensions (e.g., Abstractor’s separate key/query projections and multi-head attention).
• Biological implementation: The exact neural mechanisms for variable binding and relational isolation remain unresolved (e.g., roles of hippocampus, prefrontal cortex, cerebellum, and attention/gating), and lesion evidence is mixed.
• Mixture of abstraction and content: Human reasoning often involves both relational and non-relational factors; a graded bottleneck that admits controlled content leakage is proposed but not yet fully developed.
• Generalization guarantees: While inner-product bottlenecks reduce overfitting to content, formal guarantees of out-of-distribution generalization across all task classes are not established.
• Data and benchmarks: Comparative results rely on specific datasets and training regimes; broader, standardized benchmarks for relational abstraction would strengthen conclusions.