A framework for the emergence and analysis of language in social learning agents

The study investigates how social communication shapes task-relevant internal representations and shared abstractions that enable generalization in cooperative agents. Building on language games and emergent communication literature, the authors hypothesize that communication pressures induce neural/representation systems to form abstractions that are both individually useful and optimized for transmission. They frame language as a low-dimensional code for high-dimensional task variables and ask: (i) how agents abstract environmental variables, (ii) how these abstractions map into a common, shareable continuous language, and (iii) how message-space structure affects performance and generalization. The context spans cognitive neuroscience and multi-agent RL, emphasizing features such as interchangeability, total feedback, and productivity. The purpose is to construct a tractable framework connecting internal RL-derived representations with emergent continuous communication and to quantify structure, information content, and utility of the resulting message space.

The work situates itself in: (1) classic studies on language emergence via language games and artificial evolution; (2) deep learning–based multi-agent communication, including discrete-message referential games, translation, and policy coordination; (3) pragmatic and methodological critiques of emergent communication metrics; and (4) continuous embedding approaches and animal communication (e.g., waggle dance) that compress continuous environments into succinct message spaces. Prior frameworks often focus on performance outcomes rather than the nature of shared representations. Compared with end-to-end approaches (e.g., independent Q-learning with shared gradients), this work separates teacher training from language and student learning, employs a continuous, sparse message space, and analyzes topographic similarity and entropy of meanings-to-messages. It also contrasts supervised or symbolic methods by leveraging unsupervised, RL-derived task abstractions.

Architecture: Two agents (teacher and student) communicate via a continuous message. Teachers are trained with deep Q-learning to produce Q(s,a) for grid-world navigation tasks. Task information (Q-matrices) is compressed by a sparse autoencoder (SAE) into a K-dimensional real-valued message m, which is given to the student. The decoder reconstructs Q from m. Sparsity is encouraged via L1 regularization on m. Communication protocols: (i) No feedback—train SAE to reconstruct teacher Q; train student to use messages; (ii) With feedback—augment SAE loss with a term optimizing student goal-reaching performance, allowing bi-directional utility shaping of the message space; (iii) Closing-the-loop—encode the student’s own Q-matrices with the fixed, feedback-trained SAE and feed messages back to the student to assess information erosion and robustness. Tasks: Grid-world mazes (4×4 effective states within 6×6 including boundary walls); agents start bottom-left; four actions; rewards: step −0.1, wall −0.5, goal +2. Training uses mazes with 0 or 1 interior wall (16 worlds; total 225 tasks). Testing uses 2-wall mazes (101 configurations; 13 goals each; 1313 tasks), excluding disconnected layouts. Teacher training: DQN minimizes MSE to satisfy Bellman optimality with γ=0.99. Training uses a mixture of long-term distinct transitions and re-weighted short-term transitions. Networks: Teacher and student are MLPs with ReLU activations; student input includes message (K=5). SAE uses two conv layers encoder, a linear bottleneck (message), then linear and deconv layers to reconstruct spatial Q. Hyperparameters and layer sizes detailed in Tables 3–4. Losses: SAE loss without feedback: L_SAE = (1−κ) ||Q−Q_hat||_2 + κ ||m||_1. With feedback: add goal-finding loss that estimates probability of reaching goal within optimal steps via softmax-derived action probabilities and state occupancies; includes student Q regularization term scaled by γ (Eq. 5). κ controls reconstruction vs sparsity trade-off. Analyses: - PCA on message spaces to assess variance alignment with wall layouts vs goal locations. - Analysis of variance (ANOVA-style variances within/between groups) and F-tests comparing group separations by walls and by goals. - Topographic similarity: compare pairwise distances in message space with distances in task labels (weighted wall/goal differences) and with Frobenius distances between Q-matrices. - Shannon entropy of discretized first two PCs for teacher outputs, messages, student outputs across bin sizes. - Performance metrics: task solve rate defined as achieving goal within k_opt (shortest path) + up to 25 steps; comparisons among informed student (correct message), misinformed (random task message), smart/random walkers. - Statistical tests: one-sided t-tests vs smart random walker; two-sided t-tests vs misinformed; Bonferroni corrections applied. - Language filtering for closing-the-loop: exclude ~30% inefficient languages where informed underperforms misinformed/random on trained tasks.

- Latent structure without feedback: Messages cluster primarily by wall configurations; goal locations are secondary. PCA shows discrete groupings by maze, stratified by goal. ANOVA (Table 1): grouping by walls yields Var_within=2.88, Var_between=20.18, β=0.875, F=97.54 (significant); grouping by goals: Var_within=19.99, Var_between=3.07, β=0.133, F=2.14 (not significant at p=0.05). - With student feedback: Both walls and goals shape the message space more evenly; ANOVA (Table 1): by walls Var_within=20.06, Var_between=38.07, β=0.655, F=26.44 (significant); by goals Var_within=38.26, Var_between=19.87, β=0.342, F=7.24 (significant). Feedback increases information alignment with task-relevant features. - Topographic similarity: Positive slopes between distances in message space and (i) label-space distances (weighted wall/goal differences) and (ii) teacher Q-matrix distances. Feedback-trained languages exhibit higher slopes (greater compositionality) than no-feedback languages. - Entropy: Information decreases from teacher → messages → student. Feedback retains more information than no-feedback. Removing reconstruction loss (sparsity + performance only) further reduces entropy, with messages and student outputs collapsing to similar distributions (efficiency pressure). - Reconstruction vs sparsity: Adding student feedback lowers reconstruction error, but increases message magnitude (reduced sparsity); overall compound loss remains comparable across conditions. - Student performance and generalization: Informed students outperform misinformed and random walkers on trained goals in trained mazes; best generalization to unseen goals occurs with checkerboard training patterns; other training subsets show little improvement over random on unseen goals. Introducing new wall configurations reduces absolute performance but preserves informed > baselines. Students trained to interpret frozen feedback languages generalize better than those trained on frozen no-feedback languages. - Closing the loop (student-encoded messages): Student-generated messages exhibit variance dominated by a single component aligned with goal/initial action; wall information diminishes. Performance degrades relative to teacher-encoded messages, especially on unseen goals; informed still outperforms misinformed, indicating residual useful information. ANOVA on student messages (Table 2): by walls Var_within=1583, Var_between=54.5, β=0.033, F=0.48 (ns); by goals Var_within=367, Var_between=1270, β=0.776, F=48.15 (significant).

Findings support the hypothesis that social communication pressures shape shared, low-dimensional representations that enhance task performance and compositionality. Without feedback, the emergent language mirrors reconstruction priorities (maze structure), not necessarily student utility. Incorporating student reward reshapes the message space toward goal-relevant variability, improving topographic similarity, information retention, and reconstruction despite reduced sparsity—analogous to total feedback in natural language, where messages adapt to listener and context. Generalization benefits arise when unseen tasks can be composed/interpolated from known ones (checkerboard), highlighting the need for structured coverage of the task manifold. Closing-the-loop reveals that information can erode when agents self-encode learned policies, concentrating on goal/initial-action cues at the expense of environmental structure; yet, even degraded messages provide advantages over random communication. Overall, the work bridges internal RL abstractions and emergent continuous communication, showing that bidirectional optimization is key for learnable, generalizable shared codes.

The study introduces a tractable teacher–language–student framework where RL-derived task solutions are compressed into sparse continuous messages. It disentangles how internal task abstractions map into a shared message space and how that structure affects performance and generalization. Contributions include: (i) identifying structural features of low-dimensional embeddings linked to success and generalization; (ii) demonstrating that reward-based feedback increases compositionality, information retention, and reconstruction fidelity; and (iii) analyzing symmetric communication by feeding student-encoded messages back to the student, revealing information erosion patterns and retained utility. Future directions: extend to sequential compositional messages and discrete protocols; explore alternative architectures (RNNs/transformers) and community/channel structures; reverse teacher–student roles; integrate richer tasks and real biological data; and study effects of social graph structure on language emergence.

- Task simplicity: 4×4 grid-world mazes may limit ecological validity and the diversity of abstractions. - Continuous message space without explicit syntax/grammar; no sequential composition considered in this work. - Generalization limited when unseen goals are outside the convex hull of trained tasks; performance depends strongly on training coverage (e.g., checkerboard). - Trade-off between reconstruction and sparsity: feedback improves reconstruction at the cost of reduced sparsity. - Closing-the-loop reveals information degradation and dominance of goal/initial-action features, with reduced world-structure encoding. - Language filtering was required to exclude inefficient emergent languages in the closing-the-loop analysis (~30%), indicating sensitivity to training dynamics and potential instability of learned codes. - Affiliation and broader multi-task, multi-agent scalability not empirically explored beyond the presented toy domain.