Neural representations of situations and mental states are composed of sums of representations of the actions they afford

Predicting others' behavior requires understanding the interplay between internal mental states (moods, emotions, intentions) and external situations (environmental and social constraints). This study investigates whether the brain represents situations and mental states as weighted sums of their associated actions, drawing upon principles of ecological psychology and predictive coding. Ecological psychology emphasizes the role of affordances (action possibilities offered by the environment) in shaping behavior. Predictive coding posits that the brain actively predicts sensory inputs, and that neural representations reflect these predictions. Combining these perspectives, the hypothesis is that neural representations of situations and mental states are built from the actions they afford, facilitating social prediction. The study uses fMRI to measure neural activity associated with situations, mental states, and actions, testing whether summed action patterns, weighted by co-occurrence probabilities, can reconstruct the situation and mental state patterns.

The research draws upon existing work in social psychology (the influence of situations on behavior), personality psychology (the role of enduring traits), and affective science (the impact of momentary mental states). The theory of predictive coding is central, suggesting that the brain actively predicts sensory inputs, including social cues. Previous research demonstrates how neural representations reflect co-occurrence statistics of visual objects and how mentalizing about individuals reflects predictions of their habitual mental states. Ecological psychology, with its focus on affordances (action possibilities), provides a complementary framework. The convergence of these perspectives supports the hypothesis that action affordances are fundamental building blocks of situation and mental state representations.

The study involved three phases: stimulus selection, fMRI data collection, and data analysis. Stimulus selection employed a two-step process: 1) computational text analysis (fastText embeddings) to identify stimuli with maximal variance in co-occurrence and 2) human ratings (DIAMONDS, ACT-FAST, 3d Mind Model taxonomies) to minimize redundancy. The fMRI paradigm involved participants judging the likelihood of co-occurrences between situations, mental states, and actions (situation-action, situation-state, state-action pairs). Reliability-based feature selection identified brain regions reliably representing each stimulus type. The main analysis involved pattern summation: weighted sums of action patterns (weighted by co-occurrence ratings) were compared to situation and mental state patterns to assess reconstruction accuracy. Representational similarity analysis (RSA) provided a complementary approach, examining the relationship between neural pattern similarity and co-occurrence ratings. Several exploratory analyses were conducted to clarify the nature of the relationship between actions and situation/mental state representations, including comparisons of summed action patterns vs. single most likely actions, model comparison using AIC, and cross-validated model selection to determine the number of actions contributing to each situation/state representation.

The study found substantial overlap in brain regions representing situations, mental states, and actions, primarily within the social brain network (STS, TPJ, ATL, MPFC). The key finding is that frequency-weighted sums of action patterns successfully reconstructed situation-specific patterns (mean Z(r) difference matched-mismatched = 0.0017, d=0.49, Pcorrected = 0.043) and mental state-specific patterns (mean Z(r) difference matched-mismatched = 0.0016, d=0.50, Pcorrected = 0.039). However, summed mental state patterns did not reconstruct situation patterns. Exploratory analyses showed that the summed action patterns provided significantly better reconstruction than using only the single most likely action for each situation/state. RSA analyses further supported the hypothesis that co-occurrence rates shape neural representations, showing significant correlations between neural pattern similarity and co-occurrence ratings for all three pairwise comparisons (actions-situations, actions-mental states, situations-mental states). Additional exploratory analyses suggested a non-linear weighting function, where highly likely actions receive disproportionately high weights, and less likely actions may even receive negative weights. Cross-validated model selection indicated that approximately 40-44% of actions contribute to optimally reconstructing each situation/mental state.

The findings demonstrate that action affordances contribute significantly to the formation of situation and mental state representations. This suggests a unified framework for understanding both external and internal influences on behavior, with actions serving as the common building blocks. The results support predictive coding theories, showing that representations reflect probability distributions over actions conditioned on situational and mental state variables. The hierarchical structure of social knowledge is highlighted: actions contribute to mental state representations, which in turn contribute to person representations. The findings also relate to constructionist accounts of emotion and provide a potential explanation for dimensions in situational and mental state taxonomies. The lack of summation of mental states to situations warrants further investigation.

This study provides strong evidence that neural representations of situations and mental states are composed, at least partially, of weighted sums of the actions they afford. Actions serve as fundamental building blocks for understanding the social world, supporting predictive coding models of social cognition. Future research should explore the interaction between different types of information (e.g., mental states and situations) in shaping predictions, investigate the precise weighting functions governing action contributions to situation/state representations, and examine the role of action affordances in broader social dynamics.

The sample size, though statistically powered, was smaller than initially planned due to the COVID-19 pandemic. The sample was also not fully representative of the general population. The use of a single, non-naturalistic task may limit the generalizability of the findings. The observed reconstruction accuracies were not perfect, indicating that other factors beyond action affordances also contribute to situation and mental state representations. Future research should address these limitations using larger, more diverse samples and a wider range of tasks.