Neural representations of situations and mental states are composed of sums of representations of the actions they afford

The study investigates how the brain represents the external contexts (situations) and internal conditions (mental states) that shape human behavior to enable social prediction. Drawing from predictive coding and ecological psychology, the authors hypothesize that situations and mental states are represented as probability-weighted sums of the actions they afford. Situations provide norms, scripts, and environmental constraints that make certain actions more or less likely, while mental states (affective and cognitive) motivate and govern behavior, promoting or constraining particular actions. The central research question is whether neural representations of situations and mental states can be reconstructed from weighted sums of action representations, thereby revealing action affordances as building blocks of social cognition.

Two perspectives ground the hypothesis. Predictive coding posits that the brain is fundamentally predictive, maintaining representations that encode likely future events; neural similarity reflects co-occurrence statistics (e.g., objects that co-occur evoke similar patterns). Prior work shows the brain represents people as sums of the mental states they habitually experience, supporting a compositional, predictive structure to social representations. Ecological psychology (Gibson’s affordances) proposes that environments are defined by action opportunities; reframed representationally, this suggests situations may be represented by the actions they afford. Additional literature indicates that transition dynamics between mental states shape mental state concepts and that action co-occurrence structures can tune neural representations, motivating the test that action affordances organize situation and mental state representations.

Design and preregistration: The study was preregistered (OSF: https://osf.io/whsx8) with a priori power analysis (target n=34 based on d=0.81), behavioral task specifications, and analytic plans. Data and code are openly available (OSF: https://osf.io/qwd2k/; OpenNeuro ds004226). Ethical approval by Princeton IRB (#0000007271). Participants: fMRI sample n=28 (16 female; age 18–36, mean 20.61) after excluding one participant for motion; inclusion: right-handed/ambidextrous, 18–35 (one 36), fluent English, normal/corrected vision, no neurological issues, MRI-safe. Two online samples: (1) DIAMONDS ratings of situations (N=400; post-exclusion 393), (2) co-occurrence ratings between situations, mental states, and actions (N=900). Additional publicly available ratings: 3d Mind Model for mental states; ACT-FAST taxonomy for actions. Stimuli: Three verbal stimulus classes: 60 situations (short phrases), 60 mental states (single words), 60 actions (single-word verbs). Initial pools: actions (472→224), mental states (166), situations (166). Two-stage optimization: (1) computational text analysis using fastText 300d embeddings to maximize variance in inter-class similarities via simulations; selected top-variance 100 per class. (2) behavioral optimization to minimize redundancy using psychological rating spaces: ACT-FAST for actions, 3d Mind Model for mental states, DIAMONDS for situations; greedy search minimized stimulus redundancy and inter-dimension correlations; final sets trimmed to 60 per class. A separate online sample rated pairwise co-occurrence likelihoods on 100-point scales; averaged (~10 ratings per pair) to produce co-occurrence matrices. Task (fMRI): Participants judged likelihood of co-occurrences for pairs from different classes on each trial (situation–action, situation–state, state–action). Response scale 1–4 (not at all likely to very likely). Trial: 4.25 s, fixation ≥0.25 s + Poisson jitter (mean 1.5 s, 1.5 s steps). Ten runs; each run = 90 trials (30 per pair type). Within a run, 30 unique stimuli per class appeared twice; across 10 runs, each of the 60 stimuli per class appeared 10 times. Pairings were randomized without within-subject repetition, yielding a partially crossed design (900/10,800 possible pairings per participant). Practice used non-task stimuli. MRI acquisition: Siemens Skyra 3T, 64-channel head coil. BOLD EPI: TR=1500 ms, TE=32 ms, flip angle=70°, 2.5 mm isotropic, 52 axial slices, SMS=4. T1 MPRAGE: 1 mm isotropic, TR=2300 ms, TE=2.98 ms, FA=9°, 176 slices. Spin-echo field maps (A→P, P→A): 2.5 mm iso, TR=8000 ms, TE=66 ms. Preprocessing and GLM: Pipeline combined SPM12 (slice timing), DARTEL (motion correction, unwarping, normalization), FSL (high-pass filtering). To avoid collinearity from modeling trials twice, two separate GLMs were fit: GLM1 modeled situations (from situation–action and situation–state trials) and states (from state–action trials); GLM2 modeled states (from situation–state trials) and actions (from situation–action and state–action trials). Boxcars per condition were convolved with canonical HRF; nuisance regressors included run means/trends and 6 DOF motion. This yielded 360 whole-brain maps: 60 situations×2 contexts, 60 states×2 contexts, 60 actions×2 contexts. Feature selection: Reliability-based voxel selection performed separately for situations, states, and actions to identify reliable and overlapping representational voxels while avoiding circularity. For feature selection only, 6 mm FWHM smoothing applied to GLM betas. Voxelwise reliability computed from split conditions (e.g., situations from different trial pairings), scanning voxelwise thresholds to maximize pattern similarity reliability. Selected voxels: situations 5,788; mental states 10,294; actions 2,536. Overlap observed in canonical social brain areas (TPJ/STS/ATL, MPFC, medial parietal, lateral frontal); pairwise overlaps noted (e.g., situation–action 570 voxels; state–action 668; situation–state 1,545). Primary analysis: Pattern summation. Within overlap masks for paired classes, action patterns were weighted by online co-occurrence frequencies with each target situation/state and summed voxelwise to reconstruct target patterns. Specificity assessed by correlating reconstructed patterns with matching and non-matching targets; matched–mismatched Fisher z(r) differences averaged per participant, tested via two-tailed one-sample t-tests with maximal-statistic permutation FWER correction across confirmatory families. Confirmatory tests: actions→situations; actions→states; states→situations. Exploratory reverse-direction tests also preregistered (situations→actions, situations→states, states→actions). Additional exploratory analyses: (1) Compare weighted-sum vs single most-likely action reconstructions using voxelwise z-scored patterns and RMSE; (2) Regression model comparisons with AIC across predictors: weighted sum, single action, both; (3) Cross-validated L1-regularized regression to estimate the proportion of contributing actions per target and alignment of weights with affordances. Representational similarity analyses (RSA): Constructed cross-domain similarity matrices (actions×situations; actions×states; situations×states) within appropriate overlap masks; correlated neural similarity with co-occurrence ratings per participant and tested via one-sample t-tests with permutation FWER control. Variants included Kendall’s τ, RSA using LASSO-derived weights, and within-domain RSA relating situation/state similarities to similarity of their action-affordance profiles across full domain-voxel sets.

- Reliability-based feature selection identified robust, partially overlapping neural substrates for situations (5,788 voxels), mental states (10,294), and actions (2,536), largely within the social brain network (TPJ/STS/ATL, MPFC, medial parietal, lateral frontal cortex), with pairwise overlaps (situation–action 570 voxels; state–action 668; situation–state 1,545). - Confirmatory pattern summation analyses (matched–mismatched Fisher Z(r) differences, n=28, permutation-corrected): • Actions → Situations: mean ΔZ(r)=0.0017, d=0.49, Pcorrected=0.043. • Actions → States: mean ΔZ(r)=0.0016, d=0.50, Pcorrected=0.039. • States → Situations: mean ΔZ(r)=0.00016, d=0.078, Pcorrected=0.97 (not significant). Interpretation: Neural representations of situations and mental states are composed, at least in part, of sums of action affordances; mental states do not sum to situations. - Exploratory reverse-direction summations: • Situations → Actions: mean ΔZ(r)=0.0011, d=0.29, Pcorrected=0.37 (ns). • Situations → States: mean ΔZ(r)=−0.00011, d=−0.081, Pcorrected=1.00 (ns). • States → Actions: mean ΔZ(r)=0.0018, d=0.50, Pcorrected=0.033 (significant), suggesting action representations can be explained partly by mental states that potentiate them. - Multiple-actions contribution vs single action: • RMSE comparisons showed poorer reconstructions by the single most-likely action than by weighted sums for both targets: situations (mean RMSE difference=0.38, d=7.45, p=2.09×10^-25) and states (mean RMSE difference=0.38, d=11.00, p=6.38×10^-30). • AIC model comparisons (lower is better): single action worst (situations mean AIC=1482; states 1700); weighted sum better (situations 1480; states 1697; AIC difference for states=2.77, d=0.59, p=0.0041; situations difference=1.92, d=0.36, p=0.066); combined model (weighted sum + single action) best (situations 1470; states 1683), significantly outperforming weighted-sum alone (situations ΔAIC=10.40, d=1.55, p=8.5×10^-9; states ΔAIC=14.18, d=2.53, p=2.0×10^-13) and single-action alone (situations ΔAIC=12.32, d=2.72, p=3.5×10^-14; states ΔAIC=16.95, d=2.80, p=1.8×10^-14). • Cross-validated L1 regression indicated many actions contribute: ~44% of actions to reconstruct each situation and ~40% for each mental state; regression weights correlated with affordance ratings. Weighting appears nonlinear: highly likely actions are disproportionately weighted; unlikely actions can carry negative weights. - Representational similarity analyses: • Co-occurrence ratings predicted cross-domain neural similarity: actions–situations mean Z(r)=0.054, d=1.03, Pcorrected=0.00020; actions–states mean Z(r)=0.059, d=1.79, Pcorrected=0.00010; situations–states mean Z(r)=0.025, d=0.62, Pcorrected=0.0099. Kendall’s τ analyses produced qualitatively identical results. • Within-domain RSA across full situation-voxels showed that similarity between situations’ neural representations was predicted by similarity in their action-affordance profiles; no analogous effect for mental state representations. Overall: Findings support that neural representations of situations and mental states are composed of (nonlinearly) weighted sums of action representations, and that mental states also help explain action representations.

The results address the core hypothesis that social representations are organized by action affordances. Action patterns significantly reconstructed both situation and mental state patterns, indicating that the brain represents exogenous (situations) and endogenous (mental states) determinants of behavior as bundles of action predictions. Conversely, mental state patterns did not sum to situation patterns, suggesting that while situations and states are related in overlapping regions, their relationship is not via simple summation. Exploratory analyses showed mental states significantly summed to actions, consistent with states potentiating specific behaviors. Complementary RSA demonstrated that perceived co-occurrence structure robustly predicts cross-domain neural similarity, supporting the broader claim that co-occurrence statistics shape social representations. The findings align with predictive coding accounts in which neural representations encode priors over likely events: situations and mental states function as priors over action distributions conditioned on external and internal variables. They also provide representational support for ecological affordance theory by showing that situation representations are partly constructed from the actions they afford. Considering prior evidence that people are represented as sums of their habitual mental states, the present results suggest a hierarchical structure of social knowledge: action representations compose mental state representations, which in turn compose person representations. The weighting function appears nonlinear with positive and negative weights, implying that representations are defined both by presence of likely actions and absence of unlikely ones. The observed exception (states not summing to situations) indicates a more complex relation between internal and external determinants, warranting alternative mechanisms beyond summation (e.g., interactions, contextual gating). The authors discuss implications for affective science and concept construction: integrating information over time about action–state contingencies may underlie emotion concepts. The structure is consistent with Hidden Markov Model frameworks in which latent mental states have transition dynamics and emission probabilities (observable actions). Future work should test alignment between perceived and real-world co-occurrence rates (e.g., via experience sampling) and examine interactive predictions combining state and situational information.

This study shows that neural representations of situations and mental states are composed—at least in part—of probability-weighted sums of action representations, providing a mechanistic basis for predicting behavior from internal and external factors. The results support a predictive, compositional architecture of social cognition and suggest a hierarchical organization where actions build states, and states build person representations. Future directions include: validating perceived co-occurrence with real-world statistics (experience sampling), refining nonlinear weighting functions (including negative weights), modeling with HMMs to integrate state transitions and action emissions, examining interactive effects of situations and states on predictions, testing generalization with more naturalistic tasks, and broadening samples for external validity.

- Sampling: Convenience samples (college students, Mechanical Turk) limit representativeness; imaging sample (n=28) smaller than preregistered target due to COVID-related constraints, though still adequately powered by a priori analyses. - Task ecology: Single, non-naturalistic co-occurrence judgment task; replication with diverse, naturalistic paradigms is needed for convergent and external validity. - Effect magnitude and completeness: Although standardized effects were moderate-to-large, raw reconstruction effects were small, and representations only partially overlapped spatially; action affordances do not fully account for situation or mental state representations. - Analytical scope: Summation captures part of the relationships; other dynamics (e.g., state transition probabilities, interactions between internal and external factors) likely contribute. - Generalizability of weighting: Nonlinear and negative weights suggest complexity that may vary across contexts and individuals; further validation is needed.