Stimulus representation in human frontal cortex supports flexible control in working memory

The study addresses how stimulus information is represented and maintained across cortical regions during working memory and how these representations support flexible behavior. Prior work shows distributed, stimulus-specific representations across sensory, parietal, and frontal cortices, yet the roles of visual versus frontal regions remain debated. Human neuroimaging often finds robust stimulus maintenance in early visual cortex (EVC) during simple feature WM, with weaker, less stable representations in higher-order cortex, while EVC representations can be vulnerable to distraction. Conversely, frontal representations can be robust under prioritization, categorization, or extensive training. The authors propose a cognitive flexibility account: WM not only maintains sensory inputs but also supports active manipulation and control, generating new information (e.g., categories, rules). Given limited resources, stimulus representations may adapt to task goals, co-varying with newly generated representations. Two predictions follow: (1) frontal stimulus representation should increase with demands for active control (e.g., rule-based categorization), whereas visual cortex representations should strengthen with precise maintenance demands; (2) within regions encoding newly generated information, there should be a tradeoff between original stimulus and new (e.g., category) representations as control demands vary. The study tests these predictions using fMRI with tasks that require switching between maintenance and rule-based categorization, and uses recurrent neural network (RNN) models to probe mechanisms.

Evidence for stimulus-specific WM representations is found across cortex. Human neuroimaging shows robust decodability in EVC for simple features supporting the sensorimotor recruitment hypothesis, but these representations are susceptible to distraction and may be weaker or less stable in frontoparietal regions. Frontal representations can become robust under attention, categorization, or after extensive training. Nonhuman primate studies highlight frontal involvement but with varying task dependencies. The literature suggests that representational strength depends on both region and task demands, motivating the hypothesis that coding strategies reconfigure with cognitive control requirements, balancing high-dimensional stimulus representations for flexible readout against low-dimensional categorical abstractions for generalization.

Participants: Experiment 1 (n=24 analyzed; mean age ~24; right-handed), Experiment 2 (n=24 analyzed). All gave informed consent (CEBSIT-2020028). Tasks and design: Participants performed a flexible visual WM task switching between maintenance and categorization. Each trial presented two oriented bars (0.75 s each, 0.5 s ISI), a retro-cue (0.5 s) indicating which orientation to remember, a 1.5 s interval, then a task cue (0.5 s) indicating maintenance (letter P) or categorization (letter C), followed by an 8 s delay and a 2 s response. Maintenance trials: choose the remembered orientation from two probes. Categorization trials: report the category of the remembered orientation according to rules learned in prior training. Response mapping (keys F/J) was randomized; key labels were positioned randomly within category ranges to prevent explicit boundary cues. Inter-trial intervals: 3/5/7 s. Experiment 1: Two orthogonal categorization rules (Rule A, Rule B) were learned beforehand; within blocks of nine trials, maintenance and categorization trials were interleaved; across blocks, the active categorization rule switched (block-cued). Each participant contributed 90 trials per condition (maintenance, categorization-Rule A, categorization-Rule B). Experiment 2: Single fixed categorization rule throughout scanning; same trial structure; 90 maintenance and 180 categorization trials per subject. Behavioral training: Participants learned rules via a feedback-based task, with randomized category–response mapping and boundary placement to prevent verbal strategies; they reported learned boundaries afterward. MRI acquisition: 3T Siemens (Tim Trio; later some Prisma), 32-channel coil; EPI: TR=1000 ms, TE=30 ms, FA=40°, voxel size 3×3×3 mm, multiband=4, 60 slices, matrix 74×74. High-res T1: 1 mm isotropic. Preprocessing: AFNI; discard first 8 volumes per run; motion correction; detrending; functional runs aligned to last volume of final run, then to T1; manual quality checks. ROIs: Anatomical masks from Wang et al. probabilistic atlas, warped to native space: EVC (bilateral V1–V3), IPS (bilateral IPS0–5), sPCS (bilateral FEF). Functional ROIs: top 500 voxels per region based on GLM activity (EVC: sample display; IPS/sPCS: memory delay). Additional frontal ROIs (for completeness analyses): iPCS, IFS, MFG from HCP atlas; M1 control. Analyses: - Inverted Encoding Model (IEM): Nine equidistant orientation channels (1–161° initial centers), half-wave-rectified cosine^8 tuning; trained using all trials across conditions, leave-one-run-out cross-validation per TR; repeated with 1° shifts to tile 0–180°. Reconstruction fidelity computed by projecting channel response vectors onto true orientation and averaging. Controls: within-condition training, SVM stimulus decoding, removal of mean activation differences, matched voxel counts, M1 analyses. - MVPA: Linear SVMs for stimulus (four bins of 45°; separate oblique/cardinal classifiers; averaged) and for category. For category, trained decoders with true rule labels and with orthogonal rule labels to estimate an abstract category index (true minus orthogonal decoding). Leave-one-trial-out for balancing labels; averaged across rules (Exp 1) or halves (Exp 2). - Behavioral correlations: Time-resolved correlations (Pearson) between representational fidelity and accuracy across trials within ROI and task. - RSA and LMEM: Neural representational dissimilarity matrices (RDMs) computed from Pearson correlation distances across trials, averaged over delay. Model RDMs for graded stimulus, discrete stimulus, and abstract category. Linear mixed-effects modeling fit neural RDMs with model RDMs, subject as random effect, to estimate contributions of stimulus vs category. - Statistics: Sign-flip permutation tests (10,000 iterations) with FDR corrections across ROIs, time, tasks; defined early (5–10 s) and late (11–16 s) epochs for comparisons. RNN modeling: - Architecture: Modular recurrent neural networks with input, hidden, output layers. Hidden layer split into three modules (200 units each; 80% excitatory, 20% inhibitory; short-term synaptic plasticity), with denser within-module connectivity and constrained between-module excitatory connections (module 1 posterior ~ EVC; module 2 middle ~ IPS; module 3 anterior ~ sPCS). Only module 1 receives sensory inputs; only module 3 projects to output. - Two variants: RNN1 outputs choice via two units only; RNN2 additionally outputs the orientation via 15 orientation-tuned output units, enforcing preservation of stimulus information for readout. - Task simulation: Matches Exp 1 timing; inputs include 15 orientation-tuned units (0–180°), retro-cue, task cue; in RNN1, response options presented before test; in RNN2, category output (categorization) or null (maintenance) plus orientation output. - Training: TensorFlow (Nvidia-tensorflow 1.15); noise SD 0.01; 600 hidden units total (three modules); up to 10,000 iterations; spike penalty 0. Batches interleaved across tasks (maintenance, two categorization rules; or single rule). Stop when ≥90% accuracy on all tasks in last batch; for RNN2, category and stimulus outputs assessed separately. - Decoding from RNNs: After training, freeze weights; decode stimulus and abstract category with SVMs from time-resolved unit activity; 900 trials/condition, five-fold cross-validation; summarize over delay. RSA between human and RNN RDMs aligned to stimulus or category, Kendall’s Tau, with permutation tests for RNN comparisons.

Behavior and learning: Participants learned two orthogonal categorization rules with comparable accuracy (Rule A M=0.85±0.05; Rule B M=0.85±0.05; t(23)=0.24, p=0.813) and boundary precision (errors ~9°). In-scanner performance was matched between tasks (maintenance accuracy 0.81±0.07; categorization 0.82±0.05; t(23)=1.51, p=0.144). Categorization showed a boundary-distance effect on accuracy and RT. Stimulus representation (IEM fidelity; Exp 1): - EVC: Significant orientation representation in both tasks starting from sample; stronger during delay for maintenance than categorization after task cue, indicating precise-maintenance demands. Late-epoch (11–16 s) difference favored maintenance (p<0.00001). - IPS: Significant in both tasks; little difference between tasks across time (late-epoch difference p=0.063). - sPCS: Orientation representation persisted through delay and response in categorization, but decayed in maintenance; categorization > maintenance in late delay (late-epoch difference p=0.007), consistent with enhanced frontal stimulus representation under active control. Control analyses (condition-specific IEM, SVM decoding, mean-activity removal, voxel-count matching, M1 control) supported robustness. Behavioral relevance (Exp 1): - EVC: Early delay stimulus fidelity correlated with accuracy in both tasks. - IPS: Predictive in early delay for maintenance and throughout delay for categorization. - sPCS: Predictive of accuracy only in categorization throughout the delay; not predictive in maintenance, indicating selective role in control. Reduced control demand (Exp 2, single rule): - EVC and IPS patterns similar to Exp 1; EVC fidelity lower in categorization vs maintenance; IPS similar across tasks. - sPCS: Differential enhancement for categorization emerged earlier; significant in early epoch (5–10 s; p=0.015) but not late (11–16 s; p=0.372). Mixed ANOVA showed experiment-by-epoch interaction (F(1,46)=7.43, p=0.009), indicating temporal shift with reduced control demands. Behavioral correlations remained, with reduced task differences. Category and abstract category decoding: - Category (true rule) decodable during late epoch across ROIs in both experiments (ps<0.044). - To control stimulus similarity, decoding with orthogonal rule labels was performed; the abstract category index (true minus orthogonal) was significantly above 0 in Exp 2 across ROIs (ps<0.017) but not in Exp 1 (ps>0.14). Abstract category decoding was higher in sPCS for Exp 2 vs Exp 1 (p=0.034). LMEM on RDMs corroborated a tradeoff: in sPCS, strong stimulus (but not category) representation in Exp 1; in Exp 2, decreased stimulus and emerging category contributions. Additional frontal ROIs (iPCS, IFS, MFG) showed weaker effects; MFG showed some similar patterns in Exp 2. RNN simulations: - RNN1 (choice-only output) showed stronger stimulus decoding during maintenance than categorization in middle and anterior modules (p_middle=0.011; p_anterior=0.007), opposite to human IPS/sPCS. - RNN2 (stimulus readout enforced) reproduced human-like pattern: increased stimulus decoding in categorization selectively in anterior module (p_anterior=0.026; posterior/middle ns). - Abstract category decoding increased in fixed-rule vs flexible-rule models in RNN2 across modules (p_posterior=0.045; p_middle=0.003; p_anterior<0.001), mirroring human Exp 2 > Exp 1. Similar fixed>flexible effects also in RNN1 (all ps<0.001). - RSA: Across experiments and alignments (stimulus, category), RNN2 more closely matched human RDMs than RNN1, especially in module 3 vs sPCS. Overall, findings indicate that frontal stimulus representations strengthen with WM control demands and trade off with category representations, while visual cortex representations track precise maintenance demands. RNNs suggest that requiring preserved stimulus information at readout yields the observed frontal patterns.

The results address the central question of how stimulus representations in different cortical regions support WM functions under varying control demands. EVC supports precise maintenance: its stimulus representations are robust and predict performance, though reduced when categorization (rather than precise maintenance) is emphasized, yet a copy remains even when categorization is required. The frontal sPCS supports active control: stimulus representations there strengthen when the task requires manipulation (rule-based categorization) and predict behavior only under control demands. Varying control demands (flexible vs fixed rules) revealed a dynamic tradeoff: with lower control demands, frontal stimulus enhancement diminishes later in the delay, while abstract categorical representations increase, particularly in sPCS. This aligns with theoretical views on cognitive flexibility, balancing high-dimensional stimulus codes for flexible readout against low-dimensional abstractions for efficient generalization, and with neurophysiological data showing training-dependent categorical coding. RNN simulations show that imposing an output requirement to preserve stimulus information leads to human-like frontal patterns and the observed tradeoff, suggesting that coding strategies may be adaptively selected to facilitate flexible control and readout in changing environments. Differences between frontal and parietal regions suggest a gradient, with sPCS showing the strongest source-like effects for active control, and IPS sharing some properties but with weaker or more variable dissociations.

The study demonstrates that stimulus representations are distributed yet functionally differentiated across cortex during WM: EVC representations primarily subserve precise maintenance, whereas frontal sPCS representations support active control over WM contents, strengthening with increased control demands and predicting performance specifically during categorization. A tradeoff with category representations emerges as control demands vary. RNN models indicate that preserving stimulus information for output can produce the observed frontal dynamics and human–model representational similarities, suggesting an adaptive coding strategy for flexible behavior. Future work should examine generalization across other stimulus domains and frontal regions, probe meta-control mechanisms selecting coding strategies, and refine models to better capture visual cortex dynamics.

Model–data mismatches remain: all RNN modules showed significant abstract category decoding and fixed>flexible differences, broader than fMRI effects; RNN2 did not robustly reproduce the reduced EVC stimulus representation during categorization (negative difference was weak and only trended with IEM), suggesting additional circuitry or inputs may be needed to model fine-grained EVC coding. The sPCS emphasis may be partly stimulus-specific (orientation), with weaker effects in other frontal ROIs; generalization to other features (e.g., color) and control operations (e.g., mental rotation) is not yet established. Although mean activation differences were controlled, fMRI temporal resolution and hemodynamic lag limit precise timing inferences; sample sizes, while typical or larger than prior work, may limit detection of smaller IPS effects. Rule switching vs fixed-rule manipulations infer control demand indirectly; other manipulations of control could further validate the framework.