Maintenance and transformation of representational formats during working memory prioritization

The study asks how executive control prioritizes items in visual working memory (VWM) and whether prioritization changes the representational format of stored content. Prior work implicates ventral visual stream (VVS) regions in maintaining stimulus-specific information and prefrontal cortex (PFC) in control and prioritization, especially in retro-cue paradigms. Theories propose that prioritization involves dynamic recurrent computations and oscillatory mechanisms (notably beta for top-down control) and may transform representations from mnemonic/perceptual formats to task-optimized formats. The authors test these predictions by recording iEEG from human epilepsy patients with VVS and/or PFC electrodes during a multi-item VWM task with retro-cues, using RSA and DNN-based representational models to track format changes across encoding and maintenance.

Extensive work shows PFC and parietal regions engage in VWM prioritization, especially with retro-cues, and that recurrent dynamics in PFC support selection, integration, flexibility, interference resistance, and prioritization. Oscillations are functionally segregated: gamma conveys bottom-up information; beta supports top-down control and has been linked to reactivation of task-relevant content (rules, categories, magnitudes, decisions). Human iEEG studies show frequency-specific (theta, beta, gamma) representations carry item/category information in perception and episodic retrieval. Recent studies using DNNs suggest VWM representations transform towards formats resembling deeper (more abstract) network layers during encoding and maintenance. However, representational transformations specifically during prioritization in humans had not been tested, motivating this work.

Participants: 32 patients with medically intractable epilepsy (17 females; mean age 30±10.04 years) from centers in Freiburg (Germany), Hangzhou and Beijing (China). Ethical approvals obtained; informed consent provided. Task: Multi-item VWM with retro-cue. Each trial: encode a sequence of 3 natural images (from 6 categories; 10 exemplars each; total 60 images), then two maintenance periods separated by a retro-cue indicating to maintain either one list position (single-item, 50%) or all items in order (multi-item, 50%). Probe presented six items (3 studied + 3 lures). Behavioral performance computed per position and averaged. Analyses of neural data focus on single-item (prioritization) trials. iEEG acquisition: Depth electrodes; sampling 2000–2500 Hz, downsampled to 1000 Hz, bipolar re-referenced (N−1 virtual channels). Artifact inspection and epoch rejection performed; notch filters at 50/100/150 Hz. Electrode localization: Post-implant CT co-registered to pre-implant MRI; normalized to MNI; labels via PyLocator, 3DSlicer, FreeSurfer; white matter contacts removed. ROIs: VVS (inferior/middle/superior temporal, bankssts, fusiform, cuneus, entorhinal) and PFC (medial/lateral orbitofrontal, pars triangularis/opercularis, rostral anterior cingulate, rostral/superior frontal). Overall coverage approximately VVS ~441 electrodes (28 participants) and PFC ~147 electrodes (16 participants); RSA analyses included subjects with at least 2 electrodes per ROI (VVS n=26; PFC n=15). Lateral PFC subset defined in a control analysis. Time-frequency analysis: Complex Morlet wavelets across 3–150 Hz; 52 frequencies; cycles increased with frequency. Power values z-scored across trials within session. Representational features: For RSA, features built in 500 ms windows (centered), stepping by 100 ms. Model-based RSA computed at each individual frequency (3–150 Hz). Contrast-based RSA grouped frequencies into bands: theta (3–8), alpha (9–12), beta (13–29), low-gamma (30–75), high-gamma (75–150). RSA approaches: (1) Model-based RSA using: a categorical model RSM (same-category=1; different=0), an item model (in supplements), and DNN-derived RSMs. Neural RSMs computed from electrode×timepoint feature vectors; correlated with model RSMs using Spearman’s rho; Fisher z at group level; cluster-based permutation tests; Bonferroni across layers. (2) Contrast-based RSA: encoding-encoding similarity (EES) and encoding-maintenance similarity (EMS; performed separately for Maintenance 1 and 2), contrasting within- vs between-category correlations across all time bins to assess reoccurrence and stability; temporal generalization matrices constructed. DNN models: Feedforward AlexNet (8 layers: 5 conv, 3 FC) and two recurrent CNNs—BL-NET (7 conv layers with lateral recurrence; 8 recurrent time steps per layer) and corNET-RT (biologically inspired V1/V2/V4/IT; layer-specific recurrence: 5→2 steps). Stimuli resized/normalized per model; unit activations extracted; 60×60 RSMs computed via Spearman correlations; analyses focused on last recurrent time point per layer; additional characterization via within/between-category correlations and Category Cluster Index (CCI), MDS visualizations. Statistics: Group-level t-tests against zero for RSA fits; cluster-based permutation testing across time-frequency grids; Bonferroni corrections for network layers or frequency bands. Additional analyses included dimensionality/stress comparisons (MDS stress across 1–60 dimensions), variance of correlations, and within vs between-category structure of neural RSMs versus model RSMs. Correct vs incorrect trial-level DNN fits assessed with single-trial correlations and permutation-based multiple-comparisons control.

- Behavior: Accuracy higher in single-item than multi-item trials (0.80±0.12 vs 0.75±0.13); t(31)=3.21, p=0.0031, indicating effective prioritization. - Category model RSA (encoding): VVS showed a broad significant cluster (3–120 Hz) throughout encoding (0–0.8 s; p=0.001). PFC showed significant fits in beta (17–28 Hz; 200–800 ms; p=0.001) and theta (3–7 Hz; 200–600 ms; p=0.044). No significant fits during maintenance in either region. - Contrast-based RSA EES (encoding): Robust category-specific information in VVS across all bands (pcorr<0.005) and in PFC in theta, beta, low-gamma (pcorr<0.01). Onset latency: VVS earlier; PFC reached significance ~360 ms post-stimulus, ~290 ms after VVS; VVS>PFC differences from 150 ms (pcorr=0.007). Effects largely time-locked (diagonal) consistent with dynamic coding. - EMS for Maintenance 2 (post retro-cue): VVS showed significant reoccurrence of category-specific encoding patterns in all bands, strongest within first ~2 s after retro-cue (all pcorr<0.025). PFC showed no reoccurrence in any band (all pcorr>0.19), indicating transformation of representational format. - AlexNet RSA: During encoding, VVS representations matched AlexNet across layers in 3–75 Hz (some extending to high-gamma); no PFC fits. During maintenance, no significant fits in either region, suggesting feedforward formats do not explain prioritized representations. - BL-NET RSA: Encoding—VVS matched across layers (3–75 Hz; up to ~110 Hz in layer 7); no PFC fits. Maintenance—VVS matched BL-NET layers 4–6 in theta/alpha (3–14 Hz) late in maintenance (2.1–3.2 s) before probe (pcorr=0.035–0.014). Critically, PFC matched the final BL-NET layer in beta (15–29 Hz) starting ~200 ms after retro-cue for ~800 ms (Pcorr=0.021; other layers ns), indicating a transformed, high-level recurrent representational format time-locked to prioritization. - PFC representational geometry shift: Compared to encoding, maintenance showed higher average between-category correlations (trend t(14)=-2.12, p=0.053), significantly reduced variance of correlations (between-category t(14)=5.87, p=4.05e-05; within-category t(14)=5.37, p=9.89e-05), and higher MDS stress (i.e., higher-dimensional coding) during prioritization across 4–33 dimensions (p=0.0148). BL-NET explained PFC between-category structure during prioritization (t(14)=3.69, p=6.76e-05), whereas AlexNet did not; neither model captured within-category structure. - corNET-RT RSA: Encoding—VVS matched all layers across 3–105 Hz (all pcorr<0.004); no PFC fits. Maintenance—VVS matched IT layer in theta–alpha near probe (6–11 Hz; Pcorr=0.044). PFC matched IT layer in beta (15–29 Hz) time-locked to retro-cue for ~500 ms (Pcorr=0.016); additional V1 match observed (Pcorr=0.036). Parameter-matched feedforward controls did not reproduce PFC fits, underscoring the necessity of recurrence. - Overall: Prioritized PFC representations are transient, beta-band, and align specifically with deep layers of recurrent DNNs, while VVS maintains and later reinstates encoding-like category formats in lower frequencies (theta/alpha) prior to response.

The findings support a dissociation between sensory maintenance in VVS and executive prioritization in PFC. VVS encodes and reinstates category-specific representations across encoding and maintenance, reflecting a shared mnemonic coding scheme. In contrast, PFC does not reinstate encoding patterns after the retro-cue; instead, it exhibits a transformed representational format that aligns with deep layers of recurrent DNNs, suggesting abstract, high-level visual features become prominent when items are prioritized. This transformation is temporally locked to the retro-cue and specific to beta-band oscillations, consistent with beta’s role in top-down control and transient content reactivation. Recurrent architectures—but not feedforward—capture prioritized PFC geometry, implicating recurrent computations in PFC for WM prioritization. The distinct timings—cue-locked in PFC vs pre-probe in VVS—suggest complementary functional roles: PFC may compress and structure prioritized information for control under capacity limits, while VVS readies perceptual codes for upcoming decision. These results extend models positing rotations or subspace changes during prioritization by demonstrating specific format shifts captured by recurrent DNNs in human iEEG.

This study demonstrates that VWM prioritization recruits distinct representational formats in PFC versus VVS. VVS maintains and reinstates encoding-like category representations, whereas PFC transforms prioritized content into a high-level, recurrent-like format expressed in beta-band activity, best captured by deep layers of recurrent DNNs (BL-NET, corNET-RT). Feedforward DNNs do not explain prioritized maintenance. These results highlight the role of recurrent computations and beta oscillations in PFC for top-down control of WM content. Future work should incorporate models with explicit top-down connectivity to capture inter-areal interactions, examine multi-item representations using sequential recurrent architectures, and test how training objectives and datasets shape model–brain alignment during memory operations.

- Model training mismatch: Recurrent DNNs were trained for image classification, not working memory; nevertheless, they served as representational models of content. - Architectural constraints: BL-NET and corNET-RT include lateral but not top-down connections, limiting modeling of PFC–VVS interactions. - Task scope: Neural analyses focused on single-item prioritization trials; multi-item maintenance and interactions were not analyzed here. - Dataset dependence: DNN–brain alignment can depend on training sets; the authors controlled by evaluating ImageNet and Ecoset variants, but broader training regimes could further impact results. - Electrode sampling/power differences: Unequal ROI coverage and subject counts (more VVS than PFC electrodes) may influence sensitivity; control analyses attempted to match statistical power.