Maintenance and transformation of representational formats during working memory prioritization

Visual working memory (VWM) allows us to temporarily store and manipulate visual information. A crucial aspect of VWM is prioritization—selectively focusing attention on specific items based on task demands. Existing theories propose that prioritization involves transforming representations from a mnemonic state to a task-optimized one. Neurophysiologically, this suggests task-dependent transformations in executive control areas (like the prefrontal cortex, PFC), distinct from the mnemonic coding in sensory regions (like the ventral visual stream, VVS). This study aims to test this prediction using intracranial EEG (iEEG) data from epilepsy patients with electrodes in VVS and PFC. We utilize a multi-item VWM task with a retro-cue to prioritize specific items. Representational Similarity Analysis (RSA) combined with various deep neural network (DNN) architectures will be employed to analyze the representational format of prioritized VWM content. Understanding how the brain prioritizes information within VWM is essential for elucidating cognitive control mechanisms and memory processes. The high spatiotemporal resolution of iEEG, combined with the modeling power of DNNs and RSA, provides a powerful approach to investigate the neural underpinnings of this critical cognitive function.

Extensive research exists on the neural correlates of VWM control, with early studies focusing on pre-encoding prioritization. More recent work uses retrospective cueing paradigms, revealing that prefrontal and parietal regions involved in attention during perception are also engaged in VWM prioritization. A meta-analysis specifically highlighted selective responses to retro-cues in prefrontal areas. This literature suggests PFC involvement in VWM prioritization, but direct experimental evidence in humans is lacking. The PFC's role is often attributed to dynamic recurrent computations within local neural networks. Computational studies highlight the importance of recurrence for feature selection, integration of working memory and planning, and flexible WM. Recurrent computations may be key for selective attention by stabilizing reverberating activity, modulating the excitability of assemblies representing prioritized content. Recurrent computations are also theorized for VVS processing during perception and visual imagery, although their specific role in VWM maintenance hasn't been directly investigated. Brain oscillations, particularly gamma (bottom-up information) and beta (top-down control) frequencies, are also implicated in VWM, with studies in macaques and humans confirming their importance. Beta oscillations specifically have been linked to reactivation of stimulus-specific activity during prioritization, carrying information about task rules, stimulus categories, and magnitudes. Previous iEEG studies using multivariate techniques like RSA have identified frequency-specific representations of stimuli in various bands, crucial for episodic memory retrieval. Deep neural networks (DNNs) have been employed to investigate VWM representational formats, demonstrating transformations aligning with late layers of convolutional DNNs during encoding and maintenance. However, no prior study investigated representational transformations accompanying VWM prioritization in humans.

Thirty-two epilepsy patients (17 female, mean age 30) with iEEG electrodes implanted in VVS and/or PFC participated. They performed a three-item VWM task with a retro-cue, instructing them to maintain either a single item or all items. Stimuli consisted of 60 images from six categories (10 exemplars each). Data analysis focused on single-item trials. iEEG data (438 VVS electrodes, 146 PFC electrodes) were preprocessed to remove artifacts and noise. Time-frequency analysis using Morlet wavelets decomposed the signal into 3–150 Hz. Representational Similarity Analysis (RSA) was used with different models: a simple category model (same/different category), a feedforward DNN (AlexNet), and two recurrent DNNs (BL-NET, corNET-RT). Representational patterns included power values across electrodes, time points (five 100 ms points in a 500 ms window), and frequencies. Spearman's Rho was used to measure similarity. The category model assessed the presence of categorical information during encoding and maintenance. Contrast-based RSA (EES and EMS) compared correlations between same-category and different-category items during encoding and between encoding and maintenance, assessing representational stability. DNN-based RSA compared neural RSMs with RSMs from different DNN layers to characterize representational formats and frequency profiles. Additionally, Multidimensional Scaling (MDS) visualized representational geometry, and Category Cluster Index (CCI) quantified categorical organization in DNNs. Control analyses included examining individual exemplar representations, effects across encoding positions, and the functional relevance of representations during maintenance. Finally, parameter-matched versions of the recurrent networks without recurrence were used to evaluate the contribution of recurrent processing.

Behavioral results showed significantly better performance in single-item trials (prioritization) than in multi-item trials. Category-specific representations were present during encoding in both VVS and PFC (VVS: 3-120 Hz; PFC: beta and theta bands). During maintenance, the VVS showed reoccurrence of category-specific information across all frequency bands, while the PFC did not. DNN analysis showed that VVS encoding representations matched AlexNet representations across multiple frequencies, but no match during maintenance. In contrast, PFC representations following the retro-cue showed a significant match with the last layer of the recurrent BL-NET and corNET-RT in the beta band (15–29 Hz), suggesting a transformation into a prioritized format emphasizing high-level visual features. This transformation involved a shift towards higher dimensionality and less clustering in the representational space. Control analyses showed that recurrent computations were essential for explaining PFC representations during maintenance, while their role in VVS seemed less critical. The recurrent models only showed decreases in between-category correlations, while the AlexNet showed both an increase in within-category correlations and a decrease in between-category correlations, suggesting that recurrence specifically supports distinct representations of different categories.

The study provides evidence for distinct representational transformations in VVS and PFC during VWM prioritization. The VVS maintains a consistent representational format across encoding and maintenance, exhibiting a shared mnemonic coding scheme. The PFC, however, undergoes a representational transformation, shifting from a categorical format during encoding to a format that matches the deep layers of recurrent DNNs after the retro-cue. This reflects a dynamic prioritization coding scheme where high-level visual features are prioritized. The beta-band activity in the PFC during prioritization is consistent with its role in top-down control. These findings support capacity-limited WM models, suggesting that compressed, high-level representations are beneficial in PFC. Our results contrast with previous findings showing representational abstraction in parietal and visual cortices but not prefrontal regions, potentially due to differences in stimuli or task paradigms. The success of recurrent DNN models in capturing PFC activity highlights the importance of recurrent computations in WM prioritization, suggesting that recurrence supports the formation of distinct representations of different categories.

This research demonstrates distinct representational transformations in VVS and PFC during VWM prioritization, revealing a shift in PFC from categorical to prioritized formats matching recurrent DNNs. This transformation is linked to beta-band activity and suggests a key role for recurrent computations in flexible task-dependent memory manipulation. Future research should investigate the PFC-VVS interactions using networks with top-down connections and explore multi-item representations with sequential recurrent networks.

The study's limitations include the use of DNNs trained for image classification, not memory, and the architectures lacking top-down connections. Focusing on single-item trials limits the understanding of multi-item representations. The relatively small sample sizes of implanted electrodes in certain brain regions may have limited the statistical power for some analyses. Despite these limitations, the current study is the first to use human iEEG data to study prioritized representations.