Neural mechanisms of resource allocation in working memory

Working memory (WM) is capacity-limited: precision declines sharply as set size increases and neural noise causes trial-to-trial fluctuations, making uncertainty a key factor in memory-guided decisions. Despite these limits, humans strategically allocate WM resources, prioritizing items that are more relevant or more rewarded, yielding smaller errors and lower reported uncertainty for high-priority items. Attention and prioritization modulate neural responses in retinotopic visual maps during perception and WM, but prior work largely aggregated neural responses across trials or conditions and did not quantify the quality and uncertainty of multiple items held in WM on a trial-by-trial basis. The neural circuitry controlling trial-by-trial allocation of WM resources remains unknown. Building on probabilistic population codes—which jointly encode stimulus features and uncertainty—recent work showed spatial WM representations and their uncertainty can be decoded from extrastriate visual and parietal cortex. Here, the authors test the hypothesis that behavioral relevance sculpts population gain and that the brain jointly represents the content and uncertainty of multiple memorized items with different priorities using probabilistic population codes. They develop a generative model-based decoder to demix and decode two concurrently held items with different priorities from fMRI signals, enabling trialwise estimation of both content and uncertainty and linking neural modulations to behavioral prioritization. They preview findings that neural gain tracks priority, decoding error and uncertainty are lower for high-priority items, neural differences predict behavioral prioritization, and frontal delay activity predicts differences in decoded precision, supporting a feedback model of resource allocation.

Participants: Eleven adults (five females) with normal or corrected-to-normal vision participated, with informed consent; protocols approved by NYU IRB; compensation $30/hour. Task and stimuli: Memory-guided saccade task in the scanner. Each trial began with a precue (gray annulus divided into two semicircles by a randomly oriented divider) and colored orthogonal lines indicating which segment would contain the high-priority item. After 1 s, two items (one per segment; width 0.65°; eccentricity 12°) were presented for 500 ms. A 12-s delay followed; participants maintained fixation while remembering both locations. A response cue then indicated which item to saccade to. High-priority items were probed in 67% of trials; low-priority in the remainder. Item polar angles were pseudorandomly chosen: probed item uniformly from 0–360°; nonprobed at least 30° away; divider placed between them with each item ≥10° from divider. Each fMRI run lasted 5 min with 12 trials; on average, each participant completed 27 runs across 2–3 sessions. Setup and eye tracking: Stimuli presented via LCD projector; viewed through a mirror (52° × 31° field). Gaze tracked at 500 Hz with EyeLink 1000 Plus; 13-point calibration per session. Behavioral analysis: Automated preprocessing (iEye_ts) removed blinks, corrected drift, recalibrated trial-by-trial, identified memory-guided saccades, flagged trials for rejection. Trials excluded for behavioral analyses if primary saccade absent/ill-defined or error >5° (visual angle). Memory error computed as signed difference (decoded landing minus target) in polar angle. All trials were retained for fMRI model estimation to ensure balanced spatial sampling, but only reliable behavioral trials used for decoding performance and neural–behavioral correlations. Variable-precision modeling (updated for multiple items) fit memory errors, estimating the weight of resource allocated to the high-priority item (sum of weights constrained to 1). Retinotopic mapping and ROIs: A separate 1.5-hour session used bar apertures sweeping in four directions with RDKs; participants compared motion direction across segments at ~75% accuracy. pRF modeling with compressive spatial summation fit the data; voxels with >10% variance explained were included. ROIs defined by visual inspection of polar angle reversals: bilateral V1, V2, V3, V3AB, IPS0–IPS3, iPCS, sPCS, each with full-field representation. MRI acquisition and preprocessing: Siemens Prisma 3T, 64-channel coil; functional imaging: 2.5 mm isotropic voxels, 44 slices, SMS factor 4, FoV 200 × 200 mm, TE/TR 30/750 ms, flip angle 50°, bandwidth 2290 Hz/pixel, echo spacing 0.56 ms, P→A phase encoding. Spin-echo images with reverse phase encoding used for field map correction. Anatomical segmentation and surfaces via FreeSurfer v6.0. Functional preprocessing with AFNI: field map and reverse-blip corrections, motion correction, alignment to anatomy, projection to cortical surface and reprojected to volume, minimal smoothing perpendicular to cortex; retinotopic data smoothed 5-mm FWHM on surface. Linear trends removed; for WM experiments, time series converted to percent signal change per run and z-scored within run. Voxel selection and delay window: For model input, voxel responses were averaged over a late-delay window (5.25–12.00 s from delay onset). Voxel selection per ROI used a prior one-item WM dataset in the same participants: ANOVA across 32 evenly spaced locations selected 750 voxels with highest F per ROI. Generative (encoding) model: Multivariate voxel activity given stimuli modeled as multivariate normal. Each voxel’s polar-angle tuning function modeled as weighted sum of eight evenly tiled raised-cosine basis functions f_k(s)=|cos(s−ϕ_k)|^8 (half-wave rectified). Response to two items computed as weighted sum of item-specific channel responses with item gain weights: for voxel i, b_i(s)=∑_k W_{ik}[W_H f_k(s_H)+W_L f_k(s_L)+η_k]+v_i. High-priority gain W_H fixed at 1; low-priority gain W_L fit as a free parameter by grid search with ordinary least squares for W. Noise: channel noise η∼N(0,σ^2 I) shared across similarly tuned voxels via W, and voxel-specific noise v∼N(0,Σ), with Σ=ρθθ^T+(1+ρ)I∘ττ^T (rank-one plus diagonal). Theoretical covariance Ω_0=ρθθ^T+(1−ρ)Iττ^T+σ^2 WW^T. Empirical sample covariance Ω_sample=(1/N_train)(B−WG)(B−WG)^T. Final covariance Ω=λΩ_0+(1−λ)Ω_sample with shrinkage parameter λ. Free covariance parameters (Ω_sample, τ, λ, ρ, σ) estimated following TAFKAP. Decoding: Leave-one-run-out cross-validation. For each held-out trial, computed two-dimensional likelihood p(b|s_H,s_L;θ). Task constraints from the precue implemented by restricting s_H and s_L to their respective semicircle ranges (binary mask). Posterior P(s_H,s_L|b;θ) computed and marginalized to 1D posteriors for each item. Decoded location = circular mean; decoded uncertainty = circular SD. Bagging was not used due to computational cost for 2D decoding. Statistics: Priority effects on behavioral memory error, saccade RT, decoded uncertainty, and decoding error assessed via permutation tests (2000 permutations, labels swapped within participant). Decoder performance assessed against null distributions from permuted likelihoods. Two-way repeated-measures ANOVA (permuted) tested ROI and priority effects on decoding error. Bootstrapped single-trial correlations (2000 iterations) assessed relationships between memory error and RT, and RT with decoded uncertainty metrics; FDR correction across ROIs (q=0.05). Neural WM prioritization index per trial defined as (P_H−P_L)/(P_H+P_L), where precision P=1/σ^2; trials binned into quartiles to relate neural prioritization to behavioral prioritization. Whole-brain GLM: regressors for stimulus, response, feedback, delay (constant amplitude), and delay amplitude modulated by decoded total WM resource (sum of precisions in V3AB) or neural prioritization index (precision difference in V3AB); 12 motion regressors included; regressors convolved with gamma HRF (peak 4.6 s). Group analysis on surface: voxel-level two-tailed t-test threshold P<0.05, cluster-size corrected P<0.05 using AFNI SurfClust and slow_surf_clustsim.

Behavior: High-priority items yielded smaller memory errors than low-priority items (permutation test, P = 0.043). Saccade response times (RTs) were shorter for high- vs low-priority items (P < 0.001). Memory error correlated positively with RT across all trials (M = 0.063, bootstrap P = 0.011) and within high-priority trials (M = 0.056, bootstrap P = 0.028), consistent with RT indexing decision uncertainty. Decoding performance: All ROIs showed above-chance decoding (permutation P < 0.05). Two-way repeated-measures ANOVA on decoding error revealed a main effect of ROI [F(9,90) = 36.13, P < 0.001] and an ROI × priority interaction [F(9,90) = 5.01, P < 0.001]. Decoding error was significantly lower for high- vs low-priority items in V3AB and IPS0 (FDR-corrected permutation P < 0.05). Neural gain: Estimated gain for low-priority items was <1 in V1–V3, V3AB, and IPS0 (bootstrapping P < 0.05), indicating reduced encoding gain for lower-priority memoranda. Across participants, higher low-priority gain was associated with reduced behavioral prioritization (smaller difference in memory error between low- and high-priority items), linking gain modulation to individual differences in prioritization. Decoded uncertainty: Decoded uncertainty was lower for high- vs low-priority items [main effect of priority, F(1,90) = 23.4, P < 0.01]. A neural WM prioritization index [(P_H−P_L)/(P_H+P_L)] predicted behavioral prioritization: larger differences in decoded precision (inverse variance) between items during delay were associated with stronger behavioral prioritization in V1, V3, and V3AB (permutation P < 0.05). Participants’ average neural prioritization correlated with model-derived resource weight allocated to the high-priority item. Link to behavior via RT: Probe-specific neural uncertainty predicted RT: negative correlations between RT and probe prioritization index (precision of probed minus nonprobed item), such that lower decoded uncertainty for the probed item yielded faster RTs; similar results held when correlating RT solely with decoded uncertainty of the probed item. Whole-brain GLM: Trial-wise total WM resource (sum of precisions from V3AB) covaried with delay-period amplitude in bilateral IPS (extending posteriorly toward V3AB), right sPCS, right iPCS, right lateral prefrontal cortex (middle frontal gyrus/inferior frontal sulcus), and bilateral posterior inferior temporal cortex (PIT). WM prioritization covaried with bilateral sPCS and right PIT. No anticorrelated clusters were found. Right sPCS delay activity negatively correlated with saccade RT, overlapping with sPCS clusters linked to total resource and prioritization.

The study tested whether human cortex represents WM content and uncertainty via probabilistic population codes whose gain is modulated by behavioral relevance. A novel Bayesian decoder demixed two concurrent WM items from shared neural populations using task constraints. Findings showed that striate and extrastriate neural gain tracked item priority, with high-priority items encoded at higher gain than low-priority items, predicting individual differences in behavioral prioritization. High-priority items were decoded with less error and lower uncertainty, matching probabilistic coding theory linking gain to precision and uncertainty. Trial-wise delay activity in frontal, parietal, and temporal association cortices predicted both total decoded WM resource and its prioritization, implicating top-down control. Bilateral sPCS, likely homologous to macaque frontal eye fields, covaried with prioritization, consistent with sPCS feedback signals sculpting visual cortical WM representations according to item relevance. The PIT clusters suggest a ventro-temporal node participates in attentional control and may coordinate dorsal–ventral pathways impacting WM. Together, the results provide a mechanistic account: behavioral prioritization operates by modulating encoding gain in visual cortex via feedback, yielding lower neural uncertainty and improved memory-guided behavior.

This work extends probabilistic population coding to multi-item WM, demonstrating that behavioral relevance modulates encoding gain, reducing uncertainty and error for prioritized items, and that frontal–parietal–temporal networks, especially sPCS, broadcast control signals that allocate WM resources by sculpting gain in visual cortex. The generative model-based decoder demixed concurrent representations and linked neural uncertainty to behavior (memory errors and RTs). Future research should: (1) extend the generative model to additional features and objects, and to more naturalistic stimuli and interactions; (2) quantify total WM resource by considering joint information across distributed brain regions rather than relying on a single visual ROI; (3) further test causal roles of candidate control regions (e.g., sPCS, PIT) and delineate pathways connecting dorsal attention networks to ventral temporal cortex.

Interpretation of total WM resource is cautious: the sum of decoded precisions may reflect nonspecific fluctuations (arousal, mental effort) and not a direct neural resource measure; authors are agnostic about a direct mapping from population spiking rate to WM resource. Decoded quality measures were strongest in visual cortex and largely absent in frontal cortex, limiting direct decoding of frontal representations. Computational constraints precluded bagging in the decoding (used in TAFKAP), potentially impacting robustness. ROI voxel selection leveraged a prior single-item dataset and may bias towards voxels informative for spatial WM; decoding focused on V3AB for resource metrics, which may underrepresent contributions from other regions. Imaging coverage excluded ventral temporal poles/orbitofrontal cortex in some subjects. Model assumptions (Gaussian noise, raised-cosine basis, linear mixing of item responses, fixed high-priority gain) may not capture all neural nonlinearities. Behavioral exclusions (e.g., large saccade errors) could reduce trial counts for some analyses, and the sample size (n=11) limits generalizability. GLM findings are correlational; causal interpretations rely on converging evidence rather than direct manipulation in this study.