Neural and computational underpinnings of biased confidence in human reinforcement learning

Confidence judgments reflect the subjective probability that choices or statements are correct and influence evidence integration, speed-accuracy tradeoffs, and changes of mind. Prior neuroimaging implicates two prefrontal networks: a positive network centered on VMPFC/pregenual ACC and a negative network including dACC, DMPFC, DLPFC, and insula. However, their distinct computational roles in confidence formation remain unclear. Theoretical accounts distinguish confidence (probability of being correct) from broader uncertainty representations; these can be confounded but are separable. The authors hypothesize a neurocomputational architecture where the negative network represents objective uncertainty (often inversely related to confidence), while VMPFC aggregates a composite, subjective feeling-of-confidence that integrates decision-related uncertainty with contextual biases. Using a reinforcement-learning task that orthogonalizes outcome valence (gain vs loss) and information (partial vs complete feedback), they test whether brain regions encode: (i) objective uncertainty (higher in partial vs complete, similar in gain vs loss), (ii) condition-specific confidence (within-context improvements due to learning), or (iii) task-wide confidence (absolute confidence showing higher levels in gain than loss contexts).

Meta-analyses and studies consistently find negative correlations between confidence and activity in dACC/DMPFC/DLPFC/insula and positive correlations in VMPFC/pgACC across value-based and perceptual tasks. dACC has been linked to error detection and uncertainty computation. VMPFC has been associated with subjective valuation and confidence/self-performance evaluation. Despite robust associations, formal dissociations of these networks' roles in confidence processing are scarce. Reinforcement learning literature typically links VMPFC to option values and value-based decision signals. Recent behavioral work shows valence-induced biases in confidence during RL (higher confidence in gains than losses despite similar accuracy) and suggests confidence builds from choice difficulty and choice-congruent value signals. The present study integrates these lines to dissociate network roles and compare confidence versus value encoding in VMPFC.

Participants: 40 healthy adults (23 female; mean age 22.69±4.44). Ethics approved by FMG-UvA. Incentivized task with show-up fee and performance-based bonuses; confidence incentivized via Matching Probabilities. Task: Probabilistic instrumental learning with a 2×2 within-subject manipulation: outcome valence (Gain vs Loss) and feedback information (Partial vs Complete). Four fixed cue pairs per run map to GP, LP, GC, LC conditions; cue outcomes were ±€1 or ±€0.1 with 75%/25% probabilities. Three runs of 80 trials (each pair repeated 20 times per run). Trial sequence separated option evaluation (symbol presentation) from motor response mapping to decouple decision and action. After choice, participants rated confidence (50–100% in 5% steps, randomized starting point). Feedback displayed chosen outcome only (Partial) or both chosen and unchosen (Complete). Behavioral analyses: Repeated-measures ANOVAs on choice accuracy, confidence, overconfidence (confidence−accuracy), and RTs; supplementary GLMMs. fMRI acquisition: 3T Philips Achieva, 32-channel coil. T1 MPRAGE (1 mm isotropic). T2*-EPI: 36 axial slices, 3×3×3 mm, TR=2000 ms, TE=28 ms, flip 76°, slice gap 0.3 mm. Three runs per subject. Field maps acquired. Preprocessing: SPM12 realignment/unwarp (field maps), co-registration, segmentation, normalization to MNI, smoothing 6 mm FWHM. GLMs: Five first-level GLMs modeled symbol, choice, confidence, outcome events with parametric modulators (all z-scored per session). Motion regressors included; no serial orthogonalization; VIFs < 5. Group-level one-sample t-tests (voxel p<0.001 uncorrected; cluster pFWE<0.05); ROI analyses with ANOVAs and t-tests. - GLM1: Symbol onsets split by context (GP/LP/GC/LC), each modulated by confidence; outcomes split by context, modulated by outcome. - GLM2WID/GLM2SPE: Single symbol regressor modulated by native (task-wide) confidence or within-condition z-scored (condition-specific) confidence. - GLM3: Model-based fMRI with latent RL variables at symbol onset: chosen value (Qc), unchosen value (Qu), context value (V); outcome PE modeled. - GLM4: Symbol onset modulated by Qc, |Qc−Qu| (difficulty), previous-trial confidence. - GLM5: Symbol onset modulated by confidence and Qc simultaneously. ROIs: Derived from GLM1 confidence maps (VMPFC, DMPFC/dACC, IFG+INS) and alternative VMPFC ROIs from GLM3 Qc maps and an independent meta-analysis. Computational modeling: Choices fit with 10 RL models spanning ABS (baseline Q-learning), REL (context-dependent), ASYM (confirmatory asymmetry), RELASYM (both features). Partial-information counterfactual inference variants tested (X*=0, Qu, Rc, Ru→t). Parameters optimized via L-BFGS-B minimizing negative log-posterior with weak priors (alpha,w ~ Beta(1.1,1.1); beta ~ Gamma(1.2,5)). Model evidence via Laplace approximation; random-effects BMS (expected frequency, exceedance probability, protected EP). Confidence models: Logistic regression of confidence using RL-derived variables: intercept; difficulty |Qc−Qu|; one valence-bias term (none, EQ=Qc+Qu, Qc, or V); and an autocorrelation term Conf(t−1) for learning phase. Fit via robust regression; model evidence via BIC and random-effects BMS. Data/code: Behavioral data on OSF; MRI on Donders Repository/Neurovault; code on OSF.

Behavioral: - Choice accuracy > chance (t39=17.78, P<0.001). Accuracy affected by information but not valence (ANOVA: valence F1,39=0.00, P=0.9666; information F1,39=22.05, P<0.001; interaction F1,39=0.01, P=0.9056). - Confidence affected by valence and information with interaction (valence F1,39=36.56, P<0.001; information F1,39=6.76, P=0.0131; interaction F1,39=9.62, P=0.0036). Higher confidence in Gain vs Loss, larger valence effect under Partial vs Complete (post-hoc: Partial t39=6.93, P=2.68×10^-8; Complete t39=4.55, P=5.08×10^-5; difference t39=3.10, P=0.0451). - Calibration (confidence−accuracy): overall not different from 0 (t1,39=0.1883, P=0.8516). Significant overconfidence in Gain-Partial (t1,39=2.14, P=0.0385). Calibration improved by Loss and Complete information (valence F1,39=12.28, P=0.0012; information F1,39=14.42, P<0.001). - RTs: small valence effect (F1,39=4.77, P=0.0350); RTs negatively correlated with confidence. Valence effects on RT and on confidence were uncorrelated across individuals (slope β=−0.01±0.01, P=0.339); confidence bias present even without RT bias (intercept β=5.02±0.84, P<0.001). Neuroimaging (model-free): - Confidence encoding networks during symbol presentation: Positive correlations in VMPFC/pgACC, precentral, middle temporal gyrus; negative correlations in DLPFC/DMPFC/dACC/insula and left caudate (voxel p<0.001 uncorrected; cluster pFWE<0.05). - ROI analyses: Confidence parametric encoding similar across contexts; no main effects of valence/information on confidence-parametric slopes. - Cue-evoked baseline activity patterns (GLM1): VMPFC showed a valence effect and marginal valence×information interaction consistent with task-wide confidence (ANOVA valence F1,37=8.99, P=0.0048; interaction F1,37=3.99, P=0.0532); greater Gain>Loss difference under Partial (0.62±0.18) than Complete (0.14±0.16; t37=1.99, P=0.0532). Negative network ROIs showed no significant valence/information effects (Ps>0.08). Whole-brain Gain>Loss also activated VMPFC. - Quantitative comparison of task-wide vs condition-specific confidence (GLM2): In VMPFC, native (task-wide) confidence modulator produced larger effects than within-condition z-scored (condition-specific) confidence (t37=5.41, P<0.001; 30/38 participants). Bayesian model selection suggested negative network ROIs favored condition-specific confidence (DMPFC EP: 82.22% for GLM2SPE; IFG+INS EP: 90.34%). Computational modeling: - RL: RELASYM model (context-dependent learning + confirmatory asymmetry; with counterfactual via Rc) best explained choices (protected EP=91%). - Confidence: Qc-REG model (bias term=chosen value Qc) best explained confidence (pEP>99%). Model-based fMRI: - GLM3: Chosen value Qc correlated with BOLD: VMPFC positive (t37=3.26, P=0.0023); DMPFC negative (t37=−4.96, P<0.001); IFG+INS negative (t37=−4.43, P<0.001). Unchosen value Qu positive in DMPFC (t37=2.96, P=0.0053) and IFG+INS (t37=2.75, P=0.0091). Whole brain: Qc significant clusters in VMPFC (positive) and IFG+INS (negative). - GLM4 (confidence components in Qc-activated ROIs): VMPFC correlated with Qc (t37=3.63, P<0.001), showed trends for difficulty |Qc−Qu| (t37=1.89, P=0.0657) and negative relation with previous confidence (t37=−2.16, P=0.0370), suggesting additional confidence-related variance beyond value. - GLM5 (Qc and confidence together) within value-based VMPFC ROIs (from this study and meta-analysis): Confidence regressors robust (Ps<0.001) and larger than Qc; Qc marginal/non-significant when competing with confidence (GLM3-ROI P=0.0553; Bartra ROI P=0.2324). Paired comparisons showed stronger confidence than Qc encoding (GLM3-Qc ROI: t37=2.13, P=0.0399; Bartra ROI: t37=2.85, P=0.0070). Spatial profiling across mPFC showed dominance of confidence over value along anterior–posterior and ventral–dorsal axes.

The study dissociates two confidence-related neural systems during reinforcement learning. VMPFC integrates a task-wide, subjective feeling-of-confidence that includes contextual biases (higher in gain than loss), aligning with explicit confidence reports. In contrast, DMPFC/DLPFC encode condition-specific confidence that builds within each learning context and does not reflect the valence-induced bias. These findings suggest a hierarchical architecture where dorsal/lateral regions process local comparison/uncertainty signals that may feed into VMPFC to construct global confidence estimates. Model-based analyses confirm VMPFC’s engagement in valuation (Qc) but show that, when value and confidence signals compete, confidence better accounts for VMPFC activity than RL-derived option values, challenging the dominant view that VMPFC primarily encodes value during RL. Negative network regions show signatures consistent with value comparison (opposite signs for Qc and Qu) and with condition-specific confidence. The absence of robust information (partial vs complete) effects on confidence encoding suggests either small effect sizes, inferred counterfactuals in partial feedback, or that objective uncertainty is encoded in neural variability rather than mean BOLD. Overall, results refine the neurocomputational account of metacognition in RL, highlighting VMPFC as a hub constructing global confidence from latent decision variables and contextual affect.

This work identifies a functional dissociation in prefrontal confidence encoding during reinforcement learning: VMPFC tracks task-wide, biased confidence that mirrors subjective reports, while dorsal/lateral prefrontal regions track condition-specific confidence. Computational and neuroimaging evidence indicates VMPFC activity is better explained by confidence than by option values when both are modeled, challenging the conventional valuation-centric view of VMPFC in RL. These insights advance understanding of how metacognitive signals are constructed from latent decision variables and contextual factors. Future research should: (1) clarify mechanisms linking condition-specific and task-wide confidence; (2) determine when VMPFC polarity with respect to confidence/uncertainty may reverse (e.g., exploration vs exploitation); (3) probe how confidence and its biases guide adaptive behavior and explore–exploit trade-offs; and (4) leverage confidence readouts to improve identification of valuation networks, with potential clinical applications targeting confidence dysfunctions.

- The study did not identify regions showing both confidence encoding and sensitivity to information (partial vs complete) manipulations; effects may be small or counterfactual inference may reduce differences. - Confidence elicitation might influence valuation signals, potentially altering typical VMPFC value coding. - RL-derived value estimates may imperfectly capture true latent values; residual misfit could bias comparisons between value and confidence signals. - The affiliation to uncertainty may be reflected in neural variability (not captured by mean BOLD), limiting detectability with standard fMRI analyses. - Superscript 8 affiliations for two authors were not detailed in the provided text (does not affect scientific findings but limits reporting completeness).