Timing along the cardiac cycle modulates neural signals of reward-based learning

Prior research demonstrates that natural fluctuations in cardiac activity modulate brain activity linked to sensory stimuli and perceptual decisions, particularly concerning low-magnitude, near-threshold stimuli. However, the interaction between heart activity and internal, non-sensory representations mediating decision-making remains unclear. This study focuses on reward prediction errors (PEs), a well-studied internal representation, to determine whether the cardiac cycle influences their impact on learning. PE magnitude can be dissociated from sensory stimulus magnitude, allowing investigation of near-threshold PEs even with suprathreshold sensory events. Adaptive decisions rely on accurate subjective value estimates from past experiences. Reinforcement learning models utilize the difference between expectation and outcome (PE) to update these values. Signed PE indicates how much better or worse an outcome is than expected, while absolute PE (salience, surprise) represents the difference irrespective of direction. Separate neural networks process these PEs, with absolute PE potentially determining the extent of association adjustment. The cardiac cycle, comprising diastolic (relaxation) and systolic (contraction) phases, differentially signals to the brain via baroreceptor firing. Sensory perception and executive control are differently affected by the cardiac phase, with diastole associated with enhanced sensitivity to sensory signals and systole with enhanced executive processes. Learning is also influenced by cognitive and physiological arousal, with heart rate deceleration (longer diastole) possibly improving sensory intake and response. The study aims to examine the relationship between the cardiac cycle and quantitative learning indices (signed and absolute PEs) using model estimates, which capture cognitive and physiological fluctuations during learning. The study hypothesizes that cardiac cycle timing modulates neural representation of outcome, with near-threshold absolute PEs better represented during diastole.

Existing research highlights the influence of the cardiac cycle on sensory perception and executive functions. Studies show that individuals are more sensitive to near-threshold sensory stimuli during diastole than systole, due to changes in neural signals associated with cardiac activity. However, the impact of cardiac cycle timing on non-sensory internal representations involved in decision-making, particularly reward prediction errors, is less understood. Reinforcement learning models provide a framework for understanding how prediction errors, both signed and absolute, are used to update value estimates and guide behavior. Previous neuroimaging studies have linked signed and absolute prediction errors to different neural networks, suggesting distinct roles in learning. The interplay between these learning signals and cardiac activity, particularly concerning the salience of the absolute PE, was investigated. Analogous to cardiac cycle effects on near-threshold perception, the study examines whether cardiac timing modulates the impact of near-threshold absolute prediction errors on learning.

Thirty-two participants performed a reward-guided decision task involving predicting the color (orange or blue) associated with visual cues (faces and houses). Four association schemes were used across blocks: three predictive (highly predictive anticorrelated, highly predictive correlated, variable predictive) and one non-predictive. EEG and ECG were recorded during the outcome period (4 seconds). The researchers used four reinforcement learning models (Simple Cue, Recency Weighting, Dynamic Learning Rate, Conjunction) to model participants' choices, with Bayesian model comparison selecting the Simple Cue model as the best fit. Heartbeat-evoked potentials (HEPs) were analyzed using both event-related potential (ERP) and multivariate single-trial discriminant analysis (regularized Fisher Discriminant Analysis). ERP analyses examined HEP modulation as a function of signed PE, absolute PE, and outcome valence. Multivariate analysis identified whole-brain HEP components that predicted learning axes (signed and absolute PEs). Outcomes were categorized by whether their onset fell within systole or diastole. Analyses investigated the relationship between the HEP, model-based PEs, cardiac cycle timing and task performance (learning rates and rewards). Regression analyses investigated how the cardiac cycle influenced HEP amplitude and its relationship with learning rates and task performance.

ERP analyses revealed that HEP amplitude differed significantly between negative and positive signed prediction errors in frontocentral sites (p=0.004, Cohen's d=0.695), and between correct and incorrect outcomes (Cohen's d=0.696). HEP amplitudes also differed significantly between high and low surprise (absolute PE) trials in centro-parietal sites (p<0.003, Cohen's d=-0.724 and -1.17). Multivariate analysis showed a heart-related component reliably discriminating between high and low absolute PE outcomes (100-300ms post-R-wave), but no such component was found for signed PEs. This heart-related component, termed absPE-HEP, showed a parametric relationship with model-based absolute PEs. Only the first HEP after feedback contained significant information about absolute PE. The mean absPE-HEP was more negative during diastole compared to systole (p=0.007, Cohen's d=0.55). This difference was stronger for low-salience outcomes. A mixed-effects linear model revealed a main effect of cardiac cycle and an interaction between cardiac cycle and absolute PE on single-trial HEP variability. Higher residual variance in absPE-HEP predicted choice switching on subsequent trials. A regression analysis showed that participants with a stronger decrease in absPE-HEP during diastole compared to systole had higher learning rates and received more rewards (learning rates: p=0.037, r=0.391; reward: p=0.01, r=0.366). This relationship was only observed in blocks where learning was possible.

The findings demonstrate a clear link between HEP and absolute PEs, indicating that the cardiac cycle modulates the neural representation of how surprising an outcome is. The lack of a similar relationship with signed PEs suggests that the cardiac cycle's influence is more pronounced on attentional engagement related to the salience of the outcome. The cardiac cycle's impact on learning is analogous to its effects on decision-making, where larger cardiac responses are linked to more impactful information. The absPE-HEP might signal the need for increased attention to the outcome, given the current bodily state. The anterior insula, part of the salience network, is likely a key brain region for this interaction. The study highlights the importance of considering the timing of outcomes relative to the cardiac cycle in understanding learning and decision-making. The results suggest that, particularly when PE is small, learning is enhanced during diastole due to enhanced neuronal excitability and increased attentional breadth.

This study demonstrates that cardiac cycle timing modulates neural signals related to absolute prediction errors during reward-based learning. The stronger representation of absolute PEs during diastole, particularly for near-threshold PEs, translates into individual differences in learning rates and overall task performance. These findings offer valuable insights into brain-heart interactions and their influence on learning and decision-making. Future research should explore the precise relationship between trial-by-trial HEP amplitude changes and information integration after feedback, as well as the role of cardiac deceleration in learning.

The study's limitations include the use of a specific reward-learning task and the potential for confounding factors influencing the relationship between cardiac cycle and learning. Future studies should use different learning paradigms to improve generalizability. The study focused on absolute PEs, and more investigation is needed to clarify the cardiac cycle's interaction with signed PEs.