Caudate serotonin signaling during social exchange distinguishes essential tremor and Parkinson's disease patients

Parkinson's disease (PD) and essential tremor (ET) are clinically distinct neuromotor disorders that affect millions of individuals. PD is characterized by progressive motor symptoms (bradykinesia, rigidity, resting tremor) and non-motor impairments (sleep, neuropsychiatric vulnerability, cognitive deficits). A hallmark of PD is loss of dopaminergic neurons in the substantia nigra pars compacta, with coincident disruptions in other monoamines, including serotonin (5HT) and noradrenaline (NA). ET is predominantly a movement tremor, historically considered a motor disorder, but with increased risk of mild cognitive impairment and dementia, especially in late-onset cases affecting executive functions. Circuit-level differences include cerebellar GABAergic and excitatory balance alterations in ET versus widespread monoaminergic dysfunction in PD. Both disorders often undergo awake deep brain stimulation (DBS), with electrode trajectories passing through the caudate nucleus, a site implicated in reward processing and monoamine signaling. The study aims to characterize human caudate DA, NA, and 5HT release during a social exchange task and to test whether reward-related monoamine dynamics differ between PD and ET, potentially revealing disease-specific signatures of prediction error encoding and neurochemical boundaries between these disorders.

Prior work demonstrates monoamine systems, particularly dopamine and serotonin, encode prediction errors for reward and punishment and contribute to decision-making computations. PD involves dopaminergic neuron loss in SNc, reductions in 5HT and its metabolites, preferential declines in caudate 5HT innervation correlating with symptom severity, and noradrenergic dysfunction from locus coeruleus degeneration that can precede dopaminergic changes. These monoaminergic disruptions relate to cognitive and affective symptoms in PD. ET shows tremor with cerebellar circuit involvement (Purkinje cell changes, altered cerebellum–thalamus–motor cortex activity), and cognitive effects in attention and language, but lacks the widespread striatal monoamine deficits typical of PD, with monoamine systems largely spared except possible local cerebellar 5HT receptor changes. Human striatal DA and 5HT have been measured at sub-second timescales during decision tasks, supporting their role in prediction error signaling. Social exchange and fairness tasks (e.g., ultimatum game) recruit striatal circuits and are sensitive to monoamine modulation, making them suitable for probing disease-related alterations.

Design: Monoamine release (DA, NA, 5HT) was measured concurrently in the caudate nucleus during awake DBS surgery in patients with PD (n=12) or ET (n=6). Patients played a two-person ultimatum game (UG) as responders with 30 offers from a putative human proposer. Offers were presented in two blocks (15 trials each): low mean offers (μ=$4, σ=$1.5) and high mean offers (μ=$10, σ=$1.5), with block order counterbalanced (ET: low-to-high n=4, high-to-low n=2; PD: low-to-high n=3, high-to-low n=10). Each trial included a partner screen (4 s), offer screen (4 s), self-paced decision, and in 60% of trials an emotional rating (1=sad to 9=happy); inter-trial blank screen 2–4 s. Behavioral events were logged via custom HTML/JavaScript software. Electrochemistry: Carbon fiber microelectrodes were fabricated in-house and placed in the caudate. Voltammetry measured at 10 Hz using a triangular voltage ramp (−0.6 V to +1.4 V and back at 400 V/s; 10 ms) then held at −0.6 V for 90 ms; current sampled at 100 kHz. Precycling at 60 Hz equilibrated electrodes. In vitro training data: 76 electrodes collected datasets for DA, NA, 5HT, and pH. For each monoamine: 30 concentration vectors (0–2500 nM) plus five mixtures (840–1690 nM). pH datasets (7.0–7.8) included additional monoamine variations. Electrodes were positioned in a flow cell, precycled at 97 Hz, and recorded 65 s at 10 Hz (stable 15 s window used). Modeling: A deep convolutional neural network (InceptionTime-based with ResNet-like blocks) was adapted for multivariate regression to predict analyte concentrations from current measurements. Architecture used parallel convolutional layers (kernel sizes 1,10,20,40), batch normalization, ReLU, bottleneck layers, and global average pooling leading to a dense layer with four outputs (DA, NA, 5HT, pH). Training employed ADAM optimizer (initial LR 1e−3, halved after 5 epochs without validation loss reduction), mean squared error loss, 35 epochs, mixture-of-experts ensembles (20 models) with 90/10 train/validation split by electrode-concentration. Training set: 1,170,150 sweeps from 2,594 concentration combinations (70 electrodes); test set: 135,450 sweeps from 803 combinations (6 held-out electrodes). Tenfold cross-validation yielded high prediction accuracy (DA R²=0.997; NA R²=0.996; 5HT R²=0.998). In vivo predictions and preprocessing: Ensemble model predictions were averaged to estimate DA, NA, 5HT concentrations every 100 ms. Neurochemical time series aligned to offer onset (7 s window: 3 s pre, 4 s post); traces were linearly detrended, causally smoothed (0.5 s window), z-scored per trial, and area under the curve (AUC) computed from the 4 s post-offer window. Bayesian ideal observer: Participants were modeled as maintaining and updating beliefs that offers were drawn from a Normal distribution with unknown mean and variance, using conjugate priors (Normal-Inverse-χ²). Hyperparameters updated trial by trial; norm prediction error (NPE) computed as δ_i = s_i − ĥμ_i. Positive NPE = max(δ_i,0); Negative NPE = max(−δ_i,0). Initial hyperparameters: ĥμ_1=10, ĥσ²_1=4, k_1=4, v_1=10. Statistics: Linear mixed-effects models (R lmer, Type III ANOVA; random effect of subject) assessed behavioral and neural effects with specified models: B-M1 (Reaction time ~ Patient Group × NPE), B-M2 (Rejection rate ~ Patient Group × NPE), B-M3 (Emotional rating ~ Patient Group × NPE); neural: N-M1 (AUC_DA/NA/5HT ~ Neurotransmitter × NPE × Patient Group), N-M2 (ET: AUC ~ Neurotransmitter × NPE), N-M3 (PD: AUC ~ Neurotransmitter × NPE), N-M4 (DA: AUC ~ Patient Group × NPE), N-M5 (5HT: AUC ~ Patient Group × NPE), N-M6 (NA: AUC ~ Patient Group × NPE). Post-hoc pairwise comparisons were Bonferroni corrected. Dimensionality and classification: Singular value decomposition (SVD) applied to mean AUC differences (positive minus negative NPEs) across DA, NA, 5HT; logistic regression classifiers assessed separation of ET vs PD using NPE AUC vectors and right singular vectors; significance via 10,000-label permutation tests with AUROC evaluation.

Participants: ET n=6; PD n=12. Behavior: Reaction times did not differ by Patient Group × NPE (B-M1: F(1,555.12)=1.79, p=0.18). Negative NPE offers were rejected more frequently (B-M2 main effect of NPE: F(1,17)=41.11, p=6.44×10⁻⁸), with no Patient Group effect (F(1,17)=0.45, p=0.51). Emotional ratings were higher for positive vs negative NPEs (B-M3 main effect of NPE: F(1,323.69)=144.30, p=9.7×10⁻²⁵) and did not differ by group (F(1,17.29)=0.02, p=0.88). Neurochemistry: Overall, neurotransmitter responses varied by NPE and Patient Group (N-M1 Neurotransmitter × NPE: F(2,1669.02)=4.58, p=0.01; Neurotransmitter × NPE × Patient Group: F(2,1669.02)=3.86, p=0.021). • ET: Distinct DA/5HT patterns for positive vs negative NPEs (N-M2 Neurotransmitter × NPE: F(2,526.01)=6.28, p=0.002); DA increased and 5HT decreased for positive vs negative NPEs (DA: t(531)=2.239, p=0.026; 5HT: t(531)=−2.692, p=0.007). • PD: No significant DA/5HT patterns across NPE types (N-M3 Neurotransmitter × NPE: F(2,1143.01)=0.18, p=0.84). Across groups: DA showed higher AUC on positive vs negative NPEs (N-M4 main effect of NPE: F(1,558.31)=4.01, p=0.046), but group differences were not robust (p>0.05). 5HT differences were group-specific (N-M5 NPE × Patient Group: F(1,545.01)=4.46, p=0.035); 5HT response on positive NPE differed ET vs PD (t(81.3)=−2.281, Bonferroni p=0.025). NA showed no significant fluctuations related to NPEs in either group (N-M6: all p>0.05). Controls: Electrode placement, medication status, PD clinical features, and rejection-rate subgroups did not explain neurotransmitter differences (all p>0.05). Classification: SVD-derived latent dimensions (dominated by 5HT and DA NPE responses) separated ET and PD (logistic classifier accuracy 86%, p=0.04). Relationships between 5HT and DA release on positive and negative NPEs distinguished groups (accuracy 88%, p=0.03). Permutation tests indicated 5HT NPE AUCs drove separation; 5HT positive NPE response alone predicted disease state (AUROC=0.82, p=0.03).

Reward-evoked monoamine dynamics in the human caudate can dissociate ET from PD, with serotonin responses to norm prediction errors emerging as the key distinguishing feature. In ET, dopamine and serotonin exhibit opponent dynamics during positive NPEs (DA up, 5HT down), consistent with proposed DA–5HT opponency in appetitive versus aversive computations and with recent causal evidence in model organisms. PD lacks this opponency during the social exchange task, suggesting altered coordination of monoaminergic signaling related to reward processing. The robust group separation by 5HT responses highlights the importance of serotonergic dysfunction in PD beyond dopaminergic loss and points to potential neuromodulatory targets. Behavioral equivalence in emotional ratings across groups suggests neurochemical differences are not driven by differential emotional engagement. Overall, coordinated DA–5HT signaling appears central to encoding social reward prediction errors, and its disruption in PD may underlie cognitive and behavioral symptoms.

This study demonstrates that sub-second caudate monoamine release during social exchange, particularly serotonin responses to prediction errors, distinguishes ET from PD. ET exhibits DA–5HT opponency for positive prediction errors, absent in PD, and 5HT dynamics alone can classify disease state with high accuracy. These findings define a neurochemical boundary between the two disorders, provide insight into human reward processing in neuromotor disease, and motivate future work to map topographic monoamine dynamics, explore mechanistic contributions of serotonergic dysfunction to PD cognition, and extend intraoperative electrochemical approaches to larger cohorts and additional tasks.

The recording site was restricted to the caudate; striatal regional differences may yield distinct monoamine dynamics elsewhere. Healthy control measurements are not feasible intraoperatively, so ET cannot be assumed to reflect normative signaling despite limited evidence for broad monoaminergic dysfunction in ET. PD is heterogeneous; disease duration and symptom profiles may produce varied monoamine patterns, potentially requiring larger samples to detect subtypes. Sample size is modest but typical for intraoperative human studies. Emotional engagement differences could influence results, though emotional ratings did not differ by group. Antidepressant medication status did not affect main findings. Electrode positioning differences were examined and found not to explain group effects.