A neural speech decoding framework leveraging deep learning and speech synthesis

Speech loss due to neurological disorders severely impacts quality of life. Electrocorticography (ECoG) in epilepsy patients offers high spatiotemporal resolution neural data for developing speech decoders. Two key challenges hinder robust neural speech decoding: limited duration of paired neural–speech data for training personalized models, and substantial within-speaker variability in production (rate, intonation, pitch), which complicates modeling. Prior work spans linear models and modern deep learning approaches, but often trades off interpretability, data efficiency, speaker specificity, and naturalness. Many prior systems also employ non-causal architectures that leverage future neural context or auditory feedback, limiting real-time applicability. This study proposes a framework addressing data scarcity and real-time constraints by decoding to a compact, interpretable acoustic parameter space and using a differentiable synthesizer, with subject-specific pre-training from speech alone to guide ECoG decoder training.

Early ECoG speech decoding used linear models that achieved modest accuracy (PCC around 0.6 or lower) but were data efficient and interpretable. Deep learning advances introduced convolutional and recurrent neural networks and explored diverse intermediate representations: articulatory kinematics (robust decoding but less natural voice), direct spectrogram mapping, random latent vectors with GAN-based synthesis, and self-supervised acoustic features such as HuBERT coupled to pretrained vocoders, which can yield natural speech but often lack speaker-dependent acoustics and require post-hoc voice conversion. Many prior systems use non-causal or bidirectional models, benefiting from feedback signals (e.g., auditory cortex) unavailable in real-time prostheses, limiting practical deployment. Recent single-patient studies achieved strong performance with large data and powerful priors but relied on generic acoustic features or extensive constraints. This work positions an interpretable, compact acoustic parameter space and a differentiable synthesizer as an alternative that preserves speaker identity, supports causal decoding, and trains effectively with limited paired data.

Framework: The pipeline comprises an ECoG decoder that maps neural signals to time-varying acoustic speech parameters and a differentiable speech synthesizer that renders these parameters into spectrograms; a Griffin–Lim algorithm converts spectrograms to waveforms. A companion speech-to-speech auto-encoder (speech encoder + the same synthesizer) is pre-trained per participant using speech only to generate reference speech parameters that guide the ECoG decoder. Speech synthesizer: A fully differentiable source–filter design generates the spectrogram at each time by mixing a voiced pathway (harmonic excitation at pitch f0 filtered by a bank of six time-varying formant filters with learnable speaker-specific prototype shapes) and an unvoiced pathway (white noise passed through a broadband filter plus the same formant filters). The two are mixed with a voice weight and scaled by loudness, with a learned stationary background noise spectrum added. In total, 18 time-local speech parameters are used (including pitch, six formant center frequencies and amplitudes, broadband unvoice center frequency, bandwidth and amplitude, voice weight, and loudness). Speaker-specific, time-invariant synthesizer parameters include prototype filter shapes for each formant and the unvoiced broadband filter, plus background noise spectrum parameters. Speech encoder (pre-training): A lightweight temporal convolution + channel MLP network takes linear (and for pitch also Mel) spectrograms and outputs the 18 parameters at 125 Hz. Frequency parameters are constrained to plausible ranges via sigmoid mapping and de-normalization. Semi-supervised training minimizes multi-scale spectral loss (linear and Mel spectrogram L1 and log differences), STOI+ intelligibility loss, and supervised losses on pitch and first four formants estimated with Praat. ECoG decoders: Three interchangeable architectures were implemented with causal and non-causal variants: (1) 3D ResNet with initial temporal convolutions followed by spatiotemporal residual 3D convolutions and upsampling via transposed temporal convolutions; (2) 3D Swin Transformer with spatiotemporal shifted-window self-attention over 3D patches, staged downsampling and upsampling, with temporal attention constrained to be causal or non-causal; (3) LSTM (three layers, bidirectional for non-causal or unidirectional for causal) followed by channel MLPs to predict parameters. Training jointly uses spectral MSS loss, STOI+ loss, supervision loss on Praat-estimated pitch/formants, and a reference loss aligning decoded parameters to those from the pre-trained speech encoder. Data and tasks: N=48 native English-speaking epilepsy patients with ECoG grids over perisylvian cortex were recorded (32 left-hemisphere, 16 right-hemisphere). Five tasks elicited speech of the same 50 target words across modalities: auditory repetition, auditory naming, sentence completion, visual reading, and picture naming (400 trials/participant; ~500 ms speech per trial). Grids: 43 low-density (clinical 8×8 macro, 10-mm spacing) and 5 hybrid-density arrays with interleaved micro electrodes. Audio was synchronized to neural recordings. Preprocessing included artifact rejection, CAR referencing, extraction of 70–150 Hz high-gamma envelope via Hilbert transform, downsampling to 125 Hz, and z-scoring relative to pre-stimulus silence. Spectral denoising was applied for noisy recordings. Training protocol: For each participant, 80% of trials were used for training and 20% for testing (50 trials), with additional word-level cross-validation holding out 10 unique words entirely from all training (including auto-encoder pre-training). Auto-encoder and ECoG decoders were trained in PyTorch with Adam (lr=1e-3, β1=0.9, β2=0.999). Loss weighting was tuned empirically. Decoding quality was evaluated primarily with Pearson correlation coefficient (PCC) between decoded and ground-truth spectrograms and STOI+. Contribution analysis: An occlusion study zeroed individual electrodes during inference to quantify contribution as the PCC reduction, projected to MNI space and averaged across participants. A shuffled-control model (training on mismatched ECoG–speech pairs) provided a noise-level map; contributions below noise were set to zero.

- Robust decoding across 48 participants using causal models: - ResNet achieved mean PCC 0.806 (non-causal) and 0.797 (causal); Swin 0.792 (non-causal) and 0.798 (causal); LSTM 0.745 (non-causal) and 0.712 (causal). - No significant difference between causal and non-causal ResNet (P=0.093) or Swin (P=0.196); causal LSTM was significantly worse than non-causal (P=0.009). - ResNet and Swin causal models were not significantly different (P=0.587). - Generalization to unseen words: Word-level cross-validation holding out 10 words during all training yielded performance comparable to standard trial-based validation for causal ResNet. - Parameter-level decoding accuracy (PCC across participants): voice weight 0.781; loudness 0.571; pitch f0 0.889; first formant f1 0.812; second formant f2 0.883. - Hemisphere comparison: Right-hemisphere decoding was comparable to left for causal models (ResNet PCC=0.790, P=0.623; Swin PCC=0.798, P=0.968), suggesting feasibility for right-hemisphere prosthetics. - Electrode density: Hybrid-density grids performed similarly to low-density grids in PCC; in HB participants, using all electrodes vs only LD-in-HB electrodes showed no significant PCC difference for 4/5 participants. - Contribution analysis: Non-causal models showed enhanced reliance on auditory cortex (STG), consistent with feedback utilization; causal models reduced STG reliance and emphasized frontal/sensorimotor regions. Left and right hemispheres showed similar causal contributions in ventral sensorimotor cortex. - Latency: End-to-end synthesis achievable with sub-50 ms delay (suitable for real-time use). - Cohort-level robustness: Causal ResNet PCC ranged ~0.62–0.92; 54% of participants exceeded PCC 0.8. The approach produced natural-sounding, speaker-specific reconstructions. - Open-source pipeline released with flexible decoder architectures and metrics.

The study demonstrates that decoding to a compact, interpretable acoustic parameter space coupled with a differentiable synthesizer enables accurate, speaker-specific speech reconstruction from ECoG with limited paired data. Critically, causal convolutional and transformer decoders match non-causal performance while avoiding dependence on auditory feedback signals from STG, supporting real-time BCI applicability and recruiting sensorimotor regions more consistent with production. The framework generalizes to unseen words, performs well across low- and hybrid-density grids, and shows robust decoding from right-hemisphere coverage, suggesting a viable path for prosthetics in patients with left-hemisphere damage. Compared to prior methods relying on generic features (e.g., HuBERT) or non-interpretable latent vectors (GANs), the explicit acoustic parameterization preserves speaker identity, improves data efficiency, and provides interpretability. The open-source, modular pipeline facilitates standardized benchmarking across architectures and datasets.

This work introduces a modular neural speech decoding framework that (1) uses an explicit, low-dimensional acoustic parameter space and a novel differentiable speech synthesizer for interpretable, speaker-specific reconstruction; (2) establishes the viability of causal ECoG decoders (ResNet and Swin) with performance on par with non-causal counterparts, enabling real-time deployment; (3) achieves strong decoding from both low-density grids and right-hemisphere coverage, broadening clinical applicability; and (4) provides a publicly available pipeline to support reproducible research. Future directions include adapting training to imagined or mimed speech without paired audible speech, extending decoders to non-grid and depth electrode configurations, scaling beyond a 50-word vocabulary to continuous speech, and validating right-hemisphere decoding in patients with left-hemisphere damage.

- Requires paired ECoG–speech data for training; for patients without overt speech, strategies using imagined or mimed speech and historical/proxy speech necessitate revised training to address temporal misalignment. - Current decoders assume grid-based electrode layouts; generalization to strips and depth electrodes (sEEG) needs tailored architectures. - Experiments focus on word-level decoding with a 50-word vocabulary; generalization to sentence-level, continuous speech remains to be demonstrated. - Right-hemisphere feasibility was shown in patients with intact left hemispheres; efficacy in patients with left-hemisphere damage remains untested. - Compared to recent single-subject, large-data studies, the present work uses less data per participant; scaling and long-term stability in chronic implants warrant further study.