Combining predictive coding and neural oscillations enables online syllable recognition in natural speech

Understanding natural speech requires segmenting the continuous acoustic stream into discrete linguistic units like syllables. This process is believed to involve theta-gamma oscillation coupling, parsing syllables and encoding them in neural activity. However, speech comprehension also heavily relies on contextual cues, enabling prediction of speech structure and content. This study investigates how theta-gamma coupling influences bottom-up/top-down dynamics during online syllable identification. Existing research suggests a role for theta-gamma coupling in hierarchical speech processing, operating in a bottom-up manner without prior knowledge of syllable duration. Meanwhile, top-down predictive mechanisms, possibly linked to low-beta oscillatory activity, are essential for continuous speech perception. Predictive coding theory provides a framework where top-down predictions and bottom-up prediction errors interact to identify sensory signal causes, incorporating contextual and prior knowledge vital for speech recognition. While previous models focused either on line parsing or isolated speech segment recognition, combining predictive coding and neural oscillations offers the potential for enhanced biological realism, performance, and the exploration of orchestration between neurocentric levels of description.

The literature review highlights the established roles of neural oscillations, particularly theta and gamma coupling, in cognitive processes such as perception, memory, and attention. In the context of speech recognition, theta-gamma coupling is linked to hierarchical processing within syllables, handling their variable duration and temporal occurrence. Studies emphasize the importance of top-down predictive mechanisms leveraging contextual cues in anticipating speech content and temporal structure, potentially involving low-beta oscillatory activity. Predictive coding theory, alongside Analysis-By-Synthesis and the Bayesian Brain hypothesis, proposes internal models of sensory signal generation. This study builds upon previous neurocomputational models that demonstrated the utility of theta and gamma networks in speech pre-processing, while acknowledging the need to integrate bottom-up and top-down processes for a more complete understanding of speech perception.

The researchers designed a neurocomputational model, Process, based on the predictive coding framework. The model incorporates theta and gamma oscillatory functions to address the continuous nature of speech. The model's architecture includes a theta module driven by the slow amplitude modulations of the speech waveform, providing cues about syllable onsets and durations. This information is then used to reset and modulate gamma activity in the spectrotemporal module, which encodes the spectro-temporal structure of syllables. The model distinguishes between the 'what' (syllable identity and spectral representation) and the 'when' (syllable timing and duration), implemented through oscillatory processes. The model performance is evaluated based on its ability to correctly identify syllables in continuous speech. Several model variants were tested to assess the impact of theta-gamma coupling and the reset of accumulated evidence on syllable decoding. The TIMIT speech corpus provided the natural spoken English sentences used as input for the model. The input consisted of a time-frequency representation (auditory spectrogram) and the temporal modulation of the sound waveform. Syllable boundaries were defined using phonemic boundaries and English grammar rules, allowing for performance evaluation. The generative model used a hierarchical structure with a top level (theta module and spectrogram module) and a bottom level (amplitude fluctuations and slow amplitude modulation). The model employed Dynamic Expectation Maximization to invert the generative model, updating predictions based on prediction errors. The Ermentrout-Kopell canonical model was used to simulate theta oscillations, generating Gaussian pulses (theta triggers) that mark syllable onsets. The gamma module uses a discrete heuristic channel, where gamma unit activation provides processing windows for syllable encoding. The duration of the gamma sequence was modulated by a hidden variable, allowing for flexible adaptation to variable syllable durations. Syllable units tracked the evidence for each syllable, and the model included mechanisms for resetting accumulated evidence after processing a syllable, aiming to improve performance by preventing interference between syllables. The parameters of the model were adjusted using the first 15 consonants from the dataset; the remaining data was utilized for the performance evaluation. Model performance was measured by the percentage of correctly identified syllables and compared across different model variants. The Bayesian Information Criterion (BIC) was also used to compare models, accounting for accuracy and complexity. Simulations were performed with different speech compression rates to test the model's robustness to varying speech rates.

The Process model demonstrated above-chance performance in online syllable identification across all tested variants. However, significant differences were observed among variants. Models with theta-gamma coupling, irrespective of whether it was stimulus-driven (exogenous) or endogenously generated, outperformed models without coupling. The reset mechanism of accumulated evidence before processing new syllables significantly improved performance. Specifically, resetting the accumulated evidence based on the model’s information about the syllable content (here its spectral structure) was crucial. The model with exogenously driven theta-gamma coupling (Variant A) performed best when considering the Bayesian Information Criterion (BIC), indicating a better trade-off between accuracy and complexity. Although both exogenous and endogenous theta-gamma coupling models performed similarly with natural speech, at higher speech compression rates, the exogenously driven theta-gamma coupling model showed a significant advantage. The findings highlight the importance of theta-gamma coupling for temporal alignment of predictions with acoustic input and indicate that a stimulus-driven theta oscillation might be beneficial in challenging listening conditions. Simulations also revealed that theta-gamma coupling provides a more robust syllable parsing mechanism and improves accuracy and reduces interference between syllables. The accuracy of the onset detection played a critical role in the overall performance.

The findings support the hypothesis that predictive coding and neural oscillations work in concert for efficient online syllable recognition. The superior performance of models with theta-gamma coupling underscores the role of temporal precision in aligning internal predictions with sensory input. The significant impact of the reset mechanism points to the importance of preventing interference between neighboring syllables in continuous speech. The observation that both exogenous and endogenous theta-gamma coupling performed similarly under natural speech conditions, but diverged under high compression, suggests that a flexible, stimulus-driven system may be crucial for handling variable speech rates. The results align with experimental findings showing interactions between theta and gamma oscillations in speech processing, supporting the notion that theta-gamma coupling plays a crucial role in organizing and shaping neural encoding. The study provides a biologically plausible model of online syllable recognition, bridging between cognitive and neurophysiological levels of explanation, with implications for understanding the neural mechanisms of speech comprehension and improving automatic speech recognition (ASR) systems.

This research demonstrates that incorporating both predictive coding and neural oscillations, particularly theta-gamma coupling, significantly enhances the performance of a computational model for online syllable recognition in natural speech. The findings highlight the importance of both top-down predictive mechanisms and precise temporal alignment of internal predictions with sensory input. Future research could explore the role of other frequency bands, such as low-beta oscillations, in top-down processing and investigate how these models can be further improved to handle more complex aspects of speech processing and adverse listening conditions. The insights from this study offer valuable contributions to the fields of speech perception, cognitive neuroscience, and automatic speech recognition.

While the model successfully demonstrates the importance of theta-gamma coupling and predictive coding in syllable recognition, it is a simplified representation of the complex neural processes involved in human speech perception. The model may not fully capture the nuances of human speech processing, such as the influence of higher-level linguistic context or individual variations in speech perception. Additionally, the study primarily focuses on syllable recognition; the model could be extended to consider further linguistic units, such as words or phrases. Finally, the study used a specific speech corpus and preprocessing method; findings might vary with different datasets and methods.