Implementing machine learning techniques for continuous emotion prediction from uniformly segmented voice recordings

The ability to recognize emotions from non-verbal cues, particularly vocalizations, is crucial for human-computer interaction and artificial intelligence. While previous research focused on longer audio segments with semantic content, this study explores the feasibility of emotion recognition from short, uniformly segmented (1.5 s) audio clips. This approach simulates real-world scenarios where continuous emotion detection is needed, irrespective of sentence boundaries or clear emotional onset/offset points. The 1.5 s window is a compromise: long enough to capture meaningful acoustic features and short enough to minimize the chance of multiple emotions within a segment. This approach uses Ekman's theory of basic emotions as a framework, acknowledging its limitations but emphasizing its practical value in creating a baseline classifier which could be adapted to more nuanced frameworks in the future. This study evaluates different machine learning techniques to build a tool capable of accurately classifying emotions from these short segments, aiming to match or exceed human accuracy for practical applications and to potentially reverse-engineer aspects of human emotion recognition.

Existing studies on emotion recognition from audio primarily used longer segments (1.5–5 s) from databases like Emo-DB and RAVDESS, achieving varying accuracies (81.2–95%). These studies employed diverse methods, including neural networks and convolutional neural networks (CNNs). However, their methodologies and datasets differed significantly from the current approach. This study also notes the work of Stresemann (2021), who standardized audio recordings to 1.5 s, focusing purely on emotion recognition independent of semantic content. This work builds upon that foundation by employing more advanced machine learning techniques and a more automated process. The study's unique contribution focuses on the practical challenges of real-time emotion detection from continuous speech using very short audio clips. This is a significant improvement over prior works, which mostly focused on pre-processed or clearly defined speech units.

This study used audio data from the Emo-DB (German) and RAVDESS (English) databases. Each recording was trimmed or padded to 1.5 s. The datasets were used in three versions: combined, Emo-DB only and RAVDESS only. A variety of audio features were extracted (e.g., spectral flatness, spectral centroid, fundamental frequency, spectral rolloff, spectral bandwidth, zero-crossing rate, root mean square, MFCC, etc.), alongside spectrograms. Three machine learning models were developed: a deep neural network (DNN), a convolutional neural network (CNN) for spectrogram analysis, and a hybrid model (C-DNN) combining both DNN and CNN. Model hyperparameters were optimized using Bayesian optimization. Model performance was evaluated using 10-fold cross-validation and a Bayesian updating approach, comparing results to human performance in a forced-choice emotion identification task with 61 participants. The human performance data was taken from a prior study conducted by Stresemann (2021) and used as a benchmark for comparison.

The C-DNN model showed the highest balanced accuracy in cross-validation on the combined dataset, significantly outperforming the CNN model. Bayesian accuracy estimations showed all models outperformed random classification (p>0.99), with the DNN and C-DNN models exhibiting comparable performance. Interestingly, DNN models trained on 3-second and 5-second segments did not yield significant improvements over the 1.5-second models, suggesting the 1.5 second segment could be a sufficient window for emotion detection. Analysis of saliency maps using SHAP values revealed that specific temporal portions of the 1.5-second audio segments were more crucial for emotion detection. When comparing individual datasets, the Emo-DB dataset yielded better results for DNN and C-DNN models than RAVDESS, however, this might be due to the size and diversity of the datasets rather than any inherent differences in the quality of emotion expression in the audio. When comparing model performance to human performance, the DNN and C-DNN models showed comparable accuracy to human participants in classifying the basic emotions and neutral, while the CNN model's performance varied considerably across different emotions.

The findings indicate that accurate emotion recognition is achievable from short (1.5 s) audio segments, using appropriate machine learning models, and the performance of the DNN and C-DNN models closely resembles human performance. The success of the DNN and C-DNN models, particularly in comparison to the lower-performing CNN model, suggests that the combination of extracted audio features, rather than raw spectrograms, might be more effective for emotion classification from short segments. The consistent outperformance of random classification and the comparable results to human participants support the validity and potential of the methodology for real-time applications. The varied performance across datasets highlights the importance of large, diverse datasets for robust model training, but the overall comparable performance across English and German suggests that the models identify emotion-related patterns transcending linguistic and cultural specificity. The comparable human and model performance also suggests similar pattern recognition mechanisms may be involved.

This study demonstrates the feasibility of continuous emotion recognition from short audio segments using DNN and C-DNN models. Future work should focus on addressing limitations, such as overfitting in the CNN model and exploring alternative methods to capture temporal dynamics of emotions (e.g., overlapping windows). Expanding the dataset's scope to include a broader range of emotions, cultures, and languages is crucial for enhanced generalizability. Developing a user-friendly software application could make this technology widely accessible for diverse applications.

The use of actor-produced emotional speech, rather than naturally occurring emotional expressions, may limit the generalizability of the findings. The 1.5 s segmentation could lead to a loss of information and influence the ability to capture the temporal dynamics of emotions, as evidenced by the CNN model overfitting. The variations in performance across different datasets and models necessitate the use of larger, more diverse datasets in future work. The reliance on a single, prior study for human performance data might introduce biases. Future work needs to address these limitations and increase the amount and type of data used.