Automatic Detection of Reactions to Music via Earable Sensing

We present GrooveMeter, a novel system that automatically detects vocal and motion reactions to music via earable sensing and supports music engagement-aware applications. We use smart earbuds as sensing devices and devise reaction detection techniques leveraging an IMU and a microphone on earbuds. To explore reactions in daily music-listening situations, we collect MusicReactionSet, containing 926 minutes of IMU and audio data with 30 participants. We identify unique challenges in detecting reactions accurately and robustly using audio and motion sensing (e.g., reaction-irrelevant events, background noise and motion artifacts, and mobile processing cost) and design processing pipelines to make reaction detection accurate and efficient. Comprehensive evaluation shows GrooveMeter achieves macro F1 scores of 0.89 for vocal reaction and 0.81 for motion reaction with leave-one-subject-out cross-validation, outperforming alternatives in accuracy and robustness, especially in noisy situations (e.g., cafe). A filtering approach reduces energy overhead by 50% or more. We also demonstrate potential applications through a case study including automatic music rating, familiarity detection, and reaction-based music recommendation.