The Cyborg Philharmonic: Synchronizing interactive musical performances between humans and machines

Music has long provided a medium to express emotions, intellect, passions, and moods through a blend of tempo, rhythm, texture, pitch, and other sonic features. Traditional and cognitive musicology explore theory, evolution, social context, therapeutic benefits, and links between physics and artistic expression. With advances in AI, cognitive models are increasingly used to model musical knowledge and support machine-generated or machine-supported musical performance. Technical challenges in multi-ensemble performance include: (a) synchronized generation of the musical form, (b) dynamic, fluid division of roles (leader–follower) communicated via non-verbal gestures or a conductor, and (c) real-time understanding of gestures and non-verbal communications. These require advanced cognitive capabilities for musical robots. Proposal outline: The manuscript proposes a joint strategy to capture the core features of performance—synchronization and anticipation—via two components: (i) Mapping, for instrument control and sensing tied to sound synthesis parameters to establish and maintain synchronization; and (ii) Modelling, to define responsiveness by predicting future state changes in music generation (e.g., leader transitions, beat/tempo changes) to pre-empt transitions and support seamless human–machine collaboration.

Background and preliminaries summarize the rise of machine creativity across art domains and the maturation of AI/ML for music. The evolution of robotic musicianship spans mechatronic systems for instruments (piano, strings, percussion, wind) and algorithmic composition, with deep learning enabling advances in composition and performance. Multi-musician settings introduce synchronization complexity, with prior robotic systems exploring improvisation and interactive ensembles. Synchronization models: Treating each musician as an oscillator enables analytical treatment of synchronization with varying intrinsic frequencies. Kuramoto’s model provides a scalable approach to phase and frequency synchronization; prior work showed reduced onset time error in human–robot ensembles and robustness under noise via agent-based models. Incorporating gestural cues (e.g., head nods, body movements) improves synchronization; however, leader selection and prediction remain challenging due to mutual influence and the fluidity of roles. Kalman filter approaches to onset/pedal prediction underperform human estimates. Predictive learning models: Artificial neural networks, RNNs, and LSTMs effectively model temporal dynamics and long-term dependencies, with applications to handwriting, audio-to-motion prediction, and temporal forecasting. These suggest a path to address dual needs: feature synchronization and predicting leader–follower transitions in temporal performance by integrating oscillator coupling with deep predictive models.

Architecture (Cyborg Philharmonic): A joint strategy integrates Mapping (sensing and control for synchronization) and Modelling (predictive cognition for leader–follower transitions and beat/tempo anticipation). Training setup: Deep learning models are trained by emulating real-time ensemble performances under controlled supervision. Diverse musical pieces (varying in beats, tempo, chord progression, etc.) are annotated with leader–follower transitions and temporal changes. During simulated performances, primary inputs include musical chord notations and annotated transitions; secondary real-time inputs include auditory and visual features captured during performance. Mapping module (sensing, control, synchronization): - Inputs: (1) MIDI notations via score-following to align auditory inputs with the score and provide low-latency, global context (e.g., via MidiScore); (2) wearable sensors for body sway/beat cues (e.g., foot taps, head jerks) filtered with Kalman/complementary filters; (3) audio captured by microphones (single or arrays) with attention to placement and denoised using ICA. - Beat/tempo estimation: Online beat tracking on ICA-filtered audio yields tempo estimates; window-size induced latency is mitigated by oscillator stabilization per musician. - Phase synchronization: Each musician is modeled as a Kuramoto oscillator with intrinsic frequency (tempo) and coupling constants among oscillators. The system evolves phases toward synchronization across the ensemble. MIDI information guides note generation in sync with ensemble state. Predicted synchronization states from the Modelling module initialize oscillators for faster convergence; identified leader signals weight leader streams (audio/body sensors) more heavily. Modelling module (predictive cognition): - Leader detection: Uses visual gaze/gesture analysis (e.g., GAZE SVM on multi-party gaze features, MVCC) to distinguish leaders and followers. For large ensembles where pure visual tracking is costly or noisy, the system internally simulates a conductor via an ensemble learning model that combines visual features with MidiScore-derived dominance, computing a leadership index per instrument. Time-series models (e.g., deep ensembles) learn relationships between chord progression, phase identification (from mapping), and leader transitions to predict candidate leaders early and reduce latency. - Beat/feature prediction: LSTM-based regression predicts sonic features (e.g., BPM) conditioned on identified leaders and musical notation, improving initialization of synchronization states and reducing mapping latency. Body sway can be mapped to audio phase (e.g., via an LSTM like PhaseMod) to strengthen phase detection. Demonstrations: - Single-instrument beat tracking and synchronization: An LSTM beat predictor trained on MUSDB18 (150 full songs, 44.1 kHz, ~10 h) closely followed Aubio’s BPM output; Kuramoto oscillators synchronized to a drum beat varying 85–170 BPM in steps of +5 every 64 beats. - Multi-instrument fugue: Using Bach’s “The Art of Fugue” (flute, oboe, clarinet, bassoon) with time-varying tempos and changing leadership, the system identified leader transitions and achieved sustained phase synchronization across parts using Kuramoto plus predicted beats/leaders.

- Proposed an integrated architecture (Cyborg Philharmonic) combining Kuramoto oscillator-based synchronization (Mapping) with deep learning for prediction of beats and leader transitions (Modelling) to enable real-time human–robot ensemble performance. - Demonstrated accurate beat prediction with an LSTM model that closely follows Aubio BPM on MUSDB18 (150 songs, 44.1 kHz, ~10 h). For a controlled input drum track varying from 85 to 170 BPM (+5 every 64 beats), Kuramoto oscillators achieved phase synchronization with the dominant instrument/beat. - Showcased multi-instrument synchronization and leader tracking in Bach’s “The Art of Fugue,” with correct identification of changing leaders among four woodwinds and successful maintenance of synchronization. - Showed how predictive outputs (beat, leader identity, phase) reduce latency in beat tracking and accelerate oscillator convergence by providing informed initial states and focused attention to leader streams. - Highlighted that integrating visual gaze/gesture cues, MIDI alignment, and wearable/body sway improves robustness of synchronization compared to audio-only approaches reported in prior literature.

The work addresses the core challenges of real-time synchronization and dynamic leader–follower role transitions in human–robot musical ensembles by fusing traditional synchronization theory with modern AI prediction. Predictive modelling anticipates future state changes (beats, leader shifts), enabling the oscillator network to converge faster and remain stable despite latencies and noise. The architecture’s modular interplay—where modelling informs mapping and mapping cues feed back into modelling—supports resilient, naturalistic interaction. Potential societal impacts include enabling individuals to perform with robotic ensembles (e.g., accessibility for those with medical constraints), reducing dependence on specific human performer availability, and maintaining musical engagement during disruptions (e.g., pandemics). The approach may catalyze new compositional practices and cross-genre explorations. The authors emphasize the need for evaluation frameworks capturing both objective alignment to scores (rhythm, tempo, synchronization) and subjective experiential aspects (emotion, expressivity, enjoyment) to assess robotic musicianship in context.

The manuscript presents a potential architecture (Cyborg Philharmonic) for synchronized human–robot ensemble performance by integrating oscillator-based synchronization with deep learning for prediction of beats and leader transitions. Preliminary demonstrations indicate feasibility in both single-instrument and multi-instrument scenarios. The authors argue this is a step toward practical robotic musicianship and real-time synchronized human–computer collaboration. Future directions include exploring advanced synchronization mechanisms (e.g., Janus oscillators for explosive synchronization and multi-stability), employing Quantum Neural Networks to enhance predictive power for leader–follower self-organization, and leveraging Swarm Intelligence for fully robotic, decentralized ensembles with scalable, adaptive, and robust coordination.

- The work primarily outlines an architecture with illustrative demonstrations rather than extensive large-scale, real-world evaluations across diverse ensembles and venues. - Real-time beat tracking suffers from window-size latency; while mitigations are proposed (predictive initialization, oscillator stabilization), residual delays in dynamic settings may persist. - Microphone placement and ambient noise complicate audio capture; multi-mic arrays increase complexity and computation. - Leader detection via gaze/gesture is challenging in large ensembles, with higher computational cost and potential errors; reliance solely on subtle gestures in small groups can be error-prone and introduce latency. - Prior methods (e.g., Kalman filter estimates for onset/pedal states) are noted to be less accurate than human performance; closing this gap remains difficult. - The approach depends on annotated training data for leader transitions and chord progressions under controlled conditions, which may limit generalizability without broader datasets and adaptive learning in the wild. - No standardized human-centered evaluation metrics for subjective musical expressivity are established here, limiting assessment of artistic quality.