Exploring how a Generative AI interprets music

This research explores how artificial intelligence (AI) organizes information into human-understandable concepts, moving beyond simple memorization through sheer computing power. The study uses a Variational Auto-Encoder (VAE), specifically Google Magenta's MusicVAE, to analyze musical melodies. This approach builds upon previous research that examined symmetry in 2D images using neural networks (NNs). However, a limitation of the prior work was the need to train a new NN for each image, necessitating statistical procedures like PCA to extract relevant information. This study improves on this by employing a VAE, whose bottleneck structure naturally allows for the extraction of meaningful information from a whole dataset using a single NN. Previous work has shown the utility of VAEs in extracting underlying physical invariants and independent variables in complex systems. This study extends this concept to the domain of music, aiming to identify patterns corresponding to human-defined musical concepts such as rhythm, pitch, and melody within the latent space of MusicVAE. The MusicVAE model, trained on a massive dataset of MIDI files, provides a 512-dimensional latent space representation for short musical sequences. The research questions whether the latent vectors are organized meaningfully and if these patterns relate to human concepts of music.

The paper cites several related studies. [1] details a previous study on symmetry detection in 2D images using NNs, highlighting the limitations addressed in the current research. [3] and [4] demonstrate the use of VAEs in extracting physical invariants and identifying independent variables in complex physical systems. These papers inspire the current approach of applying VAEs to the analysis of musical data. [2] introduces MusicVAE, the core model utilized in the study. [5] describes the LMD dataset used for analysis, while [10] details the phik library, used for computing nonlinear correlations. Several other papers on machine learning in physics ([6], [7], [8]) and generative adversarial networks [9] are also mentioned, indirectly supporting the overall framework of the research.

The researchers utilize Google Magenta's MusicVAE, a Variational Auto-Encoder with a 512-dimensional latent space. They analyze the latent representations of musical sequences, focusing on the distribution of latent variables for both real music and randomly generated note sequences. The analysis involves identifying "music neurons" (latent neurons with low variance and non-zero mean activations for real music) and "noise neurons" (neurons with high variance and near-zero mean activation). They use "Twinkle Twinkle Little Star" as an initial example to illustrate the encoding process and the distinction between music and noise neurons. To study this distinction further, they analyze a large sample (around 50,000) of melodies extracted from the LMD dataset. The correlation between latent neuron activations and human-defined musical features (rhythm, pitch, and melody, quantified using the music21 library and jSymbolic features) is evaluated using the phik correlation coefficient to capture nonlinear relationships. The analysis is extended to longer (16-bar) musical sequences to investigate the representation of melody.

The study reveals a significant disparity in the utilization of the latent space within MusicVAE. A majority of the 512 latent dimensions (hundreds of "noise neurons") remain largely inactive when processing real music. Only a small subset (around 37 "music neurons" for 2-bar sequences) actively encodes musical information. For 2-bar sequences, the first two music neurons consistently encode pitch and rhythm information, demonstrating a nonlinear relationship between these human-defined features and the latent variables. The correlation analysis using phik confirms this strong correlation. Melody information, in contrast, doesn't appear as an independently encoded feature in 2-bar sequences; its representation seems intertwined with rhythm information. This finding contrasts with approaches that try to enforce a linear mapping of human defined quantities onto the latent space. In the analysis of 16-bar sequences, the number of 'music neurons' nearly doubles, with the first few neurons encoding pitch and rhythm as before. Dedicated neurons for melody appear further down the list of neurons suggesting it plays a less prominent role than pitch and rhythm in MusicVAE's representation. The analysis of random note sequences provides further insights into the distinction between music and noise neurons. Real music excites fewer noise neurons than random sequences. For 2-bar sequences, both real music and random sequences activate a similar number of music neurons, indicating that the model might not strongly differentiate between short musical patterns and randomness based on the total number of active neurons alone.

The findings demonstrate that MusicVAE employs a highly efficient encoding scheme, utilizing only a fraction of its available latent space to represent music. The nonlinear encoding of pitch and rhythm into the first few music neurons suggests that the VAE discovers principal coordinates that simplify and extend the human-defined variables, rather than directly mapping them linearly. The relatively late emergence of dedicated melody neurons in longer sequences implies that MusicVAE might prioritize pitch and rhythm in its representation of music, with melody emerging as a higher-order consequence or playing a secondary role in the model's interpretation. The comparison between real and random music shows the usefulness of looking beyond the number of activated music neurons, instead looking at the number of noise neurons activated to tell the difference between real music and noise. This result should have implications for future generative models in music and beyond.

This research offers insights into how a generative AI model processes and represents music. The model efficiently encodes musical information using a small subset of its latent space. Pitch and rhythm are predominantly encoded in the first few latent dimensions, showcasing a nonlinear relationship, while melody's representation is less independent and emerges more significantly in longer sequences. This efficient encoding scheme suggests that the model discovers fundamental, principal coordinates for representing musical data. Future work could explore different musical datasets, architectures, and longer sequences to further understand the role of melody and other musical features in the model's latent representation.

The study focuses primarily on monophonic musical sequences. The findings might not generalize fully to more complex polyphonic music. The choice of MusicVAE architecture and training data also influences the results. Furthermore, the interpretation of melody's role in the latent representation is somewhat subjective and requires further investigation. The reliance on jSymbolic features for characterizing musical attributes may lead to limitations due to the inherent assumptions embedded within those features.