Music's universal appeal and its role in society have driven research into computer-aided music creation. Generative models, including GANs, VAEs, normalizing flows, autoregressive models, and diffusion models, have shown promise in various fields like image, speech, and text generation, also making advancements in music generation. However, high-fidelity music synthesis remains challenging due to the high dimensionality of audio signals and the difficulty in capturing long-range structures. Addressing this challenge typically involves using lower-dimensional representations of the audio signal, reducing computational complexity, and allowing models to better capture salient musical features. Time-Frequency (TF) representations, such as mel spectrograms, offer a powerful alternative by capturing important musical characteristics while minimizing computational complexity. Autoregressive models and GANs, while popular, face limitations such as slow inference (autoregressive) and unstable training/low sample diversity (GANs). Diffusion models, known for fast inference, simple training, and high-quality outputs, offer a promising alternative. This paper presents Msanii, a novel approach leveraging mel spectrograms and diffusion-based modeling to synthesize high-fidelity music efficiently, generating minutes of high-quality music without relying on common techniques like concatenative synthesis or compression.

The paper reviews existing music synthesis techniques, highlighting the challenges posed by the high dimensionality of audio signals. It discusses the use of lower-dimensional representations and TF representations like mel spectrograms in addressing this complexity. The limitations of autoregressive models (slow inference) and GANs (unstable training, low diversity, computational cost) are noted. The authors then emphasize the advantages of diffusion models—fast inference, simple training, and superior quality—making them suitable for music synthesis.

Msanii uses mel spectrograms as a lower-dimensional representation of audio, addressing the computational challenges of raw waveform synthesis. The model employs a novel U-Net architecture inspired by Vision Transformers, processing mel spectrograms as a sequence of tokens to reduce context size and improve efficiency. The U-Net incorporates residual blocks with Instance Normalization and Linear Attention for capturing both local and global context. A neural vocoder, based on ISTFTNet, reconstructs the audio from the generated mel spectrograms, using the Griffin-Lim algorithm for phase approximation. The diffusion process involves a forward process that adds Gaussian noise to the data and a reverse process that learns to generate samples from noise. The loss function is based on minimizing the difference between the predicted and true noise at each timestep. The model was trained on the POP909 dataset, with data preprocessing techniques including standard scaling and min-max scaling applied to ensure that the data is appropriately prepared for input into the model. The U-Net and Neural Vocoder models were trained separately, with the U-Net being significantly larger than the Neural Vocoder. Specific hyperparameters like U-Net width, number of blocks, timestep dimensionality, optimizer, learning rate, and batch size are detailed. The sampling process uses DDIM (Denoising Diffusion Implicit Models) and Griffin-Lim for phase recovery.

Msanii successfully synthesizes 190 seconds of stereo music at 44.1 kHz, a significant achievement in long-context, high-fidelity music generation. Subjective evaluation shows good long-term coherence and diversity in the generated samples. The model demonstrates the ability to maintain coherence for approximately 3 minutes, showcasing its capacity for long-context generation. While some degradation compared to human-generated music is observed (particularly in realism and naturalness), the diversity of samples is notable, especially given the relatively small training dataset. The model also demonstrates capability in audio-to-audio style transfer, interpolating between two audio sources, and, to a lesser extent, inpainting and outpainting. In the audio-to-audio task, low noise levels preserve structure but introduce noise, while high noise levels align the output more with the training data but lose the original structure. Similar behavior is found in the interpolation experiments. Inpainting and outpainting results are less satisfactory, often resulting in a lack of rhythmic consistency and coherence with the original audio. The generated sections lack rhythm and don't capture the melody and harmony accurately. This is especially noticeable in the outpainting tasks where the generated sections are vastly different from the original audio.

Msanii's ability to generate high-fidelity, long-context music using a diffusion-based approach is a significant advancement. The model's success in generating coherent, diverse audio over extended durations demonstrates the effectiveness of the combined mel spectrogram, diffusion model, and neural vocoder approach. The limitations in realism and naturalness, potentially attributable to Griffin-Lim phase reconstruction, suggest opportunities for future improvements. The model's successful application to style transfer and interpolation highlights its versatility, while the suboptimal performance in inpainting and outpainting indicates areas needing further development. The observation that the loss function doesn't show significant improvement during training is also important and implies potential areas of improvement in the training process or loss function.

Msanii offers a promising new approach to long-context, high-fidelity music synthesis. Its ability to generate minutes of coherent audio efficiently is a significant contribution. Future work should focus on improving realism and naturalness, exploring conditional generation, evaluating the model in realistic settings, addressing the limitations in inpainting/outpainting, and exploring scaling and real-time synthesis. Investigating more complex datasets with multiple instruments and refining techniques for global coherence would further advance the model's capabilities.

The current version of Msanii has limitations in terms of realism and naturalness in the generated audio, possibly due to phase reconstruction using the Griffin-Lim algorithm. The inpainting and outpainting functionalities require improvement to better preserve the original audio's musical structure and coherence. The model's performance has only been evaluated through subjective human listening tests, lacking quantitative metrics for a more rigorous assessment. The relatively small training dataset might limit the model's generalizability to various music styles and genres. The evaluation focused mainly on unconditional generation, and the model's performance with conditional parameters was not explored.