DiffRhythm

Unlike multi-stage AI music systems that generate vocals and backing tracks separately, DiffRhythm synthesizes complete songs up to 4 minutes and 45 seconds with perfectly matched vocal and instrumental layers in approximately 10 seconds.

The model's architecture employs a non-autoregressive structure that enables parallel audio generation, combining a Variational Autoencoder (VAE) for compact latent waveform representation with a Diffusion Transformer (DiT) that operates in latent space to generate songs through iterative denoising.

This technical approach is what enables DiffRhythm's exceptional generation speed while maintaining musical coherence across the full song duration.

DiffRhythm demonstrates broad genre versatility, capable of generating pop, rock, ballads, electronic, jazz, and other styles with high musicality and natural-sounding vocal performance. The model has been validated for both English and Chinese lyrics with high intelligibility and natural pronunciation in both languages, making it suitable for multilingual music creation workflows.

The model's open-source release under the Apache 2.0 license makes it freely available for both research and commercial applications, enabling developers, researchers, and musicians to run DiffRhythm locally, integrate it into music production tools, or build applications on top of it without licensing fees.

A hosted web interface is also available at diffrhythm.ai for users who prefer not to run the model locally.

DiffRhythm 2, an updated version based on block flow matching, has been released with improvements in generation fidelity and controllability. The research backing the system has been published academically, reflecting its origins as a serious research contribution to the AI music generation field rather than a purely commercial product.