Awesome

Mug Diffusion

🎶 A charting AI for rhythm games. 🤖

MuG Diffusion is a charting AI for rhythm games based on Stable Diffusion (one of the most powerful AIGC models) with a large modification to incorporate audio waves. Given an audio file, MuG Diffusion is able to generate high-quality diverse charts, which is aligned with the music and highly controllable. Currently, it supports 4K vertical scroll rhythm game (VSRG) only, with the following control options:

• Difficulty: supporting both osu! star rating system and Etterna MSD system.
• Style: ranked beatmaps (osu!) / stable charts (Malody), or other beatmap styles.
• LNs: the ratio of the number of long notes to the total.
• Patterns: supporting all patterns in Etterna MSD system, including chordjack, stamina, stream, jumpstream, handstream and technical.

MuG Diffusion aims to support other rhythm games in the future (osu!standard, 5-8K VSRG, maimai, etc), and hopes to provide a beneficial AIGC tool for all the charters and players.

Installation and Running

Bundled Executable

I packaged a bundled executable containing all the dependencies and model weights in the Windows platform, which is available at:

• Google Drive
• hiosu

Unzip the file and double click "Mug Diffusion.exe", which will open a browser interface for controlling. It takes around 30 seconds on my computer (NVidia 3050Ti, 4GB memory) to generate four charts for a 3-minute-long audio.

Running from Source

If you use other platforms, other GPU types or want to run from source, here are the instructions.

• Install Python

• Install PyTorch

• Install other requirements:

pip install -r requirements.txt

• Install FFmpeg, make sure that ffmpeg command can execute correctly.

• Download the bundled executable, and copy the file models/ckpt/model.ckpt and models/ckpt/model.yaml to {REPOSITORY_ROOT}/models/ckpt/*.

• Run the WebUI:

python webui.py

Model Structure and Methodology

Acknowledgement

In order to ensure the fairness and transparency of training, the dataset list is published in here.

Thank all the Charters / Mappers in the community. It's you who endowed MuG Diffusion with intelligence. Besides, I would like to thank the Malody development teams (and many other supporters that cannot be listed due to space limit TAT) for the financial support.

Thank raber for webui development, RiceSS for logo design, and many testers for their support.

Special thanks:

• kangalio: for MinaCalc-standalone, which is a component of MSD controlling system.
• Evening: for reamberPy, which provides a very intuitive visualization of generated charts.

Credits

Charts created through MuG Diffusion are fully open source, explicitly falling under the CC0 1.0 Universal Public Domain Dedication. The model weights and charts created are non-commercial.

Besides, all charts created by AI are tagged with AIMode: MuG Diffusion vx.x.x in the [Meta] section. Please keep its integrity or mark it explicitly unless you modify the most of the notes, otherwise you will be at risk of abusing AI.