Home

Awesome

Python 2.7 Python 3.6

Dancing to Music

PyTorch implementation of the cross-modality generative model that synthesizes dance from music.

Paper

Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, Jan Kautz
Dancing to Music Neural Information Processing Systems (NeurIPS) 2019
[Paper] [YouTube] [Project] [Blog] [Supp]

Example Videos

<p align='left'> <img src='imgs/example.gif' width='400'/> </p> <p align='left'> <img src='imgs/multimodal.gif' width='400'/> </p> <p align='left'> <kbd> <img src='imgs/long.gif' width='300'/> </kbd> </p> <p align='left'> <img src='imgs/v2v.gif' width='800'/> </p>

Train Decomposition

python train_decomp.py --name Decomp

Train Composition

python train_comp.py --name Decomp --decomp_snapshot DECOMP_SNAPSHOT

Demo

python demo.py --decomp_snapshot DECOMP_SNAPSHOT --comp_snapshot COMP_SNAPSHOT --aud_path AUD_PATH --out_file OUT_FILE --out_dir OUT_DIR --thr THR
python demo.py -decomp_snapshot snapshot/Stage1.ckpt --comp_snapshot snapshot/Stage2.ckpt --aud_path demo/demo.wav --out_file demo/out.mp4 --out_dir demo/out_frame

Citation

If you find this code useful for your research, please cite our paper:

@inproceedings{lee2019dancing2music,
  title={Dancing to Music},
  author={Lee, Hsin-Ying and Yang, Xiaodong and Liu, Ming-Yu and Wang, Ting-Chun and Lu, Yu-Ding and Yang, Ming-Hsuan and Kautz, Jan},
  booktitle={NeurIPS},
  year={2019}
}

License

Copyright (C) 2020 NVIDIA Corporation. All rights reserved. This work is made available under NVIDIA Source Code License (1-Way Commercial). To view a copy of this license, visit https://nvlabs.github.io/Dancing2Music/LICENSE.txt.