Home

Awesome

<br/> <div align="center"> <img src="resources/xrmogen-logo.png" width="600"/> </div> <br/>

Introduction

English | 简体中文

https://user-images.githubusercontent.com/81355712/187817864-37c984a5-032d-492b-8314-493a23c27e5e.mp4

XRMoGen is a codebase for motion generation. Currently, it contains two dance generation algorithms

For environment settings, see installation.md.

<!-- TOC --> <!-- TOC -->

Datasets

We recommend using pre-extracted features for music and motion data, see dataset_preparation.md. After downloading, extract it to $PROJECT/data. In order to facilitate the synthesis of a video with music after the dance is generated, it is needed to download the original music (.mov) to the musics folder in the same directory:

xrmogen
├── mogen
├── docs
├── configs
├── data
│   ├── aistpp_train_wav
│   ├── aistpp_test_full_wav
│   ├── aistpp_music_feat_7.5fps
│   ├── aist_features_zero_start
│   ├── musics
├── ...

Build a Model

Write a new dance generation model

The model structure can be customized through config files. To implement a new method, your model need to contain following functions/medhotds to fit the training/test pipeline:

To be specific, if we want to implement a new model, there are several things to do.

  1. create a new file in mogen/models/dance_models/my_model.py.

    from ..builder import NETWORKS
from ...builder import DANCE_MODELS

@DANCE_MODELS.register_module()
class MyDanceModel(nn.Module):

    def __init__(self, model_config):
        super().__init__()

    def forward(self, ...):
        ....

    def train_step(self, data, optimizer, **kwargs):
        ....

    def val_step(self, data, optimizer=None, **kwargs):
        ....

  2. import the model in mogen/models/__init__.py

    from .my_model import MyDanceModel

  3. write a config file that defines the model as

    model = dict(
    type='MyDanceModel',
    ....

Train a Model

Epoch Controls

XRMoGen uses mmcv.runner.EpochBasedRunner to control training and test.

In the training mode, the max_epochs in config file decide how many epochs to train. In test mode, max_epochs is forced to change to 1, which represents only 1 epoch to test.

Validation frequency is set as workflow of config file:

 workflow = [('train', 20), ('val', 1)]

Train

For example, to train Bailando (Dance Revolution),

python main.py --config configs/dance_rev.py

Arguments are:

Test

To test relevant model, add --test_only tag after the config path. We provide some pretrained weights to test (see pretrained_model_list.md. Download the pretrained weights under a folder ./example, and run

python main.py --config configs/bailando_test.py --test_only

to generate the dance poses. The poses will be stored under the workdir ("./bailando_test" in this case) set in config file.

To Compute the quantitative scores:

python tools/eval_quantitative_scores.py --pkl_root ./bailando_test/test/epoch0 --gt_root data/aist_features_zero_start --music_feature_root data/aistpp_test_full_wav

The results should be aligned with benchmark.md.

Visualize

python tools/visualize_dance_from_pkl.py --pkl_root ./bailando_test/test/epoch0  --audio_path data/musics/

Tutorials

Currently, we provide some tutorials for users to learn about

Other Documents

Except for that，The document also includes the following

License

The license of our codebase is Apache-2.0. Note that this license only applies to code in our library, the dependencies of which are separate and individually licensed. We would like to pay tribute to open-source implementations to which we rely on. Please be aware that using the content of dependencies may affect the license of our codebase. Refer to LICENSE to view the full license.

Citation

If you find this project useful in your research, please consider cite:

@misc{xrmogen,
    title={OpenXRLab Motion Generation Codebase},
    author={XRMoGen Contributors},
    howpublished = {\url{https://github.com/openxrlab/xrmogen}},
    year={2022}
}

Contributing

We appreciate all contributions to improve XRMoGen. Please refer to CONTRIBUTING.md for the contributing guideline.

Acknowledgement

XRMoGen is an open source project that is contributed by researchers and engineers from both the academia and the industry. We appreciate all the contributors who implement their methods or add new features, as well as users who give valuable feedbacks. We wish that the framework and benchmark could serve the growing research community by providing a flexible framework to reimplement existing methods and develop their own new models.

Projects in OpenXRLab