Home

Awesome

Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset

<p align="center"> <a href='https://arxiv.org/abs/2307.00818'> <img src='https://img.shields.io/badge/Arxiv-2307.00818-A42C25?style=flat&logo=arXiv&logoColor=A42C25'> </a> <a href='https://arxiv.org/pdf/2307.00818pdf'> <img src='https://img.shields.io/badge/Paper-PDF-yellow?style=flat&logo=arXiv&logoColor=yellow'> </a> <a href='https://motion-x-dataset.github.io'> <img src='https://img.shields.io/badge/Project-Page-pink?style=flat&logo=Google%20chrome&logoColor=pink'></a> <a href='https://youtu.be/0a0ZYJgzdWE'> <img src='https://img.shields.io/badge/YouTube-Video-EA3323?style=flat&logo=youtube&logoColor=EA3323'></a> <a href='https://github.com/IDEA-Research/Motion-X'> <img src='https://img.shields.io/badge/GitHub-Code-black?style=flat&logo=github&logoColor=white'></a> <a href='LICENSE'> <img src='https://img.shields.io/badge/License-IDEA-blue.svg'> </a> <a href="" target='_blank'> <img src="https://visitor-badge.laobi.icu/badge?page_id=IDEA-Research.Motion-X&left_color=gray&right_color=orange"> </a> </p>

This repository contains the implementation of the following paper:

Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset <br>Jing Lin<sup>😎12</sup>, Ailing Zeng<sup>πŸ˜ŽπŸ€—1</sup>, Shunlin Lu<sup>😎13</sup>, Yuanhao Cai<sup>2</sup>, Ruimao Zhang<sup>3</sup>, Haoqian Wang<sup>2</sup>, Lei Zhang<sup>1</sup><br> <sup>😎</sup>Equal contribution. <sup>πŸ€—</sup>Corresponing author.<sup>

<sup>1</sup>International Digital Economy Academy <sup>2</sup>Tsinghua University <sup>3</sup>The Chinese University of Hong Kong, Shenzhen<sup>

πŸ₯³ News

πŸ“œ TODO

Stay tuned!

πŸ₯³ Highlight Motion Samples

<img src="assets/overview.gif" width="100%">

πŸ“Š Table of Contents

  1. General Description
  2. Dataset Download
  3. Experiments
  4. Citing

πŸ“œ General Description

We propose a high-accuracy and efficient annotation pipeline for whole-body motions and the corresponding text labels. Based on it, we build a large-scale 3D expressive whole-body human motion dataset from massive online videos and eight existing motion datasets. We unify them into the same formats, providing whole-body motion (i.e., SMPL-X) and corresponding text labels.

Labels from Motion-X:

Supported Tasks:

<div align="center"> <table cellspacing="0" cellpadding="0" bgcolor="#ffffff" border="0"> <tr> <th align="center">Dataset</th> <th align="center">Clip Number</th> <th align="center">Frame Number</th> <th align="center">Website</th> <th align="center">License</th> <th align="center">Downloading Link</th> </tr> <tr></tr> <tr> <td align="center"><b>AMASS</b></td> <td align="center">26K</td> <td align="center">5.4M</td> <td align="center"><a href="https://amass.is.tue.mpg.de/" target="_blank">AMASS<br>Website</a></td> <td align="center"><a href="https://amass.is.tue.mpg.de/license.html" target="_blank">AMASS<br>License</a></td> <td align="center"><a href="https://amass.is.tue.mpg.de/login.php" target="_blank">AMASS Data</a></td> </tr> <tr></tr> <tr> <td align="center"><b>EgoBody</b></td> <td align="center">1.0K</td> <td align="center">0.4M</td> <td align="center"><a href="https://sanweiliti.github.io/egobody/egobody.html" target="_blank">EgoBody<br>Website</a></td> <td align="center"><a href="https://egobody.ethz.ch/" target="_blank">EgoBody<br>License</a></td> <td align="center"><a href="https://egobody.ethz.ch/" target="_blank">EgoBody Data</a></td> </tr> <tr></tr> <tr> <td align="center"><b>GRAB</b></td> <td align="center">1.3K</td> <td align="center">0.4M</td> <td align="center"><a href="https://grab.is.tue.mpg.de/" target="_blank">GRAB<br>Website</a></td> <td align="center"><a href="https://grab.is.tue.mpg.de/license.html" target="_blank">GRAB<br>License</a></td> <td align="center"><a href="https://grab.is.tue.mpg.de/login.php" target="_blank">GRAB Data</a></td> </tr> <tr></tr> <tr> <td align="center"><b>IDEA400</b></td> <td align="center">12.5K</td> <td align="center">2.6M</td> <td align="center"><a href="https://motion-x-dataset.github.io/" target="_blank">IDEA400<br>Website</a> <td align="center"><a href="https://docs.google.com/document/d/1xeNQkkxD39Yi6pAtJrFS1UcZ2LyJ6RBwxicwQ2j3-Vs" target="_blank">IDEA400 License</a></td> <td align="center"><a href="https://docs.google.com/forms/d/e/1FAIpQLSeb1DwnzGPxXWWjXr8cLFPAYd3ZHlWUtRDAzYoGvAKmS4uBlA/viewform" target="_blank">IDEA400 Data</a> </td> </tr> <tr></tr> <tr> <td align="center"><b>AIST++</b></td> <td align="center">1.4K</td> <td align="center">0.3M</td> <td align="center"><a href="https://google.github.io/aistplusplus_dataset/" target="_blank">AIST++ <br>Website</a></td> <td align="center"><a href="https://google.github.io/aistplusplus_dataset/factsfigures.html" target="_blank">AIST++<br>License</a></td> <td align="center"><a href="https://docs.google.com/forms/d/e/1FAIpQLSeb1DwnzGPxXWWjXr8cLFPAYd3ZHlWUtRDAzYoGvAKmS4uBlA/viewform" target="_blank">AIST++ Data</a> </tr> <tr></tr> <tr> <td align="center"><b>HAA500</b></td> <td align="center">5.2K</td> <td align="center">0.3M</td> <td align="center"><a href="https://www.cse.ust.hk/haa/" target="_blank">HAA500<br>Website</a></td> <td align="center"><a href="https://www.cse.ust.hk/haa/index.html" target="_blank">HAA500<br>License</a></td> <td align="center"><a href="https://docs.google.com/forms/d/e/1FAIpQLSeb1DwnzGPxXWWjXr8cLFPAYd3ZHlWUtRDAzYoGvAKmS4uBlA/viewform" target="_blank">HAA500 Data</a> </tr> <tr></tr> <tr> <td align="center"><b>HuMMan</b></td> <td align="center">0.7K</td> <td align="center">0.1M</td> <td align="center"><a href="https://caizhongang.github.io/projects/HuMMan/" target="_blank">HuMMan<br>Website</a></td> <td align="center"><a href="https://caizhongang.github.io/projects/HuMMan/license.txt" target="_blank">HuMMan<br>License</a></td> <td align="center"><a href="https://docs.google.com/forms/d/e/1FAIpQLSeb1DwnzGPxXWWjXr8cLFPAYd3ZHlWUtRDAzYoGvAKmS4uBlA/viewform" target="_blank">HuMMan Data</a> </tr> <tr></tr> <tr> <td align="center"><b>BAUM</b></td> <td align="center">1.4K</td> <td align="center">0.2M</td> <td align="center"><a href="https://mimoza.marmara.edu.tr/~cigdem.erdem/BAUM1/" target="_blank">BAUM<br>Website</a> <td align="center"><a href="https://mimoza.marmara.edu.tr/~cigdem.erdem/BAUM1/" target="_blank">BAUM<br>License</a></td> <td align="center"><a href="https://docs.google.com/forms/d/e/1FAIpQLSeb1DwnzGPxXWWjXr8cLFPAYd3ZHlWUtRDAzYoGvAKmS4uBlA/viewform" target="_blank">BAUM Data</a> </td> </tr> <tr></tr> <tr> <td align="center"><b>Online Videos</b></td> <td align="center">32.5K</td> <td align="center">6.0M</td> <td align="center">---</td> <td align="center">---</a></td> <td align="center"><a href="https://docs.google.com/forms/d/e/1FAIpQLSeb1DwnzGPxXWWjXr8cLFPAYd3ZHlWUtRDAzYoGvAKmS4uBlA/viewform" target="_blank">Online Data</a> </tr> <tr></tr> <tr></tr> <tr style="background-color: lightgray;"> <td align="center"><b>Motion-X (Ours)</b></td> <td align="center">81.1K</td> <td align="center">15.6M</td> <td align="center"><a href="https://motion-x-dataset.github.io/" target="_blank">Motion-X Website</a></td> <td align="center"><a href="https://motion-x-dataset.github.io/static/license/Motion-X%20License.pdf" target="_blank">Motion-X License</a></td> <td align="center"><a href="https://docs.google.com/forms/d/e/1FAIpQLSeb1DwnzGPxXWWjXr8cLFPAYd3ZHlWUtRDAzYoGvAKmS4uBlA/viewform" target="_blank">Motion-X Data</a> </tr> </table> </div>

πŸ“₯ Dataset Download

We disseminate Motion-X in a manner that aligns with the original data sources. Here are the instructions:

1. Request Authorization

Please fill out this form to request authorization to use Motion-X for non-commercial purposes. Then you will receive an email and please download the motion and text labels from the provided downloading links. The pose texts can be downloaded from here. Please extract the body_texts folder and hand_texts folder from the downloaded motionx_pose_text.zip.(Note: We updated the Baidu Disk link of motionx_seq_face_text.zip and motionx_face_motion.zip on October 29, 2023. Thus, if you download these zips via Baidu Disk before October 29, please fill out the form and download again.οΌ‰

<summary>Please collect them as the following directory structure: </summary>
../datasets  

β”œβ”€β”€  motion_data
  β”œβ”€β”€ smplx_322
    β”œβ”€β”€ idea400
    β”œβ”€β”€ ...
β”œβ”€β”€  face_motion_data
  β”œβ”€β”€ smplx_322
    β”œβ”€β”€ humanml
    β”œβ”€β”€ EgoBody
    β”œβ”€β”€ GRAB
β”œβ”€β”€ texts
  β”œβ”€β”€  semantic_labels
    β”œβ”€β”€ idea400
    β”œβ”€β”€ ...
  β”œβ”€β”€  face_texts
    β”œβ”€β”€ humanml
    β”œβ”€β”€ EgoBody
    β”œβ”€β”€ GRAB
    β”œβ”€β”€ idea400
    β”œβ”€β”€ ...
  β”œβ”€β”€  body_texts
    β”œβ”€β”€ humanml
    β”œβ”€β”€ EgoBody
    β”œβ”€β”€ GRAB
    β”œβ”€β”€ idea400
    β”œβ”€β”€ ...
  β”œβ”€β”€  hand_texts
    β”œβ”€β”€ humanml
    β”œβ”€β”€ EgoBody
    β”œβ”€β”€ GRAB
    β”œβ”€β”€ idea400
    β”œβ”€β”€ ...

2. Non-Mocap Subsets

For the non-mocap subsets, please refer to this link for a detailed instruction, notably:

3. Mocap Subsets

For the mocap datasets (i.e., AMASS, GRAB, EgoBody), please refer to this link for a detailed instruction, notably:

The AMASS and GRAB datasets are released for academic research under custom licenses by Max Planck Institute for Intelligent Systems. To download AMASS and GRAB, you must register as a user on the dataset websites and agree to the terms and conditions of each license:

https://amass.is.tue.mpg.de/license.html

https://grab.is.tuebingen.mpg.de/license.html

<summary>Finally, the datasets folder is collected as the following directory structure:</summary>
../datasets  

β”œβ”€β”€  motion_data
  β”œβ”€β”€ smplx_322
    β”œβ”€β”€ humanml
    β”œβ”€β”€ EgoBody
    β”œβ”€β”€ GRAB
    β”œβ”€β”€ idea400
    β”œβ”€β”€ ...
β”œβ”€β”€ texts
  β”œβ”€β”€  semantic_labels
    β”œβ”€β”€ idea400
    β”œβ”€β”€ ...
  β”œβ”€β”€  face_texts
    β”œβ”€β”€ humanml
    β”œβ”€β”€ EgoBody
    β”œβ”€β”€ GRAB
    β”œβ”€β”€ idea400
    β”œβ”€β”€ ...
  β”œβ”€β”€  body_texts
    β”œβ”€β”€ humanml
    β”œβ”€β”€ EgoBody
    β”œβ”€β”€ GRAB
    β”œβ”€β”€ idea400
    β”œβ”€β”€ ...
  β”œβ”€β”€  hand_texts
    β”œβ”€β”€ humanml
    β”œβ”€β”€ EgoBody
    β”œβ”€β”€ GRAB
    β”œβ”€β”€ idea400
    β”œβ”€β”€ ...

πŸš€ Data Loading

πŸ’» Visualization

We support the visualization from the camera space and world space, please refer to this guidance.

πŸ’» Experiments

Validation of the motion annotation pipeline

Our annotation pipeline significantly surpasses existing SOTA 2D whole-body models and mesh recovery methods.

<p align="middle"> <img src="assets/motion_annot_exp.png" width=80%"> <br> </p>

Benchmarking Text-driven Whole-body Human Motion Generation

<p align="middle"> <img src="assets/motion_generation_exp.png" width=80%"> <br> </p>

Comparison with HumanML3D on Whole-body Human Motion Generation Task

<p align="middle"> <img src="assets/humanml_comp_exp.png" width=80%"> <br> </p>

Impact on 3D Whole-Body Human Mesh Recovery

<p align="middle"> <img src="assets/mesh_recovery_exp.png" width=50%"> <br> </p>

🀝 Citation

If you find this repository useful for your work, please consider citing it as follows:

@article{lin2023motionx,
  title={Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset},
  author={Lin, Jing and Zeng, Ailing and Lu, Shunlin and Cai, Yuanhao and Zhang, Ruimao and Wang, Haoqian and Zhang, Lei},
  journal={Advances in Neural Information Processing Systems},
  year={2023}
}