Awesome
Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset
<p align="center"> <a href='https://arxiv.org/abs/2307.00818'> <img src='https://img.shields.io/badge/Arxiv-2307.00818-A42C25?style=flat&logo=arXiv&logoColor=A42C25'> </a> <a href='https://arxiv.org/pdf/2307.00818pdf'> <img src='https://img.shields.io/badge/Paper-PDF-yellow?style=flat&logo=arXiv&logoColor=yellow'> </a> <a href='https://motion-x-dataset.github.io'> <img src='https://img.shields.io/badge/Project-Page-pink?style=flat&logo=Google%20chrome&logoColor=pink'></a> <a href='https://youtu.be/0a0ZYJgzdWE'> <img src='https://img.shields.io/badge/YouTube-Video-EA3323?style=flat&logo=youtube&logoColor=EA3323'></a> <a href='https://github.com/IDEA-Research/Motion-X'> <img src='https://img.shields.io/badge/GitHub-Code-black?style=flat&logo=github&logoColor=white'></a> <a href='LICENSE'> <img src='https://img.shields.io/badge/License-IDEA-blue.svg'> </a> <a href="" target='_blank'> <img src="https://visitor-badge.laobi.icu/badge?page_id=IDEA-Research.Motion-X&left_color=gray&right_color=orange"> </a> </p>This repository contains the implementation of the following paper:
Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset <br>Jing Lin<sup>π12</sup>, Ailing Zeng<sup>ππ€1</sup>, Shunlin Lu<sup>π13</sup>, Yuanhao Cai<sup>2</sup>, Ruimao Zhang<sup>3</sup>, Haoqian Wang<sup>2</sup>, Lei Zhang<sup>1</sup><br> <sup>π</sup>Equal contribution. <sup>π€</sup>Corresponing author.<sup>
<sup>1</sup>International Digital Economy Academy <sup>2</sup>Tsinghua University <sup>3</sup>The Chinese University of Hong Kong, Shenzhen<sup>
π₯³ News
- [2024.09.01] We have identified some issues with the new annotations of Motion-X++ and are working on resolving them. We have added the original Motion-X version using the same link we sent.
- [2024.4.25] We are working on a new version of Motion-X, named Motion-X++. It has the following updates: i) more paired modalities, including video, whole-body 2d keypoints, local and global whole-body SMPL-X, text, and audio (if the video has); ii) better quality, such as manual scene detection for temporally consistent video clips, more stable motion annotation, and improved semantic video caption via GPT4V and whole-body pose descriptions via Vicuna. We have released the IDEA400 subset and will release other subsets in the same directory. For detailed instructions on data preprocessing and loading, please refer to this document.
- [2024.2.6] We release the self-recorded IDEA400 videos and the corresponding SMPL-X to support (a) whole-body local or global pose estimation and (2) motion-condition video generation. Please check the email.
- [2024.1.9] We update the frame-level textual descriptions for each whole-body pose. Please download it here and refer to this usage guidance here.
- [2023.12.22] We update the sequential motion text descriptions (text_v1.1) augmented by Vicuna 1.5 to enhance the standardization and diversity of text. Please download via this link and replace it with the original file motionx_seq_text.zip. Many thanks to Linghao Chen for polishing the text labels!
- [2023.11.15] We release the rendered SMPL-X visualization of all subsets on DDS platform for quick content viewing.
- [2023.11.15] We release the HumanTOMATO motion representation (
tomato
representation) and split files. - [2023.10.26] We release the Motion-X-V1, which provides semantic text labels corresponding to SMPL-X sequences, facial expression motions, and the corresponding texts for augmenting some motions without facial expressions. Please kindly check your email!
- [2023.10.26] We release a high-quality monocular dataset named IDEA400 as a subset of Motion-X, which contains rich expressions and gestures. See this video for more details.
π TODO
- Release whole-body pose descriptions.
- Gathering more motion datasets (e.g., music-to-dance, audio-to-gesture motions).
- Release Videos after the agreement of video owners.
- Release audio and music if motions are needed.
Stay tuned!
π₯³ Highlight Motion Samples
<img src="assets/overview.gif" width="100%">π Table of Contents
π General Description
We propose a high-accuracy and efficient annotation pipeline for whole-body motions and the corresponding text labels. Based on it, we build a large-scale 3D expressive whole-body human motion dataset from massive online videos and eight existing motion datasets. We unify them into the same formats, providing whole-body motion (i.e., SMPL-X) and corresponding text labels.
Labels from Motion-X:
- Motion label: including
15.6M
whole-body poses and81.1K
motion clips annotation, represented as SMPL-X parameters. All motions have been unified in 30 fps. - Text label: (1)
15.6M
frame-level whole-body pose description and (2)81.1K
sequence-level semantic labels. - Other modalities: RGB videos, audio, and music information.
Supported Tasks:
- Text-driven 3d whole-body human motion generation
- 3D whole-body human mesh recovery
- Others: Motion pretraining, multi-modality pre-trained models for motion understanding and generation, etc.
π₯ Dataset Download
We disseminate Motion-X in a manner that aligns with the original data sources. Here are the instructions:
1. Request Authorization
Please fill out this form to request authorization to use Motion-X for non-commercial purposes. Then you will receive an email and please download the motion and text labels from the provided downloading links. The pose texts can be downloaded from here. Please extract the body_texts folder and hand_texts folder from the downloaded motionx_pose_text.zip.οΌNote: We updated the Baidu Disk link of motionx_seq_face_text.zip and motionx_face_motion.zip on October 29, 2023. Thus, if you download these zips via Baidu Disk before October 29, please fill out the form and download again.οΌ
<summary>Please collect them as the following directory structure: </summary>../datasets
βββ motion_data
βββ smplx_322
βββ idea400
βββ ...
βββ face_motion_data
βββ smplx_322
βββ humanml
βββ EgoBody
βββ GRAB
βββ texts
βββ semantic_labels
βββ idea400
βββ ...
βββ face_texts
βββ humanml
βββ EgoBody
βββ GRAB
βββ idea400
βββ ...
βββ body_texts
βββ humanml
βββ EgoBody
βββ GRAB
βββ idea400
βββ ...
βββ hand_texts
βββ humanml
βββ EgoBody
βββ GRAB
βββ idea400
βββ ...
2. Non-Mocap Subsets
For the non-mocap subsets, please refer to this link for a detailed instruction, notably:
- We do not distribute the original RGB videos. We provide the motion and text labels annotated by our team.
- Due to license and quality consideration, we do not provide NTU-RGBD120. Instead, we build IDEA400, which includes 400 daily actions (covering NTU-RGBD120). Please refer to this video for a detailed introduction.
3. Mocap Subsets
For the mocap datasets (i.e., AMASS, GRAB, EgoBody), please refer to this link for a detailed instruction, notably:
- We do not distribute the original motion data.
- We only provide the text labels and facial expressions annotated by our team.
The AMASS and GRAB datasets are released for academic research under custom licenses by Max Planck Institute for Intelligent Systems. To download AMASS and GRAB, you must register as a user on the dataset websites and agree to the terms and conditions of each license:
https://amass.is.tue.mpg.de/license.html
https://grab.is.tuebingen.mpg.de/license.html
<summary>Finally, the datasets folder is collected as the following directory structure:</summary>../datasets
βββ motion_data
βββ smplx_322
βββ humanml
βββ EgoBody
βββ GRAB
βββ idea400
βββ ...
βββ texts
βββ semantic_labels
βββ idea400
βββ ...
βββ face_texts
βββ humanml
βββ EgoBody
βββ GRAB
βββ idea400
βββ ...
βββ body_texts
βββ humanml
βββ EgoBody
βββ GRAB
βββ idea400
βββ ...
βββ hand_texts
βββ humanml
βββ EgoBody
βββ GRAB
βββ idea400
βββ ...
π Data Loading
-
To load the motion and text labels you can simply do:
import numpy as np import torch # read motion and save as smplx representation motion = np.load('motion_data/smplx_322/000001.npy') motion = torch.tensor(motion).float() motion_parms = { 'root_orient': motion[:, :3], # controls the global root orientation 'pose_body': motion[:, 3:3+63], # controls the body 'pose_hand': motion[:, 66:66+90], # controls the finger articulation 'pose_jaw': motion[:, 66+90:66+93], # controls the yaw pose 'face_expr': motion[:, 159:159+50], # controls the face expression 'face_shape': motion[:, 209:209+100], # controls the face shape 'trans': motion[:, 309:309+3], # controls the global body position 'betas': motion[:, 312:], # controls the body shape. Body shape is static } # read text labels semantic_text = np.loadtxt('semantic_labels/000001.npy') # semantic labels
π» Visualization
We support the visualization from the camera space and world space, please refer to this guidance.
π» Experiments
Validation of the motion annotation pipeline
Our annotation pipeline significantly surpasses existing SOTA 2D whole-body models and mesh recovery methods.
<p align="middle"> <img src="assets/motion_annot_exp.png" width=80%"> <br> </p>Benchmarking Text-driven Whole-body Human Motion Generation
<p align="middle"> <img src="assets/motion_generation_exp.png" width=80%"> <br> </p>Comparison with HumanML3D on Whole-body Human Motion Generation Task
<p align="middle"> <img src="assets/humanml_comp_exp.png" width=80%"> <br> </p>Impact on 3D Whole-Body Human Mesh Recovery
<p align="middle"> <img src="assets/mesh_recovery_exp.png" width=50%"> <br> </p>π€ Citation
If you find this repository useful for your work, please consider citing it as follows:
@article{lin2023motionx,
title={Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset},
author={Lin, Jing and Zeng, Ailing and Lu, Shunlin and Cai, Yuanhao and Zhang, Ruimao and Wang, Haoqian and Zhang, Lei},
journal={Advances in Neural Information Processing Systems},
year={2023}
}