Awesome
MotionLLM: Understanding Human Behaviors from Human Motions and Videos
Ling-Hao Chen<sup>š 1, 3</sup>, Shunlin Lu<sup>š 2, 3</sup>, Ailing Zeng<sup>3</sup>, Hao Zhang<sup>3, 4</sup>, Benyou Wang<sup>2</sup>, Ruimao Zhang<sup>2</sup>, Lei Zhang<sup>š¤ 3</sup>
<sup>š</sup>Co-first author. Listing order is random. <sup>š¤</sup>Corresponding author.
<sup>1</sup>Tsinghua University, <sup>2</sup>School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-SZ), <sup>3</sup>International Digital Economy Academy (IDEA), <sup>4</sup>The Hong Kong University of Science and Technology
<p align="center"> <a href='https://arxiv.org/abs/2405.20340'> <img src='https://img.shields.io/badge/Arxiv-2405.20340-A42C25?style=flat&logo=arXiv&logoColor=A42C25'> </a> <a href='https://arxiv.org/pdf/2405.20340.pdf'> <img src='https://img.shields.io/badge/Paper-PDF-yellow?style=flat&logo=arXiv&logoColor=yellow'> </a> <a href='https://lhchen.top/MotionLLM'> <img src='https://img.shields.io/badge/Project-Page-%23df5b46?style=flat&logo=Google%20chrome&logoColor=%23df5b46'></a> <a href='https://research.lhchen.top/blogpost/motionllm'> <img src='https://img.shields.io/badge/Blog-post-4EABE6?style=flat&logoColor=4EABE6'></a> <a href='https://github.com/IDEA-Research/MotionLLM'> <img src='https://img.shields.io/badge/GitHub-Code-black?style=flat&logo=github&logoColor=white'></a> <a href='http://demo.humotionx.com/'> <img src='https://img.shields.io/badge/gradio-demo-red.svg'> </a> <a href='LICENSE'> <img src='https://img.shields.io/badge/License-IDEA-blue.svg'> </a> <a href="" target='_blank'> <img src="https://visitor-badge.laobi.icu/badge?page_id=IDEA-Research.MotionLLM&left_color=gray&right_color=%2342b983"> </a> </p>š° News
- [2024-06-17]: MoVid dataset Video data is now available at HuggingFace. Have a quick view on our data here.
- [2024-06-11]: CLI mode is supported. MotionLLM is running on A100 GPU on š¤HuggingFace demo (Posts on Twitter).
- [2024-05-31]: Paper, demo, and codes are released (Posts on Twitter).
š¤© Abstract
This study delves into the realm of multi-modality (i.e., video and motion modalities) human behavior understanding by leveraging the powerful capabilities of Large Language Models (LLMs). Diverging from recent LLMs designed for video-only or motion-only understanding, we argue that understanding human behavior necessitates joint modeling from both videos and motion sequences (e.g., SMPL sequences) to capture nuanced body part dynamics and semantics effectively. In light of this, we present MotionLLM, a straightforward yet effective framework for human motion understanding, captioning, and reasoning. Specifically, MotionLLM adopts a unified video-motion training strategy that leverages the complementary advantages of existing coarse video-text data and fine-grained motion-text data to glean rich spatial-temporal insights. Furthermore, we collect a substantial dataset, MoVid, comprising diverse videos, motions, captions, and instructions. Additionally, we propose the MoVid-Bench, with carefully manual annotations, for better evaluation of human behavior understanding on video and motion. Extensive experiments show the superiority of MotionLLM in the caption, spatial-temporal comprehension, and reasoning ability.
š¤© Highlight Applications
š§ Technical Solution
š» Try it
We provide a simple online demo for you to try MotionLLM. Below is the guidance to deploy the demo on your local machine.
Step 1: Set up the environment
pip install -r requirements.txt
Step 2: Download the pre-trained model
<details> <summary><b> 2.1 Download the LLM </b></summary>Please follow the instruction of Lit-GPT to prepare the LLM model (vicuna 1.5-7B). These files will be:
./checkpoints/vicuna-7b-v1.5
āāā generation_config.json
āāā lit_config.json
āāā lit_model.pth
āāā pytorch_model-00001-of-00002.bin
āāā pytorch_model-00002-of-00002.bin
āāā pytorch_model.bin.index.json
āāā tokenizer_config.json
āāā tokenizer.model
For how to get lit_model.pth
, please refer to issue 9.
If you have any confusion, we will update a more detailed instruction in couple of days.
</details> <details> <summary><b> 2.2 Dowload the LoRA and the projection layer of the MotionLLM </b></summary>We now release one versions of the MotionLLM checkpoints, namely v1.0
(download here). Opening for the suggestions to Ling-Hao Chen and Shunlin Lu.
wget xxx
Keep them in a folder named and remember the path (LINEAR_V
and LORA
).
2.3 Run the demo
<details> <summary><b> Choice 1: gradio demo </b></summary>GRADIO_TEMP_DIR=temp python app.py --lora_path $LORA --mlp_path $LINEAR_V
If you have some error in downloading the huggingface model, you can try the following command with the mirror of huggingface.
HF_ENDPOINT=https://hf-mirror.com GRADIO_TEMP_DIR=temp python app.py --lora_path $LORA --mlp_path $LINEAR_V
The GRADIO_TEMP_DIR=temp
defines a temporary directory as ./temp
for the Gradio to store the data. You can change it to your own path.
After thiess, you can open the browser and visit the local host via the command line output reminder. If it is not loaded, please change the IP address as your local IP address (via command ifconfig
).
python cli.py --lora_path $LORA --mlp_path $LINEAR_V
During inference, you can input the video path and your question to get the answer.
# Example here
Input video path: xxx.mp4
Your question: what xxx ?
================================
The man plan to xxx.
================================
</details>
š¼ To-Do
- Release the CLI demo of MotionLLM.
- Release the video demo of MotionLLM.
- Release the motion demo of MotionLLM.
- Release the MoVid dataset and MoVid-Bench.
- Release the tuning instruction of MotionLLM.
š Acknowledgement
The author team would like to deliver many thanks to many people. Qing Jiang helps a lot with some parts of manual annotation on MoVid Bench and resolves some ethics issues of MotionLLM. Jingcheng Hu provided some technical suggestions for efficient training. Shilong Liu and Bojia Zi provided some significant technical suggestions on LLM tuning. Jiale Liu, Wenhao Yang, and Chenlai Qian provided some significant suggestions for us to polish the paper. Hongyang Li helped us a lot with the figure design. Yiren Pang provided GPT API keys when our keys were temporarily out of quota. The code is on the basis of Video-LLaVA, HumanTOMATO, MotionGPT. lit-gpt, and HumanML3D. Thanks to all contributors!
š License
This code is distributed under an IDEA LICENSE. Note that our code depends on other libraries and datasets which each have their own respective licenses that must also be followed.
If you have any question, please contact at: thu [DOT] lhchen [AT] gmail [DOT] com AND shunlinlu0803 [AT] gmail [DOT] com.
š Star History
<p align="center"> <a href="https://star-history.com/#IDEA-Research/MotionLLM&Date" target="_blank"> <img width="500" src="https://api.star-history.com/svg?repos=IDEA-Research/MotionLLM&type=Date" alt="Star History Chart"> </a> <p>š Citation
@article{chen2024motionllm,
title={MotionLLM: Understanding Human Behaviors from Human Motions and Videos},
author={Chen, Ling-Hao and Lu, Shunlin and Zeng, Ailing and Zhang, Hao and Wang, Benyou and Zhang, Ruimao and Zhang, Lei},
journal={arXiv preprint arXiv:2405.20340},
year={2024}
}