Home

Awesome

Exemplar Fine-Tuning for 3D Human Pose Fitting Towards In-the-Wild 3D Human Pose Estimation

This repository contains pseudo-GT 3D human pose data produced by Exemplar Fine-Tuning (EFT) method, published in 3DV 2021. The 3D pose data is in the form of SMPL parameters, and this can be used as a supervision to train a 3D pose estimation algiritm (e.g., SPIN or HMR). We found that our EFT dataset is sufficient to build a model that is comparable to the previous SOTA algorithms without using any other indoor 3D pose dataset. See our paper for more details.

<p> <img src="docs/example1.jpg" height="256"> <img src="docs/example2.jpg" height="256"> <img src="docs/3432.gif" height="256"> </p>

This repository also contains the pre-trained 3D pose estimation model trained with our EFT dataset and monocular motion capture demo tools. See README_bodymocap.

<p> <img src="https://github.com/jhugestar/jhugestar.github.io/blob/master/img/eft_bodymocap.gif" height="256"> </p>

News:

Installing Requirements

It is convenient and safe to use conda environment

conda create -n venv_eft python=3.6
conda activate venv_eft
pip install -r requirements.txt

Download EFT Fitting data (json formats)

This repository only provides corresponding SMPL parameters for public 2D keypoint datasets (such as COCO, MPII). You need to download images from the original dataset website.

Run the following script to download our EFT fitting data:

sh scripts/download_eft.sh 
Dataset NameSampleNumManual FilteringFile Name
COCO2014-12kp28344NoCOCO2014-Part-ver01.json
COCO2014-6kp79051NoCOCO2014-All-ver01.json
COCO2014-Val10510YesCOCO2014-Val-ver10.json
MPII14361NoMPII_ver01.json
PoseTrack28856NoPoseTrack_ver01.json
LSPet-Train2946YesLSPet_ver01.json
LSPet-Test2433YesLSPet_test_ver10.json
OCHuman-Train2495YesOCHuman_train_ver10.json
OCHuman-Test1783YesOCHuman_test_ver10.json

Download Other Required Data

Download Images from Original Public DB website

Visualize EFT Fitting Results

Run EFT Fitting by yourself

Model Zoo

Motion Capture Demo

Citation

@inproceedings{joo2020eft,
  title={Exemplar Fine-Tuning for 3D Human Pose Fitting Towards In-the-Wild 3D Human Pose Estimation},
  author={Joo, Hanbyul and Neverova, Natalia and Vedaldi, Andrea},
  booktitle={3DV},
  year={2020}
}

License

CC-BY-NC 4.0. See the LICENSE file.

References

The body mocap code is a modified version of SPIN, and the majority of this code is borrowed from it.