Home

Awesome

SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos (ECCV 2022)

This repo is the official implementation of "SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos". [Paper] [Project]

Update

It currently includes code, data, log and models for the following tasks:

Major Features

Description

When analyzing human motion videos, the output jitters from existing pose estimators are highly-unbalanced with varied estimation errors across frames. Most frames in a video are relatively easy to estimate and only suffer from slight jitters. In contrast, for rarely seen or occluded actions, the estimated positions of multiple joints largely deviate from the ground truth values for a consecutive sequence of frames, rendering significant jitters on them.

To tackle this problem, we propose to attach a dedicated temporal-only refinement network to existing pose estimators for jitter mitigation, named SmoothNet. Unlike existing learning-based solutions that employ spatio-temporal models to co-optimize per-frame precision and temporal smoothness at all the joints, SmoothNet models the natural smoothness characteristics in body movements by learning the long-range temporal relations of every joint without considering the noisy correlations among joints. With a simple yet effective motion-aware fully-connected network, SmoothNet improves the temporal smoothness of existing pose estimators significantly and enhances the estimation accuracy of those challenging frames as a side-effect. Moreover, as a temporal-only model, a unique advantage of SmoothNet is its strong transferability across various types of estimators and datasets. Comprehensive experiments on five datasets with eleven popular backbone networks across 2D and 3D pose estimation and body recovery tasks demonstrate the efficacy of the proposed solution. Our code and datasets are provided in the supplementary materials.

Results

SmoothNet is a plug-and-play post-processing network to smooth any outputs of existing pose estimators. To fit well across datasets, backbones, and modalities with lower MPJPE and PA-MPJPE, we provide THREE pre-trained models (Train on AIST-VIBE-3D, 3DPW-SPIN-3D, and H36M-FCN-3D) to handle all existing issues.

Please refer to our supplementary materials to check the cross-model validation in detail. Noted that all models can obtain lower and similar Accels than the compared backbone estimators. The differences are in MPJPEs and PA-MPJPEs.

Due to the temporal-only network without spatial modelings, SmoothNet is trained on 3D position representations only, and can be tested on 2D, 3D, and 6D representations, respectively.

3D Keypoint Results

DatasetEstimatorMPJPE (Input/Output):arrow_down:Accel (Input/Output):arrow_down:Pretrain model
AIST++SPIN107.17/95.2133.19/4.17checkpoint / config
AIST++TCMR*106.72/105.516.4/4.24checkpoint / config
AIST++VIBE*106.90/97.4731.64/4.15checkpoint / config
Human3.6MFCN54.55/52.7219.17/1.03checkpoint / config
Human3.6MRLE48.87/48.277.75/0.90checkpoint / config
Human3.6MTCMR*73.57/73.893.77/2.79checkpoint / config
Human3.6MVIBE*78.10/77.2315.81/2.86checkpoint / config
Human3.6MVideopose(T=27)*50.13/50.043.53/0.88checkpoint / config
Human3.6MVideopose(T=81)*48.97/48.893.06/0.87checkpoint / config
Human3.6MVideopose(T=243)*48.11/48.052.82/0.87checkpoint / config
MPI-INF-3DHPSPIN100.74/92.8928.54/6.54checkpoint / config
MPI-INF-3DHPTCMR*92.83/88.937.92/6.49checkpoint / config
MPI-INF-3DHPVIBE*92.39/87.5722.37/6.5checkpoint / config
MuPoTSTposeNet*103.33/100.7812.7/7.23checkpoint / config
MuPoTSTposeNet+RefineNet*93.97/91.789.53/7.21checkpoint / config
3DPWEFT90.32/88.4032.71/6.07checkpoint / config
3DPWEFT90.32/86.3932.71/6.30checkpoint / config(additional training)
3DPWPARE78.91/78.1125.64/5.91checkpoint / config
3DPWSPIN96.85/95.8434.55/6.17checkpoint / config
3DPWTCMR*86.46/86.486.76/5.95checkpoint / config
3DPWVIBE*82.97/81.4923.16/5.98checkpoint / config

2D Keypoint Results

DatasetEstimatorMPJPE (Input/Output):arrow_down:Accel (Input/Output):arrow_down:Pretrain model
Human3.6MCPN6.67/6.452.91/0.14checkpoint / config
Human3.6MHourglass9.42/9.251.54/0.15checkpoint / config
Human3.6MHRNet4.59/4.541.01/0.13checkpoint / config
Human3.6MRLE5.14/5.110.9/0.13checkpoint / config

SMPL Results

DatasetEstimatorMPJPE (Input/Output):arrow_down:Accel (Input/Output):arrow_down:Pretrain model
AIST++SPIN107.72/103.0033.21/5.72checkpoint / config
AIST++TCMR*106.95/106.396.47/4.68checkpoint / config
AIST++VIBE*107.41/102.0631.65/5.95checkpoint / config
3DPWEFT91.60/89.5733.38/7.89checkpoint / config
3DPWPARE79.93/78.6826.45/6.31checkpoint / config
3DPWSPIN99.28/97.8134.95/7.40checkpoint / config
3DPWTCMR*88.46/88.377.12/6.52checkpoint / config
3DPWVIBE*84.27/83.1423.59/7.24checkpoint / config

Getting Started

Environment Requirement

SmoothNet has been implemented and tested on Pytorch 1.10.1 with python >= 3.6. It supports both GPU and CPU inference.

Clone the repo:

git clone https://github.com/cure-lab/SmoothNet.git

We recommend you prepare the environment using conda:

# conda
source scripts/install_conda.sh

Prepare Data

All the data used in our experiment can be downloaded here.

Google Drive

Baidu Netdisk

The sructure of the repository should look like this:

|-- configs
    |-- aist_vibe_3D.yaml
    |-- ...
|-- data
    |-- checkpoints         # pretrained checkpoints
    |-- poses               # cleaned detected poses and groundtruth poses
    |-- smpl                # SMPL parameters
|-- lib
    |-- core
        |-- ...
    |-- dataset
        |-- ...
    |-- models
        |-- ...
    |-- utils
        |-- ...
|-- results                 # folders including log files, checkpoints, running config and tensorboard logs
|-- scripts
    |-- install_conda.sh
|-- eval_smoothnet.py       # SmoothNet evaluation
|-- train_smoothnet.py      # SmoothNet training
|-- README.md
|-- LICENSE
|-- requirements.txt

If you want to add your own dataset, please follow these steps (noted that this is also how the provided data is organized):

  1. Organize your data into corresponding type according to the body representation. The file structure is shown as follows:

    |-- [your dataset]\_[estimator]\_[2D/3D/smpl]
        |-- detected
            |-- [your dataset]\_[estimator]\_[2D/3D/smpl]_test.npz
            |-- [your dataset]\_[estimator]\_[2D/3D/smpl]_train.npz
        |-- groundtruth
            |-- [your dataset]\_gt\_[2D/3D/smpl]_test.npz
            |-- [your dataset]\_gt\_[2D/3D/smpl]_train.npz
    

    It is fine if you only have training or testing data. The content in each .npz file is consist of "imgname" and "human poses", which is related to the body representation you use.

    • 3D keypoints:

      • imgname: Strings containing the image and sequence name with format [sequence_name]/[image_name](string "" if the sequence_name and image_name not available).
      • keypoints_3d: 3D joint position. The shape of each sequence is corresponding_sequence_length*(keypoints_number*3). The order of it is the same with imgname
    • 2D keypoints

      • imgname: Strings containing the image and sequence name with format [sequence_name]/[image_name](string "" if the sequence_name and image_name not available).
      • keypoints_2d: 2D joint position. The shape of each sequence is corresponding_sequence_length*(keypoints_number*2). The order of it is the same with imgname
    • SMPL

      • imgname: Strings containing the image and sequence name with format [sequence_name]/[image_name](string "" if the sequence_name and image_name not available).
      • pose: pose parameters. The shape of each sequence is corresponding_sequence_length*72. The order of it is the same with imgname
      • shape: shape parameters. The shape of each sequence is corresponding_sequence_length*10. The order of it is the same with imgname
  2. If you use 3D keypoints as the body representation, add the root of all keypoints cfg.DATASET.ROOT_[your dataset]_[estimator]_3D in evaluate_config.py, train_config.py or visualize_config.py according to your purpose(test, train or visualize).

  3. Construct your own dataset following the existing dataset files. You might need to modify the detailed implementation depending on your data characteristics.

Training

Run the commands below to start training:

python train_smoothnet.py --cfg [config file] --dataset_name [dataset name] --estimator [backbone estimator you use] --body_representation [smpl/3D/2D] --slide_window_size [slide window size]

For example, you can train on 3D representation of Human3.6M using backbone estimator FCN with silde window size 8 by:

python train_smoothnet.py --cfg configs/h36m_fcn_3D.yaml --dataset_name h36m --estimator fcn --body_representation 3D --slide_window_size 8

You can easily train on multiple datasets using "," to split multiple datasets / estimator / body representation. For example, you can train on AIST++ - VIBE - 3D and 3DPW - SPIN - 3D with silde window size 8 by:

python train_smoothnet.py --cfg configs/h36m_fcn_3D.yaml --dataset_name aist,pw3d --estimator vibe,spin --body_representation 3D,3D  --slide_window_size 8

Note that the training and testing datasets should be downloaded and prepared before training.

Evaluation

Run the commands below to start evaluation:

python eval_smoothnet.py --cfg [config file] --checkpoint [pretrained checkpoint] --dataset_name [dataset name] --estimator [backbone estimator you use] --body_representation [smpl/3D/2D] --slide_window_size [slide window size] --tradition [savgol/oneeuro/gaus1d]

For example, you can evaluate MPI-INF-3DHP - TCMR - 3D and MPI-INF-3DHP - VIBE - 3D using SmoothNet trained on 3DPW - SPIN - 3D with silde window size 8, and compare the results with traditional filters oneeuro by:

python eval_smoothnet.py --cfg configs/pw3d_spin_3D.yaml --checkpoint data/checkpoints/pw3d_spin_3D/checkpoint_8.pth.tar --dataset_name mpiinf3dhp,mpiinf3dhp --estimator tcmr,vibe --body_representation 3D,3D --slide_window_size 8 --tradition oneeuro

Note that the pretrained checkpoints and testing datasets should be downloaded and prepared before evaluation.

The data and checkpoints used in our experiment can be downloaded here.

Google Drive

Baidu Netdisk

Visualization

Here, we only provide demo visualization based on offline processed detected poses of specific datasets(e.g. AIST++, Human3.6M, and 3DPW). To visualize on arbitrary given video, please refer to the inference/demo of MMHuman3D.

un the commands below to start evaluation:

python visualize_smoothnet.py --cfg [config file] --checkpoint [pretrained checkpoint] --dataset_name [dataset name] --estimator [backbone estimator you use] --body_representation [smpl/3D/2D] --slide_window_size [slide window size] --visualize_video_id [visualize sequence id] --output_video_path [visualization output video path]

For example, you can visualize the second sequence of 3DPW - SPIN - 3D using SmoothNet trained on 3DPW - SPIN - 3D with silde window size 32, and output the video to ./visualize by:

python visualize_smoothnet.py --cfg configs/pw3d_spin_3D.yaml --checkpoint data/checkpoints/pw3d_spin_3D/checkpoint_8.pth.tar --dataset_name pw3d --estimator spin --body_representation 3D --slide_window_size 32 --visualize_video_id 2 --output_video_path ./visualize

Citing SmoothNet

If you find this repository useful for your work, please consider citing it as follows:

@inproceedings{zeng2022smoothnet,
      title={SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos},
      author={Zeng, Ailing and Yang, Lei and Ju, Xuan and Li, Jiefeng and Wang, Jianyi and Xu, Qiang},
      booktitle={European Conference on Computer Vision},
      year={2022},
      organization={Springer}
}

Please remember to cite all the datasets and backbone estimators if you use them in your experiments.

License

This code is available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using this code you agree to the terms in the LICENSE. Third-party datasets and software are subject to their respective licenses.