Home

Awesome

[ECCV'22] Fast Mesh Transformer

Paper | Project Page | Model Checkpoint

intro1 intro2

<img src="./assets/occlusion_v_28.gif" width="300" height="150"> <img src="./assets/occlusion_v_14.gif" width="300" height="150">

<img src="./assets/occlusion_h_28.gif" width="300" height="150"> <img src="./assets/occlusion_h_14.gif" width="300" height="150">


Notice


Overview

Transformer encoder architectures have recently achieved state-of-the-art results on monocular 3D human mesh reconstruction, but they require a substantial number of parameters and expensive computations. Due to the large memory overhead and slow inference speed, it is difficult to deploy such models for practical use. In this paper, we propose a novel transformer encoder-decoder architecture for 3D human mesh reconstruction from a single image, called FastMETRO. We identify the performance bottleneck in the encoder-based transformers is caused by the token design which introduces high complexity interactions among input tokens. We disentangle the interactions via an encoder-decoder architecture, which allows our model to demand much fewer parameters and shorter inference time. In addition, we impose the prior knowledge of human body's morphological relationship via attention masking and mesh upsampling operations, which leads to faster convergence with higher accuracy. Our FastMETRO improves the Pareto-front of accuracy and efficiency, and clearly outperforms image-based methods on Human3.6M and 3DPW. Furthermore, we validate its generalizability on FreiHAND.

overall_architecture


Installation

We provide two ways to install conda environments depending on CUDA versions.

Please check Installation.md for more information.


Download

We provide guidelines to download pre-trained models and datasets.

Please check Download.md for more information.

<a name="model_checkpoint"></a>

(Non-Parametric) FastMETRO

ModelDatasetPA-MPJPELink
FastMETRO-S-R50Human3.6M38.8Download
FastMETRO-S-R503DPW49.1Download
FastMETRO-L-H64Human3.6M33.6Download
FastMETRO-L-H643DPW44.6Download
FastMETRO-L-H64FreiHAND6.5Download

(Parametric) FastMETRO with an optional SMPL parameter regressor

ModelDatasetPA-MPJPELink
FastMETRO-L-H64Human3.6M36.1Download
FastMETRO-L-H643DPW51.0Download

Demo

We provide guidelines to run end-to-end inference on test images.

Please check Demo.md for more information.


Experiments

We provide guidelines to train and evaluate our model on Human3.6M, 3DPW and FreiHAND.

Please check Experiments.md for more information.


Results

This repository provides several experimental results:

table2 figure1 figure4 smpl_regressor


Acknowledgments

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00290, Visual Intelligence for Space-Time Understanding and Generation based on Multi-layered Visual Common Sense; and No. 2019-0-01906, Artificial Intelligence Graduate School Program (POSTECH)).

Our repository is modified and adapted from these amazing repositories. If you find their work useful for your research, please also consider citing them:


License

This research code is released under the MIT license. Please see LICENSE for more information.

SMPL and MANO models are subject to Software Copyright License for non-commercial scientific research purposes. Please see SMPL-Model License and MANO License for more information.

We use submodules from third party (hassony2/manopth). Please see NOTICE for more information.


Contact

Junhyeong Cho (jhcho99.cs@gmail.com)

FastMETRO (fastmetro.official@gmail.com)


Citation

If you find our work useful for your research, please consider citing our paper:

@InProceedings{cho2022FastMETRO,
    title={Cross-Attention of Disentangled Modalities for 3D Human Mesh Recovery with Transformers},
    author={Junhyeong Cho and Kim Youwang and Tae-Hyun Oh},
    booktitle={European Conference on Computer Vision (ECCV)},
    year={2022}
}

This work was done @ POSTECH Algorithmic Machine Intelligence Lab