Awesome

Deep Learning for 3D Human Pose Estimation and Mesh Recovery: A Survey

Authors: Yang Liu, Changzhen Qiu, Zhiyong Zhang*

School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen, Guangdong, China

Overview

This is the regularly updated project page of Deep Learning for 3D Human Pose Estimation and Mesh Recovery: A Survey, a review that primarily concentrates on deep learning approaches to 3D human pose estimation and human mesh recovery. This survey comprehensively includes the most recent state-of-the-art publications (2019-now) from mainstream computer vision conferences and journals.

Please create issues if you have any suggestions!

Citation

Please kindly cite the papers if our work is useful and helpful for your research.

@article{liu2024deep,
      title={Deep learning for 3D human pose estimation and mesh recovery: A survey}, 
      author={Liu, Yang and Qiu, Changzhen and Zhang, Zhiyong},
      journal={Neurocomputing},
      pages={128049},
      year={2024},
      issn={0925-2312},
      doi={https://doi.org/10.1016/j.neucom.2024.128049},
      publisher={Elsevier}
}

3D human pose estimation

• Single Person
  • In Images
    • Solving Depth Ambiguity
      • Optical-aware: VI-HC [paper], Ray3D [paper]
      • Appropriate feature representation: HEMlets [paper]
      • Joint aware: JRAN [paper]
    • Solving Body Structure Understanding
      • Limb aware: Wu et al. [paper], Deep grammar network [paper]
      • Orientation keypoints: Fisch et al. [paper]
      • Graph-based: Liu et al. [paper], LCN [paper], Modulated-CNN [paper], Skeletal-GNN [paper], HopFIR [paper], RS-Net [paper]
    • Solving Occlusion Problems
      • Learnable-triangulation [paper]
      • RPMS [paper]
      • Lightweight multi-view [paper]
      • AdaFuse [paper]
      • Bartol et al. [paper]
      • 3D pose consensus [paper]
      • Probabilistic triangulation model [paper]
    • Solving Data Lacking
      • Unsupervised learning: Kudo et al. [paper], Chen et al. [paper], ElePose [paper]
      • Self-supervised learning: EpipolarPose [paper], Wang et al. [paper], MRP-Net [paper], PoseTriplet [paper]
      • Weakly-supervised learning: Hua et al. [paper], CameraPose [paper]
      • Transfer learning: Adaptpose [paper]
  • In Videos
    • Solving Single-frame Limitation
      • VideoPose3D [paper]
      • PoseFormer [paper]
      • UniPose+ [paper]
      • MHFormer [paper]
      • MixSTE [paper]
      • Honari et al. [paper]
      • HSTFormer [paper]
      • STCFormer [paper]
    • Solving Real-time Problems
      • Temporally sparse sampling: Einfalt et al. [paper]
      • Spatio-temporal sparse sampling: MixSynthFormer [paper]
    • Solving Body Structure Understanding
      • Motion loss: Wang et al. [paper]
      • Human-joint affinity: Dc-Net [paper]
      • Anatomy-aware: Chen et al. [paper]
      • Part aware attention: Xue et al. [paper]
    • Solving Occlusion Problems
      • Optical-flow consistency constraint: Cheng et al. [paper]
      • Multi-view: MTF-Transformer [paper]
    • Solving Data Lacking
      • Unsupervised learning: Yu et al. [paper]
      • Weakly-supervised learning: Chen et al. [paper]
      • Semi-supervised learning: MCSS [paper]
      • Self-supervised learning: Kundu et al. [paper], P-STMO [paper]
      • Meta-learning: Cho et al. [paper]
      • Data augmentation: PoseAug [paper], Zhang et al. [paper]
• Multi-person
  • Top-down
    • Solving Real-time Problems
      • Multi-view: Chen et al. [paper]
      • Whole body: AlphaPose [paper]
    • Solving Representation Limitation
      • VoxelTrack [paper]
    • Solving Occlusion Problems
      • Wu et al. [paper]
    • Solving Data Lacking
      • Single-shot: PandaNet [paper]
      • Optical-aware: Moon et al. [paper]
  • Bottom-up -Solving Real-time Problems - Fabbri et al. [paper]
    • Solving Supervisory Limitation.
      • HMOR [paper]
    • Solving Data Lacking
      • Single-shot: SMAP [paper], Benzine et al. [paper]
    • Solving Occlusion Problems
      • Mehta et al. [paper]
      • LCR-Net++ [paper]
  • Others
    • Single Stage
      • Jin et al. [paper]
    • Top-down & Bottom-up
      • Cheng et al. [paper]

Human Mesh Recovery

• Template-based
  • Naked
    • Multimodal Methods
      • Hybrid annotations: Rong et al. [paper]
      • Optical flow: DTS-VIBE [paper]
      • Silhouettes: LASOR [paper]
      • Cropped image and bounding box: CLIFF [paper]
    • Utilizing Attention Mechanism
      • Part-driven attention: PARE [paper]
      • Graph attention: Mesh Graphormer [paper]
      • Spatio-temporal attention: MPS-Net [paper], PSVT [paper]
      • Efficient architecture: FastMETRO [paper], Xue et al. [paper]
      • End-to-end structure: METRO [paper]
    • Exploiting Temporal Information
      • Temporally encoding features: Kanazawa et al. [paper]
      • Self-attention temporal: VIBE [paper]
      • Temporally consistent: TCMR [paper]
      • Multi-level spatial-temporal attention: MAED [paper]
      • Temporally embedded live stream: TePose [paper]
      • Short-term and long-term temporal correlations: Glot [paper]
    • Multi-view Methods
      • Confidence-aware majority voting mechanism: Dong et al. [paper]
      • Probabilistic-based multi-view: Sengupta et al. [paper]
      • Dynamic physics-geometry consistency: Huang et al. [paper]
      • Cross-view fusion: Zhuo et al. [paper]
    • Boosting Efficiency
      • Sparse constrained formulation: SCOPE [paper]
      • Single-stage model: BMP [paper]
      • Process heatmap inputs: HeatER [paper]
      • Removing redundant tokens: TORE [paper]
    • Developing Various Representations
      • Texture map: TexturePose [paper]
      • UV map: Zhang et al. [paper], DecoMR [paper], Zhang et al. [paper]
      • Heat map: Sun et al. [paper], 3DCrowdNet [paper]
      • Uniform representation: DSTFormer [paper]
    • Utilizing Structural Information
      • Part-based: holopose [paper]
      • Skeleton disentangling: Sun et al. [paper]
      • Hybrid inverse kinematics: HybrIK [paper], NIKI [paper]
      • Uncertainty-aware: Lee et al. [paper]
      • Kinematic tree structure: Sengupta et al. [paper]
      • Kinematic chains: SGRE [paper]
    • Choosing Appropriate Learning Strategies
      • Self-improving: SPIN [paper], ReFit [paper], You et al. [paper]
      • Novel losses: Zanfir et al. [paper], Jiang et al. [paper]
      • Unsupervised learning: Madadi et al. [paper], Yu et al. [paper]
      • Bilevel online adaptation: Guan et al. [paper]
      • Single-shot: Pose2UV [paper]
      • Contrastive learning: JOTR [paper]
      • Domain adaptation: Nam et al. [paper]
  • Detailed
    • With Clothes
      • Alldieck et al. [paper]
      • Multi-Garment Network (MGN) [paper]
      • Texture map: Tex2Shape [paper]
      • Layered garment representation: BCNet [paper]
      • Temporal span: H4D [paper]
    • With Hands
      • Linguistic priors: SGNify [paper]
      • Two-hands interaction: [paper]
      • Hand-object interaction: [paper]
    • Whole Body
      • PROX [paper]
      • ExPose [paper]
      • FrankMocap [paper]
      • PIXIE [paper]
      • Moon et al. [paper]
      • PyMAF [paper]
      • OSX [paper]
      • HybrIK-X [paper]
• Template-free
  • Regression-based
    • FACSIMILE [paper], PeeledHuman [paper], GTA [paper], NSF [paper]
  • Optimization-based Differentiable
    • DiffPhy [paper], AG3D [paper]
  • Implicit Representations
    • PIFu [paper], PIFuHD [paper]
    • Canonical space: ARCH [paper], ARCH++ [paper], CAR [paper]
    • Geometric priors: GeoPIFu [paper]
    • Novel representations: Peng et al. [paper], 3DNBF [paper]
  • Neural Radiance Fields
    • Volume deformation scheme [paper]
    • ActorsNeRF [paper]
  • Diffusion Models
    • HMDiff [paper]
  • Implicit + Explicit
    • HMD [paper], IP-Net [paper], PaMIR [paper], Zhu et al. [paper], ICON [paper], ECON [paper], DELTA [paper], GETAvatar [paper]
  • Diffusion + Explicit
    • DINAR [paper]
  • NeRF + Explicit
    • TransHuman [paper]
  • Gaussian Splatting + Explicit
    • Animatable 3D Gaussian [paper]

The overview of the mainstream datasets.

Comparisons of 3D pose estimation methods on Human3.6M.

Comparisons of 3D pose estimation methods on MPI-INF-3DHP.

Comparisons of human mesh recovery methods on Human3.6M and 3DPW.