Home

Awesome

manotorch: MANO Pytorch

<!-- ## :spiral_notepad: Introduction --> <br /> <br />

:rocket: Installation

Get code and dependencies

$ git clone https://github.com/lixiny/manotorch.git
$ cd manotorch

Install the dependencies listed in environment.yaml

# In a new environment,
$ conda env create -f environment.yaml

# Or in an existing conda environment,
$ conda env update -f environment.yaml

Download MANO pickle data-structures

assets/mano
    ├── info.txt
    ├── __init__.py
    ├── LICENSE.txt
    ├── models
    │   ├── info.txt
    │   ├── LICENSE.txt
    │   ├── MANO_LEFT.pkl
    │   ├── MANO_RIGHT.pkl
    │   ├── SMPLH_female.pkl
    │   └── SMPLH_male.pkl
    └── webuser
        └── ...

Optional: Install manotorch package

To be able to import and use manotorch in another project, go to your manotorch folder and run

$ pip install .
<br /> <br />

:plate_with_cutlery: Usage

we provide a simple code snippet to demonstrate the minimal usage.

import torch
from manotorch.manolayer import ManoLayer, MANOOutput

# Select number of principal components for pose space
ncomps = 15

# initialize layers
mano_layer = ManoLayer(use_pca=True, flat_hand_mean=False, ncomps=ncomps)

batch_size = 2
# Generate random shape parameters
random_shape = torch.rand(batch_size, 10)
# Generate random pose parameters, including 3 values for global axis-angle rotation
random_pose = torch.rand(batch_size, 3 + ncomps)

# The mano_layer's output contains:
"""
MANOOutput = namedtuple(
    "MANOOutput",
    [
        "verts",
        "joints",
        "center_idx",
        "center_joint",
        "full_poses",
        "betas",
        "transforms_abs",
    ],
)
"""
# forward mano layer
mano_output: MANOOutput = mano_layer(random_pose, random_shape)

# retrieve 778 vertices, 21 joints and 16 SE3 transforms of each articulation
verts = mano_output.verts  # (B, 778, 3), root(center_joint) relative
joints = mano_output.joints  # (B, 21, 3), root relative
transforms_abs = mano_output.transforms_abs  # (B, 16, 4, 4), root relative

Advanced Usage

VisualizeCompose HandError Correction
<br /> <br />

:gift: New Features

Anatomical Consistent Basis

The original MANO model is driven by a kinematic tree with 16 joints, where each joint’s rotation is represented in the form of axis-angle. To represent the joint rotation in a three-dimensional Euclidean space, we need to find an orthogonal basis (consists of three orthogonal axes) that describes the rotation. Apparently, there have infinity choices of the orthogonal basis. For example, the original MANO model adopts the same orthogonal basis as the wrist for all of its 16 joints.

We seek to find a basis whose three axes can describe three independent hand motions that satisfy the hand anatomy. Therefore we can decompose the joint rotation w.r.t. this basis and penalize the abnormal pose on that joint.

<p align="center"> <img src="doc/anatomical_consistent_basis.jpg", alt="capture"> </p>

Anatomy Loss

In our ICCV2021 work CPF, we penalize the abnormal poses by projecting the rotation axis of axis-angle into three independent axes, and then penalize the abnormal axial components on that joint. This is a effective heuristic to avoid the abnormal pose, but it is still not a perfect solution, since:

To overcome this, in the new manotorch (>= v0.0.2), we firstly use the flat hand to calculate the twist-spread-bend axes in its canonical pose. Later, we can transform these basis to the posed hand, based on the 16 $\mathbf{SE}(3)$ transformation matrices.

See manotorch/axislayer.py: AxisLayerFK for details (FK: forward kinematics).
Run: scripts/simple_app.py

python scripts/simple_app.py --mode axis
<p align="center"> <img src="doc/axis.gif", width=400> </p>

To overcome the first issue, For each joint rotation, we decompose it in any pose into the rotations of the child frame in relation to the child's anatomical consistent basis and the rotation of the the child's anatomical consistent basis in relation to the parent's predefined (MANO) coordinate basis. The latter rotation is independent of the pose of hand and is thus solved only once, for the zero pose and mean shape, it can then be used as a fixed value (denoted as TMPL_R_p_a in the AxisLayerFK module). Therefore, we only need to penalize the former rotation, which is more reliable. e.g supervise the rotation of the child frame in relation to the child's anatomical consistent basis to prevent abnormal twisting rotations (e.g., rotations around the twist (1,0,0) axis).

To overcome the second issue, we penalize the rotation in form of the euler angles, which is more robust to the small angle.

See manotorch/anatomy_loss.py: AnatomyConstraintLossEE for details (EE: euler angle).
Run: scripts/simple_anatomy_loss.py to show the pose correction.

python scripts/simple_anatomy_loss.py
<p align="center"> <img src="doc/pose_correction.gif", width=400> </p>

Composing the Hand

Based on the Anatomical Consistent Basis, we can also compose the hand from a given euler angles.

See: manotorch/axislayer.py: AxisLayerFK.compose for details (FK: forward kinematics).
Run: scripts/simple_compose.py, It shows how we specify the euler angles of joint on the index finger and compose the hand in a deterministic way.

#   transform order of right hand
#         15-14-13-\
#                   \
#*   3-- 2 -- 1 -----0   < NOTE: demo on this finger
#   6 -- 5 -- 4 ----/
#   12 - 11 - 10 --/
#    9-- 8 -- 7 --/

#  the ID: 1 joints have been rotated by pi/6 around spread-axis, and pi/2 around bend-axis
#  the ID: 2, 3 joints have been rotated by pi/2 around bend-axis

python scripts/simple_compose.py
<p align="center"> <img src="doc/simple_compose.gif", width=400> </p>

Anchor Interpolation

These anchors derive a coarse palm vertices representation to treat contact during hand-object interaction.

See manotorch/anchorlayer.py: AnchorLayer for details.
Run: scripts/simple_app.py

python scripts/simple_app.py --mode anchor
<p align="center"> <img src="doc/anchor.jpg", height=300> <img src="doc/anchor.gif", width=400> </p> <br /> <br />

:thumbsup: Citation

If you find the manotorch useful in your research, please consider citing CPF, where the manotorch is originally developed:

@inproceedings{yang2021cpf,
    title = {{CPF}: Learning a Contact Potential Field to Model the Hand-Object Interaction},
    author = {Yang, Lixin and Zhan, Xinyu and Li, Kailin and Xu, Wenqiang and Li, Jiefeng and Lu, Cewu},
    booktitle = {ICCV},
    year = {2021}
}

and the original MANO publication:


@article{MANO:SIGGRAPHASIA:2017,
    title = {Embodied Hands: Modeling and Capturing Hands and Bodies Together},
    author = {Romero, Javier and Tzionas, Dimitrios and Black, Michael J.},
    journal = {ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)},
    publisher = {ACM},
    month = nov,
    year = {2017},
    url = {http://doi.acm.org/10.1145/3130800.3130883},
    month_numeric = {11}
}