Awesome
Reposing Humans by Warping 3D Features
Markus Knoche, István Sárándi, Bastian Leibe (RWTH Aachen University)
This repository contains code for our paper “Reposing Humans by Warping 3D Features”. We warp implicitely learned volumetric features with explicit 3D transformations to change the pose of a given person into any desired target pose.
Setup
Using conda:
conda create -n warp3d_reposing
conda activate warp3d_reposing
conda install tensorflow-gpu=1.12
conda install scikit-image
conda install -c menpo opencv3
conda install numpy=1.16
Code Overview
parameters.py
contains global parameters for training, model and datasetdataset.py
loads pairs of images and their respective poses, estimates the transformations and creates the bodypart-masksmodel.py
contains generator and discriminator as well as the warping blocktrain.py
glues everything together
Datasets
The parameter params['data_dir']
points to the root directory which includes all dataset directories. Each dataset
directory contains a directory images
with all images organized in subfolders. It further must include a pickle file
poses.pkl
containing nested dictionaries with the same hierarchy as the image directory. Each image img.png
is
replaced by a dictionary key img
which points to a numpy array with one row per joint and three columns for the
coordinates in pixel space. Height and width are between 0 and 256, the depth is centered around 0.
Estimated Poses
poses.pkl for deepfashion, save as
<data_dir>/fashion3d/poses.pkl
poses.pkl for iPER, save as
<data_dir>/iPER/poses.pkl
Training
First, adapt the paths for the data root directory, the checkpoint directory and the tensorboard directory in
parameters.py
according to your system structure.
Second, download the
pose estimator checkpoint
which is used for validation. Extract the three files into pose3d_minimal/checkpoint/
.
You now have two options to start a training. You can run train.py
with command-line parameters, for example
python3 train.py name feat_weight_5 feature_loss_weight 5
.
Make sure to always include a unique name, because a new directory with this name will be created in the checkpoint directory and the tensorboard directory.
To keep track of different parameter combinations, you can also define a
JOB_ID
in parameters.py
and then for example use
python3 train.py JOB_ID 5
.
For each ablation model of our paper, a JOB_ID
is already defined.
Pretrained Models
Extract the checkpoints into <check_dir>/<model_name>/
.
iPER
name | 3D warping | 3D target pose | checkpoint |
---|---|---|---|
iPER-3d_w-3d_p | :heavy_check_mark: | :heavy_check_mark: | iPER-3d_w-3d_p.zip |
iPER-3d_w-2d_p | :heavy_check_mark: | :heavy_minus_sign: | iPER-3d_w-2d_p.zip |
iPER-2d_w-3d_p | :heavy_minus_sign: | :heavy_check_mark: | iPER-2d_w-3d_p.zip |
iPER-2d_w-2d_p | :heavy_minus_sign: | :heavy_minus_sign: | iPER-2d_w-2d_p.zip |
deepfashion
name | 3D warping | 3D target pose | checkpoint |
---|---|---|---|
fash-3d_w-3d_p | :heavy_check_mark: | :heavy_check_mark: | fash-3d_w-3d_p.zip |
fash-3d_w-2d_p | :heavy_check_mark: | :heavy_minus_sign: | fash-3d_w-2d_p.zip |
fash-2d_w-3d_p | :heavy_minus_sign: | :heavy_check_mark: | fash-2d_w-3d_p.zip |
fash-2d_w-2d_p | :heavy_minus_sign: | :heavy_minus_sign: | fash-2d_w-2d_p.zip |
BibTeX
@inproceedings{Knoche20CVPRW,
author = {Markus Knoche and Istv\'an S\'ar\'andi and Bastian Leibe},
title = {Reposing Humans by Warping 3{D} Features},
booktitle = {CVPR Workshop on Towards Human-Centric Image/Video Synthesis},
year = {2020}
}