Awesome

StackFLOW

This repository contains the official implementation for our paper: StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset.

<img alt="YouTube Video Views" src="https://img.shields.io/youtube/views/XIMLoLxmTDw?style=plastic">

Main Pipeline

Main framework for our method. (a) We use human-object offset to encode the spatial relation between the human and the object. For a human-object pair, offsets are calculated and flattened into an offset vector x. Based on all offset vectors calculated from the training set, the latent spatial relation space is constructed using principal component analysis. To get a vectorized representation for human-object spatial relation, the offset vector is projected into this latent spatial relation space by linear projection. Inversely, given a sample γ from this latent spatial relation space, we can reproject it to recover offset vector xˆ. The human-object instance can be reconstructed from xˆ by iterative optimization. (b) With pre-constructed latent spatial relation space, we use stacked normalizing flow to infer the posterior distribution of human-object spatial relation for an input image. (c) In the post-optimization stage, we further finetune the reconstruction results using 2D-3D reprojection loss and offset loss.

Pre-trained Models and Performance Comparison

On BEHAVE Dataset

On InterCap Dataset

On BEHAVE-Extended Dataset

Full-Sequence Evaluation

Run Demo

Follow these instructions to set up the environments for this project. Make sure you have downloaded the checkpoint and put it in the directory PROJECT_ROOT/outputs/stackflow/behave_aug/. Then run:

Demo Occlusion:

python ./demo_occlusion.py --cfg_file ./stackflow/configs/behave_aug.yaml --img_path ./data/demo/occlusion/3_3_basketball_none_026_0.color.jpg

Results will be written to the directory PROJECT_ROOT/outputs/demo/occlusion/.

Demo Optimization with Multi-Object:

python ./demo_multi_object.py --cfg_file ./stackflow/configs/behave_aug.yaml --post_optimization

Results will be written to the directory PROJECT_ROOT/outputs/demo/multi_objects/.

Demo Optimization Full Sequence:

Make sure you have downloaded the checkpoint and have prepared the BEHAVE-Extended dataset following this instruction.

python ./demo_sequence.py --cfg_file ./stackflow/configs/behave_extend.yaml --dataset_root_dir $BEHAVE_ROOT_DIR

We run this script on a single A100 GPU with 80GB memory. Results will be written to the directory PROJECT_ROOT/outputs/demo/multi_objects/sequences.

Train StackFLOW

Before training, make sure you have prepared BEHAVE or InterCap dataset following these instructions.

• For the BEHAVE dataset, you should go through steps #1, #3, #4, and #6. If you want to train with augmented data, you should also go through steps #2 and #5.
• For the InterCap dataset and BEHAVE-Extended dataset, you should go through all steps.

Don't forget to redirect the path _C.dataset.bg_dir (in file PROJECT_DIR/stackflow/configs/__init__.py (line 25)) to VOC_DIR in your custom setting. Then run:

python ./stackflow/train.py --cfg_file ./stackflow/configs/behave.yaml --dataset_root_dir $BEHAVE_ROOT_DIR

Our model can be trained within 2 days on a single GPU. The logs and checkpoints will be saved to the path PROJECT_DIR/outputs/stackflow.

Evaluate StackFLOW

Evaluate on BEHAVE dataset:

Download the full mask from here for BEHAVE datasets and zip it to the path BEHAVE_ROOT_DIR. Make sure the path for the checkpoint in PROJECT_DIR/stackflow/configs/behave.yaml exists. If you want to evaluate models with pose-optimization, you need to follow this instruction to prepare the keypoints of the person and the 2D-3D corresponding maps of the object.

python ./stackflow/evaluate_frames.py --cfg_file ./stackflow/configs/behave.yaml --dataset_root_dir $BEHAVE_ROOT_DIR

Evaluate on BEHAVE-extended dataset:

python ./stackflow/evaluate_sequences.py --cfg_file ./stackflow/configs/behave_extend.yaml --dataset_root_dir $BEHAVE_ROOT_DIR

The reconstruction results and evaluation metrics will be saved to the directory PROJECT_DIR/outputs/stackflow.

Acknowledgments

This work borrows some codes from ProHMR and CDPN. Thanks for these fantastic works.

Contact

If you have any questions, please feel free to put forward your issues and contact me.

Citation

@inproceedings{ijcai2023p100,
  title     = {StackFLOW: Monocular Human-Object Reconstruction by Stacked Normalizing Flow with Offset},
  author    = {Huo, Chaofan and Shi, Ye and Ma, Yuexin and Xu, Lan and Yu, Jingyi and Wang, Jingya},
  booktitle = {Proceedings of the Thirty-Second International Joint Conference on
               Artificial Intelligence, {IJCAI-23}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Edith Elkind},
  pages     = {902--910},
  year      = {2023},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2023/100},
  url       = {https://doi.org/10.24963/ijcai.2023/100},
}