Awesome
Populating 3D Scenes by Learning Human-Scene Interaction
[Project Page] [Paper]
License
Software Copyright License for non-commercial scientific research purposes. Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use the POSA data, model and software, (the "Data & Software"), including 3D meshes, images, videos, textures, software, scripts, and animations. By downloading and/or using the Data & Software (including downloading, cloning, installing, and any other use of the corresponding github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Data & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License.
Description
This repository contains the training, random sampling, and scene population code used for the experiments in POSA.
Installation
To install the necessary dependencies run the following command:
pip install -r requirements.txt
The code has been tested with Python 3.7, CUDA 10.0, CuDNN 7.5 and PyTorch 1.7 on Ubuntu 20.04.
Dependencies
POSA_dir
To be able to use the code you need to get the POSA_dir.zip. After unzipping, you should have a directory with the following structure:
POSA_dir
├── cam2world
├── data
├── mesh_ds
├── scenes
├── sdf
└── trained_models
The content of each folder is explained below:
trained_models
contains two trained models. One is trained on thecontact
only and the other one is trained oncontact
andsemantics
.data
contains the train and test data extracted from the PROX Dataset and PROX-E Dataset.scenes
contains the 12 scenes from PROX Datasetsdf
contains the signed distance field for the scenes in the previous folder.mesh_ds
contains mesh downsampling and upsampling related files similar to the ones in COMA.
SMPL-X
You need to get the SMPLx Body Model. Please extract the folder and rename it
to smplx_models
and place it in the POSA_dir
above.
AGORA
In addition, you need to get the POSA_rp_poses.zip
file from AGORA Dataset and extract in the POSA_dir
.
This file contrains a number of test poses to be used in the next steps. Note that you don't need the whole AGORA dataset.
Finally run the following command or add it to your ~/.bashrc
export POSA_dir=Path of Your POSA_dir
Inference
You can test POSA
using the trained models provided. Below we provide examples of how to generate POSA features and how to pupulate a 3D scene.
Random Sampling
To generate random features from a trained model, run the following command
python src/gen_rand_samples.py --config cfg_files/contact.yaml --checkpoint_path $POSA_dir/trained_models/contact.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --render 1 --viz 1 --num_rand_samples 3
Or
python src/gen_rand_samples.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --render 1 --viz 1 --num_rand_samples 3
This will open a window showing the generated features for the specified pkl
file. It also render the features to the folder random_samples
in POSA_dir
.
The number of generated feature maps can be controlled by the flag num_rand_samples
.
If you don't have a screen, you can turn off the visualization --viz 0
.
If you don't have CUDA installed then you can add this flag --use_cuda 0
. This applies to all commands in this repository.
You can also run the same command on the whole folder of test poses
python src/gen_rand_samples.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses --render 1 --viz 1 --num_rand_samples 3
Scene Population
Given a body mesh from the AGORA Dataset, POSA
automatically places the body mesh in 3D scene.
python src/affordance.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --scene_name MPH16 --render 1 --viz 1
This will open a window showing the placed body in the scene. It also render the placements to the folder affordance
in POSA_dir
.
You can control the number of placements for the same body mesh in a scene using the flag num_rendered_samples
, default value is 1
.
The generated feature maps can be shown by setting adding --show_gen_sample 1
You can also run the same script on the whole folder of test poses
python src/affordance.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses --scene_name MPH16 --render 1 --viz 1
To place clothed body meshes, you need to first buy the Renderpeople assets, or get the free models.
Create a folder rp_clothed_meshes
in POSA_dir
and place all the clothed body .obj
meshes in this folder. Then run this command:
python src/affordance.py --config cfg_files/contact_semantics.yaml --checkpoint_path $POSA_dir/trained_models/contact_semantics.pt --pkl_file_path $POSA_dir/POSA_rp_poses/rp_aaron_posed_001_0_0.pkl --scene_name MPH16 --render 1 --viz 1 --use_clothed_mesh 1
Testing on Your Own Poses
POSA
has been tested on the AGORA
dataset only. Nonetheless, you can try POSA
with any SMPL-X
poses you have. You just need a .pkl
file
with the SMPLX
body parameters and the gender. Your SMPL-X
vertices must be brought to a canonical form similar to the POSA training data.
This means the vertices should be centered at the pelvis joint, the x
axis pointing to the left, the y
axis pointing backward, and the z
axis pointing upwards.
As shown in the figure below. The x,y,z axes are denoted by the red, green, blue colors respectively.
See the function pkl_to_canonical
in data_utils.py for an example of how to do this transformation.
Training
To retrain POSA
from scratch run the following command
python src/train_posa.py --config cfg_files/contact_semantics.yaml
Visualize Ground Truth Data
You can also visualize the training data
python src/show_gt.py --config cfg_files/contact_semantics.yaml --train_data 1
Or test data
python src/show_gt.py --config cfg_files/contact_semantics.yaml --train_data 0
Note that the ground truth data has been downsampled to speed up training as explained in the paper. See training details in appendices.
Citation
If you find this Model & Software useful in your research we would kindly ask you to cite:
@inproceedings{Hassan:CVPR:2021,
title = {Populating {3D} Scenes by Learning Human-Scene Interaction},
author = {Hassan, Mohamed and Ghosh, Partha and Tesch, Joachim and Tzionas, Dimitrios and Black, Michael J.},
booktitle = {Proceedings {IEEE/CVF} Conf.~on Computer Vision and Pattern Recognition ({CVPR})},
month = jun,
month_numeric = {6},
year = {2021}
}
If you use the extracted training data, scenes or sdf the please cite:
@inproceedings{PROX:2019,
title = {Resolving {3D} Human Pose Ambiguities with {3D} Scene Constraints},
author = {Hassan, Mohamed and Choutas, Vasileios and Tzionas, Dimitrios and Black, Michael J.},
booktitle = {International Conference on Computer Vision},
month = oct,
year = {2019},
url = {https://prox.is.tue.mpg.de},
month_numeric = {10}
}
@inproceedings{PSI:2019,
title = {Generating 3D People in Scenes without People},
author = {Zhang, Yan and Hassan, Mohamed and Neumann, Heiko and Black, Michael J. and Tang, Siyu},
booktitle = {Computer Vision and Pattern Recognition (CVPR)},
month = jun,
year = {2020},
url = {https://arxiv.org/abs/1912.02923},
month_numeric = {6}
}
If you use the AGORA test poses, the please cite:
@inproceedings{Patel:CVPR:2021,
title = {{AGORA}: Avatars in Geography Optimized for Regression Analysis},
author = {Patel, Priyanka and Huang, Chun-Hao P. and Tesch, Joachim and Hoffmann, David T. and Tripathi, Shashank and Black, Michael J.},
booktitle = {Proceedings IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)},
month = jun,
year = {2021},
month_numeric = {6}
}
Contact
For commercial licensing (and all related questions for business applications), please contact ps-licensing@tue.mpg.de.