Awesome
Compositional Human-Scene Interaction Synthesis with Semantic Control (COINS)
This repository contains the implementation of our paper Compositional Human-Scene Interaction Synthesis with Semantic Control and the PROX-S dataset expansion.
You can find more information on our project page.
Installation
This implementation is tested on the following platform:
Python 3.7, PyTorch 1.11.0 with CUDA 11.3 and cuDNN 8.2.0, PyTorch3D 0.6.2, Ubuntu 20.04
We recommend to manage the dependencies using conda. Please first install CUDA and ensure NVCC works. You can then create a conda environment using provided yml file as following:
conda env create -n COINS -f environment.yml
External data files:
- To use the SMPL-X body models, please download the weights from SMPL-X website and set
smplx_model_folder
in config. - Please download POSA and extract the
mesh_ds
for body mesh downsampling. Please setmesh_ds_folder
in config accordingly.
(Optional): If you want off-screen image rendering from SSH session or in a headless server, please install osmesa
and set OpenGL at the start of scripts as following:
os.environ['PYOPENGL_PLATFORM'] = 'osmesa'
PROX-S dataset
The PROX-S dataset is a human-scene interaction dataset annotated on top of PROX and PROX-E, which contains:
- scene instance segmentation
- per-frame interaction semantic labels and SMPL-X body estimation
You can download PROX-S expansion here.
Please also download scenes
, cam2world
, sdf
, and body_segments
from PROX, scenes_semantics
from PROX-E, and set the paths in config.
You can render scene segmentation and log object instances by:
cd data; python scene.py
Regarding interaction data with semantic labels, we provide a script load_interaction to load interaction data of specific action or action-object pair, and visualize the interaction with object instances.
Pre-trained Models
We provide the pre-trained models for BodyVAE and PelvisVae here.
Please set checkpoint_folder
in config.
Interaction synthesis in PROX scenes
For synthesizing interactions with semantic control, please run two_stage_sample as following:
cd interaction
# sample interactions
python two_stage_sample.py --exp_name test --lr_posa 0.01 --max_step_body 100 --weight_penetration 100 --weight_pose 10 --weight_init 0.01 --weight_contact_semantic 1 --num_sample 8 --num_try 1 --visualize 1 --full_scene 1 --interaction 'sit on-chair' --scene_name 'MPH16'
python two_stage_sample.py --exp_name test --lr_posa 0.01 --max_step_body 100 --weight_penetration 100 --weight_pose 10 --weight_init 0.01 --weight_contact_semantic 1 --num_sample 8 --num_try 1 --visualize 1 --full_scene 1 --interaction 'sit on-chair+touch-table' --scene_name 'MPH16'
# compositional interaction synthesis using models trained only on atomic data
python two_stage_sample.py --exp_name test --lr_posa 0.01 --max_step_body 100 --weight_penetration 100 --weight_pose 10 --weight_init 0.01 --weight_contact_semantic 1 --num_sample 8 --num_try 1 --visualize 1 --full_scene 1 --interaction 'sit on-chair+touch-table' --scene_name 'MPH16' --composition 1 --transform_checkpoint 'pelvis_atomic.ckpt' --interaction_checkpoint 'body_atomic.ckpt'
The synthesized results can be found in ./results/two_stage
.
Currently, the script supports choosing PROX scenes by --scene_name
and interaction by --interaction
in the format of action1-object1[+action2-object2]
. The script iterates over all instances of specified category in the input scene and generates interactions for each action-instance pair.
You may freely manipulate the weights.
Training Generative Models
We use PyTorch Lightning for model training. Please refer to its documentation if you want to customize trainer features such as logging, checkpoint, resume training, etc.
To train BodyVAE, please run:
cd interaction; python interaction_trainer.py --expr_name two_contact --model InteractionVAE --weight_kl 1 --used_interaction 'all' --robust_kl 1 --batch_size 8 --latent_dim 128 --num_obj_points 8192 --num_obj_keypoints 256 --use_pointnet2 1 --body_type mesh --template_type tpose --use_annealing 1 --latent_usage memory --second_stage 2 --use_contact_feature 1 --weight_contact_rec 1 --weight_contact_dist 1 --weight_normal 0.1 --weight_edge_length 0.2 --relative_length 1 --data_overwrite 0 --include_motion 1 --weight_normal_consistency 0.05 --use_regressor 1 --contact_scene_thresh 0.01 --contact_semantic_thresh 0.05
To train PelvisVAE, please run:
cd interaction; python transform_trainer.py --expr_name floor_all --model InteractionVAE --weight_kl 1 --weight_pelvis 3 --weight_orient 1 --weight_dist 1 --weight_coord 1 --weight_penetration 3 --used_interaction 'all' --use_annealing 0 --robust_kl 1 --batch_size 8 --use_augment 1 --thresh_penetration 0.25 --second_stage 10 --num_obj_keypoints 256 --num_obj_points 8192 --num_layers 2 --embedding_dim 64 --latent_dim 6 --use_prox_single 1 --include_motion 1 --use_annotate 0 --data_overwrite 0 --use_pointnet2 1 --use_floor_height 1
The trained models with logs can be found in ./results/interaction
and ./results/transform
. You can set --debug 1
to render samples for inspection during training.
Baselines
<details> <summary>In case you want to run the baselines</summary>PiGraph-X
The code for PiGraph-X can be found in the folder pigraph. To synthesize interactions using PiGraph-X, you can run:
cd pigraph; python synthesize.py --use_penetration 0 --composition 0 --visualize 1 --gender neutral --num_results 8 --num_skeletons 8 --num_translations 32 --num_rotations 12 --interaction 'sit on-chair' --scene_name 'MPH16' --save_dir pigraph_normal
You can refer to synthesize.sh for large-scale synthesis for evaluation.
POSA-I
The POSA-I method consists of the following three steps:
Train generative model for body with contact features
Please see body_trainer.py
Sample bodies with contact feature
Please see sample_body_feature.py.
Place bodies into scenes using POSA
Please first download the POSA code and data files. Then merge the POSA folder with the POSA code from original author. Please check the instructions in orginal POSA repo and then refer to synthesize.py for synthesis.
The code of POSA-I is currently distributed in inteaction and POSA, as well as the origial POSA repo. It is currently kind of messy and will potentially be restructured.
</details>Evaluation
The evaluation folder contains the scripts for evaluation.
- load_results.py loading interaction results from different methods.
- render_results.py renders interactions in multi-views
- eval_results.py evaluates the physical plausibility, semantic contact, and diversity metrics of generated results.
License
We employ MIT license for this repository, with the exceptions of codes borrowed or modified from other works:
- mpcat40.tsv from Matterport.
- smplx_vert_segmentation.json from Meshcapade.
- chamfer_distance.py from Pytorch3D.
- loss.py from Pose2Mesh.
- mesh.py, posa_utils, viz_utils from POSA.
- eulerangles.py from transforms3d.
- pointnet2.py from pointnet_pointnet2.
- smplx_regressor.py from GraphCMR.
- transformer.py from pytorch.
We sincerely thank the authors for releasing the codes and please check their respective licenses for these parts.