Awesome

[ICCV2023] HOSNeRF: Dynamic Human-Object-Scene Neural Radiance Fields from a Single Video

This is the official repository of HOSNeRF Project page | arXiv | Video

Jia-Wei Liu, Yan-Pei Cao, Tianyuan Yang, Zhongcong Xu, Jussi Keppo, Ying Shan, Xiaohu Qie, Mike Zheng Shou

TL;DR: A novel 360° free-viewpoint rendering method that reconstructs neural radiance fields for dynamic human-object-scene from a single monocular in-the-wild video.

<img src="/assets/HOSNeRF.gif" width="1080px"/> HOSNeRF can render 360° free-viewpoint videos from a single monocular in-the-wild video. <img src="/assets/HOSNeRF.png" width="1080px"/> HOSNeRF Framework.

📢 News

[2023.12.11] We release the HOSNeRF codebase!
[2023.08.16] We release the HOSNeRF dataset!
[2023.08.12] HOSNeRF got accepted by ICCV 2023!
[2023.04.24] We release the arXiv paper!

📝 Preparation

Installation

git clone https://github.com/TencentARC/HOSNeRF.git
cd HOSNeRF
pip install -r requirements.txt

Download SMPL model

Download the gender neutral SMPL model from here, and unpack mpips_smplify_public_v2.zip.

Copy the smpl model.

SMPL_DIR=/path/to/smpl
MODEL_DIR=$SMPL_DIR/smplify_public/code/models
cp $MODEL_DIR/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl third_parties/smpl/models of 2nd_State_Conditional_Human-Object and 3rd_Complete_HOSNeRF

Follow this page to remove Chumpy objects from the SMPL model.

HOSNeRF dataset

We release the HOSNeRF dataset on link. HOSNeRF dataset consists of 6 real-world dynamic human-object-scene sequences: Backpack, Tennis, Suitcase, Playground, Dance, Lounge. Please run the optical flow estimation method using RAFT to get the optical flows of each scene.

🏋️‍️ Experiment

Training

Stage 1: Train the state-conditional background model.

$ cd 1st_State-Conditional_Scene
$ CUDA_VISIBLE_DEVICES=0,1,2,3 python run.py --ginc configs/state_mipnerf360/Backpack.gin --scene Backpack --logbase 'path to logbase'

Stage 2: Train the state-conditional dynamic human-object model.

(Please also change the datadir in configs/default.yaml and core/data/dataset_args.py)

$ cd 2nd_State_Conditional_Human-Object
$ CUDA_VISIBLE_DEVICES=0,1,2,3 python run.py --ginc configs/human-object/Backpack.gin --scene Backpack --logbase 'path to logbase' --cfg configs/human_nerf/wild/monocular/adventure.yaml --seed 777

Stage 3: Train the complete HOSNeRF model using the trained background and human-object checkpoints.

(Please also change the datadir in configs/default.yaml and core/data/dataset_args.py)

$ cd 3rd_Complete_HOSNeRF
$ CUDA_VISIBLE_DEVICES=0,1,2,3 python run.py --ginc configs/HOSNeRF/Backpack.gin --scene Backpack --logbase 'path to logbase' --cfg configs/human_nerf/wild/monocular/adventure.yaml --seed 777

Evaluation

We include the test codes in the model's test_step function. It will automatically run the test metrics (PSNR, SSIM, and LPIPS) for test images and all images after training.

Render 360° free-viewpoint videos

Please change the freeview index in the configs/default.yaml to render the free-viewpoint videos of that timestep. It will automatically render free-viewpoint videos (the model's test_step function) after training.

Render canonical human-object videos

It will automatically render the canonical human-object videos (the model's test_step function) after training.

Resume training

To resume training or resume testing after training, please add --resume_training True for each training script.

HOSNeRF checkpoints

We release the 6 HOSNeRF checkpoints on link for reference.

🎓 Citation

If you find our work helps, please cite our paper.

@inproceedings{liu2023hosnerf,
  title={Hosnerf: Dynamic human-object-scene neural radiance fields from a single video},
  author={Liu, Jia-Wei and Cao, Yan-Pei and Yang, Tianyuan and Xu, Zhongcong and Keppo, Jussi and Shan, Ying and Qie, Xiaohu and Shou, Mike Zheng},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={18483--18494},
  year={2023}
}

✉️ Contact

This repo is maintained by Jiawei Liu. Questions and discussions are welcome via jiawei.liu@u.nus.edu.

🙏 Acknowledgements

This codebase is based on HumanNeRF and NeRF-Factory. The preprocessing code is based on NeuMan. Thanks for open-sourcing!