Awesome

Animatable Neural Radiance Fields from Monocular RGB Videos

Github | Paper

Animatable Neural Radiance Fields from Monocular RGB Videos
Jianchuan Chen, Ying Zhang, Di Kang, Xuefei Zhe, Linchao Bao, Xu Jia, Huchuan Lu

Demos

• male-3-casual

  <img width="30%" src='assets/demos/male-3-casual/input.gif'></img><img width="60%" src='assets/demos/male-3-casual/male-3-casual.gif'></img>

More demos please see Demos.

Requirements

• Python3.7 or later
• PyTorch 1.6 or later
• Pytorch-lightning
• KNN_CUDA

For visualization

• pyrender
• Trimesh
• PyMCubes

Run the following code to install all pip packages:

pip install -r requirements.txt

To install KNN_CUDA, we provide two ways:

• from source

  git clone https://github.com/unlimblue/KNN_CUDA.git
  cd KNN_CUDA
  make && make install
  

• from wheel

  pip install --upgrade https://github.com/unlimblue/KNN_CUDA/releases/download/0.2/KNN_CUDA-0.2-py3-none-any.whl
  

SMPL models

To download the SMPL model go to this (male, female and neutral models).

Place them as following:

smplx
└── models
    └── smpl
        ├── SMPL_FEMALE.pkl
        ├── SMPL_MALE.pkl
        └── SMPL_NEUTRAL.pkl

Data Preparation

Thanks to @radman for providing the Colab for data preparation at Here.

People-Snapshot datasets

• Prepare images and smpls

  python -m tools.people_snapshot --data_root ${path_to_people_snapshot_datasets} --people_ID male-3-casual --gender male --output_dir data/people_snapshot
  

• Prepare template

  python -m tools.prepare_template --data_root data/people_snapshot --people_ID male-3-casual --model_type smpl --gender male --model_path ${path_to_smpl_models}
  

iPER datasets or Custom datasets

• Prepare images

  First convert video to images, and crop the images to make the person as centered as possible.

  python -m tools.video_to_images --vid_file ${path_to_iper_datasets}/023_1_1.mp4 --output_folder data/iper/iper_023_1_1/cam000/images --img_wh 800 800 --offsets -30 20
  

• Images segmentation

  Here, we use RVM to extact the foreground mask of the person.

  python -m tools.rvm --images_folder data/iper/iper_023_1_1/cam000/images --output_folder data/iper/iper_023_1_1/cam000/images
  

• SMPL estimation

  In our experiment, we use VIBE to estimate the smpl parameters.

  python -m tools.vibe --images_folder data/iper/iper_023_1_1/cam000/images --output_folder data/iper/iper_023_1_1
  

  Then convert vibe_output.pkl to the format in our experiment setup.

  python -m tools.convert_vibe --data_root data/iper --people_ID iper_023_1_1 --gender neutral
  

• Prepare template

  python -m tools.prepare_template --data_root data/iper --people_ID iper_023_1_1 --model_type smpl --gender neutral --model_path ${path_to_smpl_models}
  

Training

• Training on the training frames

  python train.py --cfg_file configs/people_snapshot/male-3-casual.yml
  

• Finetuning the smpl params on the testing frames

  python train.py --cfg_file configs/people_snapshot/male-3-casual_refine.yml train.ckpt_path checkpoints/male-3-casual/last.ckpt
  

We provide the preprocessed data and pretrained models at Here

Visualization

Novel view synthesis

python novel_view.py --ckpt_path checkpoints/male-3-casual/last.ckpt

3D reconstruction

python extract_mesh.py --ckpt_path checkpoints/male-3-casual/last.ckpt

Shape Editing

• To make the person look fatter, we can properly reduce the betas_2th parameter.

  python novel_view.py --ckpt_path checkpoints/male-3-casual/last.ckpt --betas_2th -2.0
  

• In contrast, we can increase the betas_2th parameter to make the person look slimmer.

  python novel_view.py --ckpt_path checkpoints/male-3-casual/last.ckpt --betas_2th 3.0
  

Novel pose synthesis

python novel_pose.py --ckpt_path checkpoints/male-3-casual/last.ckpt

The mixamo motion capture smpl parameters can be downloaded from here.

Testing

python test.py --ckpt_path checkpoints/male-3-casual_refine/last.ckpt --vis

Citation

If you find the code useful, please cite:

@misc{chen2021animatable,
      title={Animatable Neural Radiance Fields from Monocular RGB Videos}, 
      author={Jianchuan Chen and Ying Zhang and Di Kang and Xuefei Zhe and Linchao Bao and Xu Jia and Huchuan Lu},
      year={2021},
      eprint={2106.13629},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgements

Parts of the code were based on from kwea123's NeRF implementation: https://github.com/kwea123/nerf_pl. Some functions are borrowed from PixelNeRF https://github.com/sxyu/pixel-nerf