Awesome
Representing Volumetric Videos as Dynamic MLP Maps
Project Page | Video | Paper | Data
Representing Volumetric Videos as Dynamic MLP Maps
Sida Peng*, Yunzhi Yan*, Qing Shuai, Hujun Bao, Xiaowei Zhou (* equal contribution)
CVPR 2023
Any questions or discussions are welcomed!
Installation
Please see INSTALL.md for manual installation.
Interactive demo
<details> <summary>Interactive rendering on ZJU-MoCap</summary>Please see INSTALL.md to download the dataset. We provide the pretrained models at here.
Take the rendering on sequence 313
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/zjumocap/313/final.pth
. -
Interactive rendering demo:
python gui.py --config configs/zjumocap/dymap_313.py fast_render True
Please see INSTALL.md to download the dataset. We provide the pretrained models at here.
Take the rendering on sequence sport1
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/nhr/sport1/final.pth
. -
Interactive rendering demo:
python gui.py --config configs/nhr/sport1.py fast_render True
Run the code on ZJU-MoCap
Please see INSTALL.md to download the dataset.
We provide the pretrained models at here.
<details> <summary>Test on ZJU-MoCap</summary>Take the test on sequence 313
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/zjumocap/313/final.pth
. -
Test on unseen views:
python run.py --config configs/zjumocap/dymap_313.py mode evaluate fast_render True
Take the visualization on sequence 313
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/zjumocap/313
. -
Visualization:
- Visualize free-viewpoint videos
python run.py --config configs/zjumocap/dymap_313.py mode visualize vis_novel_view True fast_render True
- Visualize novel views of single frame
python run.py --config configs/zjumocap/dymap_313.py mode visualize vis_novel_view True fixed_time True fast_render True
- Visualize the dynamic scene with fixed camera
python run.py --config configs/zjumocap/dymap_313.py mode visualize vis_novel_view True fixed_view True fast_render True
- Visualize mesh
python run.py --config configs/zjumocap/dymap_313.py mode visualize vis_mesh True fast_render True
Take the training on sequence 313
as an example.
- Train:
# training python train_net.py --config configs/zjumocap/dymap_313.py # distributed training python -m torch.distributed.launch --nproc_per_node=4 train_net.py --config configs/zjumocap/dymap_313.py
- Post-process the trained model:
python run.py --config configs/zjumocap/dymap_313.py mode visualize occ_grid True
- Tensorboard:
tensorboard --logdir data/record/zjumocap
Run the code on NHR
Please see INSTALL.md to download the dataset.
We provide the pretrained models at here.
<details> <summary>Test on NHR</summary>Take the test on sequence sport1
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/nhr/sport1/final.pth
. -
Test on unseen views:
python run.py --config configs/nhr/sport1.py mode evaluate fast_render True
Take the visualization on sequence sport1
as an example.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/nhr/sport1
. -
Visualization:
- Visualize novel views
python run.py --config configs/nhr/sport1.py mode visualize vis_novel_view True fast_render True
- Visualize novel views of single frame
python run.py --config configs/nhr/sport1.py mode visualize vis_novel_view True fixed_time True fast_render True
- Visualize the dynamic scene with fixed camera
python run.py --config configs/nhr/sport1.py mode visualize vis_novel_view True fixed_view True fast_render True
- Visualize mesh
python run.py --config configs/nhr/sport1.py mode visualize vis_mesh True fast_render True
Take the training on sequence sport1
as an example.
- Train:
# training python train_net.py --config configs/nhr/sport1.py # distributed training python -m torch.distributed.launch --nproc_per_node=4 train_net.py --config configs/nhr/sport1.py
- Post-process the trained model:
python run.py --config configs/nhr/sport1.py mode visualize occ_grid True
- Tensorboard:
tensorboard --logdir data/record/nhr
Citation
If you find this code useful for your research, please use the following BibTeX entry.
@inproceedings{peng2023representing,
title={Representing Volumetric Videos as Dynamic MLP Maps},
author={Peng, Sida and Yan, Yunzhi and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},
booktitle={CVPR},
year={2023}
}