Awesome
News
31/07/2022
We plan to release the THUman5.0 dataset for acadamic use. The dataset contains 10 dynamic human sequences which are captured by 32 RGB cameras with resolution of 4000x3096. Please see here for more detais.
DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras
Project Page | Paper | Data
DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras
Ruizhi Shao, Zerong Zheng, Hongwen Zhang, Jingxiang Sun, Yebin Liu ECCV 2022
We plan to release the training and testing code of DiffuStereo in this repository as soon as possible. Any discussions or questions would be welcome!
Installation
Please see INSTALL.md for manual installation.
Pretrained model
We will provide the pretrained diffusion models for stereo matching including 20 degree and 45 degree as soon as possible. You can download and put them into the checkpoints/
directory.
Download: pretrained model for 20 degree stereo matching
Run the code on the THUman2.0 dataset
Please see THUman2.0 to download DEMO samples and the dataset.
Structure of DEMO samples
thuman_demo/
├── depth_db
│ └── 0001
│ ├── xxx.npz --- the depth of view xxx (reconstructed by DoubleField)
├── img
│ └── 0001
│ ├── xxx.png --- the image of view xxx
├── mask
│ └── 0001
│ ├── xxx.png --- the mask of view xxx
├── normal_db
│ └── 0001
│ ├── xxx.png --- the normal of view xxx (reconstructed by DoubleField)
└── parameter --- We use perspective camera model to render images
└── 0001
├── xxx_extrinsic.npy --- the extrinsic of view xxx (3x4 world-to-camera matrix)
├── xxx_intrinsic.npy --- the intrinsic of view xxx (3x3 matrix)
Inference on a single stereo matching pair
To inference on a single stereo matching pair of DEMO samples, please run the following script.
python -m app.inference --config configs/thuman_demo.yaml --dataroot [the directory of DEMO samples] --view_list 0 20
Visualization on a single stereo matching pair
The results will be saved in results/thuman_demo/
. To visualize the results, use MeshLab to open results/thuman_demo/fusion000.ply
and apply Possion Reconstruction with depth=11
or depth=10
.
Inference on multiple stereo matching pairs
To inference on multiple stereo matching pairs of DEMO samples, please run the following script.
python -m app.inference --config configs/thuman_demo_multi.yaml --dataroot [the directory of DEMO samples] --view_list 0 20 90 110 180 200 270 290 --use_db_normal
For cases with 45 degree, please run the following script.
python -m app.inference --config configs/thuman_demo_multi_45.yaml --dataroot [the directory of DEMO samples] --view_list 0 45 90 135 180 225 270 315 --use_db_normal
Naive multi-view fusion and visualization on the results
The results will be saved in results/thuman_demo_multi/
. To simply fuse multi-view depth point clouds, use MeshLab to open results/thuman_demo_multi/fusion000.ply
and apply Possion Reconstruction with depth=11
or depth=10
.
Run the code on the THUman5.0 dataset
Please see DATASET.md to download DEMO samples and the dataset.
Structure of DEMO samples
real_demo/
├── depth_db
│ └── 0001
│ ├── xxx.npz --- the depth of view xxx (reconstructed by DoubleField)
├── img
│ └── 0001
│ ├── xxx.jpg --- the image of view xxx (after undistortion)
├── mask
│ └── 0001
│ ├── xxx.jpg --- the mask of view xxx (after undistortion)
├── normal_db
│ └── 0001
│ ├── xxx.png --- the normal of view xxx (reconstructed by DoubleField)
└── parameter --- parameters of real world perspective camera model (after undistortion)
└── 0001
├── xxx_extrinsic.npy --- the extrinsic of view xxx (3x4 world-to-camera matrix)
├── xxx_intrinsic.npy --- the intrinsic of view xxx (3x3 matrix)
Inference on one stereo matching pair
To inference on a single stereo matching pair of DEMO samples, please run the following script.
python -m app.inference --config configs/real_demo.yaml --dataroot [the directory of DEMO samples] --view_list 0 1
Visualization on one stereo matching pair
The results will be saved in results/real_demo/
. To visualize the results, use MeshLab to open results/real_demo/fusion000.ply
and apply Possion Reconstruction with depth=11
or depth=10
.
Multi-view fusion
On account of calibration error and the complicated lighting environment in the real-world dataset, naive multi-view fusion based on Poisson Reconstruction
will generate noise and artifacts. We recommend using traditional multi-view fusion algorithm or programming our proposed light-weight multi-view fusion
to reproduce the final results.
Training on the 3D human scan dataset
Please see THUman2.0 to download DEMO samples and the dataset.
TODO
Run the code on the custom dataset
To run the code on the custom dataset, please see CUSTOM. We also clarify our camera model and provide some examples to render depth and normal maps.
Citation
If you find this code useful for your research, please use the following BibTeX entry.
@inproceedings{shao2022diffustereo,
author = {Shao, Ruizhi and Zheng, Zerong and Zhang, Hongwen and Sun, Jingxiang and Liu, Yebin},
title = {DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Cameras},
booktitle = {ECCV},
year = {2022}
}
Acknowledgments
Our project is benefit from these great resources:
- Diffusion Probabilistic Models for 3D Point Cloud Generation
- Diffusion Models Beat GANS on Image Synthesis
- RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching
- StereoPIFu: Depth Aware Clothed Human Digitization via Stereo Vision
- POP: The Power of Points for Modeling Humans in Clothing
Thanks for their sharing code.