Awesome
NerfingMVS
Project Page | Paper | Video | Data
<br/><p align='center'> <img src="imgs/demo.gif" width='80%'/> </p>NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo
Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, Jie Zhou
ICCV 2021 (Oral Presentation)
Installation
- Pull NerfingMVS repo.
git clone --recursive git@github.com:weiyithu/NerfingMVS.git
- Install python packages with anaconda.
conda create -n NerfingMVS python=3.7 conda activate NerfingMVS conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 -c pytorch pip install -r requirements.txt
- We use COLMAP to calculate poses and sparse depths. However, original COLMAP does not have fusion mask for each view. Thus, we add masks to COLMAP and denote it as a submodule. Please follow https://colmap.github.io/install.html to install COLMAP in
./colmap
folder (Note that do not cover colmap folder with the original version).
Usage
- Download 8 ScanNet scene data used in the paper here and put them under
./data
folder. We also upload final results and checkpoints of each scene here. - Run NerfingMVS
The whole procedure takes about 3.5 hours on one NVIDIA GeForce RTX 2080 GPU, including COLMAP, depth priors training, NeRF training, filtering and evaluation. COLMAP can be accelerated with multiple GPUs.You will get per-view depth maps insh run.sh $scene_name
./logs/$scene_name/filter
. Note that these depth maps have been aligned with COLMAP poses. COLMAP results will be saved in./data/$scene_name
while others will be preserved in./logs/$scene_name
Run on Your Own Data!
- Place your data with the following structure:
NerfingMVS |───data | |──────$scene_name | | | train.txt | | |──────images | | | | 001.jpg | | | | 002.jpg | | | | ... |───configs | $scene_name.txt | ...
train.txt
contains names of all the images. Images can be renamed arbitrarily and '001.jpg' is just an example. You also need to imitate ScanNet scenes to create a config file in./configs
. Note thatfactor
parameter controls the resolution of output depth maps. You also should adjustdepth_N_iters, depth_H, depth_W
inoptions.py
accordingly. - Run NerfingMVS without evaluation
Since our work currently relies on COLMAP, the results are dependent on the quality of the acquired poses and sparse reconstruction from COLMAP.sh demo.sh $scene_name
Acknowledgement
Our code is based on the pytorch implementation of NeRF: NeRF-pytorch. We also refer to mannequin challenge.
Citation
If you find our work useful in your research, please consider citing:
@inproceedings{wei2021nerfingmvs,
author = {Wei, Yi and Liu, Shaohui and Rao, Yongming and Zhao, Wang and Lu, Jiwen and Zhou, Jie},
title = {NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo},
booktitle = {ICCV},
year = {2021}
}