Home

Awesome

RPBG: Robust Point-based Graphics

RPBG: Towards Robust Neural Point-based Graphics in the Wild<br /> Qingtian Zhu<sup>1</sup>, Zizhuang Wei<sup>2,3</sup>, Zhongtian Zheng<sup>3</sup>, Yifan Zhan<sup>1</sup>, Zhuyu Yao<sup>4</sup>, Jiawang Zhang<sup>4</sup>, Kejian Wu<sup>4</sup>, Yinqiang Zheng<sup>1</sup><br /> <sup>1</sup>The University of Tokyo, <sup>2</sup>Huawei Technologies, <sup>3</sup>Peking University, <sup>4</sup>XREAL<br /> ECCV 2024 Oral

<a href='https://kaldir.vc.in.tum.de/scannetpp/benchmark/nvs'><img src='https://img.shields.io/badge/First_Place-ScanNet%2B%2B_NVS_Challenge-blue'></a> <a href='https://www.gigavision.cn'><img src='https://img.shields.io/badge/State_of_the_Art-GigaMVS_Dataset-blue'></a>

TL; DR: MVS-triangulated Splats + Image Restoration = Robust Point-based NVS

<p align="center"> <img src="doc/rpbg.png" width="800"><br><img src="doc/nreal.gif" width="400"><br>Indoor Navigation with Textureless and Transparrent Surfaces </p>

Environment

The configuration of running environment involves CUDA compiling, so please make sure NVCC has been installed (nvcc -V to check the version) and the installed PyTorch is compiled with the same CUDA version.

For example, if the system's CUDA is 11.8, run the following commands to configure the environment:

conda create -n RPBG python=3.9 -y && conda activate RPBG
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
pip install ./pcpr

To Run Your Custom Data

We provide the scripts to process custom data without camera calibration and triangulation. The typical data structure is as follows:

|-- custom_root_path
    |-- camera.xml # agisoft format of camera intrinsics & extrinsics
    |-- scene-sparse.yaml # configuration file for sparse triangulation (SfM)
    |-- scene-dense.yaml # configuration file for dense triangulation (MVS)
    |-- images # raw images (considered distorted thus not to be used in training)
    |-- sfm
        |-- sparse_pcd.ply # sparsely triangulated points
        |-- undis 
            |-- images # undistorted images (undistorted, to be used in training)
    |-- mvs
        |-- dense_pcd.ply # densely triangulated points

Data Preparation

First configure the path of your data & COLMAP installation in the script in triangulation/prepare_inputs.sh, as well as other settings if wanted, e.g., GPU indexes and distortion models, and execute it as:

sh triangulation/prepare_inputs.sh

Note that the GPU-enabled SIFT of COLMAP may not work with headless servers, check this issue for more information. By default, this script performs SfM, image undistortion, and MVS sequentially assuming all your images share one set of intrinsics including the distortion model, with unregistered images discarded.

Then please fill the relevant information in configs/paths.yaml and create a custom config file similar to configs/custom/sample.yaml, and adopting the default set of hyper-parameters will just work fine. After execution, scene-sparse.yaml, scene-dense.yaml, and camera.xml will be created under the given directory.

Following NPBG and READ, we follow the data convention of Agisoft Metashape, but we provide a useful script for converting camera parameters from one to another. It's now supporting:

For example, to convert COLMAP sparse reconstruction to Agisoft, run:

from pose_format_converter import *
COLMAP_recons = <YOUR_COLMAP_SPARSE_RECONS>
traj = load_colmap(COLMAP_recons)
traj.export_agisoft("camera.xml")

Training

To start training, please follow the scripts in scripts. We give an example as follows.

sh scripts/train.sh configs/custom/sample.yaml

Citation

@article{zhu2024rpbg,
  title={RPBG: Towards Robust Neural Point-based Graphics in the Wild},
  author={Zhu, Qingtian and Wei, Zizhuang and Zheng, Zhongtian and Zhan, Yifan and Yao, Zhuyu and Zhang, Jiawang and Wu, Kejian and Zheng, Yinqiang},
  journal={arXiv preprint arXiv:2405.05663},
  year={2024}
}

Acknowledgements

We would like to thank the maintainers of the following repositories.