Home

Awesome

DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic Exploration in the Dark

"Even with these dark eyes, a gift of the dark night, I go to seek the shining light." --Gu Cheng 1956-1993

<p align="center"> <img src="cmu_ri_logo.png" alt="Logo" width="40%""> <img src="NOAA_logo_mobile.svg" alt="Logo" width="25%"> </a> </p>

Novel-view rendering: Simulating a light cone and re-illuminating the environment.

Please check our videos (Bilibili).

<p align="center"> <img src="darkgs.gif" alt="Logo" width="100%"> </a> </p>

Sister Repo for Camera-Light calibration

The sister repo Neural Light Simulator for light-camera calibration is here.

Install

Installation generally follows vanilla Gaussian Splatting installation.

git clone git@github.com:tyz1030/darkgs.git --recursive

or

git clone https://github.com/tyz1030/darkgs.git --recursive

Conda environment setup

conda env create --file environment.yml
conda activate darkgs

Also need to install lietorch

pip install git+https://github.com/princeton-vl/lietorch.git

Data

Please find our example data on Google Drive and DropBox.

Make your own data

Please put your RAW images subfolder named "raw". To make COLMAP less struggle, I gamma-curved/manually increased the brightness of the raw images for feature extraction and matching. These corrected images are put in "input" subfolder. We only use "raw" images to build DarkGS.

python3 convert.py -s <path to your own dataset>

Light Calibration

If you are using your own light-camera setup, please calibrate your system using neural light simulator. Then put model_parameters.pth in the root directory.

Quick Start

Train

python train.py -s <path to example dataset>

Visualize with SIRB viewer:

./SIBR_remoteGaussian_app

Then you will be able to steer your light cone by pressing "JKLI" on the keyboard.\

Visualize (a checkpoint) after training:

SIBR_gaussianViewer_app is not compatible with this repo. Please try the following:

python3 viz_chkpt.py -s data/lab1/ -m output/<xxxxxx-xxx> --start_checkpoint output/<xxxxxx-xxx>/chkpnt30000.pth

then in another terminal

./SIBR_remoteGaussian_app

Relighting (I'm working on the release)

Meanwhile, there is no one-true-solution to relighting.
I have ADHD symptoms, so please forgive my slow progress.
One quick hack through is when running viz_chkpt.py, uncomment line 129 in scene/lighting.py. And you will also need to brighten, white balance and gamma correct the final render results to make it look good otherwise it is in blue-greenish RAW format.

Cite

This work is picked up by IROS 2024 as oral presentation! Arxiv

@INPROCEEDINGS{zhang2024darkgs,
  author={Tianyi Zhang and Kaining Huang and Weiming Zhi and Matthew Johnson-Roberson},
  booktitle={2024 International Conference on Intelligent Robots and Systems (IROS)}, 
  title={DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic Exploration in the Dark}, 
  year={2024}}

Acknowledgement