Awesome
The Sky’s the Limit
Official implementation of NeuSky.
Paper: The Sky's the Limit: Relightable Outdoor Scenes via a Sky-pixel Constrained Illumination Prior and Outside-In Visibility
Installation
We build on top of Nerfstudio. However, since Nerfstudio is still in very active development with fairly large codebase changes still occurring compatibility might be an issue. Pull requests and issues are very welcome.
Prerequisites
You must have an NVIDIA video card with CUDA installed on the system. This library has been tested with version 11.8 of CUDA. You can find more information about installing CUDA here
Create Environment
Nerfstudio requires python >= 3.8. We recommend using conda to manage dependencies. Make sure to install Conda before proceeding.
Install nerfstudio
conda create --name nerfstudio -y python=3.8
conda activate nerfstudio
pip install --upgrade pip
pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
git clone https://github.com/nerfstudio-project/nerfstudio.git
cd nerfstudio
pip install --upgrade pip setuptools
pip install -e .
Install NeuSky
a. Clone repo and install RENI++ repo
sudo apt install libopenexr-dev openexr
git clone https://github.com/JADGardner/ns_reni.git
cd ns_reni
pip install -e .
cd ..
b. Install NeuSky
git clone https://github.com/JADGardner/neusky.git
cd neusky
pip install -e .
c. Setup Nerfstudio CLI
ns-install-cli
d. Close and reopen your terminal and source conda environment again:
conda activate nerfstudio
Download RENI++ pre-trained models
python3 ns-reni/scripts/download_models.py output/path/for/reni_plus_plus_models/
Then update this line in the config for NeuSky to point to the chosen RENI++ directory:
Download Data
ns-download-data nerfosr --save-dir data --capture-name lk2
python neusky/scripts/download_and_copy_segmentation_masks.py lk2 /path/to/Data/NeRF-OSR
Start Training
You can now launch training for scene 'lk2'
ns-train neusky --vis wandb
If you find you run out of GPU memory you can try updating some or all of these settings in
neusky/neusky/configs/neusky_config.py
train_num_images_to_sample_from=-1, # Set to integer value if out of GPU memory
train_num_times_to_repeat_images=-1, # Iterations before resampling a new subset, set to integer value if out of GPU memory
images_on_gpu=True, # set False if out of GPU memory
masks_on_gpu=True, # set False if out of GPU memory
train_num_rays_per_batch=1024, # Lower to 512, 256, or 128 if out of GPU memory
eval_num_rays_per_batch=1024, # Lower to 512, 256, or 128 if out of GPU memory