Awesome
LaRa: Efficient Large-Baseline Radiance Fields
Project page | Paper | Data | Checkpoint |<br>
⭐ New Features
-
2024/04/05: Important updates - Now our method supports half precision training, achieving over 100% faster convergence and about 1.5dB gains with less iterations!
Model PSNR ↑ SSIM ↑ Abs err (Geo) ↓ Epoch Time(day) ckpt Paper 27.65 0.951 0.0654 50 3.5 ------ bf16 29.15 0.956 0.0574 30 1.5 Download Please download the pre-trained checkpoint from the provided link and place it in the
ckpts
folder.
Installation
git clone https://github.com/autonomousvision/LaRa.git --recursive
conda env create --file environment.yml
conda activate lara
Dataset
We used the processed gobjaverse dataset for training. A download script tools/download_dataset.py
is provided to automatically download the datasets.
python tools/download_dataset.py all
Note: The GObjaverse dataset requires approximately 1.4 TB of storage. You can also download a subset of the dataset. Please refer to the provided script for details. Please manually delete the _temp
folder after completing the download.
If you would like to process the data by yourself, we provide preprocess scripts for the gobjaverse and co3d datasets, please check tools/prepare_dataset_*
.
You can also download our preprocessed data and put them to dataset
folder:
- gobjaverse
- Google Scaned Object
- Co3D
- Instant3D - Please contact the authors of Instant3D if you wish to obtain the data for comparison.
Training
python train_lightning.py
note: You can configure the GPU id and other parameter with configs/base.yaml
.
Evaluation
Our method supports the reconstruction of radiance fields from multi-view, text, and single view inputs. We provide a pre-trained checkpoint at ckpt.
multi-view to 3D
To reproduce the table results, you can simply use:
python eval_all.py
note:
- Please double-check that the paths inside the script are correct for your specific case.
- Please specify the video_frames and save_mesh labels if you would like to output mesh or video during the evaluation
text to 3D
python evaluation.py configs/infer.yaml
infer.ckpt_path=ckpts/epoch=29.ckpt
infer.save_folder=outputs/prompts/
infer.dataset.generator_type=xxx
infer.dataset.prompts=["a car made out of sushi","a beautiful rainbow fish"]
note: This part is currently unavailable due to a permissions issue. I will look for an alternative text-to-multi-view generator later next week.
single view to 3D
python evaluation.py configs/infer.yaml
infer.ckpt_path=ckpts/epoch=29.ckpt
infer.save_folder=outputs/single-view/
infer.dataset.generator_type="zero123plus-v1"
infer.dataset.image_pathes=\["assets/examples/13_realfusion_cherry_1.png"\]
note: It supports the generator types zero123plus-v1.1
and zero123plus-v1
.
Acknowledgements
Our render is built upon 2DGS. The data preprocessing code for the Co3D dataset is partially borrowed from Splatter-Image. Additionally, the script for generating multi-view images from text and single view image is sourced from GRM. We thank all the authors for their great repos.
Citation
If you find our code or paper helps, please consider citing:
@inproceedings{LaRa,
author = {Anpei Chen and Haofei Xu and Stefano Esposito and Siyu Tang and Andreas Geiger},
title = {LaRa: Efficient Large-Baseline Radiance Fields},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2024}
}