Awesome
BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation (CVPR 2024)
Project website | Paper | Huggingface Demo
Installation
To install and activate the environment, run the following command:
conda create -n berfscene python=3.7
conda activate berfscene
pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
You may have to modify the torch version to match your hardware requirements.
Training
Dataset structure
We follow the data structure defined in Hammer. For detailed instructions on dataset preparation, please refer to the dataset preparation documentation in the Hammer repository. We provide Clevr dataset that we used for training as a reference.
Training BerfScene
Run
./scripts/training_demos/berfscene.sh 8 <DATASET_PATH> --job_name=berfscene
--keep_ckpt_num -1 --data_workers 2 --r1_gamma=1 --batch_size 8
--semantic_nc 14 --ray_start 6 --ray_end 17 --focal 4.1015625
--kernel_size 3 --hidden_dim 256
to launch the training. In order to train using a dataset with customized rendering, you may need to modify the parameters:
ray_start
: the distance to the nearest point on the ray from the camera.ray_end
: the distance to the farest point on the ray from the camera.focal
: focal length of the camera.
to align with the camera distribution.
Inference
We provide a Huggingface online demo where you can inference our model to create large-scale 3D scene.
Bibtex
@article{zhang2024berfscene,
author = {Qihang Zhang and Yinghao Xu and Yujun Shen and Bo Dai and Bolei Zhou and Ceyuan Yang},
title = {{BerfScene}: Generative Novel View Synthesis with {3D}-Aware Diffusion Models},
booktitle = {CVPR},
year = {2024}
}