Home

Awesome

Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting

Project Page | Paper | Unofficial Implementation

Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, Sida Peng
ECCV 2024

https://github.com/user-attachments/assets/f28a64bd-9932-4447-b710-9254ae5ed56f

Installation

<details> <summary>Clone this repository</summary> 
git clone https://github.com/zju3dv/street_gaussians.git
</details> <details> <summary>Set up the python environment</summary> 
# Set conda environment
conda create -n street-gaussian python=3.8
conda activate street-gaussian

# Install torch (corresponding to your CUDA version)
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116

# Install requirements
pip install -r requirements.txt

# Install submodules
pip install ./submodules/diff-gaussian-rasterization
pip install ./submodules/simple-knn
pip install ./submodules/simple-waymo-open-dataset-reader
python script/test_gaussian_rasterization.py
</details> <details> <summary>Prepare Waymo Open Dataset.</summary>

We provide the example scenes here. You can directly download the data and skip the following steps for a quick start.

Download the training and validation set of Waymo Open Dataset.

We provide the split file following EmerNeRF. You can refer to this document for download details.

<!-- Please note that `val_dynamic.txt` specify scenes from the validation set, which means you may need to change the file source [here](https://github.com/NVlabs/EmerNeRF/blob/8c051d7cccbad3b52c7b11a519c971b8ead97e1a/datasets/download_waymo.py#L31). -->

Preprocess the data

Download the tracking predictions on validation set, We provide the processed results here.

Preprocess the example scenes

python script/waymo/waymo_converter.py --root_dir TRAINING_SET_DIR --save_dir SAVE_DIR --split_file script/waymo/waymo_splits/demo.txt --segment_file script/waymo/waymo_splits/segment_list_train.txt

Preprocess the experiment scenes

python script/waymo/waymo_converter.py --root_dir VALIDATION_SET_DIR --save_dir SAVE_DIR --split_file script/waymo/waymo_splits/val_dynamic.txt --segment_file script/waymo/waymo_splits/segment_list_val.txt
--track_file TRACKER_PATH

Generating LiDAR depth

python script/waymo/generate_lidar_depth.py --datadir DATA_DIR

Generating sky mask

Install GroundingDINO following this repo and download SAM checkpoint from this link.

python script/waymo/generate_sky_mask.py --datadir DATA_DIR --sam_checkpoint SAM_CKPT
</details> <details> <summary>Prepare Custom Dataset.</summary> TODO </details> <!-- ### Configuration We build the configuration based on [3D Gaussian Splatting](https://github.com/graphdeco-inria/gaussian-splatting/blob/main/arguments/__init__.py). The parameters used are listed in `lib/config/config.py` with brief comments. -->

Training

python train.py --config configs/xxxx.yaml

Training on example scenes

bash script/waymo/train_waymo_expample.sh

Training on experiment scenes

bash script/waymo/train_waymo_exp.sh

Rendering

python render.py --config configs/xxxx.yaml mode {evaluate, trajectory}

Rendering on example scenes

bash script/waymo/render_waymo_expample.sh

Rendering on experiment scenes

bash script/waymo/render_waymo_exp.sh

Visualization

You can convert the scene at one certain frame into the format that can be viewed in SIBR_viewers.

python make_ply.py --config configs/xxxx.yaml viewer.frame_id {frame_idx} mode evaluate

Pipeline

pipeline

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{yan2024street,
    title={Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting}, 
    author={Yunzhi Yan and Haotong Lin and Chenxu Zhou and Weijie Wang and Haiyang Sun and Kun Zhan and Xianpeng Lang and Xiaowei Zhou and Sida Peng},
    booktitle={ECCV},
    year={2024}
}