Awesome

Demo

Abstract

Installation

We tested our code on Ubuntu 22.04 with an NVIDIA RTX 5000.

1. Set up conda environment

conda create --name 4dndf python=3.8
conda activate 4dndf

2. Install PyTorch

conda install pytorch==1.13.0 torchvision==0.14.0 torchaudio==0.13.0 pytorch-cuda=11.6 -c pytorch -c nvidia

The commands depend on your CUDA version. You may check the instructions here.

3. Install PyTorch3D

Follow the official instructions here to install PyTorch3D with conda.

4. Install other dependencies

pip install open3d==0.17 scikit-image tqdm pykdtree plyfile
conda install -c conda-forge quaternion

How to run it

Clone the repository

git clone https://github.com/PRBonn/4dNDF
cd 4dNDF

Sanity test and Demo

For a sanity test, run the following script to download the test data (20 frames from KITTI seq 00) :

sh ./script/download_test_data.bash

Then run:

python static_mapping.py config/test/test.yaml

After training, It will generate a static mesh in output/test and visualize the dynamics segmentation result. For the visualizer, press 'space' to start the playback of the sequence (yellow points correspond to the static parts, and red points correspond to the identified dynamics).

Evaluation

As reported in the paper, we evaluate our method in surface reconstruction and dynamic object segmentation.

Surface reconstruction

We use Co-Fusion's car4 dataset and the Newer College dataset to evaluate the quality of our reconstruction. The original Co-Fusion dataset provides depth images rendered in Blender. We convert these depth images into point clouds and use 150 frames for our experiments. (We also provide the script to convert the Co-Fusion's data into our format in script/cofusion_data_converter.py.) Run the following script to download the already converted data:

sh ./script/download_cofusion.bash

Then run the following script, which runs our pipeline (i.e., static_mapping.py) and computes the metrics:

sh ./script/run_cofusion.bash

The reconstructed mesh will be stored in the output/cofusion folder.

For the newer college dataset, we selected part of the data (1300 frames) in the yard for our experiment. Run the following script to download the pre-processed data:

sh ./script/download_newer_college.bash

And then

sh ./script/run_newer_college.bash

to run the training code and evaluation script.

The reconstructed mesh will be stored in the output/newer_college folder. Note that the ground truth point has a different cover area with input scans, so we provide the reference mesh to crop the reconstruction result automatically.

We also provide the meshes reconstructed by the baseline methods. You can download them by running:

sh ./script/download_baseline.bash

Then change the path of est_ply in eval/eval_cofusion.py and eval/eval_newercollege.py and run them to check the numbers.

Dynamic objects segmentation

We use KTH_DynamicMap_Benchmark to evaluate the result of our Dynamic objects segmentation.

Download the data from the official link here and unzip it to our data folder as:

./4dNDF/
└── data/
    └── KTH_dynamic/
            ├── 00/
            │   ├── gt_cloud.pcd
            │   ├── pcd/
            |   |    ├── 004390.pcd
            |   |    ├── 004391.pcd
            |   |    └── ...
            ├── 05/ 
            ├── av2/
            ├── semindoor/
            └── translations/

The benchmark doesn't explicitly provide the pose files, so we extract poses from the data and store them in the data/translations/ folder.

For evaluating, you need to clone (to somewhere you like) and compile the KTH_DynamicMap_Benchmark 's repo. The following commands should work if you have ROS-full installed on your machine.

git clone --recurse-submodules https://github.com/KTH-RPL/DynamicMap_Benchmark.git
cd DynamicMap_Benchmark/script
mkdir build && cd build
camke ..
make

Or check the guidance from the benchmark here.

Then, copy the Python file from 4dNDF/eval/evaluate_single_kth.py to /path/to/DynamicMap_Benchmark/scripts/py/eval/

Take sequence 00 as an example. Run:

python static_mapping.py config/kth/00.yaml

After training, the static point cloud can be found here: data/kth/00/static_points.pcd

To evaluate it, run ( change the /your/path/to and /path/to to the correct path):

cd /your/path/to/DynamicMap_Benchmark/scripts/build/
./export_eval_pcd  /path/to/4dNDF/data/KTH_dynamic/00 static_points.pcd 0.05

It will generate the eval point cloud. Finally, Run:

python /your/path/to/DynamicMap_Benchmark/scripts/py/eval/evaluate_single_kth.py /path/to/4dNDF/data/KTH_dynamic/00

to check the number. For other sequences, we need to change all the 00 to 05, av2, or semindoor. You can organize the commands as a bash script to make it more convenient.

Citation

If you use 4dNDF for your academic work, please cite:

@inproceedings{zhong2024cvpr,
  author = {Xingguang Zhong and Yue Pan and Cyrill Stachniss and Jens Behley},
  title = {{3D LiDAR Mapping in Dynamic Environments using a 4D Implicit Neural Representation}},
  booktitle = {{Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)}},
  year = 2024,
}

Contact

If you have any questions, Feel free to contact:

• Xingguang Zhong {zhong@igg.uni-bonn.de}
• Yue Pan {yue.pan@igg.uni-bonn.de}