Home

Awesome

arXiv GitHub Stars visitors

STPLS3D: A Large-Scale Synthetic and Real Aerial Photogrammetry 3D Point Cloud Dataset

Meida Chen, Qingyong Hu, Zifan Yu, Hugues Thomas, Andrew Feng, Yu Hou, Kyle McCullough, Fengbo Ren, Lucio Soibelman. <br /> [Project page] [Paper] [BMVC presentation] [Demo video] [Poster] [Urban3D workshop@ICCV2023][Urban3D workshop@ECCV2022] [Instance segmentation competition] <br />

Updates

(1) Our Focus

(2) Dataset

2.1 Download

2.2 Overview

we have built a large-scale photogrammetry 3D point cloud dataset, termed Semantic Terrain Points Labeling - Synthetic 3D (STPLS3D), which is composed of high-quality, rich-annotated point clouds from real-world and synthetic environments.

<p align="center"> <img src="imgs/STPLS3D.png" width="80%"> </p>

2.3 Data Collection

We first collect real-world aerial images using photogrammetry best practices with quadcopter drone flight at a low altitude with significant overlaps between adjacent photos. We then reconstructed point clouds with 1.27 km^2 landscape following the standard photogrammetry pipeline. Next, we follow the same UAV path and flying pattern to generate 62 synthetic point clouds with different architectural styles, vegetation types, and terrain shapes. The synthetic dataset covers about 16 km^2 of the city landscape, with up to 18 fine-grained semantic classes and 14 instance classes.

2.4 Synthetic data generation workflow demo

<p align="center"> <a href="https://youtu.be/6wYWVo6Cmfs"><img src="imgs/STPLS3D_workflow.png" width="80%"></a> </p>

2.5 Semantic Annotations

Note that not all datasets we are currently providing have all the semantic labels available, the ground points that don't have the material available (15, 18, 19) are labeled with 0.

2.6 Instance annotations

The ground is labeled with -100. Window instance is currently per building but not per window but could be post-processed using connect component algorithm. Our experiments did not include the window instances.

Only synthetic datasets v2 and v3 have the instance labels.

(3) Benchmarks

3.1 Semantic segmentation:

<p align="center"> <img src="imgs/SemanticSegmentationEvaluationOnWMSC.JPG" width="80%"> </p>

3.2 Instance segmentation:

<p align="center"> <img src="imgs/InstanceSegmentation_06202022.PNG" width="80%"> </p>

(4) Training and Evaluation

Here we provide the training and evaluation script for both semantic and instance segmentation.

4.1 Semantic segmentation:

KpConv (Ubuntu and Windows 10): The environment setup is the same as the official KpConv release. We follow the same steps as shown here to evaluate KpConv on our STPLS3D dataset.

Download the data and unzip it. Change the variable self.path of STPLS3DDataset class (here) to the place where STPLS3D is stored.

STPLS3D
├── RealWorldData
│   ├── OCCC_points.ply
│   ├── ...
│   └── WMSC_points.ply
├── Synthetic_v1
│   ├── Austin.ply
│   ├── ...
│   └── TownshipofWashington.ply
├── Synthetic_v2
│   ├── 2_points_GTv2.ply
│   ├── ...
│   └── j_points_GTv2.ply
└── Synthetic_v3
    ├── 1_points_GTv3.ply
    ├── ...
    └── 25_points_GTv3.ply
python3 train_STPLS3D.py
python3 test_models.py

Point Transformer (Ubuntu): Please refer to Point Transformer to test it on our STPLS3D dataset.

4.2 Instance segmentation:

Mask3D: Special thanks to Jonas Schult for implementing Mask3D for STPLS3D! Please refer to the official Mask3D for implementation details, and download their pretrained model.

SoftGroup: Special thanks to Thang Vu for implementing SoftGroup for STPLS3D! Please refer to the official SoftGroup for implementation details, and downloading their pretrained model.

HAIS (Ubuntu): The environment setup is the same as the official HAIS release

git clone https://github.com/meidachen/STPLS3D.git
cd STPLS3D/HAIS
conda create -n hais python=3.7
conda activate hais
pip install -r requirements.txt
conda install -c bioconda google-sparsehash
conda install libboost
conda install -c daleydeng gcc-5
cd STPLS3D/HAIS/lib/spconv
export CUDACXX= $PATH_TO_NVCC$
python setup.py bdist_wheel
cd STPLS3D/HAIS/lib/spconv/dist
pip install {wheel_file_name}.whl
cd STPLS3D/HAIS/lib/hais_ops
export CPLUS_INCLUDE_PATH={conda_env_path}/hais/include:$CPLUS_INCLUDE_PATH
python setup.py build_ext develop

Download the data, unzip it and place it under STPLS3D/HAIS/dataset.

HAIS
├── dataset
   └── Synthetic_v3_InstanceSegmentation
       ├── 1_points_GTv3.txt
       ├── 2_points_GTv3.txt
       ├── 3_points_GTv3.txt
       ├── ...
       ├── 23_points_GTv3.txt
       ├── 24_points_GTv3.txt
       └── 25_points_GTv3.txt
cd STPLS3D/HAIS/data
python prepare_data_inst_instance_stpls3d.py

By default, scene 5, 10, 15, 20, 25 are used as the validation sets. This can be changed at https://github.com/meidachen/STPLS3D/blob/6eec7abe760a45dc970714f62f6b0e555a2f44b7/HAIS/data/prepare_data_inst_instance_stpls3d.py#L179 https://github.com/meidachen/STPLS3D/blob/6eec7abe760a45dc970714f62f6b0e555a2f44b7/HAIS/data/prepare_data_inst_instance_stpls3d.py#L186

(optional) In case you are changing training data (i.e., not using data agumentation, using different ways for data agumentation, etc.), please run prepare_data_statistic_stpls3d.py to get the class_weight, class_radius_mean, and class_numpoint_mean_dict. Change them in hais_run_stpls3d.yaml, hierarchical_aggregation.cpp, and hierarchical_aggregation.cu accordingly. Make sure you rebuild the hais_ops.

CUDA_VISIBLE_DEVICES=1 python train.py --config config/hais_run_stpls3d.yaml 
CUDA_VISIBLE_DEVICES=1 python test.py --config config/hais_run_stpls3d.yaml --pretrain exp/Synthetic_v3_InstanceSegmentation/hais/hais_run_stpls3d/hais_run_stpls3d-000000500.pth

The unlabeled data can be downloaded here. Unzip it and place the three .txt files under STPLS3D/HAIS/dataset/Synthetic_v3_InstanceSegmentation.

HAIS
├── dataset
   └── Synthetic_v3_InstanceSegmentation
       ├── 26_points_GTv3.txt
       ├── 27_points_GTv3.txt
       └── 28_points_GTv3.txt

Run the preparation script again

cd STPLS3D/HAIS/data
python prepare_data_inst_instance_stpls3d.py

Set split to test https://github.com/meidachen/STPLS3D/blob/6eec7abe760a45dc970714f62f6b0e555a2f44b7/HAIS/config/hais_run_stpls3d.yaml#L71

Set save_instance to True https://github.com/meidachen/STPLS3D/blob/6eec7abe760a45dc970714f62f6b0e555a2f44b7/HAIS/config/hais_run_stpls3d.yaml#L84

Run evaluation again

CUDA_VISIBLE_DEVICES=1 python test.py --config config/hais_run_stpls3d.yaml --pretrain exp/Synthetic_v3_InstanceSegmentation/hais/hais_run_stpls3d/hais_run_stpls3d-000000500.pth

Once completed, you may find the results under exp/Synthetic_v3_InstanceSegmentation/hais/hais_run_stpls3d/result/test

You only need to keep the 300 txt files and the predicted_masks folder, and zip it to submit on our evaluation server. An example of the submission zip can be find here.

(5) Instance segmentation challenge and evaluation server

we are organizing the Urban3D@ICCV2023 - The 3rd Challenge on Large-Scale Point Clouds Analysis for Urban Scenes Understanding. The instance segmentation challenge is on CodaLab! Please feel free to submit your results to our evaluation server.

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{Chen_2022_BMVC,
author    = {Meida Chen and Qingyong Hu and Zifan Yu and Hugues THOMAS and Andrew Feng and Yu Hou and Kyle McCullough and Fengbo Ren and Lucio Soibelman},
title     = {STPLS3D: A Large-Scale Synthetic and Real Aerial Photogrammetry 3D Point Cloud Dataset},
booktitle = {33rd British Machine Vision Conference 2022, {BMVC} 2022, London, UK, November 21-24, 2022},
publisher = {{BMVA} Press},
year      = {2022},
url       = {https://bmvc2022.mpi-inf.mpg.de/0429.pdf}
}

Related Repos

Semantic segmentation:

Instance segmentation:

Data set:

Others: