Awesome

Reproducibility of "Campus3D:A Photogrammetry Point Cloud Benchmark for Outdoor Scene Hierarchical Understanding"

Introduction

The repository contains the re-implementation of this ACM MM 2020 Paper based on the repository. It also presents the reproduced results of the supported paper with trained models in MODEL ZOO. The reduced version of Campus3D dataset can be donwloaded from the official website. Compared to the original repo, the main updates of this reproducibilty repo are:

• Transfer the model implementation from TenforFlow with precompling requiremnets to pure PyTorch
• Add DGCNN and PointCNN to the benchmark models. Report the detailed benchmarks with mutiple networks and their pre-trained models.
• Upgrade the sampling method with KDTree, more details can be found in dataset/sampling/sampler_collections.py

Installation

The whole package can be downloaded by the following command.

git clone https://github.com/Yuqing-Liao/reproduce-campus3d.git

Dependencies can be installed using the provided script.

cd reproduce-campus3d
pip install -r requirements.txt

Compressed Campus3D dataset file campus3d-reduce.zip can be downloaded from official website. Put it into data/ and unzip with below script.

cd reproduce-campus3d/data
unzip campus3d-reduce.zip

Training & Evaluation

Train from scratch

To apply training of the model, please first check the configuration files in config/. Particularly you need to change the value of IS_PRETRAINED to False and then run experiments, eg:

cd reproduce-campus3d
python run.py --model 'pointnet2' --mc_level -1 --exp_name 'EXP_NAME'

The 'EXP_NAME' is the user-defined name. In this way, the models will be saved in checkpoints/EXP_NAME/models, and other output files will be saved in checkpoints/EXP_NAME.

Train from pretrained model

Pretrained models are available on Google Drive, and they can be downloaded through the link presented in the following table. You can train either from the downloaded models or from your own pretrained models. To apply training of the model, please first check the configuration files in config/. Particularly you need to change the value of IS_PRETRAINED to True, PRETRAINED_MODEL_PATH to the path of the model to train and then run experiments, eg:

cd reproduce-campus3d
python run.py --model 'pointnet2' --mc_level -1 --exp_name 'EXP_NAME'

In this way, the models will be saved in checkpoints/EXP_NAME/models, and other output files will be saved in checkpoints/EXP_NAME.

Evaluation

To apply evaluation of the model on the test set, please first check the configuration files in config/. Particularly you need to change the value of PRETRIANED_MODEL_PATH to the path of the model to evaluate and then run experiments, eg:

cd reproduce-campus3d
python run.py --eval true --model 'pointnet2' --mc_level -1 --exp_name 'EXP_NAME'

In this way, the output files will be saved in check/EXP_NAME.

Experiments

Hierarchical Learning (HL) Experiments

The hierarchical learning experiments were proposed to present the effectiveness of the Multi-task and Hierarchical Esemble(MT+HE) method. Multi-classifiers(MC) in each level were also proposed for comparison. To run the training, the argument --mc_level can be set as 0-4 and -1 for MC experiments in 0-4 levels and MT+HE experiments in all levels respectively. In addition, the MT training contains two stage Multi-task Learning without consistency loss(MT_nc) and Multi-task Learning with consistency loss(MT), of which the MT is trained based on the pretrained MT_nc model.

To run the evaluation, the argument --mc_level in run script can be set as any one from 0-4 to test 5 MC models altogether for MC and MC+HE results, or set as -1 to test the target model for MT_nc or MT(+HE) results. Particularly for MC evaluation, the PRETRAINED_MODEL_PATH is supposed to be set as the list of 5 model paths in the config file.

Benchmark Experiments

The semantic segmentation bechmark were built with three models PointNet++, PointCNN and DGCNN. They are all conducted via the MT+HE method for hierarchical learning on the Campus3D dataset. To run different models, one can change the argument --model as the indicated model. Following are the reference repository for PyTorch implementation of 3D deep models.

PointNet++ GitHub Link

PointCNN GitHub Link

DGCNN GitHub Link

MODEL ZOO

Models

Benchmark Experiments Results

Semantic segmentation benchmarks (mIoU% and OA%) for models with MT+HE method

These results are produced by model No.8, No.9 and No.10.

Hierarchical Learning Experiments Results

Semantic segmentation benchmarks(OA%) for different HL methods with model PointNet++

These results are produced by model No.0-4, No.5 and No.8. They demonstrate the effectiveness of the MT+HE method for HL problem. Results with detailed per-class IoU are displayed below.

Semantic segmentation benchmarks(IoU%) for different HL methods with model PointNet++