Home

Awesome

<div align='center'>

ALSO: Automotive Lidar Self-supervision by Occupancy estimation

Alexandre Boulch<sup>1</sup>    Corentin Sautier<sup>1,2</sup>    Björn Michele<sup>1,3</sup>    Gilles Puy<sup>1</sup>    Renaud Marlet<sup>1,2</sup>

<sub> <sup>1</sup> Valeo.ai, Paris, France <sup>2</sup> LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Vallée, France

<sup>3</sup> CNRS, IRISA, Univ. Bretagne Sud, Vannes, France </sub>

<br/>

Arxiv

<br/>

ALSO has been accepted at CVPR 2023

<br/>

Overview

</div> <br/>

This is the reference PyTorch implementation for training and testing self-supervision for automotive lidar point cloud using the method described in our paper ALSO: Automotive Lidar Self-supervision by Occupancy estimation

Overview


Citation

Please acknowledge our work in your publications:

@InProceedings{ALSO,
author = {Alexandre Boulch and Corentin Sautier and Björn Michele and Gilles Puy and Renaud Marlet},
title = {{ALSO}: Automotive Lidar Self-supervision by Occupancy estimation},
booktitle = {International Conference on Computer Vision and Pattern Recognition (CVPR)},
year = 2023,
}

Dependencies

We give the library version(s), we used for experiments.

Semantic segmentation specific libraries

Detection specific libraries

Installation

At root of the folder run:

pip install -ve .

Then all scripts can be run from also_selfsup folder:

cd also_selfsup

Datasets

ONCE

We follow intruction from ONCE dataset to generate the information files of the raw_small dataset.

Semantic segmentation

Pre-training

python train_selfsupervised.py cfg=nuscenes

for a pre-training on nuScenes.

for other datasets, replace cfg=nuscenes by the desired dataset

Parameters

We use Hydra for easy configuration of trainings. The pretraining parameters are set for pre-training with a NVidia-V100 16Gb. In order to pre-train on other configurations, one can modify parameters directly in the command line, e.g., setting the batch size to 8:

python train_selfsupervised.py ... cfg.training.batch_size=8

or the backbone to SPVCNN:

python train_selfsupervised.py ... cfg.network.backbone=SPVCNN

Downstream

From scratch


# 100%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=30 cfg.val_interval=5 cfg.downstream.skip_ratio=1

# 50%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=50 cfg.val_interval=5 cfg.downstream.skip_ratio=2

# 10%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=100 cfg.val_interval=10 cfg.downstream.skip_ratio=10

# 1%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=500 cfg.val_interval=50 cfg.downstream.skip_ratio=100

# 0.1%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=1000 cfg.val_interval=100 cfg.downstream.skip_ratio=1000

With pretrained models

First, convert the pre-trained model to be loaded

python convert_models.py --ckpt path_to_pretraining_checkpoint

Second, run downstream training

python train_downstream_semseg.py cfg=nuscenes cfg.downstream.checkpoint_dir='path_to_checkpoint_directory' cfg.downstream.checkpoint_name='pretrained_backbone_XXX.ckpt'

Evaluation of downstream models

First, convert the trained model to be loaded

python convert_models.py --downstream --ckpt path_to_downstream_checkpoint

Second, run the evaluation script

python eval.py --split val --config path_to_downstream_model/config.yaml --ckpt path_to_downstream_checkpoint/trained_model_XXX.ckpt

Detection

Pre-training

Similar to semantic segmentation, pre-training for detection can simply be run using:

python train_selfsupervised.py cfg=kitti3d_second
python train_selfsupervised.py cfg=nuscenes_second_kitti

Downstream

First, convert the trained model to be loaded

python convert_models.py --ckpt path_to_pretraining_checkpoint

Second, use the pre-trained model with the official OpenPCDet code:

cd path_to_OpenPCDet/tools/
python train.py --cfg_file cfgs/kitti_models/second.yaml --extra_tag --pretrained_model path_to_pretrained_model/pretrained_backbone_XXX.ckpt

Pre-trained models

TaskPre-training datasetModelLinkNotes
Semantic segmentationnuScenesMinkUNet34download
  SPVCNNdownload
 SemanticKITTIMinkUNet18SCdownload
  SPVCNNdownload
DetectionKITTI3DSECOND/PVRCNNdownloadtrained for KITTI3D
 nuScenesSECOND/PVRCNNdownloadtrained for KITTI3D
 KITTI360SECOND/PVRCNNdownloadtrained for KITTI3D
 ONCE SmallSECOND/PVRCNNdownloadtrained for ONCE

Here are links to pre-trained models

Acknowledgments

This project would not have been possible without many community resources and repositories. Among them:

Please, consider acknowleding these projects.