Awesome
<div align='center'>ALSO: Automotive Lidar Self-supervision by Occupancy estimation
Alexandre Boulch<sup>1</sup> Corentin Sautier<sup>1,2</sup> Björn Michele<sup>1,3</sup> Gilles Puy<sup>1</sup> Renaud Marlet<sup>1,2</sup>
<sub> <sup>1</sup> Valeo.ai, Paris, France <sup>2</sup> LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Vallée, France<sup>3</sup> CNRS, IRISA, Univ. Bretagne Sud, Vannes, France </sub>
<br/> <br/>ALSO has been accepted at CVPR 2023
<br/> </div> <br/>This is the reference PyTorch implementation for training and testing self-supervision for automotive lidar point cloud using the method described in our paper ALSO: Automotive Lidar Self-supervision by Occupancy estimation
Overview
- Citation
- Dependencies
- Installation
- Semantic segmentation
- Pre-trained models
- Performances
- Acknowledgments
Citation
Please acknowledge our work in your publications:
@InProceedings{ALSO,
author = {Alexandre Boulch and Corentin Sautier and Björn Michele and Gilles Puy and Renaud Marlet},
title = {{ALSO}: Automotive Lidar Self-supervision by Occupancy estimation},
booktitle = {International Conference on Computer Vision and Pattern Recognition (CVPR)},
year = 2023,
}
Dependencies
We give the library version(s), we used for experiments.
- Hydra (1.2.0)
- Pytorch-geometric (2.1.0)
- Pytorch-Lightning (1.7.2)
Semantic segmentation specific libraries
- torchsparse (1.4.0)
Detection specific libraries
- spconv (2.1.21)
- openpcdet (0.6.0)
Installation
At root of the folder run:
pip install -ve .
Then all scripts can be run from also_selfsup
folder:
cd also_selfsup
Datasets
ONCE
We follow intruction from
ONCE dataset to generate the information files of the raw_small
dataset.
Semantic segmentation
Pre-training
python train_selfsupervised.py cfg=nuscenes
for a pre-training on nuScenes.
for other datasets, replace cfg=nuscenes
by the desired dataset
- nuScenes:
cfg=nuscenes
- SemanticKITTI:
cfg=semantickitti
- SemanticPOSS: To be added
- LivoxSimuDataset: To be added
Parameters
We use Hydra for easy configuration of trainings. The pretraining parameters are set for pre-training with a NVidia-V100 16Gb. In order to pre-train on other configurations, one can modify parameters directly in the command line, e.g., setting the batch size to 8:
python train_selfsupervised.py ... cfg.training.batch_size=8
or the backbone to SPVCNN:
python train_selfsupervised.py ... cfg.network.backbone=SPVCNN
Downstream
From scratch
# 100%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=30 cfg.val_interval=5 cfg.downstream.skip_ratio=1
# 50%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=50 cfg.val_interval=5 cfg.downstream.skip_ratio=2
# 10%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=100 cfg.val_interval=10 cfg.downstream.skip_ratio=10
# 1%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=500 cfg.val_interval=50 cfg.downstream.skip_ratio=100
# 0.1%
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.max_epochs=1000 cfg.val_interval=100 cfg.downstream.skip_ratio=1000
With pretrained models
First, convert the pre-trained model to be loaded
python convert_models.py --ckpt path_to_pretraining_checkpoint
Second, run downstream training
python train_downstream_semseg.py cfg=nuscenes cfg.downstream.checkpoint_dir='path_to_checkpoint_directory' cfg.downstream.checkpoint_name='pretrained_backbone_XXX.ckpt'
Evaluation of downstream models
First, convert the trained model to be loaded
python convert_models.py --downstream --ckpt path_to_downstream_checkpoint
Second, run the evaluation script
python eval.py --split val --config path_to_downstream_model/config.yaml --ckpt path_to_downstream_checkpoint/trained_model_XXX.ckpt
Detection
Pre-training
Similar to semantic segmentation, pre-training for detection can simply be run using:
python train_selfsupervised.py cfg=kitti3d_second
python train_selfsupervised.py cfg=nuscenes_second_kitti
Downstream
First, convert the trained model to be loaded
python convert_models.py --ckpt path_to_pretraining_checkpoint
Second, use the pre-trained model with the official OpenPCDet code:
cd path_to_OpenPCDet/tools/
python train.py --cfg_file cfgs/kitti_models/second.yaml --extra_tag --pretrained_model path_to_pretrained_model/pretrained_backbone_XXX.ckpt
Pre-trained models
Task | Pre-training dataset | Model | Link | Notes |
---|---|---|---|---|
Semantic segmentation | nuScenes | MinkUNet34 | download | |
SPVCNN | download | |||
SemanticKITTI | MinkUNet18SC | download | ||
SPVCNN | download | |||
Detection | KITTI3D | SECOND/PVRCNN | download | trained for KITTI3D |
nuScenes | SECOND/PVRCNN | download | trained for KITTI3D | |
KITTI360 | SECOND/PVRCNN | download | trained for KITTI3D | |
ONCE Small | SECOND/PVRCNN | download | trained for ONCE |
Here are links to pre-trained models
Acknowledgments
This project would not have been possible without many community resources and repositories. Among them:
Please, consider acknowleding these projects.