Home

Awesome

CalibAnything

This package provides an automatic and target-less LiDAR-camera extrinsic calibration method using Segment Anything Model. The related paper is Calib-Anything: Zero-training LiDAR-Camera Extrinsic Calibration Method Using Segment Anything. For more calibration codes, please refer to the link <a href="https://github.com/PJLab-ADG/SensorsCalibration" title="SensorsCalibration">SensorsCalibration</a>.

Prerequisites

Compile

git clone https://github.com/OpenCalib/CalibAnything.git
cd CalibAnything
# mkdir build
mkdir -p build && cd build
# build
cmake .. && make

Run Example

We provide examples of two dataset. You can download the processed data at Google Drive or BaiduNetDisk:

# baidunetdisk
Link: https://pan.baidu.com/s/1qAt7nYw5hYoJ1qrH0JosaQ?pwd=417d 
Code: 417d

Run the command:

cd CalibAnything
./bin/run_lidar2camera ./data/kitti/calib.json # kitti dataset
./bin/run_lidar2camera ./data/nuscenes/calib.json # nuscenes dataset

Test your own data

Data collection

Preprocessing

Generate masks

Follow the instructions in Segment Anything and generate masks of your image.

  1. First download a model checkpoint. You can choose vit-l.

  2. Install SAM

# environment: python>=3.8, pytorch>=1.7, torchvision>=0.8

git clone git@github.com:facebookresearch/segment-anything.git
cd segment-anything; pip install -e .
pip install opencv-python pycocotools matplotlib onnxruntime onnx
  1. Run
python scripts/amg.py --checkpoint <path/to/checkpoint> --model-type <model_type> --input <image_or_folder> --output <path/to/output>

# example(recommend parameter)
python scripts/amg.py --checkpoint sam_vit_l_0b3195.pth --model-type vit_l --input ./data/kitti/000000/images/  --output ./data/kitti/000000/masks/ --stability-score-thresh 0.9 --box-nms-thresh 0.5 --stability-score-offset 0.9

Data folder

The hierarchy of your folders should be formed as:

YOUR_DATA_FOLDER
├─calib.json
├─pc
|   ├─000000.pcd
|   ├─000001.pcd
|   ├─...
├─images
|   ├─000000.png
|   ├─000001.png
|   ├─...
├─masks
|   ├─000000
|   |   ├─000.png
|   |   ├─001.png
|   |   ├─...
|   ├─000001
|   ├─...

Processed masks

For large masks, we only use part of it near the edge.

python processed_mask.py -i <YOUR_DATA_FOLDER>/masks/ -o <YOUR_DATA_FOLDER>/processed_masks/

Edit the json file

<details><summary>Content description</summary> </details>

Calibration

./bin/run_lidar2camera <path-to-json-file>

Output

Citation

If you find this project useful in your research, please consider cite:

@misc{luo2023calibanything,
      title={Calib-Anything: Zero-training LiDAR-Camera Extrinsic Calibration Method Using Segment Anything}, 
      author={Zhaotong Luo and Guohang Yan and Yikang Li},
      year={2023},
      eprint={2306.02656},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}