Awesome

ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding

Overview

Installation

The code requires python>=3.8, as well as pytorch>=1.7 and torchvision>=0.8. For this project, python=3.8, pytorch=1.11.0, and torchvision=0.12.0 are used; run the following command in the root directory of this project:

cd ./ASI
pip install -r requirements.txt

Dataset

The datasets we used in our experiments are endoivs 2018 and endoivs 2017.

For EndoVis2017, we use robot-surgery-segmentation as our pre-processing strategies and cross-validation splits.

For EndoVis2018, we use ISINet as the instrument type segmentation annotation.

Checkpoints

In ASI-Seg, we used vit_h for SAM (Segmentation Anything Model), and CLIP (Contrastive Language-Image Pre-Training).

Please see the download link of the checkpoint of SAM in the vit_h version here.

Please run the following command in the root directory of this project to download CLIP:

cd ./ASI
git clone https://github.com/openai/CLIP.git

Train

Run the following command in the root directory:

cd ./ASI
python train.py

Inference

Run the following command in the root directory:

cd ./ASI
python inference.py

Citation

If you use our code or paper in your work, please cite our paper.

@inproceedings{chen2024iros,
	title={{ASI-Seg: Audio-Driven Surgical Instrument Segmentation with Surgeon Intention Understanding}},
	author={Zhen Chen, Zongming Zhang, Wenwu Guo, Xingjian Luo, Long Bai, Jinlin Wu,
Hongliang Ren, Hongbin Liu},
	booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
	year={2024}
}