Awesome
Zero-Shot Surgical Tool Segmentation in Monocular Video Using Segment Anything Model 2
🚨 News
The paper is available: Arxiv
🧐 Installation & Usage
Directly use .ipynb file
🎯 Example Cases
EndoNeRF Dataset
| Prompt (first frame) | Video segmentation |
|---|---|
| <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/cutting_prompt.png" width="360" alt="Result"/></div> | <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/cutting_output.gif" width="360" alt="Result"/></div> |
| <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/pulling_prompt.png" width="360" alt="Result"/></div> | <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/pulling_output.gif" width="360" alt="Result"/></div> |
EndoVis'17 Segmentation Dataset
| Prompt (first frame) | Video segmentation |
|---|---|
| <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/endovis17_1_prompt.png" width="360" alt="Result"/></div> | <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/endovis17_1.gif" width="360" alt="Result"/></div> |
| <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/endovis17_3_prompt.png" width="360" alt="Result"/></div> | <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/endovis17_3.gif" width="360" alt="Result"/></div> |
SurgToolLoc Dataset
| Prompt (first frame) | Video segmentation |
|---|---|
| <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/surgtoolloc22_prompt.png" width="360" alt="Result"/></div> | <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/surgtoolloc22.gif" width="360" alt="Result"/></div> |
Cochlear Implant Dataset (We focus more on the insertion and driling tools)
| Prompt (first frame) | Video segmentation |
|---|---|
| <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/driling_vumc_prompt.png" width="360" alt="Result"/></div> | <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/driling_vumc.gif" width="360" alt="Result"/></div> |
| <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/insertion_vumc_prompt.png" width="360" alt="Result"/></div> | <div align=center><img src="https://github.com/AngeLouCN/SAM-2_Surgical_Video/blob/main/result/insertion_vumc.gif" width="360" alt="Result"/></div> |
📝 Cite
@misc{lou2024zeroshotsurgicaltoolsegmentation,
title={Zero-Shot Surgical Tool Segmentation in Monocular Video Using Segment Anything Model 2},
author={Ange Lou and Yamin Li and Yike Zhang and Robert F. Labadie and Jack Noble},
year={2024},
eprint={2408.01648},
archivePrefix={arXiv},
primaryClass={eess.IV},
url={https://arxiv.org/abs/2408.01648},
}