Awesome
OSTRA
OSTRA is a novel segmentation-then-reconstruction method for segmenting complex open objects in 3D point clouds. This method uses a Segment-Anything Model (SAM) to segment target objects and video object segmentation (VOS) technology to continuously track video frame segmentation targets. Our pipeline enables a complete segmentation process from videos to 3D cloud points and meshes in different level(semantic segmentation, instance segmentation and part segmentation).
You can check our detailed tutorials here!
:computer:Getting Started
Clone OSTRA
This project is tested under python3.9, cuda11.5 and pytorch1.11.0. An equivalent or higher version is recommended.
#Clone OSTRA
git clone --recursive https://github.com/ganlab/OSTRA.git
cd OSTRA
#Install dependencies:
pip install -r requirements.txt
Install Colmap
Our reconstruction process is based on Colmap. Please follow the instruction and install Colmap first.
Prepare Model
All these models are required for OSTRA: SAM: the default model is sam_vit_b_01ec64.pth DeAOT:the default model is R50_DeAOTL_PRE_YTB_DAV.pth XMem:the default model is XMem-s012.pth Grounding-DINO:the default model is groundingdino_swint_ogc.pth
WebUI
We developed WebUI that user can easily access.
python app.py --device cuda:0
:film_projector: Demo
Two samples of complex object segmentation:
<div align=center class='img_top'> <img src="./assets/top.jpg" width="90%"/> </div> <div align=center class='img_top'> <img src="./assets/result.gif" width="90%"/> </div>:book:Citation
Please considering cite our paper if you find this work useful!
@misc{xu2023stop,
title={A One Stop 3D Target Reconstruction and multilevel Segmentation Method},
author={Jiexiong Xu and Weikun Zhao and Zhiyan Tang and Xiangchao Gan},
year={2023},
eprint={2308.06974},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
:clap: Acknowledgements
This work is based on Segment Anything, Track Anything, Segment and Track Anything, Colmap and Open3D.