Awesome

<img src="figures/gem (3).png" width="30"> GEM: Glass-Segmentor

Jing Hao, Moyun Liu, Jinrong Yang, Kuo Feng Hung.

This repository is the official implementation of the GEM: Boost Simple Network for Glass Surface Segmentation via Vision Foundation Models. Our code is based on MaskDINO.

• Release code and GEM-Tiny checkpoint

• Release S-GSD dataset

• Release multiple pre-trained models

• Release GEM-Base model

Features

• A simple but accurate segmentation framework, named GEM, for glass surface segmentation.
• Automatically construct a large-scale synthesized glass surface dataset with precise mask annotation, termed S-GSD.
• Surpasses the previous state-of-the-art methods by a large margin (IoU +2.1%).

Installation

See Mask DINO.

Getting Started

See Inference Demo with Pre-trained Model

See Results.

See Preparing Datasets for GEM.

See Getting Started.

Results

[07/15/2024] All pre-trained models can also download from Huggingface

[07/18/2024] S-GSD-1x dataset can also download from Huggingface

The download link of S-GSD-5x is blow here, it be divided into three parts:

Part1: https://pan.baidu.com/s/1CL0x8s1LXdIIoOjcr5wirw?pwd=2ff4

Part2: https://pan.baidu.com/s/18uCDKFmzy7vSmWm1IF4JFQ?pwd=9jqb

Part3: https://pan.baidu.com/s/1Z8pePl9Ps3QtrZB7WQ7aoQ?pwd=8fku

The S-GSD-10x and S-GSD-20x are not released because of the large disk storage, if you want to get these two large-scale datasets, feel free to contact me via isjinghao@gmail.com.

Getting Started

In the above tables, the corresponding model checkpoints can pre-trained models can be downloaded from the link.

If your dataset files are not under this repo, you need to add export DETECTRON2_DATASETS=/path/to/your/data or use Symbolic Link ln -s to link the dataset into this repo before the following command first.

Evalaluate our pretrained models

• You can download our pretrained models and evaluate them with the following commands.

  python train_net.py --eval-only --num-gpus 8 --config-file config_path MODEL.WEIGHTS /path/to/checkpoint_file
  

Train GEM to reproduce results

• Use the above command without eval-only will train the model. For MobileSAM/SAM backbones, you need to download its weight from () and specify the path of the pretrained backbones with MODEL.WEIGHTS /path/to/pretrained_checkpoint

  python train_net.py --num-gpus 8 --config-file config_path MODEL.WEIGHTS /path/to/checkpoint_file
  

You can also refer to Getting Started with Detectron2 for full usage.

<a name="CitingMaskDINO"></a>Citing GEM

If you find our work helpful for your research, please consider citing the following BibTeX entry.

@article{hao2024gem,
  title={GEM: Boost Simple Network for Glass Surface Segmentation via Segment Anything Model and Data Synthesis},
  author={Hao, Jing and Liu, Moyun and Hung, Kuo Feng},
  journal={arXiv preprint arXiv:2401.15282},
  year={2024}
}

If you find the code useful, please also consider the following BibTeX entry.

@misc{li2022mask,
      title={Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation}, 
      author={Feng Li and Hao Zhang and Huaizhe xu and Shilong Liu and Lei Zhang and Lionel M. Ni and Heung-Yeung Shum},
      year={2022},
      eprint={2206.02777},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

Many thanks to these excellent opensource projects

• Mask DINO
• Segment-Anything