Home

Awesome

RGB-T-Glass-Segmentation

Code for this paper Glass Segmentation with RGB-Thermal Image Pairs

Dong Huo, Jian Wang, Yiming Qian, Yee-Hong Yang

Overview

This paper proposes a new glass segmentation method utilizing paired RGB and thermal images. Due to the large difference between the transmission property of visible light and that of the thermal energy through the glass where most glass is transparent to the visible light but opaque to thermal energy, glass regions of a scene are made more distinguishable with a pair of RGB and thermal images than solely with an RGB image. To exploit such a unique property, we propose a neural network architecture that effectively combines an RGB-thermal image pair with a new multi-modal fusion module based on attention. As well, we have collected a new dataset containing 5551 RGB-thermal image pairs with ground-truth segmentation annotations. The qualitative and quantitative evaluations demonstrate the effectiveness of the proposed approach on fusing RGB and thermal data for glass segmentation.

Motivation

Most glass is transparent to the visible light but opaque to thermal energy, glass regions of a scene are made more distinguishable with a pair of RGB and thermal images than solely with an RGB image.

<p align="center"> <img width="400" src="./images/motivation.png"> </p>

Architecture

Our architecture follows the standard encoder-decoder framework with skip-connections, which consists of two encoding branches, one decoding branch and a multi-modal fusion module (MFM) as the bridge.

<p align="center"> <img width="800" src="./images/architecture.png"> </p>

Datasets

The datasets utilized in our paper can be downloaded via the links below:

Prerequisites

Training

RGB-T: python main.py --rgbt_path your_data_path

RGB-only: python main.py --rgbt_path your_data_path --is_rgbt False

Modify the arguments in parse_args()

Testing

RGB-T: python main.py --rgbt_path your_data_path --resume your_checkpoints_path --eval

RGB-only: python main.py --rgbt_path your_data_path --is_rgbt False --resume your_checkpoints_path --eval

Download the well-trained models (RGB-T, RGB-only)

Compared results

Download the compared results from the link

Citation

If you use this code and data for your research, please cite our paper.

@article{huo2023glass,
  title={Glass segmentation with RGB-thermal image pairs},
  author={Huo, Dong and Wang, Jian and Qian, Yiming and Yang, Yee-Hong},
  journal={IEEE Transactions on Image Processing},
  volume={32},
  pages={1911--1926},
  year={2023},
  publisher={IEEE}
}