Awesome

<p align=center>GCoNet+: A Stronger Group Collaborative Co-Salient Object Detector</p>

This repo is the official implementation of "GCoNet+: A Stronger Group Collaborative Co-Salient Object Detector" (T-PAMI 2023).

Note that this work has more than 70% new staff (methods, discussion, etc.) and much better performance compared with the GCoNet (the conference version).

Authors: Peng Zheng, Huazhu Fu, Deng-Ping Fan, Qi Fan, Jie Qin, Yu-Wing Tai, Chi-Keung Tang, & Luc Van Gool.

[arXiv] [IEEE] [code] [stuff] [中文版]

Try our online demo for inference:

Try our FPS benchmark of previous CoSOD methods: CoSOD_fps_collection.

Abstract

In this paper, we present a novel end-to-end group collaborative learning network, termed GCoNet+, which can effectively and efficiently (250 fps) identify co-salient objects in natural scenes. The proposed GCoNet+ achieves the new state-of-the-art performance for co-salient object detection (CoSOD) through mining consensus representations based on the following two essential criteria: 1) intra-group compactness to better formulate the consistency among co-salient objects by capturing their inherent shared attributes using our novel group affinity module (GAM); 2) inter-group separability to effectively suppress the influence of noisy objects on the output by introducing our new group collaborating module (GCM) conditioning on the inconsistent consensus. To further improve the accuracy, we design a series of simple yet effective components as follows: i) a recurrent auxiliary classification module (RACM) promoting the model learning at the semantic level; ii) a confidence enhancement module (CEM) helping the model to improve the quality of the final predictions; and iii) a group-based symmetric triplet (GST) loss guiding the model to learn more discriminative features. Extensive experiments on three challenging benchmarks, i.e., CoCA, CoSOD3k, and CoSal2015, demonstrate that our GCoNet+ outperforms the existing 12 cutting-edge models. Code has been released at https://github.com/ZhengPeng7/GCoNet_plus.

Framework Overview

The figure of network architecture is drawn by Inkscape (0.92.5) as a .svg file. You can download and modify it if you find it useful.

Result

• Comparison with the previous state-of-the-art methods with different training sets:

• Ablation study:

Prediction

To see the better performance of our GCoNet+, we select the currently latest and top models (UFO-arXiv2022, DCFM-CVPR2022, and CADC-ICCV2021) for the qualitative comparison.

We not only show the selected extremely hard samples in the test sets but also simply put the unscreened samples (the first 10 samples in the first group in CoCA) for more objective and fair qualitative comparisons.

• The first^2 samples:

• The extremely hard cases:

Usage

1. Environment

   GPU: V100 x 1
   Install Python 3.7, PyTorch 1.8.2
   pip install requirements.txt
   
   

2. Datasets preparation

   Download necessary datasets:
   from my google-drive: DUTS_class, COCO-9k, COCO-SEG, and CoSOD_testsets, or
   from my BaiduDisk: DUTS_class, COCO-9k, COCO-SEG, and CoSOD_testsets.
   The CoSOD_testsets contains CoCA, CoSOD3k and CoSal2015.

   The file directory structure on my machine is as follows:

   +-- datasets
   |   +-- sod
   |       +-- images
   |           +-- DUTS_class
   |           +-- COCO-9k
   |           ...
   |           +-- CoSal2015
   |       +-- gts
   |           +-- DUTS_class
   |           +-- COCO-9k
   |           ...
   |           +-- CoSal2015
   |   ...
   ...
   +-- codes
   |   +-- sod
   |       +-- GCoNet_plus
   |       ...
   |   ...
   ...
   

3. Update the paths

   Replace all /root/datasets/sod/GCoNet_plus and /root/codes/sod/GCoNet_plus in this project to /YOUR_PATH/datasets/sod/GCoNet_plus and /YOUR_PATH/codes/sod/GCoNet_plus, respectively.

4. Training + Test + Evaluate + Select the Best

   ./gco.sh

   If you can apply more GPUs on the DGX cluster, you can ./sub_by_id.sh to submit multiple times for more stable results.

   If you have the OOM problem, plz decrease batch_size in config.py.

5. Adapt the settings of modules in config.py

   You can change the weights of losses, try various backbones or use different data augmentation strategies. There are also some modules coded but not used in this work, like adversarial training, the refiner in BASNet, weighted multiple output and supervision used in GCoNet and GICD, etc.

   

Download

​ Find well-trained models + predicted saliency maps and all other stuff on my google-drive folder for this work:

Acknowledgement

We appreciate the codebases of GICD, GCoNet. Thanks for the CoSOD evaluation toolbox provided in eval-co-sod. Thanks for the drawing codes of figure 1 from DGNet.

Citation

@article{zheng2022gconet+,
  author={Zheng, Peng and Fu, Huazhu and Fan, Deng-Ping and Fan, Qi and Qin, Jie and Tai, Yu-Wing and Tang, Chi-Keung and Van Gool, Luc},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={GCoNet+: A Stronger Group Collaborative Co-Salient Object Detector}, 
  year={2023},
  volume={45},
  number={9},
  pages={10929-10946},
  doi={10.1109/TPAMI.2023.3264571}
}
@inproceedings{fan2021gconet,
  title = {Group Collaborative Learning for Co-Salient Object Detection},
  author = {Fan, Qi and Fan, Deng-Ping and Fu, Huazhu and Tang, Chi-Keung and Shao, Ling and Tai, Yu-Wing},
  booktitle = CVPR,
  year = {2021}
}

Contact

Any question, discussion or even complaint, feel free to leave issues here or send me e-mails (zhengpeng0108@gmail.com).