Home

Awesome

JSM Source Code

Source repository for our paper entilted "Joint Semantic Mining for Weakly Supervised RGB-D Salient Object Detection" accepted by NeurIPS 2021 (poster).

Pre-Implementation

  1. vim ./DenseCRF/README.md
  2. DenseCRF Installation: Refer to DenseCRF Readme.md, and run demo successfully.
  3. Pytorch Environment: Run conda install pytorch torchvision cudatoolkit=10.1 -c pytorch.
  4. Run pip install tqdm.
  5. Run pip install pandas.
  6. Run pip install tensorboardX.
  7. Run pip install fairseq. Possible Question "SyntaxError: invalid syntax", please see FAQ-Q1 below.
  8. Run pip install scipy.
  9. Run pip install matplotlib.

Dataset & Evaluation

  1. The proposed CapS dataset: you can download directly (Baidu Cloud (Passworde: 5okc) or Google Drive), including initial pseudo-labels, captions, tags, and etc. More details are approached in this link.
  2. RGB-D SOD benchmarks: you can download directly for realted RGBD SOD test sets.
  3. We use this toolbox for evaluating all SOD models.

Our JSM Implementation (Weakly Supervised)

  1. Modify the path of dataset in python demo_test.py and python demo_train.py.
  2. Inference stage: python demo_test.py; Using Pre-trained Model (Baidu Cloud (Passworde: vs85) or Google Drive).
  3. Training stage: CUDA_VISIBLE_DEVICES=0 python demo_train.py
  4. Check the log file: cat ./result.txt
  5. Load the training details: tensorboard --logdir=/YourComputer/JSM_model/runs/*

Saliency Results

Our weakly-supervied saliency results can be approached in Baidu Cloud (Passworde: m10a) or Google Drive. If you want to use our JSM to test on your own dataset, you can load our pretrained ckpt and run python demo_test.py directly.

Bibtex

@InProceedings{li2021joint,
    author    = {Li, Jingjing and Ji, Wei and Bi, Qi and Yan, Cheng and Zhang, Miao and Piao, Yongri and Lu, Huchuan and Cheng, Li},
    title     = {Joint Semantic Mining for Weakly Supervised RGB-D Salient Object Detection},
    booktitle = {Advances in Neural Information Processing Systems},
    month     = {December},
    year      = {2021}
}

Contact Us

If you have any questions, please contact us ( wji3@ualberta.ca ).

Question1: When installing fairseq， post an 'SyntaxError: invalid syntax'

Answer1: You can directly update python version, e.g., conda install python=3.7. More details can be found in this channel.

Question2: You should replace the inplace operation by an out-of-place one.

Answer2: This is because *= is not compatible with Python 3.9. q *= self.scaling -> q = q * self.scaling