Home

Awesome

LSCM-Refseg

Code of our ECCV 2020 paper Linguistic Structure Guided Context Modeling for Referring Image Segmentation.

Tianrui Hui, Si Liu, Shaofei Huang, Guanbin Li, Sansi Yu, Faxi Zhang, and Jizhong Han.

Interpretation of LSCM

Interpretation

We construct a Dependency Parsing Tree suppressed Word Graph (DPT-WG) to model multimodal context in three steps.

Experimental Results

We modify the way of feature concatenation in the end of LSCM module and achieve higher performances than the results reported in our paper on most splits. New experimental results are summarized in the table bellow. You can download our trained checkpoints to test on the four datasets. The link to the checkpoints is: Baidu Drive, pswd: kbhb.

MethodUNC valUNC testAUNC testBUNC+ valUNC+ testAUNC+ testBG-Ref valReferIt test
STEP-ICCV19 [1]60.0463.4657.9748.1952.3340.4146.4064.13
Ours-ECCV2061.4764.9959.5549.3453.1243.5048.0566.57
Ours-Updated61.6564.4259.9250.2153.0443.8349.5067.23

Setup

We recommended the following dependencies.

This code is derived from RRN [2]. Please refer to it for more details of setup.

Data Preparation

We conduct experiments on 4 datasets of referring image segmentation, including UNC, UNC+, Gref and ReferIt. After downloading these datasets, you can run the following commands for data preparation:

python build_batches.py -d Gref -t train
python build_batches.py -d Gref -t val
python build_batches.py -d unc -t train
python build_batches.py -d unc -t val
python build_batches.py -d unc -t testA
python build_batches.py -d unc -t testB
python build_batches.py -d unc+ -t train
python build_batches.py -d unc+ -t val
python build_batches.py -d unc+ -t testA
python build_batches.py -d unc+ -t testB
python build_batches.py -d referit -t trainval
python build_batches.py -d referit -t test

Please first download GloVe embedding (glove.840B.300d.zip) and save it to data/. Then run the following commands to generate embeddings for the above four datasets (UNC, UNC+ and G-Ref use the same GloVe embedding):

 python scripts/embedding_generate.py -d referit
 python scripts/embedding_generate.py -d Gref

Training and Evaluation

For convenience, we further provide an example bash script to train on UNC train set and evaluate on val set by running the following command:

./trainval.sh

Reference

[1] Chen, Ding-Jie, et al. "See-through-text grouping for referring image segmentation." Proceedings of the IEEE International Conference on Computer Vision. 2019.

[2] Li, Ruiyu, et al. "Referring image segmentation via recurrent refinement networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.