Home

Awesome

<!-- ``` #创建conda环境 conda create --name deop python=3.7 conda activate deop conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 -c pytorch conda install -c conda-forge yacs conda install tensorboard conda install pydot conda install iopath tabulate iopath<0.1.9 and >=0.1.7 fvcore<0.1.4 and >=0.1.3 timm wandb python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.8/index.html ``` -->

[ICCV2023] Open Vocabulary Semantic Segmentation with Decoupled One-Pass Network

This repo is for our paper Open Vocabulary Semantic Segmentation with Decoupled One-Pass Network. It is based on the official repo of SimBaseline. The project page is Project Page.

<!-- <iframe src="resources/framework6.pdf" width="500" height="375"></iframe> -->

Results

Results on COCO-Stuff and Pascal VOC in the open-vocabulary setting. <a name="resdataset"></a> Results on Pascal VOC, Pascal Context and ADE20K in the cross-dataset setting

Guideline

Setup environment

<details> <summary> Environment </summary>
conda create --name deop python=3.7
conda activate deop
conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 -c pytorch
pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.8/index.html
pip install mmcv==1.3.14

#FurtherMore, install the modified clip package.
cd third_party/CLIP
python -m pip install -Ue .

</details>

Data Preparation

<details> <summary> Data Preparation</summary>

In our experiments, four datasets are used. For Cityscapes and ADE20k, follow the tutorial in MaskFormer.

</details>

Run demo

The demo is almost training-free, we only train the learnable text prompt. You can download the weights from text prompt which is trained on COCO-Stuff-156 dataset.

python3 demo.py --input dataset/test/000000000285.jpg --output ./output

or you can add --class-names to set classes.

python3 demo.py --input dataset/test/000000000285.jpg --output ./output --class-names bear other

Evaluation

We release the weights of DeOP in model weight. The results are shown on Results.

sh deop_verifycoco.sh

Train

# 1.Train a learnable text prompt model.
python train_net.py --config-file configs/coco-stuff-164k-156/zero_shot_proposal_classification_learn_prompt_bs32_10k.yaml --num-gpus 8 

# 2. Train a MaskFormer model.
python train_net.py --config-file configs/coco-stuff-164k-156/zero_shot_maskformer_R101c_bs32_60k.yaml --num-gpus 8 MODEL.CLIP_ADAPTER.PROMPT_CHECKPOINT ${TRAINED_PROMPTS}
# 3. Train DeOP.
sh deop_train.sh

Cite

If you find this project useful for your research, please consider citing the following BibTeX entry.

@proceedings{Han2023ZeroShotSS,
  title={Zero-Shot Semantic Segmentation with Decoupled One-Pass Network},
  author={Cong Han and Yujie Zhong and Dengjie Li and Kai Han and Lin Ma},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2023},
}