Awesome
<div align="center"> <h1>Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation</h1>Muzhi Zhu<sup>1*</sup>, Yang Liu<sup>1*</sup>, Zekai Luo<sup>1*</sup>, Chenchen Jing<sup>1</sup>, Hao Chen<sup>1</sup>, Guangkai Xu<sup>1</sup>, Xinlong Wang<sup>2</sup>, Chunhua Shen<sup>1</sup>
<sup>1</sup>Zhejiang University, <sup>2</sup>Beijing Academy of Artificial Intelligence
NeurIPS 2024
</div>🚀 Overview
<div align="center"> <img width="800" alt="image" src="figs/method.png"> </div>📖 Description
We systematically study four crucial elements of applying the Diffusion Model to Few-shot Semantic Segmentation. For each of these aspects, we propose several reasonable solutions and validate them through comprehensive experiments.
Building upon our observations, we establish the DiffewS framework, which maximally retains the generative framework and effectively utilizes the pre-training prior. Notably, we introduce the first diffusion-based model dedicated to Few-shot Semantic Segmentation, setting the groundwork for a diffusion-based generalist segmentation model.
🚩 Plan
<!-- - [ ] Release the weights. -->- Release the weights.
- Release the inference code.
- Release the training code.
👻 Getting Started
Installation
Preparing the environment following GenPercept.
conda create -n diffews python=3.10
conda activate diffews
pip install -r requirements.txt
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
Dataset
Preparing the dataset following Matcher You only need to download the COCO 2014 dataset.
Training
Evaluation
Download the pre-trained model weights from here.
CUDA_VISIBLE_DEVICES=0 bash scripts/eval_coco2014_rthres_1shot_nosample.sh weight/coco_fold0
CUDA_VISIBLE_DEVICES=0 bash scripts/eval_coco2014_rthres_5shot_nosample.sh weight/coco_fold0
CUDA_VISIBLE_DEVICES=0 bash scripts/eval_coco2014_rthres_1shot_nosample_fold0.sh weight/incontext
🎫 License
For academic use, this project is licensed under the 2-clause BSD License. For commercial use, please contact Chunhua Shen.
🖊️ Citation
If you find this project useful in your research, please consider to cite:
@article{zhu2024unleashing,
title={Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation},
author={Zhu, Muzhi and Liu, Yang and Luo, Zekai and Jing, Chenchen and Chen, Hao and Xu, Guangkai and Wang, Xinlong and Shen, Chunhua},
journal={arXiv preprint arXiv:2410.02369},
year={2024}
}