Home

Awesome

DIT

Python PyTorch

<p align="center"> <img src="./DIT.png" width="550"> </p> This is the official implementation of "Deep Instruction Tuning for Segment Anything Model", which propose two simple yet effective deep instruction tuning (DIT) methods for text-guided SAM.

News

Installation

pip install -r requirements.txt
wget https://github.com/explosion/spacy-models/releases/download/en_vectors_web_lg-2.1.0/en_vectors_web_lg-2.1.0.tar.gz -O en_vectors_web_lg-2.1.0.tar.gz
pip install en_vectors_web_lg-2.1.0.tar.gz

Training and Evaluation

  1. Prepare your settings. To train a model, you should modify ./config/config.yaml to adjust the settings you want.
  2. Train the model. run train.py under the main folder to start training:
python train.py --config ./config/config.yaml
  1. Test the model. Then, you can run test.py by
python test.py --eval-weights ./logs/dit/1/weights/seg_best.pth
  1. Training log. Logs are stored in ./logs directory, which records the detailed training curve and accuracy per epoch. If you want to log the visualizations, please set LOG_IMAGE to True in config.yaml.

Model Weights

Following the steps of Data preparation and Training, you can reproduce and get better results in our paper. We provide the model weights for RefCOCO, RefCOCO+, RefCOCOg and GRES.

  1. RefCOCO Download link
valtest Atest B
76.277.8573.53
  1. RefCOCO+ Download link
valtest Atest B
65.9469.7858.89
  1. RefCOCOg Download link
valtest
67.468.07
  1. GRES Download link
valtest Atest B
63.7667.1961.85

Citation

@inproceedings{
	huang2024deep,
	title={Deep Instruction Tuning for Segment Anything Model},
	author={Xiaorui Huang and Gen Luo and Chaoyang Zhu and Bo Tong and Yiyi Zhou and Xiaoshuai Sun and Rongrong Ji},
	booktitle={ACM Multimedia 2024},
	year={2024}
}

Acknowledgement

Thanks a lot for the nicely organized code from the following repos