Awesome
HGCLIP
👀Introduction
This repository contains the code for our paper HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understanding
. [arXiv]
Created by Peng Xia, Xingtong Yu, Ming Hu, Lie Ju, Zhiyong Wang, Peibo Duan, Zongyuan Ge.
💡Requirements
Environment
- Python 3.8.*
- CUDA 12.2
- PyTorch
- TorchVision
Install
Create a virtual environment and activate it.
conda create -n hgclip python=3.8
conda activate hgclip
The code has been tested with PyTorch 1.13 and CUDA 12.2.
pip install -r requirements.txt
⏳Dataset
Please first download the required datasets. Follow prepare_datasets.md to install the datasets.
📦Usage
Training & Evaluation
To train or evaluation our HGCLIP, you need to first generate and save the prototypes.
python generate_prototypes.py \
--dataset 'air' \
--batch_size 64 \
--gpu_id 0
Then run
cd hgclip
python main.py \
--config configs/air/train_gnn.py
Zero-Shot Evaluation
To evaluation the performance of zero-shot CLIP, run
cd zsclip
python zero_shot.py \
--config configs/air/zero_shot_clip.py
Quick Start
The main script for training and evaluating model performance is hgclip/main.py
. Here are the list of key arguments:
-
--config
: In this configuration file, you can define all the arguments included below. -
--device
:cuda
orcpu
-
--gpu_id
: gpu id -
--dataset
: dataset name -
--batch_size
-
--epochs
-
--trainer
: training method-
CoOp
-
CoCoOp
-
MaPLe
-
VPT
-
PromptSRC
-
KgCoOp
-
HGCLIP
-
-
--ctx_init
: context initialization -
--n_ctx
: context length for random initialization -
--lr
: learning rate -
--optimizer
: optimizer-
sgd
-
adam
-
adamw
-
🙏Acknowledgements
We use code from MaPLe, CoCoOp-CoOp and CLIP. We thank the authors for releasing their code.
📧Contact
If you have any questions, please create an issue on this repository or contact at richard.peng.xia@gmail.com.
📝Citing
If you find this code useful, please consider to cite our work.
@article{xia2023hgclip,
title={HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understanding},
author={Xia, Peng and Yu, Xingtong and Hu, Ming and Ju, Lie and Wang, Zhiyong and Duan, Peibo and Ge, Zongyuan},
journal={arXiv preprint arXiv:2311.14064},
year={2023}
}