Home

Awesome

Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models (AAAI2024)

Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models<br> Yubin Wang, Xinyang Jiang, De Cheng, Dongsheng Li, Cairong Zhao

PWC PWC PWC PWC PWC

Official implementation of the paper "Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models".

<hr />

๐Ÿ“ข News

โœจ Highlights

main figure

<p align="justify"> Abstract: Prompt learning has become a prevalent strategy for adapting vision-language foundation models to downstream tasks. As large language models (LLMs) have emerged, recent studies have explored the use of category-related descriptions as input to enhance prompt effectiveness. Nevertheless, conventional descriptions fall short of structured information that effectively represents the interconnections among entities or attributes linked to a particular category. To address this limitation and prioritize harnessing structured knowledge, this paper advocates for leveraging LLMs to build a graph for each description to model the entities and attributes describing the category, as well as their correlations. Preexisting prompt tuning methods exhibit inadequacies in managing this structured knowledge. Consequently, we propose a novel approach called Hierarchical Prompt Tuning (HPT), which enables simultaneous modeling of both structured and conventional linguistic knowledge. Specifically, we introduce a relationship-guided attention module to capture pair-wise associations among entities and attributes for low-level prompt learning. In addition, by incorporating high-level and globallevel prompts modeling overall semantics, the proposed hierarchical structure forges cross-level interlinks and empowers the model to handle more complex and long-term relationships. Extensive experiments demonstrate that our HPT shows strong effectiveness and generalizes much better than existing SOTA methods. </p>

:rocket: Contributions

๐Ÿ“Š Results

Base-to-New Generalization

Results reported below show average accuracy for base and new classes across 11 recognition datasets averaged over 3 seeds. Please refer to our paper for more numerical results

NameBase AccuracyNew AccuracyHarmonic Mean
CLIP69.3474.2271.70
CoOp82.6963.2271.66
CoCoOp80.4771.6975.83
MaPLe82.2875.1478.55
HPT84.32 (+2.04)76.86 (+1.72)80.23 (+1.68)

Cross-Dataset Evaluation

Results reported below show accuracy for the source dataset ImageNet and 4 ImageNet-variant datasets averaged over 3 seeds.

ImNetCaltechPetsCarsFlowersFoodAircraftSUN397DTDEuroSATUCFAverage
CLIP71.5193.7089.1464.5168.7185.3018.4764.1541.9246.3966.5563.88
CoCoOp71.0294.4390.1465.3271.8886.0622.9467.3645.7345.3768.2165.74
MaPLe70.7293.5390.4965.5772.2386.2024.7467.0146.4948.0668.6966.30
HPT71.7294.2092.6366.3374.8486.2125.6868.7550.8747.3670.5067.74

Domain Generalization

Results reported below show accuracy for the source dataset ImageNet and the other 10 target datasets averaged over 3 seeds.

ImageNetImageNetV2ImageNet-SImageNet-AImageNet-RAverage
CLIP66.7360.8346.1547.7773.9657.17
CoOp71.5164.2047.9949.7175.2159.28
CoCoOp71.0264.0748.7550.6376.1859.90
MaPLe70.7264.0749.1550.9076.9860.26
HPT71.7265.2549.3650.8577.3860.71

๐Ÿ› ๏ธ Installation

For installation and other package requirements, please follow the instructions detailed in INSTALL.md.

๐Ÿ“ฆ Model Zoo

Download checkpoints here and put them at ./results/B2N to reproduce the benchmark results.

๐Ÿ—‚๏ธ Data preparation

Please follow the instructions at DATASETS.md to prepare all datasets.

๐Ÿงช Training and Evaluation

Please refer to RUN.md for detailed instructions on training and evaluating.

๐Ÿ” Citation

If you use our work, please consider citing:

@misc{wang2023learning,
      title={Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models}, 
      author={Yubin Wang and Xinyang Jiang and De Cheng and Dongsheng Li and Cairong Zhao},
      year={2023},
      eprint={2312.06323},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

๐Ÿ“ง Contact

If you have any questions, please create an issue on this repository or contact us at wangyubin2018@tongji.edu.cn or zhaocairong@tongji.edu.cn.

๐Ÿ˜ƒ Acknowledgments

Our code is based on CoCoOp and CoOp repository. We thank the authors for releasing their code. If you use our model and code, please consider citing these works as well.