Awesome

Visual-Language Prompt Tuning with Knowledge-guided Context Optimization [CVPR 2023]

Visual-Language Prompt Tuning with Knowledge-guided Context Optimization<br> Hantao Yao, Rui Zhang, Changsheng Xu

Highlights

We introduce a novel Knowledge-guided Context Optimization (KgCoOp) to enhance the generalization ability of the learnable prompt for unseen classes. The key insight of KgCoOp is that the forgetting about essential knowledge can be alleviated by reducing the discrepancy between the learnable prompt and the hand-crafted prompt. Especially, KgCoOp minimizes the discrepancy between the textual embeddings generated by learned prompt and the hand-crafted prompts. Extensive evaluation shows that KgCoOp is an efficient method, i.e., achieves better performance with less training time.

Main Contributions

We found that the the degree of performance degradation on the New classes is consistent with the distance between the learnable textual embedding and the hand-crafted textual embedding, which is the core motivation of the KgCoOp.

Running code

1. Dassl.ProGrad.pytorch is the modified toolbox of Dassl.pytorch.
2. KgCoOp. To get the results in our paper, follow the README.md under KgCoOp/ to set the environment.

Citation

If you use our work, please consider citing:

@inproceedings{kgcoop23,
    title={Visual-Language Prompt Tuning with Knowledge-guided Context Optimization},
    author={Hantao Yao, Rui Zhang, Changsheng Xu},
    booktitle={The IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2023}
}

Acknowledgements

Our code is based on Co-CoOp and CoOp and ProGrad repository. We thank the authors for releasing their code. If you use our model and code, please consider citing these works as well.