Home

Awesome

PLOT: Prompt Learning with Optimal Transport for Vision-Language Models

Created by Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, Kun Zhang

This repository contains PyTorch implementation for ICLR2023 spotlight paper PLOT:Prompt Learning with Optimal Transport for Vision-Language Models [arXiv]

PLOT is a method to jointly learn multiple comprehensive prompts to describe diverse characteristics of categories such as intrinsic attributes or extrinsic contexts. To solve the degradation problem of multiple prompts, we introduce optimal transport to match the multiple local patterns of vision and text modalities. Specifically, we first model images and the categories with visual and textual feature sets. Then, we apply a two-stage optimization strategy to learn the prompts. In the inner loop, we optimize the optimal transport distance to align visual features and prompts by the Sinkhorn algorithm, while in the outer loop, we learn the prompts by this distance from the supervised data.

intro

Updates

PLOT for Prompt Learning

See PLOT-CoOp.

PLOT for Adapter-based Methods

See PLOT-Adapter.

PLOT for Multi-modal Prompt Learning

See PLOT-PP.

License

MIT License

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{chen2023plot,
  title={Prompt Learning with Optimal Transport for Vision-Language Models},
  author={Chen, Guangyi and Yao, Weiran and Song, Xiangchen and Li, Xinyue and Rao, Yongming and Zhang, Kun}
  booktitle={ICLR},
  year={2023}
}