Home

Awesome

Hierarchy-CLIP

[CVPR 2023] Improving Zero-shot Generalization and Robustness of Multi-modal Models

Improving Zero-shot Generalization and Robustness of Multi-modal Models <br> Yunhao Ge*, Jie Ren*, Andrew Gallagher, Yuxiao Wang, Ming-Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Lakshminarayanan, Jiaping Zhao ( * =equal contribution) <br> IEEE/ CVF International Conference on Computer Vision and Pattern Recognition (CVPR), 2023

Project Page | Video | Paper

<div align="center"> <img src="./docs/Fig-1.png" alt="Editor" width="1500"> </div>

Figure: Our zero-shot classification pipeline consists of 2 steps: confidence estimation via self-consistency (left block) and top-down and bottom-up label augmentation using the WordNet hierarchy (right block).

<div align="center"> <img src="./docs/Fig-2.png" alt="Editor" width="1500"> </div>

Figure: Typical failure modes in the cases where top-5 prediction was correct but top-1 was wrong.

Getting Started

Installation

git clone https://github.com/gyhandy/Hierarchy-CLIP.git
cd Hierarchy-CLIP
git clone https://github.com/google-research/scenic.git
cd scenic
pip install .

Load dataset:

Download WordNet hierarchy information to build top-down and bottom-up prompt augmentation:

Code

We provide a colab code, all details are in the following:

Hierarcy_Clip.ipynb

Contact / Cite

Got Questions? We would love to answer them! Please reach out by email! You may cite us in your research as:

@inproceedings{ge2023improving,
  title={Improving Zero-shot Generalization and Robustness of Multi-modal Models},
  author={Ge, Yunhao and Ren, Jie and Gallagher, Andrew and Wang, Yuxiao and Yang, Ming-Hsuan and Adam, Hartwig and Itti, Laurent and Lakshminarayanan, Balaji and Zhao, Jiaping},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={11093--11101},
  year={2023}
}