Awesome

🔍 FIND: Interfacing Foundation Models' Embeddings

:grapes: [Read our arXiv Paper]   :apple: [Try our Demo]   :orange: [Walk through Project Page]

We introduce FIND that can INterfacing Foundation models' embeDDings in an interleaved shared embedding space. Below is a brief introduction to the generic and interleave tasks we can do!

by Xueyan Zou, Linjie Li, Jianfeng Wang, Jianwei Yang, Mingyu Ding, Junyi Wei, Zhengyuan Yang, Feng Li, Hao Zhang, Shilong Liu, Arul Aravinthan, Yong Jae Lee*, Lijuan Wang*,

** Equal Advising **

:rocket: Updates

• [2024.8.20] We have released an updated version arXiv with comprehensive user guide on GitHub!
• [2023.12.3] We have a poster session @ NeurIPS24 for SEEM, feel free to visit us during 5:00-7:00pm (CT)!
• [2023.12.2] We have released all the training, evaluation, and demo code!

:bookmark_tabs: Catalog

• Demo Code
• Model Checkpoint
• Comprehensive User Guide
• Dataset
• Training Code
• Evaluation Code

:hammer: Getting Started

Build Environment

conda create --name find python=3.10
conda activate find
conda install -c conda-forge mpi4py
conda install -c conda-forge cudatoolkit=11.7
conda install -c nvidia/label/cuda-11.7.0 cuda-toolkit
pip install torch==2.0.1+cu117 torchvision==0.15.2+cu117 torchaudio==2.0.2+cu117 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r assets/requirements/requirements.txt
pip install -r assets/requirements/requirements_custom.txt
cd modeling/vision/encoder/ops
sh make.sh
cd ../../..

Build Dataset

Explore through 🤗 Hugging Face: FIND-Bench.

Download Raw File:

• To run the demo, files/folders within * * are required, please download COCO dataset and FIND-Bench annotations entity_val2017.json and entity_val2017_long.json.

• To run evaluation, please additionally prepare panoptic_val2017 according to Mask2Former.

• To run training, please additionally download and prepare all other files.

Run Demo

Run Evaluation

Run Training

⛳ Interleave Checkpoint

:framed_picture: FIND-Bench Visualization

🔗 Citation

If you find this repo useful for your research and applications, please cite using this BibTeX:

@misc{zou2022xdecoder,
      title={Generalized decoding for pixel, image, and language}, 
      author={Zou*, Xueyan and Dou*, Zi-Yi and Yang*, Jianwei and Gan, Zhe and Li, Linjie and Li, Chunyuan and Dai, Xiyang and Behl, Harkirat and Wang, Jianfeng and Yuan, Lu and Peng, Nanyun and Wang, Lijuan and Lee†, Yong Jae and Gao†, Jianfeng},
      publisher={CVPR},
      year={2023},
}

@misc{zou2023seem,
      title={Segment everything everywhere all at once}, 
      author={Zou*, Xueyan and Yang*, Jianwei and Zhang*, Hao and Li*, Feng and Li, Linjie and Wang, Jianfeng and Wang, Lijuan and Gao†, Jianfeng and Lee†, Yong Jae},
      publisher={NeurIPS},
      year={2023},
}

@misc{zou2024find,
      title={Interfacing Foundation Models' Embeddings}, 
      author={Zou, Xueyan and Li, Linjie and Wang, Jianfeng and Yang, Jianwei and Ding, Mingyu and Yang, Zhengyuan and Li, Feng and Zhang, Hao and Liu, Shilong and Aravinthan, Arul and Lee†, Yong Jae and Wang†, Lijuan},
      publisher={arXiv preprint arXiv:2312.07532},
      year={2024},
}

📚 Acknowledgement

This research project has benefitted from the Microsoft Accelerate Foundation Models Research (AFMR) grant program.