Home

Awesome

TagAlign - Official Pytorch Implementation

TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification <br> Qinying Liu, Kecheng Zheng, Wei Wu, Zhan Tong, Yu Liu, Wei Chen, Zilei Wang, Yujun Shen<br>

Arxiv Dataset

PWC PWC PWC PWC PWC PWC PWC PWC

<div align="center"> <img src="figs/pipeline.png" width="100%"> </div>

📜 News

[2024/12/11] Release tag.

[2023/12/25] The paper and project page are released!

💡 Highlights

👨‍💻 Todo

🛠️ Usage

Installation

Data Preparation

For the training phase, we utilize the CC12M dataset. Researchers can procure the CC12M dataset either directly from its source or by employing the img2dataset tool.

For evaluation, refer to the GroupVit to properly prepare the datasets. Make sure to update the image directories in 'segmentation/configs/base/datasets/*.py' as necessary.

Train and Evaluate

  1. Modify the 'tagalign.yml'. We provide the processed tag_file (object_list.csv) and label_file (CC12M_url_object_index.json). We provide the URLs of the images; please change them to your local paths.

  2. Train the TagAlign model by run

    torchrun --rdzv_endpoint=localhost:6000 --nproc_per_node=auto main.py --cfg configs/tagalign.yml
    
  3. You can evaluate the TagAlign model by running the command below.

    torchrun --rdzv_endpoint=localhost:6000 --nproc_per_node=auto main.py --cfg configs/eval.yml --eval --resume $WEIGHT
    

    $WEIGHT is the path of the pre-trained checkpoints. We provide our pre-trained weights in weights(TODO).

✒️ Citation

If you find our work to be useful for your research, please consider citing.

@article{liu2023tagalign,
  title={TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification},
  author={Liu, Qinying and Zheng, Kecheng and Wei, Wu and Tong, Zhan and Liu, Yu and Chen, Wei and Wang, Zilei and Shen, Yujun},
  journal={arXiv preprint arXiv:2312.14149},
  year={2023}
}

❤️ Acknowledgements