Awesome

Implementation of "Data-efficient Large Vision Models through Sequential Autoregression" [ICML 2024].

<p align="center"> <img src="figs/DeLVM.PNG" > </p> <p align="center"> </p> Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended to not only comprehend but also seamlessly transit to out-of-domain tasks. However, current endeavors are hamstrung by an over-reliance on colossal models, exemplified by models with upwards of 3B parameters, and the necessity for an extensive corpus of visual data, often comprising a staggering 400B tokens. In this paper, we delve into the development of an efficient, autoregression-based vision model, innovatively architected to operate on a limited dataset. We meticulously demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding during the testing phase. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint, and a marked decrease in training data requirements, thereby paving the way for more sustainable and accessible advancements in the field of generalist vision models.

TODO List

Code about training models.
Code about inferencing models.
Huggingface & InternLM ckpts.
Code about data generation.

Set up

based on InternLM-v0.2.1dev20231121

Install: https://github.com/InternLM/InternLM/blob/v0.2.1dev20231121/doc/en/install.md

Put your training data to /path/to/data/vision.

Training command: torchrun --nproc_per_node 8 train.py --config ./configs/pretrain_300m.py --launcher torch

Training via KD command: torchrun --nproc_per_node 8 train.py --config ./configs/kd_1b_to_300m.py --launcher torch

Convert model and inference example: ./tools

The corresponding huggingface ckpt can be downloaded at LLaMA-1b-hf Onedrive / LLaMA-1b-hf Baidu Disk and LLaMA-300m-hf.

Data generation

Please refer to data_generation/README.md.

Citation

If you find this project useful in your research, please consider cite:

@article{guo2024dataefficient,
  title={Data-efficient Large Vision Models through Sequential Autoregression},
  author={Guo, Jianyuan and Hao, Zhiwei and Wang, Chengcheng and Tang, Yehui and Wu, Han and Hu, Han and Han, Kai and Xu, Chang},
  journal={arXiv preprint arXiv:2402.04841},
  year={2024}
}

Acknowledgement

We maily follow the directon of project LVM. And this repo is based on InternLM, huggingface.co/transformers, and huggingface.co/openMUSE.