Awesome
LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models
Yuzhang Shang*, Mu Cai*, Bingxin Xu, Yong Jae Lee^, Yan Yan^
*Equal Contribution, ^Equal Advising
[Paper] [Project Page]
<div align="center"> <img src="https://llava-prumerge.github.io/images/architecture.png" alt="Our approach" width="50%"> </div>How to run
Step.0: Set the environment the same as LLaVA-1.5
Note that the core of our proposed module is here in the CLIP image encoder.
Step.1 (for inference): Download Checkpoints
Download the checkpoints (LoRA Version) from Yuzhang's Huggingface Homepage to checkpoints/llava-v1.5-7b-lora-prunemerge.
Step.2 (for inference): Change the methods (PruMerge or PruMerge+).
Change the call function of token reduction from here in the CLIP image encoder.
Step.3 (for inference): Run the script.
For example, the evaluation for TextVQA is:
CUDA_VISIBLE_DEVICES=7 XDG_CACHE_HOME='/data/shangyuzhang/' bash scripts/v1_5/eval/testvqa.sh
For other inference scripts, refer to LLaVA Evaluation.