Home

Awesome

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

dataset

dataset

Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, Hongsheng Li

This repository contains code for the paper Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning and it was built based on LLaVA

We collect and introduce the Visual CoT dataset comprising 438k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions. The work also proposes a multi-turn processing pipeline for the multi-modal language model that dynamically focuses on visual inputs and provides interpretable thoughts. Importantly, the released benchmark is capable of evaluating MLLMs in scenarios requiring specific local region identification.

Project Page, Dataset, arXiv

News

Contents

Install

  1. Clone this repository and navigate to Visual-CoT folder
git clone https://github.com/deepcs233/Visual-CoT.git
cd Visual-CoT
  1. Install Package
conda create -n viscot python=3.10 -y
conda activate viscot
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
  1. Install additional packages for training cases
pip install -e ".[train]"
pip install flash-attn --no-build-isolation

Demo

Please refer to https://github.com/haotian-liu/LLaVA/tree/main?tab=readme-ov-file#demo

Model Zoo

pipeline

We propose a novel multi-turn processing pipeline for MLLMs that can dynamically focus on visual inputs and provide intermediate interpretable thoughts.

The model weights below are merged weights. You do not need to apply delta. The usage of VisCoT checkpoints should comply with the base LLM's model license.

VersionSizeResolutionCheckpoint
VisCoT7B224deepcs233/VisCoT-7b-224
VisCoT7B336deepcs233/VisCoT-7b-336
VisCoT13B224deepcs233/VisCoT-13b-224
VisCoT13B336deepcs233/VisCoT-13b-336

Train

Our training steps are largely consistent with LLaVA; for further details, please refer to the LLaVA documentation/issues.

VisCoT training consists of two stages: (1) feature alignment stage: use our 558K subset of the LAION-CC-SBU dataset to connect a frozen pretrained vision encoder to a frozen LLM; (2) visual instruction tuning stage: use 665K dataset including multimodal instruction-following data and academic VQA tasks from LLaVA-1.5, 1.4M dataset with positional annotations from Shikra, and 373K visual CoT dataset from ours, to teach the model to follow multimodal instructions and obtain the visual CoT ability.

VisCoT is trained on 8 A100 GPUs with 80GB memory. To train on fewer GPUs, you can reduce the per_device_train_batch_size and increase the gradient_accumulation_steps accordingly. Always keep the global batch size the same: per_device_train_batch_size x gradient_accumulation_steps x num_gpus.

Hyperparameters

We use a similar set of hyperparameters as Vicuna in finetuning. Both hyperparameters used in pretraining and finetuning are provided below.

  1. Pretraining
HyperparameterGlobal Batch SizeLearning rateEpochsMax lengthWeight decay
LLaVA-v1.5-13B2561e-3120480
  1. Finetuning
HyperparameterGlobal Batch SizeLearning rateEpochsMax lengthWeight decay
LLaVA-v1.5-13B1282e-5120480

Download Vicuna checkpoints (automatically)

Our base model Vicuna v1.5, which is an instruction-tuned chatbot, will be downloaded automatically when you run our provided training scripts. No action is needed.

Pretrain (feature alignment)

In our work, we directly use the project's weight from LLaVA-1.5. If you do not need to train it by yourself, projector weights can be downloaded here: https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md#projector-weights

Please download the 558K subset of the LAION-CC-SBU dataset with BLIP captions used in LLaVA here.

Pretrain takes around 5.5 hours for VisCoT-13B on 8x A100 (80G), due to the increased resolution to 336px. It takes around 3.5 hours for VisCoT-7B.

Training script with DeepSpeed ZeRO-2: pretrain.sh.

Visual Instruction Tuning

  1. Prepare data

Please download the annotation of our mixed instruction tuning data viscot_mixed_2m.json to ./playground/data. We provide our 363k visual CoT dataset viscot_363k.json for building your own dataset. Additionally, the ./viscot_dataset directory contains metadata for the visual CoT dataset, which includes detailed information requiring further postprocessing before it can be used for training. The ./cot_with_detailed_reasoning_steps directory contains 98k data pairs, each accompanied by detailed reasoning steps. The train/validation split is maintained consistent with the original dataset. Necessary scripts are available in the ./tools directory. For instance, the script ./tools/convert_data_to_llava_format.py can convert the meta JSONL file into the required format for training. Please download the images for constituting datasets, and some of them may need to register/complete the form first.

We also prepare the image file in this link, you need to merge these split archive files and then extract them.

After downloading all of them, organize the data as follows in ./playground/data,

ā”œā”€ā”€ coco
ā”‚   ā””ā”€ā”€ train2017
ā”‚   ā””ā”€ā”€ train2014
ā”œā”€ā”€ gqa
ā”‚   ā””ā”€ā”€ images
ā”œā”€ā”€ ocr_vqa
ā”‚   ā””ā”€ā”€ images
ā”œā”€ā”€ textvqa
ā”‚   ā””ā”€ā”€ train_images
ā””ā”€ā”€ vg
ā”‚   ā”œā”€ā”€ VG_100K
ā”‚   ā””ā”€ā”€ VG_100K_2
ā””ā”€ā”€ v7w
ā”‚   ā””ā”€ā”€ images
ā””ā”€ā”€ flickr30k
ā”‚   ā””ā”€ā”€ images
ā””ā”€ā”€ cot
ā”‚   ā””ā”€ā”€ flickr30k
ā”‚   ā””ā”€ā”€ docvqa
ā”‚   ā””ā”€ā”€ gqa
ā”‚   ā””ā”€ā”€ infographicsvqa
ā”‚   ā””ā”€ā”€ openimages
ā”‚   ā””ā”€ā”€ textvqa
ā”‚   ā””ā”€ā”€ vsr
ā”‚   ā””ā”€ā”€ dude
ā”‚   ā””ā”€ā”€ sroie
ā”‚   ā””ā”€ā”€ cub
  1. Start training!

We have prepared LLaVA's pretrained projectors in our repo (checkpoints/llava_7b_mm_projector.bin and checkpoints/llava_13b_mm_projector.bin). It is not recommended to use legacy projectors, as they may be trained with a different version of the codebase, and if any option is off, the model will not function/train as we expected.

Visual instruction tuning takes around 60 hours for VisCoT-7b-224 on 8x A100 (80G).

Training script with DeepSpeed ZeRO-3: finetune.sh.

If you are interested in finetuning LLaVA model to your own task/data, please check out Finetune_Custom_Data.md怂

Some options to note:

Evaluation

Visual CoT Benchmark

  1. Single-GPU inference, VisCoT-7b-336 can be changed to other model names saved in the ./checkpoints/
bash scripts/v1_5/eval/cot_benchmark.sh VisCoT-7b-336
  1. Obtain the score using ChatGPT-3.5, the API KEY need to be set in llava/eval/eval_cot_score.py
bash scripts/v1_5/eval/cot_score.sh VisCoT-7b-336
  1. Stat the overall score
python tools/cot_get_result.py VisCoT-7b-336
  1. Stat the detection accuracy of visual CoT bounding boxes (optional)
python tools/cot_detection_get_result.py VisCoT-7b-336

RefCoCo

  1. Single-GPU inference, VisCoT-7b-336 can be changed to other model names saved in the ./checkpoints/
bash scripts/v1_5/eval/refcoco.sh VisCoT-7b-336
  1. Stat the overall accuracy
python tools/refcoco_get_result.py VisCoT-7b-336

General Benchmarks

Please refer to LLaVA's scripts.

Visualization

demo1 demo2 demo4

Acknowledgements

This implementation is based on code from several repositories.

Citation

If you find our repo, dataset or paper useful, please cite us as

@misc{shao2024visual,
      title={Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models}, 
      author={Hao Shao and Shengju Qian and Han Xiao and Guanglu Song and Zhuofan Zong and Letian Wang and Yu Liu and Hongsheng Li},
      year={2024},
      eprint={2403.16999},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Disclaimer

This dataset was collected and released solely for research purposes, with the goal of making the MLLMs dynamically focus on visual inputs and provide intermediate interpretable thoughts. The authors are strongly against any potential harmful use of the data or technology to any party.

Intended Use

The data, code, and model checkpoints are intended to be used solely for (I) future research on visual-language processing and (II) reproducibility of the experimental results reported in the reference paper. The data, code, and model checkpoints are not intended to be used in clinical care or for any clinical decision making purposes.

Primary Intended Use

The primary intended use is to support AI researchers reproducing and building on top of this work. \shortname{} and its associated models should be helpful for exploring various vision question answering (VQA) research questions.

Out-of-Scope Use

Any deployed use case of the model --- commercial or otherwise --- is out of scope. Although we evaluated the models using a broad set of publicly-available research benchmarks, the models and evaluations are intended for research use only and not intended for deployed use cases.

License

All code within this repository is under Apache License 2.0.