Awesome
HallE-Control: Controlling Object Hallucination in Large Mutimodal Models
[Paper] [Project Page] <br> Bohan Zhai*, Shijia Yang*, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, Manling Li
Release
- [3/28] We release the full training and eval code. Welcome any question!
- [3/27] 🔥 We released the new version HallE-Control: Controlling Object Hallucination in LMMs. Checkout the paper.
- [12/13] We add CCEval's code for evaluation object existence hallucination.
- [12/3] 🔥 We released HallE-Switch: Controlling Object Hallucination in LVLMs. Checkout the paper.
Contents
Install
- Clone this repository and navigate to HallE_Control folder
git clone https://github.com/bronyayang/HallE_Control.git
cd HallE_Control
- Install Package
conda create -n halle python=3.10 -y
conda activate halle
bash scripts/run.sh
Training
- Prepare data
Follow LLaVA to prepare data. Download controller data here and put in ./data folder.
- Start training
- Train controller: Model can output less hallucinated/more imagination captions based on $\epsilon$
bash scripts/v1_5/tune_controller.sh
Make sure the output_dir contains the word "controller" for correct inference behavior.
- Train indication: Model can output caption with [object] indication on imagined objects
bash scripts/v1_5/finetune_indication.sh
Evaluation
Here, we provide the procedure of evaluating any model on CCEval.
-
Download VisualGenome images part 1, part 2, and objects; unzip to ./data folder
-
Obtain captions of your model for 100 VG images. For example, you can obtain captions of controller model by using the following script:
bash scripts/v1_5/model_control_eval.sh
- Get CCEval results (without coverage) by running:
python3 cceval.py --cap_file [YOUR_CAPTION_FILE_PATH] --key [YOUR_OPENAI_API_KEY]
Citation
If you find HallE-Control useful for your research and applications, please cite using this BibTeX:
@misc{zhai2023halleswitch,
title={HallE-Switch: Controlling Object Hallucination in Large Vision Language Models},
author={Bohan Zhai and Shijia Yang and Chenfeng Xu and Sheng Shen and Kurt Keutzer and Manling Li},
year={2023},
eprint={2310.01779},
archivePrefix={arXiv},
primaryClass={cs.CV}
}