Awesome
HaELM
An automatic MLLM hallucination detection framework
1. Installing
Install peft
$ pip install git+https://gitclone.com/github.com/huggingface/peft.git -i https://pypi.mirrors.ustc.edu.cn/simple --trusted-host=pypi.mirrors.ustc.edu.cn
2. Preparing
Download the checkpoint of llama-7b-hf
3. Training
We provide the hallucination training dataset in "data/train_data.jsonl" and the manually labeled validation set in "data/eval_data.jsonl". If you want to:
- Retrain
- Use another scale of llama
- Use llama-2
- Use additional data
see here.
- Modify the path in lines 19-21 of finetune.py
- Run the command below
python finetune.py
4. Interface
We provide interface templates populated by the output of mPLUG-Owl in "LLM_output/mPLUG_caption.jsonl".
- Modify the path in lines 14-16 of interface.py
- Run the command below
python interface.py
5. Citation
@article{wang2023evaluation,
title={Evaluation and Analysis of Hallucination in Large Vision-Language Models},
author={Wang, Junyang and Zhou, Yiyang and Xu, Guohai and Shi, Pengcheng and Zhao, Chenlin and Xu, Haiyang and Ye, Qinghao and Yan, Ming and Zhang, Ji and Zhu, Jihua and others},
journal={arXiv preprint arXiv:2308.15126},
year={2023}
}