Awesome

R-Bench

This repo is for the paper Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models (ICML2024).

@InProceedings{pmlr-v235-wu24l,
  title = 	 {Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models},
  author =       {Wu, Mingrui and Ji, Jiayi and Huang, Oucheng and Li, Jiale and Wu, Yuhang and Sun, Xiaoshuai and Ji, Rongrong},
  booktitle = 	 {Proceedings of the 41st International Conference on Machine Learning},
  pages = 	 {53553--53570},
  year = 	 {2024},
  editor = 	 {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix},
  volume = 	 {235},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {21--27 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://raw.githubusercontent.com/mlresearch/v235/main/assets/wu24l/wu24l.pdf},
  url = 	 {https://proceedings.mlr.press/v235/wu24l.html},
  }

Data

Download R-Bench. The main annotation files include:

- image-level_filterd.json
- instance-level_filterd.json
- nocaps_pope_obj_random_image.json
- nocaps_pope_obj_popular_image.json
- nocaps_pope_obj_adversarial_image.json
- web_data
- instance_mask

These files contain annotations for image-level, instance-level(box and mask share same questions), pope-object, and web-data questions. For image-level and instance-level questions, we randomly sampled five subsets into the [type]_ids_[subset].json files.

Download the images from image (source from Open Image validation set (v4)).

Eval

Step1: To run LVLM on R-Bench using the official inference script of the LVLMs.

For Image-level, pseudocode is as follows,

for line in questions:
    question_id = line['question_id']
    question = line['text']
    image = open(line['image'])
    text = model(question, image)
    answer_file.write(json.dumps("question_id": question_id, "text":text))

For Instance-level, pseudocode is as follows, set instance_level_box=True or instance_level_mask=True to get result for Instance-Level result with Box or Mask.

import instance_qs_construct, draw_box, draw_mask

for line in questions:
    question_id = line['question_id']
    question = instance_qs_construct(line, type='mask' if instance_level_mask else 'box')
    if instance_level_box:
      image = draw_box(line)
    elif instance_level_mask:
      image = draw_mask(line)
    text = model(question, image)
    answer_file.write(json.dumps("question_id": question_id, "text":text))

and format the result.json file as follows:

{"question_id": 0, "text":[model output]}
{"question_id": 1, "text":[model output]}
...

Tips: We provide instance-level question tools in utils.py. Please use the draw_mask and draw_box functions to draw the mask or box on input images, respectively. Additionally, use the instance_qs_construct function to reformat the instance questions.

Step2: Eval with,

sh eval.sh

Tips: You can just replace --result-file with the file you generated, and the eval.py script will automatically calculate the average results for different subsets. The results obtained through our script are for the Subset, and the results for Image(All) in the paper are for reference only.

Acknowledge

The evaluation code is based on POPE.