Awesome

FTIIBench

(ARXIV24) This is the official code repository for "FTII-Bench: A Comprehensive Multimodal Benchmark for Flow Text with Image Insertion."

Dataset

The text of FTII-Bench could be download from Google Drive

The images of FTII-Bench could be download from Google Drive

Note that the data is only used for research purposes!

Evaluation

1. Set the appropriate paths in the run_eval_fi and run_eval_sc scripts.

   bash run_eval_fi.sh # for flow insertion tasks
   bash run_eval_sc.sh # for single choice tasks
   

2. For evaluating with BGE models You can run ./mllm_eval/bge_eval.ipynb in the Jupyter environment.

Acknowledgement

Thanks to the open-source code from Mantis