Awesome

M3Exam: A Multilingual 🌏, Multimodal 🖼, Multilevel 📈 Benchmark for LLMs

This is the repository for M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models.

TL;DR: We introduce M3Exam, a novel benchmark sourced from real and official human exam questions for evaluating LLMs in a multilingual, multimodal, and multilevel context.

Data

Access the data

You can download the data from here.
The downloaded folder will be encrypted (to prevent some automatic crawling scripts). Please get the password from the bottom of this page.
After unzipping the file, you will see the following file structure:

data/
    multimodal-questions/         <- questions requiring images
        xx-questions-image.json   <- file containing the questions, xx is a language
        iamges-xx/                <- folder containg all the images for xx
    text-questions/               <- questions with pure text
        xx-questions-dev.json     <- held-out data (e.g., can be used as in-context examples)
        xx-questions-test.json    <- main test data for evaluation

Data format

Questions are stored in json format, you can read each json file to check the data. For example:

with open(f'./data/text-question/{lang}-questions-dev.json', 'w') as f:
    data = json.load(f)  # data is a list of questions

Each question is stored in json format:

{
    'question_text': 'Which Civil War event occurred first?',
    'background_description': [],
    'answer_text': '2',
    'options': ['(1) battle of Gettysburg',
    '(2) firing on Fort Sumter',
    '(3) assassination of President Lincoln',
    '(4) Emancipation Proclamation'],
    'need_image': 'no',
    'language': 'english',
    'level': 'mid',
    'subject': 'social',
    'subject_category': 'social-science',
    'year': '2006'
}

Evaluation

first you need to fill in your OpenAI API key in the bash files:

python main.py \
--setting zero-shot \
--model chat \
--use_api \
--selected_langs "['english']" \
--api_key #put your key here

then you can quickly check by running quick_run.sh, which will run on 10 English questions and produce english-pred.json in the corresponding output folder
to evaluate, you can also run eval.sh to check the performance on this 10 examples!
to run on more data, you can refer to run.sh for more detailed settings

python main.py \
--setting zero-shot \
--model chat \
--use_api \
--selected_langs "['english']" \
--selected_levels "['low', 'mid', 'high']" \
--num_samples all \
--api_key #put your key here

* specify the languages you want to run through `--selected_langs`
* running on all questions, set `--num_samples all`

Citation

If you find this useful in your research, please consider citing it:

@article{zhang2023m3exam,
      title={M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models},
      author={Wenxuan Zhang and Sharifah Mahani Aljunied and Chang Gao and Yew Ken Chia and Lidong Bing},
      year={2023},
      eprint={2306.05179},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

password: 12317