Home

Awesome

MathVista: Evaluating Math Reasoning in Visual Contexts

MathQA Mathematical Reasoning Multi-Modal ScienceQA
Claude-4 ChatGPT GPT-4 Gemini GPT-4V

Code for the Paper "MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts".

For more details, please refer to the project page with dataset exploration and visualization tools: https://mathvista.github.io/.

:bell: If you have any questions or suggestions, please don't hesitate to let us know. You can comment on the Twitter, or post an issue on this repository.

[Webpage] [Paper] [Huggingface Dataset] [Leaderboard] [Visualization] [Result Explorer] [Twitter]

<p align="center"> <img src="assets/logo_v1.png" width="40%"> <br> Tentative logo for <b>MathVista</b>. Generated by DALLยทE 3 prompted by <br>"A photo-based logo with a gradient of soft blue and modern typography, accompanied by the title 'MathVista'". </p>

Outlines

๐Ÿ’ฅ Spotlight: Performance Update (Sept 12, 2024) ๐Ÿ’ฅ

๐Ÿ’ฅ News ๐Ÿ’ฅ

๐Ÿ‘€ About MathVista

Large Language Models (LLMs) and Large Multimodal Models (LMMs) exhibit impressive problem-solving skills in many tasks and domains, but their ability in mathematical reasoning in visual contexts has not been systematically studied. To bridge this gap, we present MathVista, a benchmark designed to combine challenges from diverse mathematical and visual tasks. It consists of 6,141 examples, derived from 28 existing multimodal datasets involving mathematics and 3 newly created datasets (i.e., IQTest, FunctionQA, and PaperQA). Completing these tasks requires fine-grained, deep visual understanding and compositional reasoning, which all state-of-the-art foundation models find challenging.

<p align="center"> <img src="assets/data-composition.png" width="40%"> <br> Source dataset distribution of <b>MathVista</b>. </p>

In October 2023, we conducted a comprehensive, quantitative evaluation of 12 prominent foundation models with MathVista. The best-performing GPT-4V model achieved an overall accuracy of 49.9%, substantially outperforming Bard, the second-best performer, by 15.1%. Our in-depth analysis revealed that the superiority of GPT-4V is mainly attributed to its enhanced visual perception and mathematical reasoning. However, GPT-4V still falls short of human performance by 10.4%, as it often struggles to understand complex figures and perform rigorous reasoning. This significant gap underscores the critical role that MathVista will play in the development of general-purpose AI agents capable of tackling mathematically intensive and visually rich real-world tasks.

<p align="center"> <img src="assets/score_leaderboard_gpt4v.png" width="70%"> <br> Accuracy scores the testmini set (1,000 examples) of <b>MathVista</b>. </p>

We further explore the new ability of self-verification, the use of self-consistency, and the goal-directed multi-turn human-AI dialogues, highlighting the promising potential of GPT-4V for future research.

<p align="center"> <img src="assets/tease_scores_version4_gemini.png" width="80%"> <br> Accuracy scores of one leading LLM (i.e., PoT GPT-4), four primary LMMs, random chance, and human performance on <b>MathVista</b>. </p> <details> <summary>๐Ÿ” See the accuracy scores without Gemini Ultra</summary> <p align="center"> <img src="assets/tease_scores_gpt4v.png" width="80%"> <br> Accuracy scores of one leading LLM (i.e., PoT GPT-4), four primary LMMs, random chance, and human performance on <b>MathVista</b>. </p> </details>

For more details, you can find our project page here and our paper here.

๐Ÿ† Leaderboard ๐Ÿ†

Contributing the Leaderboard

๐Ÿšจ๐Ÿšจ The leaderboard is continuously being updated.

The evaluation instructions are available at ๐Ÿ”ฎ Evaluations on MathVista and ๐Ÿ“ Evaluation Scripts of Our Models.

To submit your results to the leaderboard on the testmini subset, please send to this email with your result json file and score json file, referring to the template files below:

To submit your results to the leaderboard on the test subset, please send to this email with your result file (we will generate the score file for you), referring to the template file below:

Leaderboard on the testmini subset

Accuracy scores on the testmini subset (1,000 examples):

#ModelMethodSourceDateALLFQAGPSMWPTQAVQAALGARIGEOLOGNUMSCISTA
-Human Performance*-Link2023-10-0360.359.748.473.063.255.950.959.251.440.753.864.963.9
1OpenAI o1 ๐Ÿฅ‡LMM ๐Ÿ–ผ๏ธLink2024-09-1273.9------------
2Grok-2 ๐ŸฅˆLMM ๐Ÿ–ผ๏ธLink2024-08-1369.0------------
3Grok-2 mini ๐Ÿฅ‰LMM ๐Ÿ–ผ๏ธLink2024-08-1368.1------------
4Claude 3.5 SonnetLMM ๐Ÿ–ผ๏ธLink2024-06-2067.7------------
5LLaVA-OneVisionLMM ๐Ÿ–ผ๏ธLink2024-08-0667.5------------
6InternVL2-ProLMM ๐Ÿ–ผ๏ธLink2024-09-0466.870.665.476.971.548.066.562.363.627.040.365.681.1
7TextGrad (GPT-4o)LMM ๐Ÿ–ผ๏ธLink2024-07-0866.1------------
8Gemini 1.5 Pro (May 2024)LMM ๐Ÿ–ผ๏ธLink2024-05-1763.9------------
9GPT-4oLMM ๐Ÿ–ผ๏ธLink2024-05-1363.8------------
10InternVL-Chat-V1.2-PlusLMM ๐Ÿ–ผ๏ธLink2024-02-2259.951.761.179.652.557.054.563.261.116.248.655.760.8
11Gemini 1.5 Flash (May 2024)LMM ๐Ÿ–ผ๏ธLink2024-05-1758.4------------
12GPT-4T 2024-04-09LMM ๐Ÿ–ผ๏ธLink2024-05-1358.1------------
13Pixtral 12BLMM ๐Ÿ–ผ๏ธLink2024-09-1158.0------------
14InternLM-XComposer2-VL-7BLMM ๐Ÿ–ผ๏ธLink2024-01-2257.655.063.073.756.339.756.652.462.38.142.459.064.1
15Gemini 1.0 UltraLMM ๐Ÿ–ผ๏ธLink2023-12-0653.049.156.253.869.040.258.445.955.621.638.962.359.5
16Grok-1.5VLMM ๐Ÿ–ผ๏ธLink2024-04-1252.8------------
17Gemini 1.5 Pro (Feb 2024)LMM ๐Ÿ–ผ๏ธLink2024-02-1552.1------------
18Claude 3 OpusLMM ๐Ÿ–ผ๏ธLink2024-03-0450.5------------
19GPT-4V (Playground)LMM ๐Ÿ–ผ๏ธLink2023-10-1549.943.150.557.565.238.053.049.051.021.620.163.155.8
20Claude 3 SonnetLMM ๐Ÿ–ผ๏ธLink2024-03-0447.9------------
21InternVL-Chat-V1.2LMM ๐Ÿ–ผ๏ธLink2024-02-2247.750.961.130.648.144.752.336.558.218.930.654.951.8
22Math-LLaVA-13BLMM ๐Ÿ–ผ๏ธLink2024-06-2546.637.257.756.551.333.553.040.256.516.233.349.243.9
23LLaVA-NeXT-34BLMM ๐Ÿ–ผ๏ธLink2024-01-3046.5------------
24Claude 3 HaikuLMM ๐Ÿ–ผ๏ธLink2024-03-0446.4------------
25Gemini 1.0 ProLMM ๐Ÿ–ผ๏ธLink2023-12-0645.247.640.439.261.439.145.238.841.010.832.654.956.8
26Phi-3-Vision-128K-InLMM ๐Ÿ–ผ๏ธLink2024-05-2144.5------------
27Phi-3.5-Vision 4.2BLMM ๐Ÿ–ผ๏ธLink2024-04-2243.9------------
28Qwen-VL-PlusLMM ๐Ÿ–ผ๏ธLink2023-12-2143.354.638.531.255.134.139.132.039.318.926.459.056.1
29Mini-Gemini-HD (Hermes-2-Yi-34B)LMM ๐Ÿ–ผ๏ธLink2024-03-2743.3------------
30SPHINX-MoEMoE ๐Ÿค–Link2024-01-1242.349.831.242.546.839.731.741.630.516.227.150.850.8
31Mini-Gemini (Mixtral-8x7B)LMM ๐Ÿ–ผ๏ธLink2024-03-2741.8------------
32MM1-7B-MoE-ChatLMM ๐Ÿ–ผ๏ธLink2024-03-1440.9------------
33MiniCPM-V-2 (2.8B)LMM ๐Ÿ–ผ๏ธLink2024-04-1440.653.226.037.144.339.128.533.128.010.839.648.451.8
34MM1-30B-ChatLMM ๐Ÿ–ผ๏ธLink2024-03-1439.4------------
35SPHINX-PlusMoE ๐Ÿค–Link2024-01-1236.8------------
36SPHINX (V2)LMM ๐Ÿ–ผ๏ธLink2023-11-1736.754.616.423.141.843.020.633.417.624.321.543.451.5
37MM1-7B-ChatLMM ๐Ÿ–ผ๏ธLink2024-03-1435.9------------
38SPHINX-Intern2MoE ๐Ÿค–Link2024-01-1235.5------------
39OmniLMM-12BLMM ๐Ÿ–ผ๏ธLink2024-02-0134.945.017.826.944.939.123.132.320.918.927.845.944.2
40Multimodal BardLMM ๐Ÿ–ผ๏ธLink2023-10-0334.826.047.129.648.726.846.528.647.813.514.947.533.0
41LLaVA-NeXT-Vicuna-7BLMM ๐Ÿ–ผ๏ธLink2024-01-3034.6------------
42PoT GPT-4 (Caption+OCR)Tool ๐Ÿ› ๏ธLink2023-10-0333.930.139.430.639.931.337.431.741.018.920.144.337.9
43CoT Claude (Caption+OCR)Tool ๐Ÿ› ๏ธLink2023-10-0333.227.529.336.049.429.131.032.931.016.217.450.837.2
44CoT GPT4 (Caption+OCR)Tool ๐Ÿ› ๏ธLink2023-10-0333.227.931.731.251.928.533.530.932.213.512.558.237.9
45CoT ChatGPT (Caption+OCR)Tool ๐Ÿ› ๏ธLink2023-10-0333.226.031.735.548.130.232.432.333.016.217.454.936.2
46MM1-3B-MoE-ChatLMM ๐Ÿ–ผ๏ธLink2024-03-1432.6------------
47MM1-3B-ChatLMM ๐Ÿ–ผ๏ธLink2024-03-1432.0------------
48Gemini 1.0 Nano 2LMM ๐Ÿ–ผ๏ธLink2023-12-0630.628.623.630.641.831.827.129.826.810.820.840.233.5
49LLaVA-1.5-13BLMM ๐Ÿ–ผ๏ธLink2024-01-3027.6------------
50SPHINX (V1)LMM ๐Ÿ–ผ๏ธLink2023-11-0927.523.423.121.539.934.125.628.123.416.217.440.223.6
51Gemini 1.0 Nano 1LMM ๐Ÿ–ผ๏ธLink2023-12-0627.330.921.623.729.130.723.825.521.313.520.827.930.9
52PoT ChatGPT (Caption+OCR)Tool ๐Ÿ› ๏ธLink2023-10-0326.824.526.423.733.527.927.826.128.018.913.233.629.9
53SPHINX-TinyMoE ๐Ÿค–Link2024-01-1226.4------------
54LLaVA (LLaMA-2-13B)LMM ๐Ÿ–ผ๏ธLink2023-10-0326.126.829.316.132.326.327.320.128.824.318.337.325.1
55InstructBLIP (Vicuna-7B)LMM ๐Ÿ–ผ๏ธLink2023-10-0325.323.120.718.332.335.221.827.120.718.920.433.023.1
56LLaVARLMM ๐Ÿ–ผ๏ธLink2023-10-0325.221.925.016.734.830.724.222.123.013.515.342.621.9
57LLaMA-Adapter-V2 (7B)LMM ๐Ÿ–ผ๏ธLink2023-10-0323.921.225.511.332.331.826.320.424.324.313.929.518.3
58miniGPT4 (LLaMA-2-7B)LMM ๐Ÿ–ผ๏ธLink2023-10-0323.118.626.013.430.430.228.121.024.716.216.725.417.9
59mPLUG-Owl (LLaMA-7B)LMM ๐Ÿ–ผ๏ธLink2023-10-0322.222.723.610.227.227.923.619.223.913.512.726.321.4
60IDEFICS (9B-Instruct)LMM ๐Ÿ–ผ๏ธLink2023-10-0319.821.621.16.525.924.022.115.019.818.99.924.618.1
61Random Chance-Link2023-10-0317.915.524.14.523.424.325.813.822.713.48.815.814.3

Some notations in the table:

๐Ÿ”” The automatic evaluation on CodaLab are under construction.

๐Ÿ“Š Dataset Examples

Examples of our newly annotated datasets: IQTest, FunctionQA, and PaperQA:

<p align="center"> <img src="assets/our_new_3_datasets.png" width="60%"> <br> </p> <details> <summary>๐Ÿ” Click to expand/collapse more examples</summary>

Examples of seven mathematical reasoning skills:

  1. Arithmetic Reasoning
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/ari.png" style="zoom:40%;" />
  1. Statistical Reasoning
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/sta.png" style="zoom:40%;" />
  1. Algebraic Reasoning
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/alg.png" style="zoom:40%;" />
  1. Geometry Reasoning
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/geo.png" style="zoom:40%;" />
  1. Numeric Commonsense Reasoning
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/num.png" style="zoom:40%;" />
  1. Scientific Reasoning
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/sci.png" style="zoom:40%;" />
  1. Logical Reasoning
<img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/skills/log.png" style="zoom:40%;" /> </details>

๐Ÿ“– Dataset Usage

Data Source

The MathVista dataset is derived from three newly collected datasets: IQTest, FunctionQA, and Paper, as well as 28 other source datasets. Details can be found in the source.json file. All these source datasets have been preprocessed and labeled for evaluation purposes.

Data Downloading

All the data examples were divided into two subsets: testmini and test.

You can download this dataset by the following command (make sure that you have installed Huggingface Datasets):

from datasets import load_dataset

dataset = load_dataset("AI4Math/MathVista")

Here are some examples of how to access the downloaded dataset:

# print the first example on the testmini set
print(dataset["testmini"][0])
print(dataset["testmini"][0]['pid']) # print the problem id 
print(dataset["testmini"][0]['question']) # print the question text 
print(dataset["testmini"][0]['query']) # print the query text
print(dataset["testmini"][0]['image']) # print the image path
print(dataset["testmini"][0]['answer']) # print the answer
dataset["testmini"][0]['decoded_image'] # display the image

# print the first example on the test set
print(dataset["test"][0])

We have uploaded a demo to illustrate how to access the MathVista dataset on Hugging Face, available at hugging_face_dataset_demo.ipynb.

Data Format

The dataset is provided in json format and contains the following attributes:

{
    "question": [string] The question text,
    "image": [string] A file path pointing to the associated image,
    "choices": [list] Choice options for multiple-choice problems. For free-form problems, this could be a 'none' value,
    "unit": [string] The unit associated with the answer, e.g., "m^2", "years". If no unit is relevant, it can be a 'none' value,
    "precision": [integer] The number of decimal places the answer should be rounded to,
    "answer": [string] The correct answer for the problem,
    "question_type": [string] The type of question: "multi_choice" or "free_form",
    "answer_type": [string] The format of the answer: "text", "integer", "float", or "list",
    "pid": [string] Problem ID, e.g., "1",
    "metadata": {
        "split": [string] Data split: "testmini" or "test",
        "language": [string] Question language: "English", "Chinese", or "Persian",
        "img_width": [integer] The width of the associated image in pixels,
        "img_height": [integer] The height of the associated image in pixels,
        "source": [string] The source dataset from which the problem was taken,
        "category": [string] The category of the problem: "math-targeted-vqa" or "general-vqa",
        "task": [string] The task of the problem, e.g., "geometry problem solving",
        "context": [string] The visual context type of the associated image,
        "grade": [string] The grade level of the problem, e.g., "high school",
        "skills": [list] A list of mathematical reasoning skills that the problem tests
    },
    "query": [string] the query text used as input (prompt) for the evaluation model
}

Data Visualization

๐ŸŽฐ You can explore the dataset in an interactive way here.

<details> <summary>Click to expand/collapse the visualization page screenshot.</summary> <img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/data_visualizer.png" style="zoom:40%;" /> </details>

Usage Demos

We offer a few demo examples for using the dataset, as follows:

Stay tuned for more demos coming soon!

๐Ÿ”ฎ Evaluations on MathVista

Requirements (Optional)

Install the Python dependencies if you would like to reproduce our results for ChatGPT, GPT-4, Claude-2, and Bard:

pip install openai # for ChatGPT and GPT-4
pip install anthropic # for Claude-2
pip install bardapi # for Bard

For more details, please refer to:

If you are considering evaluating your own model, these dependencies might be optional.

Downloading Images (Optional)

We provide images in the JPG format. You can download and unzip them using the following commands:

cd data
wget https://huggingface.co/datasets/AI4Math/MathVista/resolve/main/images.zip
unzip images.zip && rm images.zip

This step might be optional if you prefer to use the Hugging Face format of the data.

Evaluation Pipelines

Recent foundation models have been trained to generate longer responses instead of brief text. As such, we propose a new strategy for benchmarking MathVista. This evaluation process comprises three stages:

(Step 1) Response Generation (generate_response.py): The models generate responses based on the given input query (prompt). This input query integrates the task description, the question, choices, and metadata. Such a design encourage the models yield responses in the desired format, subsequently enhancing the overall evaluation scores. An example of such an input query is:

Hint: Please answer the question and provide the correct option letter, e.g., A, B, C, D, at the end.
Question: Find $m\\angle H$
Choices:
(A) 97
(B) 102
(C) 107
(D) 122

The task description is defined as follows:

Question typeAnswer typeTask instruction
Multiple-choiceTextPlease answer the question and provide the correct option letter, e.g., A, B, C, D, at the end.
Free-formIntegerPlease answer the question requiring an integer answer and provide the final value, e.g., 1, 2, 3, at the end.
Free-formFloat (1)Please answer the question requiring a floating-point number with one decimal place and provide the final value, e.g., 1.2, 1.3, 1.4, at the end.
Free-formFloat (2)Please answer the question requiring a floating-point number with two decimal places and provide the final value, e.g., 1.23, 1.34, 1.45, at the end.
Free-formListPlease answer the question requiring a Python list as an answer and provide the final list, e.g., [1, 2, 3], [1.2, 1.3, 1.4], at the end.

(Step 2) Answer Extraction (extract_answer.py): Next, the short answer text is extracted from the detailed response. We propose an answer extractor based on LLMs such as GPT-4. A preliminary study of 200 examples shows that GPT-4 can extract the answer text with more than 99.5% accuracy. Below are examples of extracting short answers from long responses:

# Example 1
Hint: Please answer the question requiring an integer answer and provide the final value,
e.g., 1, 2, 3, at the end.
Question: Which number is missing?

Model response: The number missing in the sequence is 14.

Extracted answer: 14

# Example 2
Hint: Please answer the question and provide the correct option letter, e.g., A, B, C,
D, at the end.
Question: What fraction of the shape is blue?
Choices: 
(A) 3/11 
(B) 8/11 
(C) 6/11 
(D) 3/5

Model response: The correct answer is (B) 8/11.

Extracted answer: B

(Step 3) Score Calculation (calculate_score.py): Finally, the extracted answer is normalized to a required answer format (e.g., an option letter or an integer), and the target metric scores are computed.

๐Ÿ“ Evaluation Scripts of Our Models

To execute the evaluation scripts in our paper, ensure your data folder has the following structure:

โ”œโ”€โ”€ query.json
โ”œโ”€โ”€ test.json
โ”œโ”€โ”€ testmini.json
โ”œโ”€โ”€ images
    โ”œโ”€โ”€ 1.jpg
    โ”œโ”€โ”€ 2.jpg
    โ””โ”€โ”€ ...
โ””โ”€โ”€ texts
    โ”œโ”€โ”€ captions_bard.json
    โ””โ”€โ”€ ocrs_easyocr.json

Additionally, ensure that the API keys for ChatGPT, GPT-4, Claude-2, and Bard are properly set up.

Evaluating Multimodal Bard

If you have setted Multimodal Bard, you can run the following commands:

Generate the response on the testmini subset:

cd evaluation

python generate_response.py \
--model bard \
--output_dir ../results/bard \
--output_file output_bard.json

Extract the short answer text for score calculation on the testmini subset:

python extract_answer.py \
--output_dir ../results/bard \
--output_file output_bard.json 

Calculate the final score on the testmini subset:

python calculate_score.py \
--output_dir ../results/bard \
--output_file output_bard.json \
--score_file scores_bard.json

Generate the response of the test subset:

python generate_response.py \
--model bard \
--input_file test.json \
--output_dir ../results/bard \
--output_file output_bard_test.json

Extract the short answer text for score calculation on the test subset:

python extract_answer.py \
--output_dir ../results/bard \
--output_file output_bard_test.json 

Evaluating Chain-of-Thought GPT-4

Generate the response on the testmini subset:

cd evaluation

python generate_response.py \
--model gpt-4-0613 \
--output_dir ../results/gpt4 \
--output_file output_gpt4_2shot_solution_use_caption_ocr.json \
--shot_num 2 \
--shot_type solution \
--use_caption \
--use_ocr \
--caption_file ../data/texts/captions_bard.json \
--ocr_file ../data/texts/ocrs_easyocr.json 

Extract the short answer text for score calculation on the testmini subset:

python extract_answer.py \
--output_dir ../results/gpt4 \
--output_file output_gpt4_2shot_solution_use_caption_ocr.json

Calculate the final score on the testmini subset:

python calculate_score.py \
--output_dir ../results/gpt4 \
--output_file output_gpt4_2shot_solution_use_caption_ocr.json \
--score_file scores_gpt4_2shot_solution_use_caption_ocr.json

Generate the response of the test subset:

python generate_response.py \
--model gpt-4-0613 \
-input_file test.json \
--output_dir ../results/gpt4 \
--output_file output_test_gpt4_2shot_code_use_caption_ocr.json \
--shot_num 2 \
--shot_type solution \
--use_caption \
--use_ocr \
--caption_file ../data/texts/captions_bard.json \
--ocr_file ../data/texts/ocrs_easyocr.json 

Extract the short answer text for score calculation on the test subset:

python extract_answer.py \
--output_dir ../results/bard \
--output_file output_test_gpt4_2shot_code_use_caption_ocr.json 

Evaluating Program-of-Thought GPT-4

Generate the response on the testmini subset:

cd evaluation

python generate_response.py \
--model gpt-4-0613 \
--output_dir ../results/gpt4 \
--output_file output_gpt4_2shot_code_use_caption_ocr.json \
--shot_num 2 \
--shot_type code \
--use_caption \
--use_ocr \
--caption_file ../data/texts/captions_bard.json \
--ocr_file ../data/texts/ocrs_easyocr.json 

Extract the short answer text for score calculation on the testmini subset:

python extract_answer.py \
--output_dir ../results/gpt4 \
--output_file output_gpt4_2shot_code_use_caption_ocr.json \
--response_label execution

Calculate the final score on the testmini subset:

python calculate_score.py \
--output_dir ../results/gpt4 \
--output_file output_gpt4_2shot_code_use_caption_ocr.json \
--score_file scores_gpt4_2shot_code_use_caption_ocr.json

Generate the response of the test subset:

python generate_response.py \
--model gpt-4-0613 \
--input_file test.json \
--output_dir ../results/gpt4 \
--output_file output_test_gpt4_2shot_code_use_caption_ocr.json \
--shot_num 2 \
--shot_type code \
--use_caption \
--use_ocr \
--caption_file ../data/texts/captions_bard.json \
--ocr_file ../data/texts/ocrs_easyocr.json 

Extract the short answer text for score calculation on the test subset:

python extract_answer.py \
--output_dir ../results/gpt4 \
--output_file output_test_gpt4_2shot_code_use_caption_ocr.json \
--response_label execution

Evaluating More Settings

For additional settings for large language models and other baselines, please refer to the running scripts available in the scripts directory.

Evaluating Large Multimodal Models

We thank Hritik Bansal and the VisIT-Bench project for providing easy-to-use codes for evaluating most of the large multimodal models included in our paper.

๐Ÿ“ˆ Evaluation Results

<details> <summary>Click to expand/collapse the examples.</summary> <img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/results_examples/5.png" style="zoom:40%;" /> </details> <details> <summary>Click to expand/collapse the examples.</summary> <img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/results_examples/6.png" style="zoom:40%;" /> </details> <details> <summary>Click to expand/collapse the example.</summary> <img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/results_examples/48.png" style="zoom:40%;" /> </details> <details> <summary>Click to expand/collapse the example.</summary> <img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/results_examples/50.png" style="zoom:40%;" /> </details> <details> <summary>Click to expand/collapse the example.</summary> <img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/results_examples/52.png" style="zoom:40%;" /> </details> <details> <summary>Click to expand/collapse the example.</summary> <img src="https://raw.githubusercontent.com/lupantech/MathVista/main/assets/results_examples/53.png" style="zoom:40%;" /> </details> We stored the result files from different models in the [results](https://github.com/lupantech/MathVista/tree/main/results/) directory.

๐Ÿ™ For visualization of these results, visit our exploration page.

๐Ÿ“œ License

The new contributions to our dataset are distributed under the CC BY-SA 4.0 license, including

The copyright of the images and the questions belongs to the original authors, and the source of every image and original question can be found in the metadata field and in the source.json file. Alongside this license, the following conditions apply:

:coffee: Stay Connected!

Fantastic! I'm always open to engaging discussions, collaborations, or even just sharing a virtual coffee. To get in touch, visit Pan Lu's homepage for contact information.

:white_check_mark: Cite

If you find MathVista useful for your your research and applications, please kindly cite using this BibTeX:

@inproceedings{lu2024mathvista,
  title={MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts},
  author={Lu, Pan and Bansal, Hritik and Xia, Tony and Liu, Jiacheng and Li, Chunyuan and Hajishirzi, Hannaneh and Cheng, Hao and Chang, Kai-Wei and Galley, Michel and Gao, Jianfeng},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2024}
}

๐Ÿง  Related Work

Explore our additional research on large language models and large multimodal models , focusing on mathematical reasoning, scientific reasoning, and multimodal reasoning:

๐Ÿค Contributors

Here are the key contributors to this project:

Pan Lu<sup>1</sup>, Hritik Bansal<sup>1</sup>, Tony Xia<sup>1</sup>, Jiacheng Liu<sup>2</sup>, Chunyuan Li<sup>3</sup>, Hannaneh Hajishirzi<sup>2</sup>, Hao Cheng<sup>3</sup>, Kai-Wei Chang<sup>1</sup>, Michel Galley<sup>3</sup>, Jianfeng Gao<sup>3</sup>

<sup>1</sup>University of California, Los Angeles, <sup>2</sup>University of Washington, <sup>3</sup>Microsoft Research