Awesome

English | 简体中文

🎯 Who Should Pay Attention to Our Work?

• If you are developing a Benchmark, you can use our xFinder to replace traditional RegEx methods for extracting key answers from LLM responses. This will help you improve the accuracy of your evaluation results, enabling more reliable and meaningful comparisons and validation of model performance.
• If you are a designer of evaluation frameworks, you can integrate our xFinder into your framework's answer extraction component to enhance the robustness and reliability of the evaluation process.

[!IMPORTANT]

🌟 Star Us! By starring our project on GitHub, you'll receive all release notifications instantly. We appreciate your support!

:loudspeaker: News

• [2024/08] We updated xFinder: The model now supports processing both English and Chinese.
• [2024/05] We released xFinder: Robust and Pinpoint Answer Extraction for Large Language Models. Check out the paper.

:sparkles: Overview

We summarize our primary contributions as follows:

• We provide a comprehensive review of LLM evaluation processes in the industry, identifying critical factors that can lead to unreliable evaluation results.
• We introduce xFinder, a model specifically designed for key answer extraction. The KAF dataset supports its effective training and evaluation.
• In our extensive experiments, we demonstrate that RegEx-based evaluation methods are unreliable, while our xFinder model significantly improves reliability.

As shown in the figure, instances where evaluation frameworks such as LM Eval Harness and OpenCompass failed to extract key answers are illustrated. Specifically, A/T/C/M represent tasks with alphabet / short text / categorical label / math options, respectively.

:zap: Quick Start

1. Ensure Compatibility: Ensure you have Python 3.10.0+.
2. Create Benchmark Dataset: To facilitate the evaluation of benchmark datasets using xFinder, we have standardized various mainstream benchmark datasets into a unified JSON format. For details, see create_benchmark_dataset.py. Additionally, if you want to use xFinder to evaluate your own datasets, you can refer to the provided script template benchmark_dataset_template.py for format conversion.
3. Prepare QA pairs & LLM Outputs: Prepare the LLM outputs that you want to evaluate.
   • provide a .json file including original question, key answer type (alphabet / short_text / categorical_label / math), LLM output, standard answer range.
   • For a detailed example of the expected format, refer to demo/example.json.
4. Deploy the xFinder Model: Choose between two models for deployment, xFinder-qwen1505 or xFinder-llama38it.
5. Finish Configuration: Compile the above details into a configuration file. For configuration details, see demo\xfinder_config.yaml.

After setting up the configuration file, you have two methods to proceed with the evaluation:

1. Use with install:

> git clone  git@github.com:IAAR-Shanghai/xFinder.git
> cd xFinder
> conda create -n xfinder_env python=3.11 -y
> conda activate xfinder_env
> pip install -e .
> xfinder $PATH_TO_CONFIG

2. Use without install:

> git clone  git@github.com:IAAR-Shanghai/xFinder.git
> cd xFinder
> pip install -r requirements.txt
> python
>>> from xfinder.eval import calc_acc
>>> calc_acc($PATH_TO_CONFIG)

Note: We provide scripts for fine-tuning xFinder in xfinder_training.

:sun_with_face: Examples: RegEx vs. xFinder

We demonstrate instances across four types of questions where RegEx fails to extract or frequently extracts incorrect answers, whereas xFinder accurately extracts the key answers.

{
    "key_answer_type": "alphabet option",
    "question": "A man is seen playing guitar on a stage with others playing instruments behind him. The man grabs a guitar from the audience and begins playing both one after the other ...",
    "llm_output": "Option A is the correct choice as it describes ...",
    "standard_answer_range": "[['A', 'strums the guitar in the end, continues playing the guitar with the crowd following him as well as lining up next to him.'], ['B', 'continues playing the instruments and ends by waving to the crowd and walking off stage.'], ['C', 'then turns to the audience and gives a stuffed toy to the audience and continues playing.'], ['D', 'finally stops playing and moves his hands for the crowd to see.']]",
    "gold_label": "A",
    "xFinder_output": "A",
},
{
    "key_answer_type": "short text",
    "question": "If you really wanted a grape, where would you go to get it? Answer Choices: winery / fruit stand / field / kitchen / food",
    "llm_output": "The answer is winery / fruit stand / field / kitchen / food ...",
    "standard_answer_range": "[\"winery\", \"fruit stand\", \"field\", \"kitchen\", \"food\"]",
    "gold_label": "[No valid answer]",
    "xFinder_output": "[No valid answer]",
},
{
    "key_answer_type": "categorical label",
    "question": "How tall is the Sears Building ?",
    "llm_output": "The Sears Building is a specific structure, so the answer would be a Location ...",
    "standard_answer_range": "['Abbreviation', 'Entity', 'Description', 'Person', 'Location', 'Number']",
    "gold_label": "Location",
    "xFinder_output": "Location",
},
{
    "key_answer_type": "math",
    "question": " Mike made 69 dollars mowing lawns over the summer. If he spent 24 dollars buying new mower blades, how many 5 dollar games could he buy with the money he had left? ",
    "llm_output": "To find out how many 5 dollar ... Let's calculate that:\n\n$45 / $5 = 9\n\nSo, Mike could buy 9 5 dollar games with the money he had left.",
    "standard_answer_range": "a(n) number / set / vector / matrix / interval / expression / function / equation / inequality",
    "gold_label": "9",
    "xFinder_output": "9",
}

:trophy: Results of Extraction Accuracy

Baseline: OpenCompass, LM Eval Harness, UltraEval, GPT-4. Our Method: xFinder-qwen1505, xFinder-qwen1518, xFinder-gemma7, xFinder-chatglm36base, xFinder-llama38, xFinder-llama38it.

We evaluated their accuracy in extracting key answers from both the KAF test set and generalization sets. The metric in the table is accuracy.

:memo: Citation

@article{xFinder,
      title={xFinder: Robust and Pinpoint Answer Extraction for Large Language Models}, 
      author={Qingchen Yu and Zifan Zheng and Shichao Song and Zhiyu Li and Feiyu Xiong and Bo Tang and Ding Chen},
      journal={arXiv preprint arXiv:2405.11874},
      year={2024},
}

:triangular_flag_on_post: TODOs

• feat: customized configuration on attributes' names.
• feat: support quick replacement of RegEx in OpenCompass.
• feat: support parallel (or offline batched) evaluation.
• feat: demo deployment.
• feat: API deployment.
• codes: upload data construction pipeline codes.
• docs: add video tutorial.

:star: Star Trends