Awesome

<img src="assets/logo.png" alt="Loong" width="120" align="left"><div align="center"><h1>  Loong: Benchmarking Long-Context LLMs with Extended Multi-Doc QA</h1></div>

👀Overview

This repository contains code for our paper Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA. We propose a novel long-context benchmark, 🐉 Loong, aligning with realistic scenarios through extended multi-document question answering (QA). Loong typically consists of 11 documents per test instance on average, spanning three real-world scenarios in English and Chinese: (1) Financial Reports, (2) Legal Cases, and (3) Academic Papers. Meanwhile, Loong introduces new evaluation tasks from the perspectives of Spotlight Locating, Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic and comprehensive evaluation of long-context understanding. Furthermore, Loong features inputs of varying lengths (e.g., 10K-50K, 50K-100K, 100K-200K, beyond 200K) and evaluation tasks of diverse difficulty, enabling fine-grained assessment of LLMs across different context lengths and task complexities.

Please find more details of this work in our paper.

Showcase of four evaluation tasks in Loong (<di>...</di> marks the content of the i-th document). (a) Spotlight Locating: Locate the evidence. (b) Comparison: Locate and compare the evidence. (c) Clustering: Locate and cluster the evidence into groups. (d) Chain of Reasoning: Locate and reasoning along a logical chain.

📰News

[2024-09-20] 📰Our paper has been accepted to the EMNLP Main Conference.

[2024-07-30] 🤖The performance of phi-3, llama-3.1-8B, gpt-4o-mini on Loong are updated.

[2024-07-03] 🔥The code and benchmark are releasing. If you encounter any issues, please feel free to contact us.

[2024-06-25] 👨‍💻The code is currently being refined, and we plan to release the evaluation code and benchmark within the next one or two weeks. If you encounter any issues, please feel free to contact me at wangminzheng2023@ia.ac.cn.

🏆Leaderboard

Overall results on four evaluation tasks. For each task, the indicator on the left represents the Avg Scores(0～100), while the right one represents the Perfect Rate(0~1).

The performance of LLMs on four evaluation tasks with different length sets. For each task, the indicator on the left represents the Avg Scores(0～100), while the right one represents the Perfect Rate(0~1).

• Following previous work, we prompt GPT-4 as a judge to evaluate the model's output based on the golden answer and the question's requirements from three aspects: Accuracy, Hallucinations, and Completeness, scoring from 0 to 100. For a detailed prompt, please refer to our paper.
• We design two indicators: (1) Avg Scores: the average value of scores given by GPT-4 for all questions; (2) Perfect Rate: the proportion of cases scoring 100 out of the total cases. The latter is a more stringent evaluation metric compared to the former.
• We set temperature = 0 to eliminate randomness and keep other hyper-parameters default. For API-Based LLMs, we directly utilize the official API for testing. Since the Kimi-Chat-200k currently does not provide an interface, we manually input content on the web. As for open-source models, we conduct experiments on a server with 8 $\times$ A100 80GB.

🔧Evaluate long-context LLMs

Step1 Download Loong benchmark and docs

git clone https://github.com/MozerWang/Loong.git
cd Loong
wget -P data/ http://alibaba-research.oss-cn-beijing.aliyuncs.com/loong/doc.zip
unzip data/doc.zip -d data/

Step2 Create a conda environment and Install other dependencies.

conda create --name loong python=3.9 -y
conda activate loong
pip install -r requirements.txt

Step3 Preparing the Model

1. (Must) Set up your OPENAI key in config/models/gpt4.yaml

api_key: "Your OPENAI key"

2. If you are using API-based LLM

# Firstly, Set up your key in config/models/*.yaml
api_key: "Your API key"

3. If you are using Open-sourced LLM

# We recommend using vLLM. And we use HTTP server that implements OpenAI’s Completions and Chat API.
# We have provided using example for Qwen2 and GLM4. See details in Loong/src/vllm_eample.sh
cd src
sh vllm_example.sh

Step4 Evaluate

cd src
sh run.sh

Things To Know

• We provide a complete evaluation process:
  step1_load_data.py # Data loading
  step2_model_generate.py # Model generation
  step3_model_evaluate.py # GPT-4 evaluation
  step4_cal_metric.py # Result statistics

• For step2_model_generate.py, you can design the model generation part yourself, modifying it to use your own model's inference method. Just make sure the input and output interfaces in src/utils/generate.py remain consistent:

# Input
generate(prompts, config, output_path, process_num, tag)

# Output
result = prompt.copy() # for prompt in prompts
result[tag] = response_content # Your LLM's response
with open(output_path, 'a', encoding='utf-8') as fw:
    fw.write(json.dumps(result, ensure_ascii=False) + '\n')

• In data/loong.jsonl, the level key means task:
  level1 # Spotlight Locating
  level2 # Comparison
  level3 # Clustering
  level4 # Chain of Reasoning

Citation

@inproceedings{wang2024loong,
  title={Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA},
  author={Minzheng Wang, Longze Chen, Cheng Fu, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, Yongbin Li},
  year={2024}
  booktitle ={Proceedings of EMNLP},
  url={https://aclanthology.org/2024.emnlp-main.322},
  pages={5627--5646},
}