Awesome

⚖️LAiW: A Chinese Legal Large Language Models Benchmark

| English | Chinese

LAiW：A Comprehensive Benchmark for Chinese Legal Large Language Models (LLMs)

🔥 LAiW Leaderboard

🔥 Technical Report and Official Paper

News

🔄 Recent Updates

• [2024-04-19] The official paper has been updated.

📅 Earlier News

• [2024/1/22] Added evaluation results for the general LLMs Baichuan-7B.
• [2024/1/14] Provided more detailed information on the evaluation dataset here, along with the calculation method for the model evaluation metric SCULAiW.
• [2024/1/12] Further confirmed and improved relevant evaluation results, optimized the layout of the evaluation leaderboard SCULAiW, and supplemented more detailed information on evaluated models.
• [2024/1/10] Added evaluations for commercial LLMs GPT-4 and general LLMs Llama-7B, Llama13B, Chinese-LLaMA-13B.
• [2024/1/2] Announced the scoring mechanism for the legal capabilities of LLMs in here and published the evaluation scores for LLMs in here.
• [2024/1/2] Released test datasets for 14 foundational tasks here.
• [2024/1/1] Updated the legal capability evaluation results for SCULAiW.
• [2024/12/31] Completed legal capability evaluations for mainstream LLMs. During the evaluation process, in addition to the models mentioned earlier, general LLMs ChatGLM and legal LLMs Lawyer-LLaMA, Fuzi-Mingcha, Wisdom-Interrogatory, LexiLaw were added.
• [2023/10/12] Published the initial version of the LAiW Technical Report.
• [2023/10/08] Released the first phase evaluation system for LAiW capabilities here.
• [2023/10/08] Completed the first phase evaluation of the Basic Information Retrieval capabilities of LLMs, including commercial LLMs: ChatGPT; general LLMs: Llama2, Ziya-LLaMA, Chinese-LLaMA, Baichuan2; and legal LLMs: HanFei, ChatLaw, LaWGPT.
• [2023/10/08] Released evaluation scores and calculation methods for legal capabilities and foundational tasks.

Contents

• ⚖️LAiW: A Chinese Legal Large Language Models Benchmark
  • News
  • Contents
    • Evaluation structure diagram
    • Scores for LLMs
    • Tasks
    • Datasets
    • Scoring Mechanism
    • Run
      • 1.Preparation
      • 2.Output of LLM
    • Contributors
    • Disclaimer
    • Acknowledgements
    • Cite

Evaluation structure diagram

<img src="https://github.com/Dai-shen/LAiW/blob/main/resources/task_framework_en.png" width="70%" height="70%"></img>

Scores for LLMs

According to the calculation method of the large models' scoring mechanism, we have evaluated 7 mainstream legal large models and 6 general large models at this stage. The model scores are as follows:

The overall scores and scores for each level of legal capability of LLMs are ranked as follows:

Tasks

With the joint efforts of legal experts and artificial intelligence experts, we categorize the Legal Capabilities of LLMs into three levels, ranging from easy to difficult: Basic Information Retrieval (BIR), Legal Foundation Inference (LFI), and Cplex Legal Application (CLA), totaling 14 foundational tasks. The diagram above shows the structure of these three capability levels.

• Basic Information Retrieval. The capability of LLMs aims to address some fundamental tasks in the field of law that can be directly transferred from NLP, as well as some simple yet crucial pre-tasks in the legal domain. It includes 5 foundational tasks: Legal Article Recommendation (AR), Element Recognition (ER), Named Entity Recognition (NER), Judicial Summarization (JS), and Case Recognition (CR).
• Legal Foundation Inference. This capability aims to test some basic legal applications for LLMs. It includes 6 foundational tasks: Controversial Focus Mining (CFM), Similar Case Matching (SCM), Charge Prediction (CP), prison Term Prediction (PTP), Civil Trial Prediction (CTP), and Legal Question Answering (LQA).
• Legal Foundation Inference. We consider the challenging tasks that LLMs may face, such as complex reasoning in the legal field and aligning with real legal logic. Here, we focus on three tasks: Judicial Reasoning Generation (JRG), Case Understanding (CU), and Legal Consultation (LC).

Below is a brief description to each evaluation task.

Datasets

We have reorganized and constructed the evaluation datasets for the aforementioned tasks based on existing publicly available Chinese legal datasets. These datasets are collectively referred to as the Legal Evaluation Dataset (LED). We present the evaluation datasets for each foundational task. For more detailed information about the datasets, please refer to here.

Scoring Mechanism

⭐️ socres for each task

$$ S_{(Task)} = \begin{cases} F1 * 100, & \text{If }\quad Task\quad\in\quad Classification \ \frac{1}{3}*(R1 + R2 + RL) * 100, & \text{If }\quad Task \quad\in\quad Text\quad Generation \ Acc * 100, & \text{If }\quad Task\quad\in\quad NER \end{cases} $$

Currently, our evaluation benchmarks mainly consist of three types of tasks: classification tasks, text generation tasks and named entity recognition. For classification tasks, we use the F1 score. For text generation tasks, we use the average of Rouge1, Rouge2, and RougeL scores. Specifically, for legal Named Entity Recognition tasks, we use the extraction accuracy of legal entities as their score.

🌟 Scores for each LLM

For individual LLM, we first calculate the average score of tasks at each level as its legal capability score for that level. Then, we take the average of these three legal capability scores as the final evaluation score for the LLM. Model evaluation scores can be found here.

Run

We will continue to evaluate the performance of existing LLMs on these tasks according to the structure diagram of the 14 foundational tasks. For details, please refer to the leaderboard.

1.Preparation

git clone git clone https://github.com/Dai-shen/LAiW.git --recursive
cd LAiW
pip install -r requirements.txt
cd LAiW/src/financial-evaluation
pip install -e .[multilingual]

2.Output of LLM

We select the model and legal tasks to be evaluated. By running the following code, we can obtain the model's output.

export CUDA_VISIBLE_DEVICES="1,2"
python eval.py \
    --model "hf-causal-experimental" \
    --model_args "use_accelerate=True,pretrained=$pretrained_model,tokenizer=$pretrained_model,use_fast=False,trust_remote_code=True" \
    --tasks "legal_ar,legal_er,legal_js" \
    --no_cache \
    --num_fewshot 0 \
    --write_out \
    --output_base_path ""

Parameter Description

• model: Model interface type, optional parameters can be found in src/financial-evaluation/lm_eval/models/__init__.py
• tasks: Predefined task names, you can define your own tasks in src/tasks/_init_.py and src/tasks/legal.py
• pretrained_model: Path to the large model (Hugging Face space or local model path)
• output_base_path: Model saving path

Contributors

• Sichuan University: Yongfu Dai, Duanyu Feng, Haochen Jia, Yifang Zhang and Hao Wang
• Wuhan University: Qianqian Xie, Weiguang Han and Jimin Huang
• Southwest Petroleum University: Wei Tian

Disclaimer

This project is provided for academic and educational purposes only. We do not take responsibility for any issues, risks, or adverse consequences that may arise from the use of this project.

Acknowledgements

This project is built upon the following open-source projects, and we are really thankful for them:

• LLMindCraft
• Awesome Chinese Legal Resources

Cite

If this project has been helpful to your research, please consider citing our project.

@article{dai2023laiw,
  title={LAiW: A Chinese legal large language models benchmark},
  author={Dai, Yongfu and Feng, Duanyu and Huang, Jimin and Jia, Haochen and Xie, Qianqian and Zhang, Yifang and Han, Weiguang and Tian, Wei and Wang, Hao},
  journal={arXiv preprint arXiv:2310.05620},
  year={2023}
}