Awesome

English | 简体中文

Introduction

Welcome to CFBenchmark

In recent years, with the rapid development of Large Language Models~(LLMs), outstanding performance has been achieved in various tasks by existing LLMs. However, we notice that there is currently a limited amount of benchmarks focused on assessing the performance of LLMs in specific domains.

The "InternLM·JiShi" Chinese Financial Evaluation Benchmark (CFBenchmark) basic version consists of data from CFBenchmark-Basic and CFBenchmark-OpenFinData, focusing on evaluating the capabilities and safety of related large models in practical financial applications in the following aspects:

• Financial Natural Language Processing, mainly focusing on the model's understanding and generation capabilities of financial texts, such as financial entity recognition, industry classification, research report summarization, and risk assessment;
• Financial Scenario Calculation, focusing on assessing the model's calculation and reasoning capabilities in specific financial scenarios, such as risk assessment and investment portfolio optimization;
• Financial Analysis and Interpretation Tasks, testing the model's ability to understand complex financial reports, predict market trends, and assist in decision-making;
• Financial Compliance and Security Checks, assessing the model's potential compliance risks, such as the privacy, content safety, and financial compliance capabilities of generated content.

In the future, the "InternLM·JiShi" Chinese Financial Evaluation Benchmark will continue to deepen the construction of the financial big model evaluation system, including assessing the accuracy, timeliness, safety, privacy, and compliance of the model-generated content in the financial industry application process.

News

[2024.03.17] Added the evaluation on the financial dataset CFBenchmark-OpenFinData and released an implementation of the evaluation of the subjective questions in this dataset. In addition, we also reported the test results of 9 LLMs on the OpenFinData dataset.

OpenFinData is publicly released by EastMoney.com and Shanghai AI Lab. See details Github.

[2023.11.10] We released CFBenchmark-Basic and the corresponding technical report, mainly focusing on a comprehensive evaluation of large models in financial natural language tasks and financial text generation tasks.

Contents

• CFBenchmark-Basic
• QuickStart
• Performance of Existing LLMs
• Acknowledgements
• To-Do
• License
• Citation

CFBenchmark-Basic

CFBenchmark-Basic includes 3917 financial texts spanning three aspects and eight tasks, organized from three aspects, financial recognition, financial classification, and financial generation.

• Recognition-Company: Recognize the company names associated with financial documents (273).
• Recognition-Product: Recognize the product names associated with financial documents (297).
• Classification-Sentiment: Classify the sentiment associated with financial documents (591).
• Classification-Event: Classify the event categories associated with financial documents (577).
• Classification-Industry: Classify the industry categories associated with financial documents (402).
• Generation-Suggestion: Generate investment suggestions based on the provided financial document (593).
• Generation-Risk: Generate risk alerts based on the provided financial document (591).
• Generation-Summary: Generate a content summary based on the provided financial document (593).

We provide two examples to reveal how the few-shot setting and zero-shot setting work during evaluation.

Example 1 Fewshot Input:

Example 2 Zeroshot Input：

QuickStart

Installation

Below are the steps for quick installation.

   conda create --name CFBenchmark python=3.10
   conda activate CFBenchmark

    git clone https://github.com/TongjiFinLab/CFBenchmark
    cd CFBenchmark
    pip install -r requirements.txt

Evaluation

CFBenchmark-Basic

We have prepared the testing and evaluation codes for you in repo CFBenchmark-Basic/src.

To begin the evaluation, you can run the following code from the command line:

cd CFBenchmark-Basic/src
python -m run.py

You can enter CFBenchmark-Basic/src/run.py to modify the parameters in it to make the code running path meet your requirements.

from CFBenchmark import CFBenchmark
if __name__=='__main__':

    # EXPERIMENT SETUP
    modelname = 'YOUR-MODEL-NAME'
    model_type= 'NORMAL' #NORMAL or LoRA
    model_path= 'YOUR-MODEL-PATH'
    peft_model_path= ''#PASS YOUR OWN PATH OF PEFT MODEL IF NEEDED
    fewshot_text_path= '../fewshot'#DEFAULT PATH
    test_type='few-shot'#LET'S TAKE THE FEW-SHOT TEST AS AN EXAMPLE
    response_path='../cfbenchmark-response'#PATH TO RESERVE THE RESPONSE OF YOUR MODEL
    scores_path='../cfbenchmark-scores'	#PATH TO RESERVE THE SCORE OF YOUR MODEL
    embedding_model_path='../bge-zh-v1.5' #PASS YOUR OWN PATH OF BGE-ZH-V1.5
    benchmark_path='../data' #DEFAULT PATH

    #generate Class CFBenchmark
    cfb=CFBenchmark(
        model_name=modelname,
        model_type=model_type,
        model_path=model_path,
        peft_model_path=peft_model_path,
        fewshot_text_path=fewshot_text_path,
        test_type=test_type,
        response_path=response_path,
        scores_path=scores_path,
        embedding_model_path=embedding_model_path,
        benchmark_path=benchmark_path,
    )
    
    cfb.generate_model()# TO GET RESPONSE FROM YOUR MODEL
    cfb.get_test_scores()# TO GET YOUR MODEL SCORES FROM RESPONSE

We defined a class CFBenchmark to do the evaluation.

class CFBenchmark:
    def __init__(self,
                 model_name,
                 model_type,
                 model_path,
                 peft_model_path,
                 fewshot_text_path,
                 test_type,
                 response_path,
                 scores_path,
                 embedding_model_path,
                 benchmark_path
                 ) -> None:

• You can use the arguments to set paths for models. If you want to use a LoRA fine-tuned model, set model_type`` toLoRAand pass your peft model path throughpeft_model_path```.
• You can set test-type to 'zero-shot' or 'few-shot' to do different evaluations.
• embedding_model_path is set for bzh-zh-v1.5 for calculating cosine-similarity.
• You can modify the hyperparameters in CFBenchmark.generate_model() for text generations.
• We provide CFBenchmark saved as a Dataset data type in both Hugging Face and Github.

CFBenchmark-OpenFinData

In the CFBenchmark-OpenFinData directory, we have prepared the code and data for testing and evaluation. The design of the evaluation code is similar to Fineva1.0, where the mode of calling the evaluation model is defined through CFBenchmark-OpenFinData/src/evaluator, and the key parameters are configured and experimented with through the bash files in CFBenchmark-OpenFinData/run_scripts.

To run the evaluation, you can execute the following code in the command line:

cd CFBenchmark-OpenFinData/run_scripts
sh run_baichuan2_7b.sh

It is important to note that since the evaluation process of OpenFinData involves subjective judgement, our evaluation framework utilizes ERNIE to evaluate financial interpretation and analysis problems as well as financial compliance issues. To smoothly use the ERNIE API for evaluation, please set BAIDU_API_KEY and BAIDU_SECRET_KEY in your environment variables, so that the get_access_token function in CFBenchmark-OpenFinData/src/get_score.py can run successfully.

def get_access_token():
    """
    使用 API Key，Secret Key 获取access_token，替换下列示例中的应用API Key、应用Secret Key
    """

    url = "https://aip.baidubce.com/oauth/2.0/token?grant_type=client_credentials&client_id={}&client_secret={}".format(os.environ.get("BAIDU_API_KEY"), os.environ.get("BAIDU_SECRET_KEY"))
    
    payload = json.dumps("")
    headers = {
        'Content-Type': 'application/json',
        'Accept': 'application/json'
    }
    
    response = requests.request("POST", url, headers=headers, data=payload)
    return response.json().get("access_token")

Performance of Existing LLMs

We utilize different metrics to evaluate the performance of LLMs in the financial domain on our CFBenchmark.

For recognition and classification tasks in CFBenchmark-Basic, we employ the F1 score as the evaluation metric, which balances precision and recall.

For the generation tasks in CFBenchmark-Basic, we utilize cosine similarity between the vector representation (generated by bge-zh-v1.5) of ground truth and the generated answer to measure the generation ability.

For the knowledge, calculation, and identification in CFBenchmark-OpenFinData, we calculate the accuracy of multi-choice question.

For the explanation, analysis, and compliance in CFBenchmark-OpenFinData, we exploit the ERNIE-Bot-4 as socrer to judge the correctness of generated answer with ground truth.

The performance of LLMs are demonstrated below:

CFBenchmark-Basic

CFBenchmark-OpenFinData

CFBenchmark

Acknowledgements

CFBenchmark has referred to the following open-source projects. We want to express our gratitude and respect to the researchers of the projects.

• tiiuae/falcon LLM series(https://huggingface.co/tiiuae/falcon-7b)
• bigscience/bloomz LLM series(https://huggingface.co/bigscience/bloomz-7b1)
• QwenLM/Qwen LLM series(https://github.com/QwenLM/Qwen)
• THUDM/ChatGLM2-6b(https://github.com/THUDM/ChatGLM2-6B)
• baichuan-inc/Baichuan2 LLM series(https://github.com/baichuan-inc/Baichuan2)
• InternLM/InternLM LLM series(https://github.com/InternLM/InternLM)
• ssymmetry/BBT-FinCUGE-Applications(https://github.com/ssymmetry/BBT-FinCUGE-Applications)
• chancefocus/PIXIU(https://github.com/chancefocus/PIXIU)
• SUFE-AIFLM-Lab/FinEval(https://github.com/SUFE-AIFLM-Lab/FinEval)
• alipay/financial_evaluation_dataset(https://github.com/alipay/financial_evaluation_dataset)
• open-compass/OpenFinData(https://github.com/open-compass/OpenFinData)
• QwenLM/Qwen(https://github.com/QwenLM/Qwen)

To-Do

• CFBenchmark-Advanced:
  • In various scenarios of Chinese financial usage, propose more evaluation tasks to enrich the CFBenchmark series.

License

CFBenchmark is a research preview intended for non-commercial use only, subject to the Terms of Use of the data generated by OpenAI. Please contact us if you find any potential violations. The code is released under the Apache License 2.0.

Thanks To Our Contributors :

Citation

@misc{lei2023cfbenchmark,
      title={{CFBenchmark}: Chinese Financial Assistant Benchmark for Large Language Model}, 
      author={Lei, Yang and Li, Jiangtong and Cheng, Dawei and Ding, Zhijun and Jiang, Changjun},
      year={2023},
      eprint={2311.05812},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}