Awesome

Overview

This repository contains resources for accessing the official benchmarks, codes, and checkpoints of the paper: "Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations".

This work pioneers exploring and building powerful Multilingual Math Reasoning (xMR) LLMs. To accomplish this, we make the following works:

• MGSM8KInstruct, the first multilingual math reasoning instruction dataset, encompassing ten distinct languages, thus addressing the issue of training data scarcity in xMR tasks.
• MSVAMP, an out-of-domain xMR test dataset, to conduct a more exhaustive and comprehensive evaluation of the model’s multilingual mathematical capabilities.
• MathOctopus, our effective Multilingual Math Reasoning LLMs, training with different strategies, which notably outperform conventional open-source LLMs and exhibit superiority over ChatGPT in few-shot scenarios.

News

• [2024/01] 🔥 🔥 We are adding our MathInstruct-RFT data. Please patient.
• [2024/01] 🔥 🔥 We add Mistral LLM as our backbone, which achieves the SOTA results among open-source LLMs in Multilingual Math Reasoning, which achieves about 50.2 and 43.9 on MGSM and MSVAMP. See HF Checkpoint

🐙🐙MathOctopus 🐙🐙

This repo contains the code, data, and models for "Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations"

Official Website

🤗Datasets

Our collected MGSM8KInstruct training dataset and MSVAMP testset both contain 10 languages:

• Spanish
• French
• German
• Russian
• Chinese
• Japanese
• Thai
• Swahili
• Bengali
• Telugu

MGSM8KInstruct

MSVAMP

Usage

Our dataset and models are available at Huggingface!

🦑🦑🦑

🤗 MGSM8KInstruct_Parallel Dataset

🤗 MGSM8KInstruct_Cross Dataset

🤗 MSVAMP Dataset

🤗Models

*-Parallel refers to our model trained with the parallel-training strategy.

*-Cross refers to our model trained with cross-training strategy.

*-xRFT means we train the model with multilingual rejection sampling.

Results

We evaluate our models in two datasets: MGSM and MSVAMP

Overall Results on MGSM

Overall Results on MSVAMP

MathOctopus in English

Table of Contents

• ℹ Introduction
• ⚙️ Installation
• 🛠️ Training and Inference
• 📜 License
• 📖 Citation

Introduction

We introduce 🐙 MathOctopus, a series of open-source large language models (LLMs) specifically tailored for multilingual math problem-solving. The MathOctopus models are trained on 🤗 MGSM8KInstruct Dataset, encompassing ten distinct languages. MathOctopus notably outperforms conventional open-source LLMs and exhibits superiority over ChatGPT in few-shot scenarios.

Installation

Clone this repository and install the required packages:

git clone https://github.com/nuochenpku/MathOctopus.git
cd MathOctopus
pip install -r requirements.txt

Training and Inference

Data Loading

Run the following command to preprocess the data from our Instruct in the Parallel-training setting:

from datasets import load_dataset

dataset = load_dataset("https://huggingface.co/datasets/Mathoctopus/GSM8KInstruct_Parallel")

Quick Start

To play with our model, run:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Mathoctopus/Parallel_7B")

alpaca_template = "Below is an instruction that describes a task. Write a response that appropriately completes the request in {lang}. Please answer in {lang}. \n### Instruction:\n{query}\n\n### Response:"

query = "Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?"



### You can let MathOctopus output in a specific language.
language = English

input = alpaca_template.format(lang= language, query = query)

output = pipeline(input)[0]['generated_text']
print(output)

Evaluation

To replicate the experimental results of MGSM in our paper, run:

CUDA_VISIBLE_DEVICES=1 python3  generate_and_eval.py --model_path $MODEL_PATH \
    --streategy Parallel \
    --batch_size 32 \
&> $MODEL_PATH/mgsm_Parallel_testbf16.log 

To replicate the experimental results of MSVAMP in our paper, run:

CUDA_VISIBLE_DEVICES=1 python3  svamp_test.py --model_path $MODEL_PATH \
    --streategy Parallel \
    --batch_size 32 \
&> $MODEL_PATH/svamp_parallel_testbf16.log &

--strategy should be Parallel or Cross. If you want to test in a specific language, you can add --lang_only $lang.

Large-scale Evaluation

If you have four GPUs, you can get test models with parallel/cross strategy in two datasets via:

bash test_xmath.sh

Fine-tuning

To train the 7B/13B model, run:

cd step1_supervised_finetuning
bash training_scripts/single_node/run_llama2.sh

which consists of the following commands:

#!/bin/bash
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: Apache-2.0
# local/xjsonfile/rftV2
# DeepSpeed Team
OUTPUT=$1
ZERO_STAGE=$2
MODEL_PATH=$3
DATA_PATH=$4
if [ "$OUTPUT" == "" ]; then
    OUTPUT=./OUTPUT
fi
if [ "$ZERO_STAGE" == "" ]; then
    ZERO_STAGE=3
fi
mkdir -p $OUTPUT

deepspeed --include localhost:0,1,2,3 --master_port=29500 main.py  \
   --data_path $DATA_PATH \
   --data_split 10,0,0 \
   --model_name_or_path $MODEL_PATH \
   --per_device_train_batch_size 8 \
   --per_device_eval_batch_size 8 \
   --max_seq_len 512 \
   --learning_rate 2e-5  \
   --weight_decay 0. \
   --num_train_epochs 3  \
   --gradient_accumulation_steps 4 \
   --lr_scheduler_type cosine \
   --num_warmup_steps 0 \
   --seed 1234 \
   --gradient_checkpointing \
   --zero_stage $ZERO_STAGE \
   --deepspeed \
   --output_dir $OUTPUT \
   &> $OUTPUT/training.log

To train the 34B/70B model, run:

cd step1_supervised_finetuning
bash training_scripts/single_node/run_llama_30b.sh

Citation

Please cite our paper if you use our data, model or code. Please also kindly cite the original dataset papers.

@misc{chen2023breaking,
      title={Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations}, 
      author={Nuo Chen and Zinan Zheng and Ning Wu and Linjun Shou and Ming Gong and Yangqiu Song and Dongmei Zhang and Jia Li},
      year={2023},
      eprint={2310.20246},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}