Home

Awesome

LLM Decontaminator

| Paper | Blog |

<img src="./assets/overview.png" alt="img" width="800"/>

In this package, you can use LLM decontaminator to quantify a dataset's rephrased samples relative to a benchmark. Based on the detection results, you can estimate the contamination of rephrased samples in the dataset and remove them from the training set.

Contents

Install

git clone https://github.com/lm-sys/llm-decontaminator.git
cd llm-decontaminator
conda create -n llm-detect python=3.9 -y
conda activate llm-detect
pip install -r requirement.txt

Detect

Pre-Process

Please process the train set and test set into a jsonl format, with each line containing {"text": data}

import json
from datasets import load_dataset

# Load dataset
dataset = load_dataset('bigcode/starcoderdata', data_dir="python", split="train", streaming=True)

# Extract up to 500,000 samples
subset_size = 500000
codes = [sample['content'] for _, sample in zip(range(subset_size), dataset)]

# Write to file
with open("starcoderdata.jsonl", "w") as fout:
    for code in codes:
        fout.write(json.dumps({"text": code}) + "\n")

End2End

# export OPENAI_API_KEY=sk-xxx
# run llm-decontaminator
python3 main.py --train_path ./data/train/CodeAlpaca-20k.jsonl \
    --test_path ./data/test/HumanEval.jsonl \
    --output_path ./data/database/CodeAlpacaDB.jsonl \
    --data-type code \
    --top_k 1

Contamination in Real-world Dataset

Training SetBenchmarkTrain Set SizeTest Set SizeRephrased SamplesPercentage (%)
The Stack (4G subset)HumanEval500k1643118.9
StarCoder-Data (2.4G subset)HumanEval500k1642615.9
CodeExercise-PythonHumanEval27k1642615.9
CodeAlpacaHumanEval20k1642112.8
RedPajama-Data-1T (16G subset)HumanEval1625k164148.5
Evol-Instruct-CodeHumanEval78.3k164137.9
rossetacodeHumanEval4.26k16442.4
MATHInstruct (before Sep 30)MATH Test262k500076915.
MATH TrainMATH Test7.5k5000791.6
FLAN CoTMMLU184k14042760.5
WizardLM-Evol-InstructMMLU143k14042750.5

Dataset and Training Code

Reproduce Llama-rephraser with this document.

F1 Score

Reproduce paper's Table 5 & 6

# MMLU
python3 f1score/mmlu/f1_emb.py
python3 f1score/mmlu/f1_llm.py

# HumanEval
python3 f1score/humaneval/f1_emb.py
python3 f1score/humaneval/f1_llm.py

Table 5:

<img src="./assets/MMLU-f1score.png" alt="img" width="800"/>

Table 6:

<img src="./assets/HumanEval-f1score.png" alt="img" width="400"/>

Citation

Please cite the following paper if you find the code or datasets helpful.

@misc{yang2023rethinking,
      title={Rethinking Benchmark and Contamination for Language Models with Rephrased Samples}, 
      author={Shuo Yang and Wei-Lin Chiang and Lianmin Zheng and Joseph E. Gonzalez and Ion Stoica},
      year={2023},
      eprint={2311.04850},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}