Awesome

This is a repository to help all readers who are interested in the data assessment and selection methods for Instruction Tuning of LLMs.

Most recent studies in 2023 and 2024 are investigated under the present survey. If your papers are missing or you have other requests, please contact to yuleichin@126.com. We will update this repository and paper on a regular basis to maintain up-to-date.

News📰

• Aug. 2024: 🔥🔥🔥Come and check out our survey paper for the latest updates on data evaluation and selection! [arxiv version]

Citation🎓

@article{qin2024unleashingpowerdatatsunami,
     title={Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models}, 
     author={Yulei Qin and Yuncheng Yang and Pengcheng Guo and Gang Li and Hang Shao and Yuchen Shi and Zihan Xu and Yun Gu and Ke Li and Xing Sun},
     year={2024},
     url={https://arxiv.org/abs/2408.02085},
}

Papers📑

Overview📊

We present a unified organization of existing researches and categorize them in terms of the dimensionality of data assessment and selection.

List of Papers with Categorization

All Papers are sorted chronologically according to three categories above, so that you can find related papers more quickly.

[Index: Quality-based Selection, Diversity-based Selection, Importance-based Selection]

Comprehensive Data Assessment and Selection
|--- A. Quality-based Selection
     |--- A.1. Hand-crafted Indicators: Vocabulary; Linguistics; Lexical and Semantic Analysis.
     |--- A.2. Model-based Indicators: Perplexity; Learning Complexity; Reward Score; Error Norm; Memorization; Instruction Following Difficulty; Predictability; Uncertainty.
     |--- A.3. GPT Score: Direct Scoring; Pairwise Ranking; Justification.
     |--- A.4. Human Evaluation: Instruction and Response Annotation; Direct Scoring; Pairwise Ranking.
|--- B. Diversity-based Selection
     |--- B.1. Hand-crafted Indicators: Lexical and Semantic Diversity; Statistics.
     |--- B.2. Model-based Indicators: Entropy; Simpson's Index; Vendi Score; Fisher Information Matrix; Tagging.
     |--- B.3. Geometry-based Coreset Sampling: __K__-center Greedy; Herding; Clustering.
     |--- B.4. Bilevel Optimization-based Coreset Sampling: Retrieve; Glister; Consistency Regularization; Entropy Regularization; Refined Minimal Size.
|--- C. Importance-based Selection
     |--- C.1. Hand-crafted Indicators: Readability Indices; Education-level Difficulty.
     |--- C.2. Model-based Indicators: Uncertainty; Reward Score; Datamodels.
     |--- C.3. Loss and Error-based Coreset Sampling: Forgetting; Memorization; Influence by Loss.
     |--- C.4. Gradient-based Coreset Sampling: Gradient Matching; Influence by Gradient.

Other Topics related to Data Curation in Training Language Models

We also provide list of papers for practical data curation in language modeling, ranging from dataset construction, dataset deduplication, and data mixture.

[Index: Dataset Construction and Synthesis, Dataset Deduplication, Dataset Mixture]

Tools

Some frequently used tools are collected for efficient processing of NLP corpus.

[Index: Tools]

Related Work

We provide related surveys on data measurement and selection for reference.

[Index: Surveys]

<a name="A"></a>

A. Quality-based Selection🏋️

A.1 Hand-crafted Indicators

A.2 Model-based Indicators

A.3. GPT Score

A.4. Human Evaluation

<a name="B"></a>

B. Diversity-based Selection

B.1. Hand-crafted Indicators

B.2. Model-based Indicators

B.3. Geometry-based Coreset Sampling

B.4. Bilevel Optimization-based Coreset Sampling

<a name="C"></a>

C. Importance-based Selection🏗️

C.1. Hand-crafted Indicators

C.2. Model-based Indicators

C.3. Loss and Error-based Coreset Sampling

C.4. Gradient-based Coreset Sampling

<a name="D"></a>

D. Dataset Construction and Synthesis🚧

Self-Instruct🤳

• Self-instruct: Aligning language models with self-generated instructions - One most effective way to expand dataset is to bootstrap off generations from large language models themselves. It uses input-first or output-first approach to generate new instructions and filters out low-quality or highly duplicated/similar ones.

• Muffin: Curating multi-faceted instructions for improving instruction following - Three paradigms for expanding the instruction datasets: 1) scaling inputs; 2) scaling input-free tasks; and 3) scaling tasks per input. To choose appropriate input samples, input contents are selected from different domains with controled similar frequency. The entire scaling pipeline includes facets recognition, instruction brainstorm, instruction rematching, and output annotation.

• Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models - The two-stage instruction-followng data synthesis method is composed of: 1) instruction augmentation and verification and 2) query augmentation and verification. In the first step, human-verified seed instructions are provided for the supervisor model (e.g., larger powerful LLMs) to perform self-instruct. The augmented instructions are then used to generate their verification python codes and test cases, where only high quality, excutable triplets (test cases, instructions, verification functions) are kept via filtering (e.g., back translation check). In the second step, real-world queries are randomly selected from ShareGPT dataset and concatenated with the generated instructions as complexity-augmented instructions. These instructions are scored by a LLM to filter out contradictory ones (e.g., query imcompatible with instruction). Only valid instructions are sent to the supervisory model for response generation and verification. Correct, high quality responses can be used for supervised fine-tuning and their invalid counterparts can be employed as negative preferences in DPO alignment.

Evo-Instruct🐛

• Wizardlm: Empowering large language models to follow complex instructions - The evolution method to increase the complexity of existing instruction datasets is to add more variation (e.g., deepening, reasoning, concretizing, constraints, in-breadth extension, and complicated inputs) to the original samples, where the differences of the responses with respect to the nuances of inputs benefit the model's ability of following instructions.

• Tree-Instruct: A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment - The complexity of instructions can be represented as the number and diversity of nodes of their semantic trees. Therefore, the pipeline of improving instruction complexity includes: 1) tree construction; 2) nodes expansion; and 3) tree sentencization (remapping from tree to instruction).

• Infiniy Instruct - The infinity instruct method first collects high-quality open instruction datasets and perform filtering and tagging to give detailed description about each instruction (e.g., domain, knowledge prerequisite, abilities, fine-grained categories). Based on these attributes, high-quality instructions are chosen as seeds for expansion via Evo-Instruct. Finally, for each evolved instruction, multi-rounds conversations are generated.

<a name="E"></a>

E. Dataset Deduplication🦄

Exact Match🟰

• URL normalization for de-duplication of web pages - URL removel for efficient data-deduplication.

• CCNet: Extracting high quality monolingual datasets from web crawl data - Hash code of each document is calculated and compared for deduplication.

• Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures - A non-collision resistant hashing algorithm isdeveloped to remove duplidates.

• MinHash MinHashLSH SimHash - Efficient approximation methods of the hashing algorithm are devloped to handle massive corpora.

Semantics🔤

• Semdedup: Data-efficient learning at web-scale through semantic deduplication - The removal of redundant data pairs that are semantically similar but not identical can speed up training but also preserve performance.
• D4: Improving llm pretraining via document de-duplication and diversification - The pruning of datasets can be simply achieved by SemDeDup (semantic deduplication) and prototypicality filtering. Such dedupped and diversified subsets improve downstreaming performance even with repeating training epochs.

<a name="F"></a>

F. Dataset Mixture🥄

• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts - The mixture weights of datasets from different domains are set according to the performance of a small model trained independently on each component dataset. Small-yet-high-quality datasets such as Wikipedia are not over-sampled.

• SlimPajama-DC: Understanding Data Combinations for LLM Training - The proportions of different domains in the deduplicated SlimPajama/RefinedWeb dataset are studied with 7 different configurations where their total number of tokens remains the same. The results of 7 configurations in terms of both downstream metrics and losses suggests appropriate weights for datasets from different sources.

• Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance - It discovers the quantitative predictability of model performance regarding data mixtures, and formulates it as the data mixing laws. It proposes a pipeline to predict model performance on different mixture proportions through nested use of the scaling laws of training steps, model sizes, and data mixing laws.

• RegMix: Data Mixture as Regression for Language Model Pre-training - The paper proposes the RegMix method to automatically identify an effective data mixture for language model pre-training by formulating it as a regression task. It involves training small models with diverse data mixtures and fitting a regression model to predict their performance. The top-ranked mixture is then used to train a large-scale model. The experiments show that RegMix is superior to human selection, achieves comparable or better results than DoReMi with only 10% of the compute budget. It also finds that data mixtures significantly impact performance, web corpora have a stronger correlation with downstream performance, domains interact complexly, and the data mixture effects transcend scaling laws.

• Data Mixing Made Efficient: A Bivariate Scaling Law for Language Model Pretraining - BiMix adopts the bivariate scaling law to reveal the impact of data quantity and mixing proportions using entropy proxies. The scaling behavior is disentangled into two dimensions: 1) scaling step with fixed proportions, and 2) scaling proportion at fixed steps.

• How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition - Different skills require different strategies of supervised fine-tuning. Sequential training results in catastrophic forgetting and the proposed dual-stage mixed fine-tuning strategy alleviates such forgetting and keeps the general capabilities simultaneously. Specifically, the specialized domain datasets are trained in the first stage and then mixed with general datasets for fine-tuning in the second stage.

• Doremi: Optimizing data mixtures speeds up language model pretraining - The mixture of datasets across diverse domains can be determined by a small proxy model (e.g., 280M params) using group distributionally robust optimization. The weights of datasets are used for resampling the pretrained datasets for training a much larger model (e.g., 8B) with lower perplexity across all domains.

• Codegen2: Lessons for training llms on programming and natural languages - One important lesson in organizing coding datasets for pretraining is to permute a portion of sequences for causal LLMs to get the code infilling ability. However, such permutation is not for free, where the performance (with a mixture loss of causal language modeling and prefix/suffix/middle sequence re-ordering) drops compared with the vanilla method (only with the causal language modeling loss).

<a name="G"></a>

G. Tools🛠️

• FastText Classifier - Bag of Tricks for Efficient Text Classification (can be used for topic/domain/quality classification).

• FASTTEXT.ZIP - Compressed FastText classification models.

• An Open Dataset of High-Quality Mathematical Web Text - A fastText-based classifier that evaluates the MathScore of any given contents for selection of mathematic corpus.

<a name="H"></a>

H. Surveys📝

• A survey on data selection for language models - A systematic overview for data pipeline construction of language models. Any selection method, either via distribution matching or diversification, can be composed of: 1) utility function; 2) selection mechanism. During different stages of the pipeline (e.g., language filtering, data quality, domain knowledge, deduplication, toxic and explicit content removal, and data mixing), the selection method should be adjusted according to different selection objectives.

• A Survey on Data Selection for LLM Instruction Tuning - Existing methods on collectiong instruction tuning datasets include: 1) reformulating the discriminative NLP datasets into generative ones; 2) self-instruct with seed prompts; 3) prompt mapping and evol-instruct. Popular methods on dataset selection can be simply classified as: 1) system of indicators; 2) trainable LLMs; 3) powerful LLMs; and 4) small models. The validation of different selection methods is often performed via direct scoring by GPT4 and human evaluation.

• Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models - This study presents a three-stage scheme for data selection: 1) data preprocess, 2) data selector construction, and 3) data selector evaluation. It finds that the targeted method with data-specific and model-specific quality labels have higher efficiency while irrelevant noisy information should be avoided for developing selection algorithms.

• The devil is in the details: A deep dive into the rabbit hole of data filtering - The pruning is conducted mainly via distribution alignment (between pretraining multi-modal datapoints and downstream datapoints): cluster-importance resampling; quality scoring-based reweighting and sampling; semantic deduplication (thresholding); enhancement on selecting samples from specific domains (e.g., digits).

• Deepcore: A comprehensive library for coreset selection in deep learning - Task-agnostic data sampling (coreset selection) methods include: 1) geometry-based methods (e.g., herding, kcenter-greedy); 2) uncertainty-based methods (e.g., least confidence/entropy/margin); 3) error/loss-based methods (forgetting; GraND/EL2N; importance resampling); 4) decision boundary-based (adversarial deepfool; contrastive active learning); 5) gradient matching-based (gradient approximation towards full set); 6) bi-level optimization-based (inner loop of model optimization and outer loop of datapoint selection); 7) sub-modularity-based (e.g., graph cut; facility location); 8) proxy-based (preference of a small model on data selection).

:man_astronaut: Show your support☕️

Give a ⭐️ if this project helped you!