Home

Awesome

AcTune

This is the code repo for our paper `AcTune: Uncertainty-Based Active Self-Training for Active Fine-Tuning of Pretrained Language Models' (In Proceedings of NAACL 2022 Main Conference, Oral Presentation).

Requirements

python 3.7
transformers==4.2.0
pytorch==1.6.0
tqdm
scikit-learn
faiss-cpu==1.6.4

Datasets

Main Experiments

We use the following four datasets for the main experiments.

DatasetTaskNumber of ClassesNumber of Train/Test
SST-2Sentiment260.6k/1.8k
AG NewsNews Topic4119k/7.6k
Pubmed-RCTMedical Abstract5180k/30.1k
DBPediaOntology Topic14280k/70k

The processed data can be found at this link. The folder to put these datasets will be discribed in the following parts.

Weak Supervision Experiments

Most of the dataset are from the WRENCH benchmark. Please checkout their repo for dataset details.

Training

Please use the commands in commands folder for experiments. Take AG News dataset as an example, run_agnews_finetune.sh is used for running the experiment of standard active learning approaches, and run_agnews.sh is used for running active self-training experiments as unlabeled data is also used during fine-tuning.

Here, we suppose there is a folder for storing datasets as in ../datasets/, and a folder for logging the experiment results as in ../exp.

Hyperparameter Tuning

The key hyperparameter for our approach includes pool, pool_min, gamma, gamma_min, self_training_weight sample_per_group and n_centroids.

Citation

Please cite the following paper if you are using our datasets/tool. Thanks!

@inproceedings{yu2022actune,
    title = "{A}c{T}une: Uncertainty-Based Active Self-Training for Active Fine-Tuning of Pretrained Language Models",
    author = "Yu, Yue and Kong, Lingkai and Zhang, Jieyu and Zhang, Rongzhi and Zhang, Chao",
    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.naacl-main.102",
    pages = "1422--1436",
}