Awesome
BERT and ROBERTA Codebase
This codebase is designed for fine-tuning pre-trained BERT and ROBERTA models on MR, AG NEWS, and SST2 datasets.
Quick Start
Workspace
- Main Workspace Directory:
/adv/
- Main Python File:
/adv/high_level_at.py
- Bash Scripts for Testing:
/adv/bash_scripts/
Dependencies
One crucial modification we made is to the trainer.py
file found in:
adv/training_utils/trainer_branch.py
Environment
- Python Version: 3.8.5
- Conda Environment Export:
SemRoDe.yaml
We also have the individual requirement list in requirements.txt
Install with requirements.txt or yml
Installation (requirements.txt)
Create a new conda environment with python 3.8.5 and then:
pip install -r requirements.txt
Installation (YML)
conda env create -f SemRoDe.yml --name SemRoDe
To accommodate some changes, you'll need to install our version of TextAttack and transformers locally:
pip install -e src/TextAttack/
pip install -e src/transformers/
In case you want to run the DSRM baseline you'll also need to install higher locally
pip install -e higher
Running Tests
A straightforward example test is as follows:
CUDA_VISIBLE_DEVICES=0 python ./adv/high_level_at.py --attack_train TextFooler --attack_evaluate TextFooler --dataset 'MR' --model 'BERT' --method 'MMD' --method_type None --method_val 1 --save_space 'mmd_test' --GPU '1.1' --frozen 'True' --eval_method 'epoch' --epochs 7 --online_epochs 7 --batch_size 64 --data_ratio 0.1 --debug;
This command will initiate the MMD with counter-fitted embeddings, where only 10% (--data_ratio 0.1
) of the dataset is utilized to generate the adversarial dataset. The generated adversarial data will be saved in /adv/caches/
.
The process includes training for 7 epochs on the BERT model using the MR dataset. After training, the saved weights from each epoch are evaluated using the TextFooler attack. To adjust the evaluation to occur only at the last epoch, change --eval_method 'epoch'
to --eval_method 'last'
.
Results
- The results are stored in:
/adv/LOCAL_Test/GLUE/MR/BERT/MMD_Test
- If
--eval_method 'last'
is used, performance metrics by TextFooler for the last epoch are stored in the fileR.txt
. - If
--eval_method 'epoch'
is selected, you'll find the results stored asE1.txt
,E2.txt
, etc., representing each epoch.
Extra notes
Most tests were conducted on 1 Nvidia V100 with 32GB of memory.
TODO
Hyper parameter to increase, decrease parallelism, at the moment it's set up to generate the adversarial samples on 1 GPU with 8 model instances in parallel, some users may run out of memory.