Awesome

DecT

Source code for ACL 2023 paper Decoder Tuning

Installation

Our code is based on PyTorch, HuggingFace Transformers, and OpenPrompt, please install dependencies by

pip install -r requirements.txt

Download Datasets

Download the 10 datasets with the following scripts

cd datasets
bash download_datasets.sh
cd ..

Run DecT

Then you can run DecT by running run_dect.py, for example

python src/run_dect.py \
	--model roberta \
	--size large \
	--type mlm \
	--model_name_or_path roberta-large \
	--shot 1 \
	--dataset sst2 \
	--proto_dim 128 \
	--model_logits_weight 1 \

In run_dect.py we provide instructions for each argument. To reproduce the results in paper, please run the following combinations

python src/run_dect.py \
	--shot [1, 4, 16] \
	--dataset [sst2, imdb, yelp, agnews, dbpedia, yahoo, rte, snli, mnli-m, mnli-mm, fewnerd] \
	--seed [0, 1, 2, 3, 4] \

Configure Models

You can configure different models by setting model, type, size, model_name_or_path parameters.

• model: Model name. We now support plms in OpenPrompt, LLaMA, Alpaca and Vicuna.
• type: mlm, lm or chat. This will determine the prompt template. For lm type models, we put the [mask] token at the end of the template. For chat models, we implement the chat template for Vicuna v1.1. You may change the template if you use other models.
• size: Model size. Currently, it is used to set the hidden state dimension for LLaMA models.
• model_name_or_path: Path to model weights.

You can also modify the load_model function in src/run_dect.py to support more models!