Awesome
Continual Prompt Tuning for Dialog State Tracking
This is the official code for "Continual Prompt Tuning for Dialog State Tracking" (ACL 2022).
Requirements
- transformers==4.6.1
- torch==1.8.1
- pytorch-lightning==1.2.5
Datasets
We conduct experiments on Schema-Guided Dataset. We choose 15 services from the dataset and do most experiments on them. To prepare dataset, download the dataset and put it in data/ path.
If you are interested in the pre-processing, please check utils/preprocess.py and utils/dataloader.py.
Reproduce results in our paper
Our baseline scripts are contained in baseline_scripts and our prompt-tuning scripts are in prompt_scripts.
To reproduce results in Table 1, refer to scripts:
| Methods | avg. JGA | Script Path |
|---|---|---|
| Fine-tuning | 14.3 | baseline_scripts/run_t5_vanilla.sh |
| EWC | 13.9 | baseline_scripts/run_t5_ewc.sh |
| Replay | 58.6 | baseline_scripts/run_t5_replay.sh |
| AdapterCL (20x) | 49.8 | baseline_scripts/run_t5_adapter.sh |
| AdapterCL (1x) | 30.6 | baseline_scripts/run_t5_small_adapter.sh |
| Prompt Tuning | 48.1 | prompt_scripts/run_train_pt_randinit.sh |
| Continual Prompt Tuning | 59.5 | prompt_scripts/run_train_cpt_augq.sh |
| w/ memory | 60.7 | prompt_scripts/run_train_cpt_w_mem_augq.sh |
| w/ memory & backward | 61.2 | prompt_scripts/run_train_cpt_w_mem_augq_bwd.sh |
| Multi-Task Prompt Tuning | 64.0 | prompt_scripts/run_multiprompt.sh |
To reproduce results in Table 2, refer to scripts:
| Techniques | avg. JGA | Script Path |
|---|---|---|
| - | 29.6 | prompt_scripts/run_train_noMSR.sh |
| CLInit | 41.8 | prompt_scripts/run_train_noMSR_cl.sh |
| MSR | 48.1 | prompt_scripts/run_train_pt_randinit.sh |
| MSR+CLInit | 57.6 | prompt_scripts/run_train_cpt.sh |
| MSR+CLInit+QF | 59.5 | prompt_scripts/run_train_cpt_augq.sh |
| MSR+CLInit+MR | 60.4 | prompt_scripts/run_train_cpt_w_mem.sh |
| MSR+CLInit+QF+MR | 60.7 | prompt_scripts/run_train_cpt_w_mem_augq.sh |
To reproduce results in Table 3, refer to scripts:
| Initialization | avg. JGA on 5 seeds | Script Path |
|---|---|---|
| RandomInit | 48.1 | prompt_scripts/run_train_pt_randinit.sh |
| SelectInit | 54.5 | prompt_scripts/run_train_pt_selectinit.sh |
| CLInit | 57.6 | prompt_scripts/run_train_cpt.sh |
To reproduce results in Table 4, refer to prompt_scripts/run_train_cpt.sh and set --dataset_order=30/1/31/32.
To reproduce results in Figure 3 and Table 5, refer to prompt_scripts/run_train_cpt_augq.sh and set --model_name_or_path to paths to your t5-base or t5-large pre-trained weights. We use RTX-2080 GPU with 11GB RAM. In our experiments, we choose 4 batch size and 4 accumulation steps for t5-base
and 2 batch size, 8 accumulation steps for t5-large. We use fp32 training for both of them.
To reproduce results in Table 6 and Table 7, set the MEM_SIZE accordingly. To make domain's memory size proportional to its training data size, use --no_memory_uniform_dist flag.
Evaluation
For evaluation for non-prompt experiments, refer to gather_res_baseline.py
For evaluation for prompt-tuning experiments, refer to gather_res_prompt.py
For FWT calculation for non-prompt experiments, refer to gather_res_baseline_fwt.py
For FWT calculation for prompt-tuning experiments, refer to gather_res_prompt_fwt.py
Citation
@inproceedings{zhu-etal-2022-cpt4dst,
title = "Continual Prompt Tuning for Dialog State Tracking",
author = "Zhu, Qi and Li, Bing and Mi, Fei and Zhu, Xiaoyan and Huang, Minlie",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics",
publisher = "Association for Computational Linguistics",
}