Awesome
This repo contains the instructions on how to reproduce results in the paper.
Updates on March 10th, 2022.
People are requesting data pre-processing code for MATRES and TBD pickle files. The original pickles files were produced by the internal NLP annotation tools used by the Information Sciences Institute at USC. Due to contract restriction, we are not able to make those tools public. However, we made effort to replicate those files using public tools: https://github.com/rujunhan/TEDataProcessing. There are some unavoidable but minor differences between the two versions.
0. Environment
Some key packages.
- pytorch=1.6.0
- transformer=3.1.0
- cudatoolkit=10.1.243
- apex=0.1
- A detailed lisf of packages used can be found in env.yml
1. Continual Pretraining
For replication purpose, we provide several checkpoints for our pretrained models here: https://drive.google.com/drive/folders/1otj3NjBfra9bzPNWGXOvMsB7SQL4WSDr?usp=sharing
Under the pretrained_models/
folder,
- contrastive_lm_roberta-large*: final ECONET model based on RoBERTa-large.
pytorch_model_ep3.bin
is the one picked (as stated in the paper). - contrastive_lm_bert-large-uncased*: final ECONET model based on BERT-large-uncased.
pytorch_model_ep1.bin
is the one picked. - generator_lm_roberta-large*: generator-only model based on RoBERTa-large.
pytorch_model_ep1.bin
is the one picked. - random_mask_roberta-large*: model pretrained with random mask objective based on RoBERTa-large.
pytorch_model_ep0.bin
is the one picked. - random_mask_bert-large-uncased*: model pretrained with random mask objective based on BERT-large-uncased.
pytorch_model_ep0.bin
is the one picked.
If you are interested in re-training ECONET (and its variants), here are the instructions.
1.1 Pretraining Data
We also released our pre-processed pretraining data using the same Google Drive link above.
Under the data/
folder,
- Vocabulary for temporal indicators:
temporal_indicators.txt
; for events:all_vocab.json
. - Data for continual pretraining BERT-large,
samples_temporal_100K_bert.json
andsamples_event_100K_bert.json
- Data for continual pretraining RoBERTa-large,
samples_temporal_100K_roberta.json
andsamples_event_100K_roberta.json
- Data for continual pretraining with random masks,
random_mask_bert_tokens.json
andrandom_mask_roberta_tokens.json
1.2 Generator + Discriminator (ECONET)
Save all of the above data objects in the local .data/
folder.
- For BERT-large, run
./code/train_contrastive_lm_bert.sh
. Models for each epoch (25K steps) will be saved in./output/
- For RoBERTa-large, run
./code/train_contrastive_lm_roberta.sh
.
1.3 Generator Only
- We only experimented with RoBERTa-large. Run
./code/train_joint_lm.sh
.
1.4 Random Masks
- For BERT-large, run
./code/train_random_lm_bert.sh
. - For RoBERTa-large, run
./code/train_random_lm_roberta.sh
.
2. Replicating Fine-tuning Results
- We provide finetuned model objects for the reported RoBERTa-large and RoBERTa-large + ECONET results.
- Download and save them in
./output/
- Run the evaluation script below to replicate results.
2.1 TORQUE.
- Data:
https://allennlp.org/torque.html
. Download, process and save data in./data/
- Run
bash ./code/evaluate_torque.sh
, setmodel_dir
to the correct folder in./output/
- In the bash script,
pretrained=roberta-large_0
indicates finetuning original RoBERTa-large; otherwise, ECONET.
2.2 TBD / MATRES / RED
- Data: provided in
./data/
- Run
bash ./code/evaluate_ere_data.sh
, setmodel_dir' to the correct folder in
./output/` - In the bash script,
task=no_transfer
indicates finetuning RoBERTa-large;task=transfer
means RoBERTa-large + ECONET.
2.3 MCTACO
- Finetuning details can be found in 3.3 below.
- We did not save trained models, but simply save the outputs as the original code does. These outputs are names as
eval_outputs.txt
in the associated folders. - As in 2.2,
no_transfer
indicates finetuning RoBERTa-large;transfer
means finetuning RoBERTa-large + ECONET.
3. Re-run Fine-tuning
If you are interested in re-running our fine-tuing, we also provide instructions here.
3.1 TORQUE.
Fine-tuning with ECONET: run bash ./code/finetune_torque.sh
A couple parameters you will have to set,
model={roberta-large | bert-large-uncased}
(default=roberta-large)pretrained_dir
models pretrained from Step 1checkpoint
: number of epochs (0-9) in pretraining
Fine-tuning with RoBERTa-large or BERT-large: run bash ./code/finetune_torque.sh
with parameters,
model={roberta-large | bert-large-uncased}
pretrained_dir=${model}
checkpoint=0
3.2 TBD / MATRES / RED
Fine-tuning with ECONET: run bash ./code/finetune_ere_data.sh
A couple parameters you will have to set,
model={roberta-large | bert-large-uncased}
(default=roberta-large)pretrained_dir
models pretrained from Step 1checkpoints
: number of epochs (0-9) in pretrainingdata={tbd | matres | red}
(default=tbd)task=transfer
Fine-tuning with RoBERTa-large or BERT-large: run bash ./code/finetune_ere_data.sh
with parameters,
model={roberta-large | bert-large-uncased}
(default=roberta-large)checkpoints=0
data={tbd | matres | red}
(default=tbd)task=no_transfer
3.3 MCTACO
- Data and source code can be found here: https://github.com/CogComp/MCTACO
- To finetune with ECONET, simply modify the code here: https://github.com/CogComp/MCTACO/tree/master/experiments/bert by loading our pretrained models.
- We do not repeat their provided instruction here.
4. TacoLM Baseline
- The recommended pretrained model used in our paper can be found here: https://drive.google.com/drive/folders/1kx5Vc8iFYorWHrxHndkUzOssdeOm8oYC.
- You can load it to the finetuning code above by setting
pretrained_dir
to it and then run the same scripts above.