Awesome

Probabilistic Transformer

The code base for project Probabilistic Transformer, a model of contextual word representation from a syntactic and probabilistic perspective. The paper "Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation" was accepted to ACL2023 Findings.

Warning
In this git branch, the codes are developed in a way that is easy to integrate with all kinds of modules, but not well-optimized for speed. The repo structure is a bit messy and the framework it uses (flair) is outdated.

Preparation

Code Environment

To prepare the code environment, use

cd src
pip install -r requirements.txt

Due to package compatibility, it will install pytorch with version 1.7.1. Feel free to upgrade it with the command:

pip install torch==1.10.2

Or this command:

pip install --upgrade torch

This work is developed under torch==1.10.2.

Dataset

Our code will automatically download the dataset if it finds the dataset you want to use is missing. Some datasets require license/purchase, and the code would throw an error telling you where to download the dataset. We also provide detailed instructions in the template config file and doc strings.

How to run

Training

Simply run the following commands:

cd src
python train.py

By default, it will use the config file src/config.toml. To use other config files, use -c or --config to specify the configuration file:

cd src
python train.py -c ../path/to/config.toml

Prediction / Inference

To do inference on a sentence, run python predict.py. The usage is exactly the same with training, just use a config that has just been used for training before. To modify the sentence for inference, please modify codes in predict.py.

Drawing Dependency Parse Trees

To visualize the dependency parse trees produced by our models, run python draw.py. The usage is the same as inference. It will generate dep_graph.tex in your working directory. You may compile the latex file and get the figures in PDF.

There are 3 options at the top of the file draw.py:

• SENTENCE: The sentence for dependency parsing. It will be tokenized with a white space tokenizer.
• ALGORITHM: The algorithm for generating dependency parse trees. Options: argmax, nonprojective, projective.
  • argmax: Use the most probable head for each token. It doesn't care whether the generated parse tree is connected or not.
  • nonprojective: Parse with the Chu-Liu-Edmonds algorithm. The produced parse tree is valid but not necessarily projective. If ROOT is not considered in the model ([CLS] could be seen as a counterpart of ROOT), then all scores for ROOT will be zero.
  • projective: Parse with the Eisner algorithm. The produced parse tree is projective. If ROOT is not considered in the model ([CLS] could be seen as a counterpart of ROOT), then all scores for ROOT will be zero.
• MODE: Whether to combine results in different heads (transformers) / channels (probabilistic transformers). Options: all, average:
  • all: Draw parse trees for each layer/iteration and each head/channel.
  • average: Draw parse trees for each layer/iteration and use the average score in different heads/channels as the score in this layer/iteration.

If we take the attention scores in transformers as the dependency edge scores, then we may also draw dependency parse trees from transformers.

Evaluation for Unsupervised Dependency Parsing Task

To do unsupervised dependency parsing, run python evaluate.py. The usage is the same as drawing. It will print the UAS (Unlabeled Attachment Score) to the console.

There are 4 options at the top of the file draw.py:

• TEST_FILE: Path to conll-style file as the dependency parsing test set. None for UD-English ewt test set. Type: None or str.
• ALGORITHM: Same as drawing.
• MODE: How to do evaluation.
  • average: Use the average of all channels' probabilities as the score matrix.
  • hit: Each channel produce one parse tree. So each word has multiple head options. If any one hits the gold answer, then we take it as correct.
  • best: Each channel produce one parse tree. We evaluate them seperately, then choose the best channel's result as final result.
  • left: All left arcs.
  • right: All right arcs.
  • random: Random heads.
• ITERATION: Use the dependency head distribution from which iteration. Use numbers 1, 2, ... or -1 for the last iteration.

Result

We provide the config files in configs/best. To reproduce the results, please use the following command

cd src
python train.py -c ../configs/best/<CONFIG_FILE>

where <CONFIG_FILE> should be replaced by the config file in the tables below.

Note
Part of the results presented below was not contained in our paper.

Probabilistic Transformer

Transformer

Universal Transformer

_{<i>* "Universal Transformer" only means weight sharing between layers in transformers. See details in Ontanón et al. (2021).</i>}
_{<i>** The training speed and time are for reference only. The speed data is randomly picked during the training and the product of speed and time is not equal to the number of samples.</i>}
_{<i>*** The random seeds for the 5 runs are: 0, 1, 2, 3, 4.</i>}

Questions

1. I am working on a cluster where the compute node does not have Internet, so I cannot download the dataset before training. What should I do?

That is simple. Go to src/train.py and add exit(0) before training (line 105). Execute the training command in the login node (where you have access to the Internet). It will download the dataset without training the model. Finally, remove the line of code you added and train the model in the compute node.

2. Which type of positional encoding do you use for transformers?

We use absolute positional encoding for transformers in our experiments. Though the computation graph of probabilistic transformers is closer to that of transformers with relative positional encoding, we empirically find that positional encoding hardly makes any difference to the performance of transformers.

3. Why not test on the GLUE dataset?

GLUE is a standard benchmark for language understanding, and most recent works with strong pre-trained word representations choose to test their models on this dataset. Our work does not involve pre-training, which indicates a weak ability for language understanding. To better evaluate the ability of word representation for our model, we think it might be more suitable to compare our model with a vanilla transformer on MLM and POS tagging tasks than GLUE.

4. How strong is your baseline?

To make sure our baseline (transformer) implementation is strong enough, part of our experiments use the same setting as previous works:

• Our experiment for task MLM, dataset PTB has the same setting as that of StructFormer.
• Our experiment for task MLM, dataset BLLIP-XS has the same setting as that of UDGN, though they did not conduct experiments on this split. The reason we do not follow StructFormer is that the code for this dataset is not open-sourced.
• Our experiment for task SYN, dataset COGS has the same setting as that of Compositional.
• Our experiment for task SYN, dataset CFQ has the same setting as that of Edge Transformer.

5. I have trouble understanding / running the code. Could you help me with it?

Sure. Welcome to create an issue or email me at wuhy1@shanghaitech.edu.cn.