Home

Awesome

What does BERT learn about the structure of language?

Code used in our ACL'19 paper for interpreting BERT model.

Dependencies

Quick Start

Phrasal Syntax (Section 3 in paper)

cd chunking/
wget https://www.clips.uantwerpen.be/conll2000/chunking/train.txt.gz
gunzip train.txt.gz

The last command replaces train.txt.gz file with train.txt file.

python extract_features.py --train_file train.txt --output_file chunking_rep.json
python visualize.py --feat_file chunking_rep.json --output_file_prefix tsne_layer_

This would create one t-SNE plot for each BERT layer and stores as pdf (e.g. tsne_layer_0.pdf).

python cluster.py --feat_file chunking_rep.json

Probing Tasks (Section 4)

cd probing/
python extract_features.py --data_file tree_depth.txt --output_file tree_depth_rep.json

In the above command, append --untrained_bert flag to extract untrained BERT features.

python classifier.py --labels_file tree_depth.txt --feats_file tree_depth_rep.json --layer 0

We use the hyperparameter search space recommended by SentEval.

Subject-Verb Agreement (SVA) (Section 5)

cd sva/
python extract_features.py --data_file agr_50_mostcommon_10K.tsv --output_folder ./
python classifier.py --input_folder ./ --layer 0

We use the hyperparameter search space recommended by SentEval.

Compositional Structure (Section 6)

cd tpdn/
python extract_features.py --input_folder . --output_folder .
python approx.py --input_folder . --output_folder . --layer 0

Check --role_scheme and --rand_tree flags for setting the role scheme.

python induce_dep_trees.py --sentence text "The keys to the cabinet are on the table" --head_id 11 --layer_id 2 --sentence_root 6 

Acknowledgements

This repository would not be possible without the efforts of the creators/maintainers of the following libraries:

License

This repository is GPL-licensed.