Home

Awesome

Polbert - Polish BERT

Polish version of BERT language model is here! It is now available in two variants: cased and uncased, both can be downloaded and used via HuggingFace transformers library. I recommend using the cased model, more info on the differences and benchmark results below.

PolBERT image

Cased and uncased variants

Pre-training corpora

Below is the list of corpora used along with the output of wc command (counting lines, words and characters). These corpora were divided into sentences with srxsegmenter (see references), concatenated and tokenized with HuggingFace BERT Tokenizer.

Uncased

TablesLinesWordsCharacters
Polish subset of Open Subtitles23663540814311996017628097730
Polish subset of ParaCrawl84709501766708851163505275
Polish Parliamentary Corpus9799859121154785938896963
Polish Wikipedia - Feb 202080142061320679861015849191
Total262920423186109325710746349159

Cased

TablesLinesWordsCharacters
Polish subset of Open Subtitles (Deduplicated) 419989422135906561424873235
Polish subset of ParaCrawl84709501766708851163505275
Polish Parliamentary Corpus9799859121154785938896963
Polish Wikipedia - Feb 202080142061320679861015849191
Total682839606464791974543124667

Pre-training details

Uncased

Cased

Usage

Polbert is released via HuggingFace Transformers library.

For an example use as language model, see this notebook file.

Uncased

from transformers import *
model = BertForMaskedLM.from_pretrained("dkleczek/bert-base-polish-uncased-v1")
tokenizer = BertTokenizer.from_pretrained("dkleczek/bert-base-polish-uncased-v1")
nlp = pipeline('fill-mask', model=model, tokenizer=tokenizer)
for pred in nlp(f"Adam Mickiewicz wielkim polskim {nlp.tokenizer.mask_token} był."):
  print(pred)
# Output:
# {'sequence': '[CLS] adam mickiewicz wielkim polskim poeta był. [SEP]', 'score': 0.47196975350379944, 'token': 26596}
# {'sequence': '[CLS] adam mickiewicz wielkim polskim bohaterem był. [SEP]', 'score': 0.09127858281135559, 'token': 10953}
# {'sequence': '[CLS] adam mickiewicz wielkim polskim człowiekiem był. [SEP]', 'score': 0.0647173821926117, 'token': 5182}
# {'sequence': '[CLS] adam mickiewicz wielkim polskim pisarzem był. [SEP]', 'score': 0.05232388526201248, 'token': 24293}
# {'sequence': '[CLS] adam mickiewicz wielkim polskim politykiem był. [SEP]', 'score': 0.04554257541894913, 'token': 44095}

Cased

model = BertForMaskedLM.from_pretrained("dkleczek/bert-base-polish-cased-v1")
tokenizer = BertTokenizer.from_pretrained("dkleczek/bert-base-polish-cased-v1")
nlp = pipeline('fill-mask', model=model, tokenizer=tokenizer)
for pred in nlp(f"Adam Mickiewicz wielkim polskim {nlp.tokenizer.mask_token} był."):
  print(pred)
# Output:
# {'sequence': '[CLS] Adam Mickiewicz wielkim polskim pisarzem był. [SEP]', 'score': 0.5391148328781128, 'token': 37120}
# {'sequence': '[CLS] Adam Mickiewicz wielkim polskim człowiekiem był. [SEP]', 'score': 0.11683262139558792, 'token': 6810}
# {'sequence': '[CLS] Adam Mickiewicz wielkim polskim bohaterem był. [SEP]', 'score': 0.06021466106176376, 'token': 17709}
# {'sequence': '[CLS] Adam Mickiewicz wielkim polskim mistrzem był. [SEP]', 'score': 0.051870670169591904, 'token': 14652}
# {'sequence': '[CLS] Adam Mickiewicz wielkim polskim artystą był. [SEP]', 'score': 0.031787533313035965, 'token': 35680}

See the next section for an example usage of Polbert in downstream tasks.

Evaluation

Thanks to Allegro, we now have the KLEJ benchmark, a set of nine evaluation tasks for the Polish language understanding. The following results are achieved by running standard set of evaluation scripts (no tricks!) utilizing both cased and uncased variants of Polbert.

ModelAverageNKJP-NERCDSC-ECDSC-RCBDPolEmo2.0-INPolEmo2.0-OUTDYKPSCAR
Polbert cased81.793.693.493.852.787.471.159.198.685.2
Polbert uncased81.490.193.993.555.088.168.859.498.885.4

Note how the uncased model performs better than cased on some tasks? My guess this is because of the oversampling of Open Subtitles dataset and its similarity to data in some of these tasks. All these benchmark tasks are sequence classification, so the relative strength of the cased model is not so visible here.

Bias

The data used to train the model is biased. It may reflect stereotypes related to gender, ethnicity etc. Please be careful when using the model for downstream task to consider these biases and mitigate them.

Acknowledgements

Author

Darek Kłeczek - contact me on Twitter @dk21

Citation

@inproceedings{Kleczek2020,
  author = {Dariusz Kłeczek},
  title = {Polbert: Attacking Polish NLP Tasks with Transformers},
  booktitle = {Proceedings of the PolEval 2020 Workshop},
  year = {2020},
  editor = {Maciej Ogrodniczuk and Łukasz Kobyliński},
  publisher = {Institute of Computer Science, Polish Academy of Sciences},
 }

References