Home

Awesome

🇹🇷 BERTurk

<p align="center"> <img alt="Logo provided by Merve Noyan" title="Awesome logo from Merve Noyan" src="https://raw.githubusercontent.com/stefan-it/turkish-bert/master/merve_logo.png"> </p>

DOI

We present community-driven BERT, DistilBERT, ELECTRA and ConvBERT models for Turkish 🎉

Some datasets used for pretraining and evaluation are contributed from the awesome Turkish NLP community, as well as the decision for the BERT model name: BERTurk.

Logo is provided by Merve Noyan.

Changelog

Pretraining Corpora Stats

The current version of the model is trained on a filtered and sentence segmented version of the Turkish OSCAR corpus, a recent Wikipedia dump, various OPUS corpora and a special corpus provided by Kemal Oflazer.

The final training corpus has a size of 35GB and 4,404,976,662 tokens.

Thanks to Google's TensorFlow Research Cloud (TFRC) we can train both cased and uncased models on a TPU v3-8. You can find the TensorBoard outputs for the training here:

We also provide cased and uncased models that aŕe using a larger vocab size (128k instead of 32k).

A detailed cheatsheet of how the models were trained, can be found here.

C4 Multilingual dataset (mC4)

We've also trained an ELECTRA (cased) model on the recently released Turkish part of the multiligual C4 (mC4) corpus from the AI2 team.

After filtering documents with a broken encoding, the training corpus has a size of 242GB resulting in 31,240,963,926 tokens.

We used the original 32k vocab (instead of creating a new one).

Turkish Model Zoo

Here's an overview of all available models, incl. their training corpus size:

Model nameModel hub linkPre-training corpus size
ELECTRA Small (cased)here35GB
ELECTRA Base (cased)here35GB
ELECTRA Base mC4 (cased)here242GB
ELECTRA Base mC4 (uncased)here242GB
BERTurk (cased, 32k)here35GB
BERTurk (uncased, 32k)here35GB
BERTurk (cased, 128k)here35GB
BERTurk (uncased, 128k)here35GB
DistilBERTurk (cased)here35GB
ConvBERTurk (cased)here35GB
ConvBERTurk mC4 (cased)here242GB
ConvBERTurk mC4 (uncased)here242GB

DistilBERTurk

The distilled version of a cased model, so called DistilBERTurk, was trained on 7GB of the original training data, using the cased version of BERTurk as teacher model.

DistilBERTurk was trained with the official Hugging Face implementation from here.

The cased model was trained for 5 days on 4 RTX 2080 TI.

More details about distillation can be found in the "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter" paper by Sanh et al. (2019).

ELECTRA

In addition to the BERTurk models, we also trained ELECTRA small and base models. A detailed overview can be found in the ELECTRA section.

ConvBERTurk

In addition to the BERT and ELECTRA based models, we also trained a ConvBERT model. The ConvBERT architecture is presented in the "ConvBERT: Improving BERT with Span-based Dynamic Convolution" paper.

We follow a different training procedure: instead of using a two-phase approach, that pre-trains the model for 90% with 128 sequence length and 10% with 512 sequence length, we pre-train the model with 512 sequence length for 1M steps on a v3-32 TPU.

More details about the pre-training can be found here.

mC4 ELECTRA

In addition to the ELECTRA base model, we also trained an ELECTRA model on the Turkish part of the mC4 corpus. We use a sequence length of 512 over the full training time and train the model for 1M steps on a v3-32 TPU.

Evaluation

In 2024 we ran new evaluations on PoS tagging, NER and sentiment classification datasets. Prior evaluation results can be found here.

All evaluations are performed with the awesome Flair library and the evaluation code and configs can be found in the `experiments folder of this repository.

PoS Tagging

The Model Zoo is evaluated on (the concatenation) of the following PoS Tagging datasets from Universal Dependencies:

We perform a hyper-parameter search over the following configurations:

ParameterValues
Batch Size[16, 8]
Learning Rate[3e-5, 5e-5]
Epoch[3]

And report averaged Accuracy over 5 runs (with different seeds):

Model NameBest ConfigurationBest Development ScoreBest Test Score
BERTurk (cased, 128k)bs16-e3-lr5e-0593.93 ± 0.0494.50 ± 0.07
BERTurk (uncased, 128k)bs8-e3-lr5e-0593.84 ± 0.0494.41 ± 0.13
BERTurk (cased, 32k)bs16-e3-lr5e-0593.95 ± 0.0594.57 ± 0.04
BERTurk (uncased, 32k)bs16-e3-lr5e-0593.84 ± 0.0494.38 ± 0.03
ConvBERTurk (cased)bs8-e3-lr5e-0594.03 ± 0.0794.58 ± 0.06
ConvBERTurk mC4 (cased)bs8-e3-lr5e-0593.90 ± 0.0894.52 ± 0.04
DistilBERTurk (cased)bs8-e3-lr5e-0593.52 ± 0.0394.19 ± 0.04
ELECTRA Base (cased)bs16-e3-lr5e-0593.89 ± 0.0594.45 ± 0.05
ELECTRA Base mC4 (cased)bs16-e3-lr5e-0593.88 ± 0.0594.53 ± 0.11
ELECTRA Base mC4 (uncased)bs8-e3-lr5e-0593.80 ± 0.0994.41 ± 0.04
ELECTRA Small (cased)bs8-e3-lr5e-0593.15 ± 0.0493.88 ± 0.06

Named Entity Recognition

The Model Zoo is evaluated on the Turkish split of the WikiANN dataset, using the following hyper-parameter search:

ParameterValues
Batch Size[16, 8]
Learning Rate[3e-5, 5e-5]
Epoch[10]

Averaged F1-Score over 5 runs (with different seeds):

Model NameBest ConfigurationBest Development ScoreBest Test Score
BERTurk (cased, 128k)bs8-e10-lr3e-0593.92 ± 0.0793.92 ± 0.16
BERTurk (uncased, 128k)bs16-e10-lr3e-0593.59 ± 0.0593.29 ± 0.11
BERTurk (cased, 32k)bs8-e10-lr3e-0593.36 ± 0.0493.26 ± 0.14
BERTurk (uncased, 32k)bs8-e10-lr3e-0593.13 ± 0.1992.96 ± 0.06
ConvBERTurk (cased)bs8-e10-lr3e-0593.93 ± 0.0793.93 ± 0.05
ConvBERTurk mC4 (cased)bs8-e10-lr3e-0593.68 ± 0.1393.58 ± 0.15
DistilBERTurk (cased)bs8-e10-lr5e-0591.8 ± 0.0591.17 ± 0.03
ELECTRA Base (cased)bs8-e10-lr3e-0593.58 ± 0.1293.60 ± 0.09
ELECTRA Base mC4 (cased)bs16-e10-lr3e-0593.51 ± 0.0993.42 ± 0.11
ELECTRA Base mC4 (uncased)bs16-e10-lr5e-0593.01 ± 0.1292.94 ± 0.13
ELECTRA Small (cased)bs8-e10-lr5e-0591.42 ± 0.0991.07 ± 0.09

Sentiment Classification

The Model Zoo is additionally evaluated on the OffensEval-TR 2020 dataset for sentiment classification.

The following parameters are used for a hyper-parameter search:

ParameterValues
Batch Size[16, 8]
Learning Rate[3e-5, 5e-5]
Epoch[3]

Averaged Macro F1-Score over 5 runs (with different seeds) is reported:

Model NameBest ConfigurationBest Development ScoreBest Test Score
BERTurk (cased, 128k)bs16-e3-lr3e-0581.30 ± 0.6181.72 ± 0.47
BERTurk (uncased, 128k)bs16-e3-lr3e-0580.31 ± 0.5482.16 ± 0.27
BERTurk (cased, 32k)bs16-e3-lr5e-0579.64 ± 0.5080.65 ± 0.40
BERTurk (uncased, 32k)bs16-e3-lr3e-0580.87 ± 0.2281.68 ± 0.37
ConvBERTurk (cased)bs16-e3-lr3e-0582.22 ± 0.4182.29 ± 0.34
ConvBERTurk mC4 (cased)bs16-e3-lr3e-0581.69 ± 0.2981.81 ± 0.37
DistilBERTurk (cased)bs16-e3-lr3e-0578.54 ± 0.5579.12 ± 0.17
ELECTRA Base (cased)bs16-e3-lr3e-0579.76 ± 0.2481.69 ± 0.38
ELECTRA Base mC4 (cased)bs8-e3-lr3e-0580.34 ± 0.6782.14 ± 0.27
ELECTRA Base mC4 (uncased)bs16-e3-lr5e-0580.46 ± 0.8081.52 ± 0.56
ELECTRA Small (cased)bs16-e3-lr5e-0577.25 ± 0.4779.89 ± 0.28

Overall

The following table shows the performance of all models over all datasets:

Model NameOverall DevelopmentOverall Test
BERTurk (cased, 128k)89.7290.05
BERTurk (uncased, 128k)89.2589.95
BERTurk (cased, 32k)88.9889.49
BERTurk (uncased, 32k)89.2889.67
ConvBERTurk (cased)90.0690.27
ConvBERTurk mC4 (cased)89.7689.97
DistilBERTurk (cased)87.9588.16
ELECTRA Base (cased)89.0889.91
ELECTRA Base mC4 (cased)89.2490.03
ELECTRA Base mC4 (uncased)89.0989.62
ELECTRA Small (cased)87.2788.28

Model usage

All trained models can be used from the DBMDZ Hugging Face model hub page using their model name.

Example usage with 🤗/Transformers:

tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-turkish-cased")

model = AutoModel.from_pretrained("dbmdz/bert-base-turkish-cased")

This loads the BERTurk cased model. The recently introduced ELECTRA base model can be loaded with:

tokenizer = AutoTokenizer.from_pretrained("dbmdz/electra-base-turkish-cased-discriminator")

model = AutoModelWithLMHead.from_pretrained("dbmdz/electra-base-turkish-cased-discriminator")

Citation

You can use the following BibTeX entry for citation:

@software{stefan_schweter_2020_3770924,
  author       = {Stefan Schweter},
  title        = {BERTurk - BERT models for Turkish},
  month        = apr,
  year         = 2020,
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.3770924},
  url          = {https://doi.org/10.5281/zenodo.3770924}
}

Acknowledgments

Thanks to Kemal Oflazer for providing us additional large corpora for Turkish. Many thanks to Reyyan Yeniterzi for providing us the Turkish NER dataset for evaluation.

We would like to thank Merve Noyan for the awesome logo!

Research supported with Cloud TPUs from the awesome TRC program.

Many thanks for providing access to the TPUs ❤️