Home

Awesome

AraBERTv2 / AraGPT2 / AraELECTRA

visitors PyPI version PyPI - Downloads

<p align="middle"> <img src="https://github.com/aub-mind/arabert/blob/master/arabert_logo.png" width="150" align="left"/> <img src="https://github.com/aub-mind/arabert/blob/master/AraGPT2.png" width="150"/> <img src="https://github.com/aub-mind/arabert/blob/master/AraELECTRA.png" width="150" align="right"/> </p>

This repository now contains code and implementation for:

If you want to clone the old repository:

git clone https://github.com/aub-mind/arabert/
cd arabert && git checkout 6a58ca118911ef311cbe8cdcdcc1d03601123291

Update

Installation

Install AraBERT from PyPI:

pip install arabert

then use it as follows:

from arabert import ArabertPreprocessor
from arabert.aragpt2.grover.modeling_gpt2 import GPT2LMHeadModel

AraBERTv2

What's New!

AraBERTv0.2-Twitter-base/large are two new models for Arabic dialects and tweets, trained by continuing the pre-training using the MLM task on ~60M Arabic tweets (filtered from a collection on 100M).

The two new models have had emojies added to their vocabulary in addition to common words that weren't at first present. The pre-training was done with a max sentence length of 64 only for 1 epoch.

Models

AraBERT comes in 6 variants:

More Detail in the AraBERT folder and in the README and in the AraBERT Paper

ModelHuggingFace Model NameSize (MB/Params)Pre-SegmentationDataSet (Sentences/Size/nWords)
AraBERTv0.2-Twitter-basebert-base-arabertv02-twitter543MB / 136MNoSame as v02 + 60M Multi-Dialect Tweets
AraBERTv0.2-Twitter-largebert-large-arabertv02-twitter1.38G / 371MNoSame as v02 + 60M Multi-Dialect Tweets
AraBERTv0.2-basebert-base-arabertv02543MB / 136MNo200M / 77GB / 8.6B
AraBERTv0.2-largebert-large-arabertv021.38G / 371MNo200M / 77GB / 8.6B
AraBERTv2-basebert-base-arabertv2543MB / 136MYes200M / 77GB / 8.6B
AraBERTv2-largebert-large-arabertv21.38G / 371MYes200M / 77GB / 8.6B
AraBERTv0.1-basebert-base-arabertv01543MB / 136MNo77M / 23GB / 2.7B
AraBERTv1-basebert-base-arabert543MB / 136MYes77M / 23GB / 2.7B

All models are available in the HuggingFace model page under the aubmindlab name. Checkpoints are available in PyTorch, TF2 and TF1 formats.

Better Pre-Processing and New Vocab

We identified an issue with AraBERTv1's wordpiece vocabulary. The issue came from punctuations and numbers that were still attached to words when we trained the wordpiece vocab. We now insert a space between numbers and characters and around punctuation characters.

The new vocabulary was learnt using the BertWordpieceTokenizer from the tokenizers library, and now supports the Fast tokenizer implementation from the transformers library.

P.S.: All the old BERT codes should work with the new BERT, just change the model name and check the new preprocessing function

Please read the section on how to use the preprocessing function

Bigger Dataset and More Compute

We used ~3.5 times more data, and trained for longer. For Dataset Sources see the Dataset Section

ModelHardwarenum of examples with seq len (128 / 512)128 (Batch Size/ Num of Steps)512 (Batch Size/ Num of Steps)Total StepsTotal Time (in Days)
AraBERTv0.2-baseTPUv3-8420M / 207M2560 / 1M384/ 2M3M36
AraBERTv0.2-largeTPUv3-128420M / 207M13440 / 250K2056 / 300K550K7
AraBERTv2-baseTPUv3-8420M / 207M2560 / 1M384/ 2M3M36
AraBERTv2-largeTPUv3-128520M / 245M13440 / 250K2056 / 300K550K7
AraBERT-base (v1/v0.1)TPUv2-8-512 / 900K128 / 300K1.2M4

AraGPT2

More details and code are available in the AraGPT2 folder and README

Model

ModelHuggingFace Model NameSize / Params
AraGPT2-basearagpt2-base527MB/135M
AraGPT2-mediumaragpt2-medium1.38G/370M
AraGPT2-largearagpt2-large2.98GB/792M
AraGPT2-megaaragpt2-mega5.5GB/1.46B
AraGPT2-mega-detector-longaragpt2-mega-detector-long516MB/135M

All models are available in the HuggingFace model page under the aubmindlab name. Checkpoints are available in PyTorch, TF2 and TF1 formats.

Dataset and Compute

For Dataset Source see the Dataset Section

ModelHardwarenum of examples (seq len = 1024)Batch SizeNum of StepsTime (in days)
AraGPT2-baseTPUv3-1289.7M1792125K1.5
AraGPT2-mediumTPUv3-89.7M801M15
AraGPT2-largeTPUv3-1289.7M256220k3
AraGPT2-megaTPUv3-1289.7M256800K9

AraELECTRA

More details and code are available in the AraELECTRA folder and README

Model

ModelHuggingFace Model NameSize (MB/Params)
AraELECTRA-base-generatoraraelectra-base-generator227MB/60M
AraELECTRA-base-discriminatoraraelectra-base-discriminator516MB/135M

Dataset and Compute

ModelHardwarenum of examples (seq len = 512)Batch SizeNum of StepsTime (in days)
ELECTRA-baseTPUv3-8-2562M24

Dataset

The pretraining data used for the new AraBERT model is also used for AraGPT2 and AraELECTRA.

The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars (before applying Farasa Segmentation)

For the new dataset we added the unshuffled OSCAR corpus, after we thoroughly filter it, to the previous dataset used in AraBERTv1 but with out the websites that we previously crawled:

Preprocessing

It is recommended to apply our preprocessing function before training/testing on any dataset. Install farasapy to segment text for AraBERT v1 & v2 pip install farasapy

from arabert.preprocess import ArabertPreprocessor

model_name = "aubmindlab/bert-base-arabertv2"
arabert_prep = ArabertPreprocessor(model_name=model_name)

text = "ولن نبالغ إذا قلنا: إن 'هاتف' أو 'كمبيوتر المكتب' في زمننا هذا ضروري"
arabert_prep.preprocess(text)
>>>"و+ لن نبالغ إذا قل +نا : إن ' هاتف ' أو ' كمبيوتر ال+ مكتب ' في زمن +نا هذا ضروري"

You can also use the unpreprocess() function to reverse the preprocessing changes, by fixing the spacing around non alphabetical characters, and also de-segmenting if the model selected need pre-segmentation. We highly recommend unprocessing generated content of AraGPT2 model, to make it look more natural.

output_text = "و+ لن نبالغ إذا قل +نا : إن ' هاتف ' أو ' كمبيوتر ال+ مكتب ' في زمن +نا هذا ضروري"
arabert_prep.unpreprocess(output_text)
>>>"ولن نبالغ إذا قلنا: إن 'هاتف' أو 'كمبيوتر المكتب' في زمننا هذا ضروري"

The ArabertPreprocessor class:

ArabertPreprocessor(
  model_name= "",
  keep_emojis = False,
  remove_html_markup = True,
  replace_urls_emails_mentions = True,
  strip_tashkeel = True,
  strip_tatweel = True,
  insert_white_spaces = True,
  remove_non_digit_repetition = True,
  replace_slash_with_dash = None,
  map_hindi_numbers_to_arabic = None,
  apply_farasa_segmentation = None
)

Examples Notebooks

TensorFlow 1.x models

You can find the PyTorch, TF2 and TF1 models in HuggingFace's Transformer Library under the aubmindlab username

If you used this model please cite us as :

AraBERT

Google Scholar has our Bibtex wrong (missing name), use this instead

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

AraGPT2

@inproceedings{antoun-etal-2021-aragpt2,
    title = "{A}ra{GPT}2: Pre-Trained Transformer for {A}rabic Language Generation",
    author = "Antoun, Wissam  and
      Baly, Fady  and
      Hajj, Hazem",
    booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
    month = apr,
    year = "2021",
    address = "Kyiv, Ukraine (Virtual)",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.wanlp-1.21",
    pages = "196--207",
}

AraELECTRA

@inproceedings{antoun-etal-2021-araelectra,
    title = "{A}ra{ELECTRA}: Pre-Training Text Discriminators for {A}rabic Language Understanding",
    author = "Antoun, Wissam  and
      Baly, Fady  and
      Hajj, Hazem",
    booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
    month = apr,
    year = "2021",
    address = "Kyiv, Ukraine (Virtual)",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.wanlp-1.20",
    pages = "191--195",
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs, couldn't have done it without this program, and to the AUB MIND Lab Members for the continous support. Also thanks to Yakshof and Assafir for data and storage access. Another thanks for Habib Rahal (https://www.behance.net/rahalhabib), for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07 (AT) mail (DOT) aub (DOT) edu | wissam.antoun (AT) gmail (DOT) com

Fady Baly: Linkedin | Twitter | Github | fgb06 (AT) mail (DOT) aub (DOT) edu | baly.fady (AT) gmail (DOT) com