Home

Awesome

BanglaBERT

This repository contains the official release of the model "BanglaBERT" and associated downstream fine-tuning code and datasets introduced in the paper titled "BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla" published in Findings of the Association for Computational Linguistics: NAACL 2022.

Updates

Table of Contents

Models

The pretrained model checkpoints are available at Huggingface model hub.

To use these models for the supported downstream tasks in this repository see Training & Evaluation.

Note: These models were pretrained using a specific normalization pipeline available here. All finetuning scripts in this repository uses this normalization by default. If you need to adapt the pretrained model for a different task make sure the text units are normalized using this pipeline before tokenizing to get best results. A basic example is available at the model page.

Datasets

We are also releasing the Bangla Natural Language Inference (NLI) and Bangla Question Answering (QA) datasets introduced in the paper.

Please fill out this Google Form to request access to the Bangla2B+ pretraining corpus.

Setup

For installing the necessary requirements, use the following bash snippet

$ git clone https://github.com/csebuetnlp/banglabert
$ cd banglabert/
$ conda create python==3.7.9 pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.0 cudatoolkit=10.2 -c pytorch -p ./env
$ conda activate ./env # or source activate ./env (for older versions of anaconda)
$ bash setup.sh 

Training & Evaluation

To use the pretrained model for finetuning / inference on different downstream tasks see the following section:

Benchmarks

ModelParamsSC (macro-F1)NLI (accuracy)NER (micro-F1)QA (EM/F1)BangLUE score
mBERT180M27.0562.2239.2759.01/64.1850.35
XLM-R (base)270M42.0372.1845.3755.03/61.8355.29
XLM-R (large)550M49.4978.1356.4871.13/77.7066.59
BanglishBERT110M48.3975.2655.5672.87/78.6366.14
ModelParamsSC (macro-F1)NLI (accuracy)NER (micro-F1)QA (EM/F1)BangLUE score
mBERT180M67.5975.1368.9767.12/72.6470.29
XLM-R (base)270M69.5478.4673.3268.09/74.2772.82
XLM-R (large)550M70.9782.4078.3973.15/79.0676.79
sahajBERT18M71.1276.9270.9465.48/70.6971.03
BanglishBERT110M70.6180.9576.2872.43/78.4075.73
BanglaBERT (small)13M69.2976.7573.4163.30/69.6570.38
BanglaBERT110M72.8982.8077.7872.63/79.3477.09
BanglaBERT (large)335M71.9483.4179.2076.10/81.5078.43

The benchmarking datasets are as follows:

Acknowledgements

We would like to thank Intelligent Machines and Google TFRC Program for providing cloud support for pretraining the models.

License

Contents of this repository are restricted to non-commercial research purposes only under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).

<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a>

Citation

If you use any of the datasets, models or code modules, please cite the following paper:

@inproceedings{bhattacharjee-etal-2022-banglabert,
    title = "{B}angla{BERT}: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in {B}angla",
    author = "Bhattacharjee, Abhik  and
      Hasan, Tahmid  and
      Ahmad, Wasi  and
      Mubasshir, Kazi Samin  and
      Islam, Md Saiful  and
      Iqbal, Anindya  and
      Rahman, M. Sohel  and
      Shahriyar, Rifat",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2022",
    month = jul,
    year = "2022",
    address = "Seattle, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.findings-naacl.98",
    pages = "1318--1327",
    abstract = "In this work, we introduce BanglaBERT, a BERT-based Natural Language Understanding (NLU) model pretrained in Bangla, a widely spoken yet low-resource language in the NLP literature. To pretrain BanglaBERT, we collect 27.5 GB of Bangla pretraining data (dubbed {`}Bangla2B+{'}) by crawling 110 popular Bangla sites. We introduce two downstream task datasets on natural language inference and question answering and benchmark on four diverse NLU tasks covering text classification, sequence labeling, and span prediction. In the process, we bring them under the first-ever Bangla Language Understanding Benchmark (BLUB). BanglaBERT achieves state-of-the-art results outperforming multilingual and monolingual models. We are making the models, datasets, and a leaderboard publicly available at \url{https://github.com/csebuetnlp/banglabert} to advance Bangla NLP.",
}