Home

Awesome

thai2fit (formerly thai2vec)

ULMFit Language Modeling, Text Feature Extraction and Text Classification in Thai Language. Created as part of pyThaiNLP with ULMFit implementation from fast.ai

Models and word embeddings can also be downloaded via Dropbox.

We pretrained a language model with 60,005 embeddings on Thai Wikipedia Dump (perplexity of 28.71067) and text classification (micro-averaged F-1 score of 0.60322 on 5-label classification problem. Benchmarked to 0.5109 by fastText and 0.4976 by LinearSVC on Wongnai Challenge: Review Rating Prediction. The language model can also be used to extract text features for other downstream tasks.

random word vectors

Dependencies

Version History

v0.1

v0.2

v0.3

I remember someone doing a script but I can’t find it. For both, you just have to map the old names of the weights to the new ones. Note that:

in language models, there is a bias in the decoder in fastai v1 that you probably won’t have
in the classifier, the order you see for the layers is artificial (it’s the pytorch representation that takes the things in the order you put them in __init__ when not using Sequential) but the two models (old and new) apply batchnorm, dropout and linear in the same order
tokenizing is done differently in fastai v1, so you may have to fine-tune your models again (we add an xxmaj token for words beginning with a capital for instance)
for weight dropout, you want the weights you have put both in '0.rnns.0.module.weight_hh_l0' and 0.rnns.0.weight_hh_l0_raw (the second one is copied to the first with dropout applied anyway)

v0.31

v0.32

v0.4 (In Progress)

Text Classification

We trained the ULMFit model implemented bythai2fit for text classification. We use Wongnai Challenge: Review Rating Prediction as our benchmark as it is the only sizeable and publicly available text classification dataset at the time of writing (June 21, 2018). It has 39,999 reviews for training and validation, and 6,203 reviews for testing.

We achieved validation perplexity at 35.75113 and validation micro F1 score at 0.598 for five-label classification. Micro F1 scores for public and private leaderboards are 0.59313 and 0.60322 respectively, which are state-of-the-art as of the time of writing (February 27, 2019). FastText benchmark based on their own pretrained embeddings has the performance of 0.50483 and 0.49366 for public and private leaderboards respectively. See ulmfit_wongnai.ipynb for more details.

Text Feature Extraction

The pretrained language model of thai2fit can be used to convert Thai texts into vectors, after which said vectors can be used for various machine learning tasks such as classification, clustering, translation, question answering and so on. The idea is to train a language model that "understands" the texts then extract certain vectors that the model "thinks" represents the texts we want. You can access this functionality easily via pythainlp

from pythainlp.ulmfit import *
document_vector('วันนี้วันดีปีใหม่',learn,data)
>> array([ 0.066298,  0.307813,  0.246051,  0.008683, ..., -0.058363,  0.133258, -0.289954, -1.770246], dtype=float32)

Language Modeling

The goal of this notebook is to train a language model using the fast.ai version of AWD LSTM Language Model, with data from Thai Wikipedia Dump last updated February 17, 2019. Using 40M/200k/200k tokens of train-validation-test split, we achieved validation perplexity of 27.81627 with 60,004 embeddings at 400 dimensions, compared to state-of-the-art as of October 27, 2018 at 42.41 for English WikiText-2 by Yang et al (2018). To the best of our knowledge, there is no comparable research in Thai language at the point of writing (February 17, 2019). See thwiki_lm for more details.

Word Embeddings

We use the embeddings from v0.1 since it was trained specifically for word2vec as opposed to latter versions which garner to classification. The thai2vec.bin 51,556 word embeddings of 300 dimensions, in descending order by their frequencies (See thai2vec.vocab). The files are in word2vec format readable by gensim. Most common applications include word vector visualization, word arithmetic, word grouping, cosine similarity and sentence or document vectors. For sample code, see thwiki_lm/word2vec_examples.ipynb.

Word Arithmetic

You can do simple "arithmetic" with words based on the word vectors such as:

word arithmetic

Word Grouping

It can also be used to do word groupings. For instance:

word grouping

Cosine Similarity

Calculate cosine similarity between two word vectors.

cosine similarity

Citation

@software{charin_polpanumas_2021_4429691,
  author       = {Charin Polpanumas and
                  Wannaphong Phatthiyaphaibun},
  title        = {thai2fit: Thai language Implementation of ULMFit},
  month        = jan,
  year         = 2021,
  publisher    = {Zenodo},
  version      = {v0.3},
  doi          = {10.5281/zenodo.4429691},
  url          = {https://doi.org/10.5281/zenodo.4429691}
}

NLP Workshop at Chiangmai University