Awesome

NLP for Hindi

This repository contains State of the Art Language models and Classifier for Hindi language (spoken in Indian sub-continent).

The models trained here have been used in Natural Language Toolkit for Indic Languages (iNLTK)

Dataset

Created as part of this project

1. Hindi Wikipedia Articles - 172k

2. Hindi Wikipedia Articles - 55k

3. Hindi Movie Reviews Dataset

4. Hindi Text Short Summarization Corpus

5. Hindi Text Short and Large Summarization Corpus

Open Source Datasets

1. BBC News Articles : Sentiment analysis corpus for Hindi documents extracted from BBC news website.

2. IIT Patna Product Reviews : Sentiment analysis corpus for product reviews posted in Hindi.

3. IIT Patna Movie Reviews : Sentiment analysis corpus for movie reviews posted in Hindi.

Results

Language Model Perplexity (on validation set)

Note: Nirant has done previous SOTA work with Hindi Language Model and achieved perplexity of ~46. The scores above aren't directly comparable with his score because his train and validation set were different and they aren't available for reproducibility

Classification Metrics

ULMFiT

Visualizations

Word Embeddings

Sentence Embeddings

Results of using Transfer Learning + Data Augmentation from iNLTK

On using complete training set (with Transfer learning)

On using 20% of training set (with Transfer learning)

On using 20% of training set (with Transfer learning + Data Augmentation)

Pretrained Models

Language Models

Download pretrained Language Models of ULMFiT, TransformerXL trained on Hindi Wikipedia Articles - 172k and Hindi Wikipedia Articles - 55k from here

Tokenizer

Unsupervised training using Google's sentencepiece

Download the trained model and vocabulary from here