Awesome

NLP for Marathi

This repository contains State of the Art Language models and Classifier for Marathi, which is spoken predominantly by Marathi people of Maharashtra, India.

The models trained here have been used in Natural Language Toolkit for Indic Languages (iNLTK)

Dataset

Created as part of this project

1. Marathi Wikipedia Articles

2. Marathi News Dataset

Open Source Datasets

1. iNLTK Headlines Corpus - Marathi : Uses the Marathi News Dataset prepared above.

Results

Language Model Perplexity (on validation set)

Classification Metrics

ULMFiT

Visualizations

Word Embeddings

Sentence Embeddings

Results of using Transfer Learning + Data Augmentation from iNLTK

On using complete training set (with Transfer learning)

On using 5% of training set (with Transfer learning)

On using 5% of training set (with Transfer learning + Data Augmentation)

Pretrained Models

Language Models

Download pretrained Language Model from here

Tokenizer

Trained tokenizer using Google's sentencepiece

Download the trained model and vocabulary from here