Home

Awesome

NLP for Punjabi

This repository contains State of the Art Language models and Classifier for Punjabi language (spoken in Indian sub-continent)

The models trained here have been used in Natural Language Toolkit for Indic Languages (iNLTK)

Dataset

Created as part of this project

  1. Punjabi Wikipedia Articles

  2. Punjabi News Dataset

Open Source Datasets

  1. IndicNLP News Article Classification Dataset - Punjabi

Results

Language Model Perplexity (on validation set)

Architecture/DatasetPunjabi Wikipedia Articles
ULMFiT24.40
TransformerXL14.03

Classification Metrics

ULMFiT
DatasetAccuracyMCCNotebook to Reproduce results
IndicNLP News Article Classification Dataset - Punjabi97.1296.17Link

Visualizations

Word Embeddings
ArchitectureVisualization
ULMFiTEmbeddings projection
TransformerXLEmbeddings projection
Sentence Embeddings
ArchitectureVisualization
ULMFiTEncodings projection

Pretrained Models

Language Models

Download pretrained Language Models from here

Tokenizer

Unsupervised training using Google's sentencepiece

Download the trained model and vocabulary from here