Awesome
Russian AWD-LSTM language model
This is the AWD-LSTM language model [1] trained on a subset of the Taiga corpus [2]. It can be used with vanilla PyTorch or with fast.ai [3] to get sentence representations for downstream linguistic tasks, such as text classification with ULMFit [4].
You can download weights for the model here: https://drive.google.com/open?id=1_d4XCMMWdIZt57JJyH34bzY2gRSB7KTE
NOTE This model was trained a while back with an old version of fastai. If you want to use it with a newer (v1.5x+) version, run these lines before creating a learner:
config = awd_lstm_lm_config.copy()
config['n_hid'] = 1150
Metrics
The model was trained for 10 epochs. The total number of tokens was 208,006,138, of which 10% were used for validation. Validation results are:
Cross-entropy | Perplexity | Accuracy |
---|---|---|
3.09 | 21.98 | 0.43 |
References
- https://arxiv.org/abs/1708.02182
- https://tatianashavrina.github.io/taiga_site/
- https://www.fast.ai/
- https://arxiv.org/abs/1801.06146
Additional links
- Similar model trained on Wikipedia: https://github.com/ppleskov/Russian-Language-Model