Awesome
Awesome Transformers
A curated list of awesome transformer models.
If you want to contribute to this list, send a pull request or reach out to me on twitter: @abacaj. Let's make this list useful.
There are a number of models available that are not entirely open source (non-commercial, etc), this repository should serve to also make you aware of that. Tracking the original source/company of the model will help.
I would also eventually like to add model use cases. So it is easier for others to find the right one to fine-tune.
Format:
- Model name: short description, usually from paper
- Model link (usually huggingface or github)
- Paper link
- Source as company or group
- Model license
Table of Contents
- Encoder (autoencoder) models
- Decoder (autoregressive) models
- Encoder+decoder (seq2seq) models
- Multimodal models
- Vision models
- Audio models
- Recommendation models
- Grounded Situation Recognition models
<a name="encoder"></a>
Encoder models
<a name="albert"></a>
- ALBERT: "A Lite" version of BERT
- BERT: Bidirectional Encoder Representations from Transformers <a name="bert"></a>
- DistilBERT: Distilled version of BERT smaller, faster, cheaper and lighter <a name="distilbert"></a>
- DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing <a name="debertav3"></a>
- Electra: Pre-training Text Encoders as Discriminators Rather Than Generators <a name="electra"></a>
- RoBERTa: Robustly Optimized BERT Pretraining Approach <a name="roberta"></a>
<a name="decoder"></a>
Decoder models
<a name="bio-gpt"></a>
- BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
- CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis <a name="codegen"></a>
- LLaMa: Open and Efficient Foundation Language Models
<a name="llama"></a>
- Model
- Paper
- Requires approval, non-commercial
- GPT: Improving Language Understanding by Generative Pre-Training <a name="gpt"></a>
- GPT-2: Language Models are Unsupervised Multitask Learners <a name="gpt-2"></a>
- GPT-J: A 6 Billion Parameter Autoregressive Language Model <a name="gpt-j"></a>
- GPT-NEO: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow <a name="gpt-neo"></a>
- GPT-NEOX-20B: An Open-Source Autoregressive Language Model <a name="gpt-neox"></a>
- NeMo Megatron-GPT: Megatron-GPT 20B is a transformer-based language model. <a name="nemo"></a>
- OPT: Open Pre-trained Transformer Language Models
<a name="opt"></a>
- Model
- Paper
- Requires approval, non-commercial
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
<a name="bloom"></a>
- Model
- Paper
- BigScience
- OpenRAIL, use-based restrictions
- GLM: An Open Bilingual Pre-Trained Model
<a name="glm"></a>
- Model
- Paper
- Knowledge Engineering Group (KEG) & Data Mining at Tsinghua University
- Custom license, see restrictions
- YaLM: Pretrained language model with 100B parameters <a name="yalm"></a>
<a name="encoder-decoder"></a>
Encoder+decoder (seq2seq) models
<a name="bio-gpt"></a>
- T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer <a name="t5"></a>
- FLAN-T5: Scaling Instruction-Finetuned Language Models <a name="flan-t5"></a>
- Code-T5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation <a name="code-t5"></a>
- Bart: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension <a name="bart"></a>
- Pegasus: Pre-training with Extracted Gap-sentences for Abstractive Summarization <a name="pegasus"></a>
- MT5: A Massively Multilingual Pre-trained Text-to-Text Transformer <a name="mt5"></a>
- UL2: Unifying Language Learning Paradigms <a name="ul2"></a>
- FLAN-UL2: A New Open Source Flan 20B with UL2 <a name="flanul2"></a>
- EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation <a name="edgeformer"></a>
<a name="multimodal"></a>
Multimodal models
<a name="donut"></a>
- Donut: OCR-free Document Understanding Transformer
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking <a name="layoutlmv3"></a>
- TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
<a name="trocr"></a>
- Model
- Paper
- Microsoft
- Inherits MIT license
- CLIP: Learning Transferable Visual Models From Natural Language Supervision <a name="clip"></a>
- Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks <a name="unifiedio"></a>
<a name="vision"></a>
Vision models
<a name="dit"></a>
- DiT: Self-supervised Pre-training for Document Image Transformer
- Model
- Paper
- Microsoft
- Inherits MIT license
- DETR: End-to-End Object Detection with Transformers <a name="detr"></a>
- EfficientFormer: Vision Transformers at MobileNet Speed <a name="efficientformer"></a>
<a name="audio"></a>
Audio models
<a name="whisper"></a>
- Whisper: Robust Speech Recognition via Large-Scale Weak Supervision
- VALL-E: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
<a name="valle"></a>
- Model (unofficial)
- MIT but has a dependency on a CC-BY-NC library
- Model (unofficial)
- Apache v2
- Paper
- Microsoft
- Model (unofficial)
<a name="recommendation"></a>
Recommendation models
<a name="p5"></a>
- Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)
<a name="gsr"></a>
Grounded Situation Recognition models
<a name="gsrtr"></a>