Home

Awesome

Awesome-LLM Awesome

🔥 Large Language Models(LLM) have taken the NLP community AI community the Whole World by storm. Here is a curated list of papers about large language models, especially relating to ChatGPT. It also contains frameworks for LLM training, tools to deploy LLM, courses and tutorials about LLM and all publicly available LLM checkpoints and APIs.

Trending LLM Projects

Table of Content

Milestone Papers

DatekeywordsInstitutePaper
2017-06TransformersGoogleAttention Is All You Need
2018-06GPT 1.0OpenAIImproving Language Understanding by Generative Pre-Training
2018-10BERTGoogleBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
2019-02GPT 2.0OpenAILanguage Models are Unsupervised Multitask Learners
2019-09Megatron-LMNVIDIAMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
2019-10T5GoogleExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
2019-10ZeROMicrosoftZeRO: Memory Optimizations Toward Training Trillion Parameter Models
2020-01Scaling LawOpenAIScaling Laws for Neural Language Models
2020-05GPT 3.0OpenAILanguage models are few-shot learners
2021-01Switch TransformersGoogleSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
2021-08CodexOpenAIEvaluating Large Language Models Trained on Code
2021-08Foundation ModelsStanfordOn the Opportunities and Risks of Foundation Models
2021-09FLANGoogleFinetuned Language Models are Zero-Shot Learners
2021-10T0HuggingFace et al.Multitask Prompted Training Enables Zero-Shot Task Generalization
2021-12GLaMGoogleGLaM: Efficient Scaling of Language Models with Mixture-of-Experts
2021-12WebGPTOpenAIWebGPT: Browser-assisted question-answering with human feedback
2021-12RetroDeepMindImproving language models by retrieving from trillions of tokens
2021-12GopherDeepMindScaling Language Models: Methods, Analysis & Insights from Training Gopher
2022-01COTGoogleChain-of-Thought Prompting Elicits Reasoning in Large Language Models
2022-01LaMDAGoogleLaMDA: Language Models for Dialog Applications
2022-01MinervaGoogleSolving Quantitative Reasoning Problems with Language Models
2022-01Megatron-Turing NLGMicrosoft&NVIDIAUsing Deep and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
2022-03InstructGPTOpenAITraining language models to follow instructions with human feedback
2022-04PaLMGooglePaLM: Scaling Language Modeling with Pathways
2022-04ChinchillaDeepMindAn empirical analysis of compute-optimal large language model training
2022-05OPTMetaOPT: Open Pre-trained Transformer Language Models
2022-05UL2GoogleUnifying Language Learning Paradigms
2022-06Emergent AbilitiesGoogleEmergent Abilities of Large Language Models
2022-06BIG-benchGoogleBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
2022-06METALMMicrosoftLanguage Models are General-Purpose Interfaces
2022-09SparrowDeepMindImproving alignment of dialogue agents via targeted human judgements
2022-10Flan-T5/PaLMGoogleScaling Instruction-Finetuned Language Models
2022-10GLM-130BTsinghuaGLM-130B: An Open Bilingual Pre-trained Model
2022-11HELMStanfordHolistic Evaluation of Language Models
2022-11BLOOMBigScienceBLOOM: A 176B-Parameter Open-Access Multilingual Language Model
2022-11GalacticaMetaGalactica: A Large Language Model for Science
2022-12OPT-IMLMetaOPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization
2023-01Flan 2022 CollectionGoogleThe Flan Collection: Designing Data and Methods for Effective Instruction Tuning
2023-02LLaMAMetaLLaMA: Open and Efficient Foundation Language Models
2023-02Kosmos-1MicrosoftLanguage Is Not All You Need: Aligning Perception with Language Models
2023-03LRUDeepMindResurrecting Recurrent Neural Networks for Long Sequences
2023-03PaLM-EGooglePaLM-E: An Embodied Multimodal Language Model
2023-03GPT 4OpenAIGPT-4 Technical Report
2023-04LLaVAUW–Madison&MicrosoftVisual Instruction Tuning
2023-04PythiaEleutherAI et al.Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
2023-05DromedaryCMU et al.Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
2023-05PaLM 2GooglePaLM 2 Technical Report
2023-05RWKVBo PengRWKV: Reinventing RNNs for the Transformer Era
2023-05DPOStanfordDirect Preference Optimization: Your Language Model is Secretly a Reward Model
2023-05ToTGoogle&PrincetonTree of Thoughts: Deliberate Problem Solving with Large Language Models
2023-07LLaMA2MetaLlama 2: Open Foundation and Fine-Tuned Chat Models
2023-10Mistral 7BMistralMistral 7B
2023-12MambaCMU&PrincetonMamba: Linear-Time Sequence Modeling with Selective State Spaces
2024-01DeepSeek-v2DeepSeekDeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
2024-03JambaAI21 LabsJamba: A Hybrid Transformer-Mamba Language Model
2024-05Mamba2CMU&PrincetonTransformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
2024-05Llama3MetaThe Llama 3 Herd of Models

Other Papers

If you're interested in the field of LLM, you may find the above list of milestone papers helpful to explore its history and state-of-the-art. However, each direction of LLM offers a unique set of insights and contributions, which are essential to understanding the field as a whole. For a detailed list of papers in various subfields, please refer to the following link:

LLM Leaderboard

Open LLM

LLM Data

LLM Evaluation:

LLM Training Frameworks

LLM Deployment

Reference: llm-inference-solutions

LLM Applications

LLM Tutorials and Courses

LLM Books

Great thoughts about LLM

Miscellaneous

Contributing

This is an active repository and your contributions are always welcome!

I will keep some pull requests open if I'm not sure if they are awesome for LLM, you could vote for them by adding 👍 to them.


If you have any question about this opinionated list, do not hesitate to contact me chengxin1998@stu.pku.edu.cn.