Awesome

Megatron-LLM

This library enables pre-training and fine-tuning of large language models (LLMs) at scale. Our repository is a modification of the original Megatron-LM codebase by Nvidia.

Added key features include:

• architectures supported: Llama, Llama 2, Code Llama, Falcon and Mistral
• support training of large models (70B Llama 2, 65B Llama 1, 34B Code Llama, 40B Falcon and Mistral) on commodity hardware on multiple nodes
• 3-way parallelism: tensor parallel, pipeline parallel and data parallel training (inherited from Megatron)
• full pretraining, finetuning and instruct tuning support
• Support for special tokens & tokenizers
• grouped-query attention (GQA) and multi-query attention (MQA)
• Rotary Position Embeddings (RoPE), RMS layer norm, Lima dropout
• RoPE scaling for longer attention context support
• FlashAttention 2
• BF16 / FP16 training
• WandB integration
• Metrics support: Ease to add custom metrics to evaluate on the validation set while training
• Conversion to and from Hugging Face hub

Documentation

Take a look at the online documentation.

Alternatively, build the docs from source:

cd docs/
pip install -r requirements.txt
make html

Example models trained with Megatron-LLM

• TOWER: An Open Multilingual Large Language Model for Translation-Related Tasks
• Executable Code Actions Elicit Better LLM Agents
• Sailor: A suite of Open Language Models tailored for South-East Asia
• Meditron 70b: Scaling Medical Pretraining for Large Language Models
• Llama2-70b-OAsst-sft-v10
• Falcon-40b-megacode2-OAsst
• CodeLlama-13b-OAsst-sft-v10
• Meditron 7b
• ...

(Let us know about yours!)

Citation

If you use this software please cite it: