Awesome

Table of Contents

• Table of Contents
• What is LLMOps?
• Prompt Engineering
• Models
• Optimization
• Tools (GitHub)
• Tools (Other)
• RLHF
• Awesome
• Contributing

What is LLMOps?

LLMOps is a part of MLOps practices, specialized form of MLOps that focuses on managing the entire lifecycle of large language models(LLM).

Starting in 2021, as LLMs evolved rapidly and the technology matured, we began to focus on practices for managing LLMs efficiently, and LLMOps, which are adaptations of traditional MLOps practices to LLMs, began to be talked about.

LLMOps vs MLOps

:arrow_up: Go to top

Prompt Engineering

• PromptBase - Marketplace of the prompt engineering
• PromptHero - The website for prompt engineering
• Prompt Search - The search engine for the prompt engineering
• Prompt Perfect - Auto Prompt Optimizer
• Learn Prompting - The tutorial website for the prompt engineering
• Blog: Exploring Prompt Injection Attacks
• Blog: Prompt Leaking
• Paper: Prefix-Tuning: Optimizing Continuous Prompts for Generation

:arrow_up: Go to top

Models

:arrow_up: Go to top

Optimization

• Blog: A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and bitsandbytes
• Blog: Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU
• Blog: Handling big models for inference
• Blog: How To Fine-Tune the Alpaca Model For Any Language | ChatGPT Alternative
• Paper: LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
• Gist: Script to decompose/recompose LLAMA LLM models with different number of shards

:arrow_up: Go to top

Tools (GitHub)

• Stanford Alpaca - - A repository of Stanford Alpaca project, a model fine-tuned from the LLaMA 7B model on 52K instruction-following demonstrations.
• LoRA - - An implementation of "LoRA: Low-Rank Adaptation of Large Language Models".
• Dolly - - A large language model trained on the Databricks Machine Learning Platform.
• DeepSpeed - - A deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
• LMFlow - - An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Model for All.
• Promptify - - An utility / tookit for Prompt engineering.
• Auto-GPT - - An experimental open-source attempt to make GPT-4 fully autonomous.
• Jarvis - - A system to connect LLMs with ML community, a composite model connector via the LLM interface.
• dalai - - The cli tool to run LLaMA on the local machine.
• haystack - -an open source NLP framework to interact with the data using Transformer models and LLMs.
• langchain - - The library which assists in the development of applications with LLM.
• langflow - - An UI for LangChain, designed with react-flow to provide an effortless way to experiment and prototype flows.
• deeplake - - Data Lake for Deep Learning. Build, manage, query, version, & visualize datasets.
• alpaca-lora - - Instruct-tune LLaMA on consumer hardware.
• bosquet - - LLMOps for Large Language Model based applications.
• llama_index - - A project that provides a central interface to connect your LLM's with external data.
• gradio - - An UI helper for the machine learning model.
• sharegpt - - An open-source Chrome Extension for you to share your wildest ChatGPT conversations with one click.
• keras-nlp - - A natural language processing library that supports users through their entire development cycle.
• Snowkel AI - - The data platform for foundation models.
• promptflow - - A toolkit that simplifies the development of LLM-based AI applications, from ideation to deployment.

:arrow_up: Go to top

Tools (Other)

• PaLM2 API - An API service that makes PaLM2, Large Language Models (LLMs), available to Google Cloud Vertex AI.
• Perspective API - A tool that can help mitigate toxicity and ensure healthy dialogue online.

:arrow_up: Go to top

RLHF

• evals - - A curated list of reinforcement learning with human feedback resources.
• trlx - - A repo for distributed training of language models with Reinforcement Learning via Human Feedback. (RLHF)
• PaLM-rlhf-pytorch - - Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture.

:arrow_up: Go to top

Awesome

• Hannibal046/Awesome-LLM
• KennethanCeyer/awesome-llm
• f/awesome-chatgpt-prompts
• promptslab/Awesome-Prompt-Engineering
• tensorchord/awesome-open-source-llmops
• opendilab/awesome-RLHF

:arrow_up: Go to top

Contributing

We welcome contributions to the Awesome LLMOps list! If you'd like to suggest an addition or make a correction, please follow these guidelines:

1. Fork the repository and create a new branch for your contribution.
2. Make your changes to the README.md file.
3. Ensure that your contribution is relevant to the topic of LLMOps.
4. Use the following format to add your contribution:

[Name of Resource](Link to Resource) - Description of resource

5. Add your contribution in alphabetical order within its category.
6. Make sure that your contribution is not already listed.
7. Provide a brief description of the resource and explain why it is relevant to LLMOps.
8. Create a pull request with a clear title and description of your changes.

We appreciate your contributions and thank you for helping to make the Awesome LLMOps list even more awesome!

:arrow_up: Go to top