Home

Awesome

👋 Awesome Local LLMs

There are an overwhelming number of open-source tools for local LLM inference - for both proprietary and open weights LLMs. These tools generally lie within three categories:

  1. LLM inference backend engine
  2. LLM front end UI
  3. All-in-one desktop application

However these tools can overlap in scope with new features are constantly being added so I have chosen not to manually categorize or label features of each project.

GitHub repository metrics, like number of stars, contributors, issues, releases, and time since last commit, have been collected as a proxy for popularity and active maintenance.

Contributions are welcome! Feel free to suggest open-source repos that I have missed either in the Issues of this repo or run the script in the script branch and update the README and make a pull request.

For full table with all metrics go to this Google Sheet.

For my thoughts on local LLM tooling: https://vinlam.com/posts/local-llm-options/

Note the condensed table below has two filters applied:

  1. Repositories need more than 100 stars
  2. Repositories require a commit within the last 60 days

Open-Source Local LLM Projects

Last Updated: 01/11/2024

#RepoAboutStarsForksIssuesContributorsReleasesLicenseTime Since Last Commit
1transformers🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.134,27726,8471,426431169Apache License 2.00 days, 15 hrs, 45 mins
2ollamaGet up and running with Llama 3.2, Mistral, Gemma 2, and other large language models.95,7757,6001,43730895MIT License0 days, 14 hrs, 28 mins
3ChatGPT-Next-WebA cross-platform ChatGPT/Gemini UI (Web / PWA / Linux / Win / MacOS). 一键拥有你自己的跨平台 ChatGPT/Gemini 应用。76,17759,02545522873MIT License0 days, 8 hrs, 40 mins
4gpt4allGPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.70,3657,68361511428MIT License0 days, 15 hrs, 20 mins
5llama.cppLLM inference in C/C++67,1729,6395544652,494MIT License0 days, 10 hrs, 41 mins
6gpt_academic为GPT/GLM等LLM大语言模型提供实用化交互接口，特别优化论文阅读/润色/写作体验，模块化设计，支持自定义快捷按钮&函数插件，支持Python和C++等项目剖析&自译解功能，PDF/LaTex论文翻译&总结功能，支持并行问询多种LLM模型，支持chatglm3等本地模型。接入通义千问, deepseekcoder, 讯飞星火, 文心一言, llama2, rwkv, claude2, moss等。65,2998,0383729031GNU General Public License v3.02 days, 1 hrs, 15 mins
7gpt4freeThe official gpt4free repository, various collection of powerful language models60,95313,29222218153GNU General Public License v3.02 days, 5 hrs, 57 mins
8privateGPTInteract with your documents using the power of GPT, 100% privately, no data leaks54,0497,2642399010Apache License 2.015 days, 4 hrs, 9 mins
9open-webuiUser-friendly AI Interface (Supports Ollama, OpenAI API, ...)44,8945,46613923866MIT License0 days, 11 hrs, 37 mins
10lobe-chat🤯 Lobe Chat - an open-source, modern-design AI chat framework. Supports Multi AI Providers( OpenAI / Claude 3 / Gemini / Ollama / Azure / DeepSeek), Knowledge Base (file upload / knowledge management / RAG ), Multi-Modals (Vision/TTS) and plugin system. One-click FREE deployment of your private ChatGPT/ Claude application.44,0299,8453961591,020Other0 days, 14 hrs, 22 mins
11text-generation-webuiA Gradio web UI for Large Language Models.40,3915,29528732952GNU Affero General Public License v3.03 days, 17 hrs, 59 mins
12vllmA high-throughput and memory-efficient inference and serving engine for LLMs29,4514,4152,21845441Apache License 2.00 days, 8 hrs, 58 mins
13anything-llmThe all-in-one Desktop & Docker AI application with built-in RAG, AI agents, and more.25,3862,566185788MIT License0 days, 21 hrs, 57 mins
14LocalAI:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed inference24,3051,86338510963MIT License0 days, 17 hrs, 14 mins
15janJan is an open source alternative to ChatGPT that runs 100% offline on your computer. Multiple engine support (llama.cpp, TensorRT-LLM)23,0631,3361745533GNU Affero General Public License v3.00 days, 9 hrs, 14 mins
16chatboxUser-friendly Desktop Client App for AI Models/LLMs (GPT, Claude, Gemini, Ollama...)21,4722,1693752944GNU General Public License v3.09 days, 2 hrs, 7 mins
17llamafileDistribute and run LLMs with a single file.20,0701,0061344630Other1 days, 2 hrs, 1 mins
18localGPTChat with your documents on your local device using GPT models. No data leaves your device and 100% private.20,0392,238476440Apache License 2.04 days, 11 hrs, 25 mins
19mlc-llmUniversal LLM Deployment Engine with ML Compilation19,0861,5632001271Apache License 2.01 days, 12 hrs, 14 mins
20LibreChatEnhanced ChatGPT Clone: Features Anthropic, AWS, OpenAI, Assistants API, Azure, Groq, o1, GPT-4o, Mistral, OpenRouter, Vertex AI, Gemini, Artifacts, AI model switching, message search, langchain, DALL-E-3, ChatGPT Plugins, OpenAI Functions, Secure Multi-User System, Presets, completely open-source for self-hosting. Actively in public development.18,5073,11217616347MIT License1 days, 0 hrs, 59 mins
21ChuanhuChatGPTGUI for ChatGPT API and many LLMs. Supports agents, file-based QA, GPT finetuning and query with web search. All with a neat UI.15,2332,2941225125GNU General Public License v3.010 days, 7 hrs, 54 mins
22web-llmHigh-performance In-browser LLM Inference Engine13,50487071421Apache License 2.010 days, 20 hrs, 22 mins
23h2ogptPrivate chat with local GPT with document, images, video, etc. 100% private, Apache 2.0. Supports oLLaMa, Mixtral, llama.cpp, and more. Demo: https://gpt.h2o.ai/ https://gpt-docs.h2o.ai/11,3881,248281692Apache License 2.00 days, 10 hrs, 36 mins
24chathubAll-in-one chatbot client10,0161,022281120GNU General Public License v3.014 days, 8 hrs, 53 mins
25OpenLLMRun any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.9,9616352131131Apache License 2.03 days, 8 hrs, 51 mins
26FlexGenRunning large language models on a single GPU for throughput-oriented scenarios.9,18054857190Apache License 2.04 days, 11 hrs, 51 mins
27text-generation-inferenceLarge Language Model Text Generation Inference8,9791,05912411749Apache License 2.00 days, 12 hrs, 52 mins
28TensorRT-LLMTensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.8,543969790169Apache License 2.02 days, 23 hrs, 33 mins
29serverThe Triton Inference Server provides an optimized cloud and edge inferencing solution.8,2691,47362412072BSD 3-Clause "New" or "Revised" License0 days, 14 hrs, 46 mins
30SillyTavernLLM Frontend for Power Users.8,0722,38624616684GNU Affero General Public License v3.00 days, 19 hrs, 50 mins
31llama-cpp-pythonPython bindings for llama.cpp8,027951525156276MIT License0 days, 21 hrs, 6 mins
32chat-uiOpen source codebase powering the HuggingChat app7,5281,10527410814Apache License 2.01 days, 17 hrs, 47 mins
33openplaygroundAn LLM playground you can run on your laptop6,24348688150MIT License4 days, 8 hrs, 53 mins
34big-agiGenerative AI suite powered by state-of-the-art models and providing advanced AI/AGI functions. It features AI personas, AGI functions, multi-model chats, text-to-image, voice, response streaming, code highlighting and execution, PDF import, presets for developers, much more. Deploy on-prem or in the cloud.5,4481,2472094316MIT License3 days, 11 hrs, 40 mins
35inferenceReplace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.5,2614242067787Apache License 2.00 days, 8 hrs, 28 mins
36koboldcppRun GGUF models easily with a KoboldAI UI. One File. Zero Install.5,16935324346388GNU Affero General Public License v3.00 days, 21 hrs, 40 mins
37llmAccess large language models from the command-line4,5802532422229Apache License 2.02 days, 20 hrs, 36 mins
38lmdeployLMDeploy is a toolkit for compressing, deploying, and serving LLMs.4,5544093177637Apache License 2.00 days, 9 hrs, 8 mins
39lollms-webuiLord of Large Language Models Web User Interface4,3185431553823Apache License 2.00 days, 13 hrs, 45 mins
40exllamav2A fast inference library for running LLMs locally on modern consumer-class GPUs3,619279904634MIT License11 days, 19 hrs, 30 mins
41LLamaSharpA C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.2,6333431405622MIT License0 days, 8 hrs, 2 mins
42cortex.cppLocal AI API Platform2,02411412032138Apache License 2.00 days, 8 hrs, 18 mins
43nitroLocal AI API Platform2,02411412032138Apache License 2.00 days, 8 hrs, 16 mins
44page-assistUse your locally running AI models to assist you in your web browsing1,469140981220MIT License5 days, 19 hrs, 34 mins
45maidMaid is a cross-platform Flutter app for interfacing with GGUF / llama.cpp models locally, and with Ollama and OpenAI models remotely.1,434159172131MIT License3 days, 7 hrs, 29 mins
46LLMFarmllama and other large language models on iOS and MacOS offline using GGML library.1,3148419132MIT License1 days, 19 hrs, 35 mins
47oterma text-based terminal client for Ollama1,0376181235MIT License0 days, 17 hrs, 26 mins
48amicaAmica is an open source interface for interactive communication with 3D characters with voice synthesis and speech recognition.70011336164MIT License3 days, 5 hrs, 46 mins
49ChatterUISimple frontend for LLMs built in react-native.5002712145GNU Affero General Public License v3.01 days, 1 hrs, 47 mins
50exuiWeb UI for ExLlamaV2436413380MIT License22 days, 16 hrs, 37 mins
51avaAll-in-one desktop app for running LLMs locally.41715330Other6 days, 23 hrs, 57 mins
52tenere🤖 TUI interface for LLMs written in Rust35182713GNU General Public License v3.057 days, 4 hrs, 16 mins
53web-llm-chatChat with AI large language models running natively in your browser. Enjoy private, server-free, seamless AI conversations.30550101810Apache License 2.027 days, 18 hrs, 53 mins
54mikupadLLM Frontend in a single html file24827241040Creative Commons Zero v1.0 Universal15 days, 17 hrs, 35 mins
55emeltalLocal ML voice chat using high-end models.1428110MIT License5 days, 5 hrs, 5 mins

Inspired By