Home

Awesome

Serge - LLaMA made easy ๐Ÿฆ™

License Discord

Serge is a chat interface crafted with llama.cpp for running GGUF models. No API keys, entirely self-hosted!

๐ŸŽฅ Demo:

demo.webm

โšก๏ธ Quick start

๐Ÿณ Docker:

docker run -d \
    --name serge \
    -v weights:/usr/src/app/weights \
    -v datadb:/data/db/ \
    -p 8008:8008 \
    ghcr.io/serge-chat/serge:latest

๐Ÿ™ Docker Compose:

services:
  serge:
    image: ghcr.io/serge-chat/serge:latest
    container_name: serge
    restart: unless-stopped
    ports:
      - 8008:8008
    volumes:
      - weights:/usr/src/app/weights
      - datadb:/data/db/

volumes:
  weights:
  datadb:

Then, just visit http://localhost:8008, You can find the API documentation at http://localhost:8008/api/docs

๐ŸŒ Environment Variables

The following Environment Variables are available:

Variable NameDescriptionDefault Value
SERGE_DATABASE_URLDatabase connection stringsqlite:////data/db/sql_app.db
SERGE_JWT_SECRETKey for auth token encryption. Use a random stringuF7FGN5uzfGdFiPzR
SERGE_SESSION_EXPIRYDuration in minutes before a user must reauthenticate60
NODE_ENVNode.js running environmentproduction

๐Ÿ–ฅ๏ธ Windows

Ensure you have Docker Desktop installed, WSL2 configured, and enough free RAM to run models.

โ˜๏ธ Kubernetes

Instructions for setting up Serge on Kubernetes can be found in the wiki.

๐Ÿง  Supported Models

CategoryModels
Alfred40B-1023
BioMistral7B
Code13B, 33B
CodeLLaMA7B, 7B-Instruct, 7B-Python, 13B, 13B-Instruct, 13B-Python, 34B, 34B-Instruct, 34B-Python
Codestral22B v0.1
Gemma2B, 1.1-2B-Instruct, 7B, 1.1-7B-Instruct, 2-9B, 2-9B-Instruct, 2-27B, 2-27B-Instruct
GorillaFalcon-7B-HF-v0, 7B-HF-v1, Openfunctions-v1, Openfunctions-v2
Falcon7B, 7B-Instruct, 11B, 40B, 40B-Instruct
LLaMA 27B, 7B-Chat, 7B-Coder, 13B, 13B-Chat, 70B, 70B-Chat, 70B-OASST
LLaMA 311B-Instruct, 13B-Instruct, 16B-Instruct
LLaMA Pro8B, 8B-Instruct
Mathstral7B
Med4270B, v2-8B, v2-70B
Medalpaca13B
MedicineChat, LLM
Meditron7B, 7B-Chat, 70B, 3-8B
Meta-LlaMA-33-8B, 3.1-8B, 3.2-1B-Instruct, 3-8B-Instruct, 3.1-8B-Instruct, 3.2-3B-Instruct, 3-70B, 3.1-70B, 3-70B-Instruct, 3.1-70B-Instruct
Mistral7B-V0.1, 7B-Instruct-v0.2, 7B-OpenOrca, Nemo-Instruct
MistralLite7B
Mixtral8x7B-v0.1, 8x7B-Dolphin-2.7, 8x7B-Instruct-v0.1
Neural-Chat7B-v3.3
Notus7B-v1
Notux8x7b-v1
Nous-Hermes 2Mistral-7B-DPO, Mixtral-8x7B-DPO, Mistral-8x7B-SFT
OpenChat7B-v3.5-1210? 8B-v3.6-20240522
OpenCodeInterpreterDS-6.7B, DS-33B, CL-7B, CL-13B, CL-70B
OpenLLaMA3B-v2, 7B-v2, 13B-v2
Orca 27B, 13B
Phi2-2.7B, 3-mini-4k-instruct, 3.1-mini-4k-instruct, 3.1-mini-128k-instruct,3.5-mini-instruct, 3-medium-4k-instruct, 3-medium-128k-instruct
Python Code13B, 33B
PsyMedRP13B-v1, 20B-v1
Starling LM7B-Alpha
SOLAR10.7B-v1.0, 10.7B-instruct-v1.0
TinyLlama1.1B
Vicuna7B-v1.5, 13B-v1.5, 33B-v1.3, 33B-Coder
WizardLM2-7B, 13B-v1.2, 70B-v1.0
Zephyr3B, 7B-Alpha, 7B-Beta

Additional models can be requested by opening a GitHub issue. Other models are also available at Serge Models.

โš ๏ธ Memory Usage

LLaMA will crash if you don't have enough available memory for the model

๐Ÿ’ฌ Support

Need help? Join our Discord

๐Ÿงพ License

Nathan Sarrazin and Contributors. Serge is free and open-source software licensed under the MIT License and Apache-2.0.

๐Ÿค Contributing

If you discover a bug or have a feature idea, feel free to open an issue or PR.

To run Serge in development mode:

git clone https://github.com/serge-chat/serge.git
cd serge/
docker compose -f docker-compose.dev.yml up --build

The solution will accept a python debugger session on port 5678. Example launch.json for VSCode:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Remote Debug",
            "type": "python",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}/api",
                    "remoteRoot": "/usr/src/app/api/"
                }
            ],
            "justMyCode": false
        }
    ]
}