Awesome

cortex.llamacpp

cortex.llamacpp is a high-efficiency C++ inference engine for edge computing.

It is a dynamic library that can be loaded by any server at runtime.

Repo Structure

.
├── base -> Engine interface
├── examples -> Server example to integrate engine
├── llama.cpp -> Upstream llama C++
├── src -> Engine implementation
├── third-party -> Dependencies of the cortex.llamacpp project

Build from source

This guide provides step-by-step instructions for building cortex.llamacpp from source on Linux, macOS, and Windows systems.

Clone the Repository

First, you need to clone the cortex.llamacpp repository:

git clone --recurse https://github.com/janhq/cortex.llamacpp.git

If you don't have git, you can download the source code as a file archive from cortex.llamacpp GitHub.

Build library with server example

• On Windows Install choco Install make

  choco install make -y
  

• On Linux, and Windows:

  make build-example-server CMAKE_EXTRA_FLAGS=""
  

• On MacOS with Apple Silicon:

  make build-example-server CMAKE_EXTRA_FLAGS="-DGGML_METAL_EMBED_LIBRARY=ON"
  

• On MacOS with Intel processors:

  make build-example-server CMAKE_EXTRA_FLAGS="-DGGML_METAL=OFF"
  

• On Linux with CUDA:

  make build-example-server CMAKE_EXTRA_FLAGS="-DGGML_CUDA=ON"
  

Quickstart

Step 1: Downloading a Model

mkdir model && cd model
wget -O llama-2-7b-model.gguf https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q5_K_M.gguf?download=true

Step 2: Start server

• On MacOS:

  mkdir engines\cortex.llamacpp\
  cp ../../build/libengine.dylib engines/cortex.llamacpp/
  ./server
  

• On Linux:

  mkdir engines\cortex.llamacpp\
  cp ../../build/libengine.so engines/cortex.llamacpp/
  ./server
  

• On Windows:

  cd Release
  mkdir engines\cortex.llamacpp\
  copy ..\..\..\build\Release\engine.dll engines\cortex.llamacpp\
  server.exe
  

Step 3: Load model

curl http://localhost:3928/loadmodel \
  -H 'Content-Type: application/json' \
  -d '{
    "llama_model_path": "/model/llama-2-7b-model.gguf",
    "model_alias": "llama-2-7b-model",
    "ctx_len": 512,
    "ngl": 100,
    "model_type": "llm"
  }'

Step 4: Making an Inference

curl http://localhost:3928/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Who won the world series in 2020?"
      },
    ],
    "model": "llama-2-7b-model"
  }'

Table of parameters