Home

Awesome

Go Report Card codecov Go Test GoDoc License: MIT

OllamaFarm

OllamaFarm is a Go package that manages multiple Ollama instances, providing a convenient way to interact with a farm of Ollama servers. It offers features like automatic offline detection and failover, model availability tracking, and server selection based on criteria such as model.

Installation

To install OllamaFarm, use the following command:

go get github.com/presbrey/ollamafarm

Usage

Here's an example of how to use OllamaFarm with multiple Ollamas in the same group and different priorities:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/presbrey/ollamafarm"
    "github.com/ollama/ollama/api"
)

func main() {
    farm := ollamafarm.New()

    // Register Ollama servers in the same group with different priorities
    farm.RegisterURL("http://ollama1:11434", &ollamafarm.Properties{Group: "4090", Priority: 1})
    farm.RegisterURL("http://ollama2:11434", &ollamafarm.Properties{Group: "4090", Priority: 2})
    farm.RegisterURL("http://ollama3:11434", &ollamafarm.Properties{Group: "3090", Priority: 1})

    // Select an Ollama instance
    ollama := farm.First(&ollamafarm.Where{Model: "llama3.1:8b-instruct-fp16"})
    if ollama != nil {
        // Perform a Chat call
        req := &api.ChatRequest{
            Model: "llama3.1:8b-instruct-fp16",
            Messages: []api.Message{
                {Role: "user", Content: "How many letter R are in the word Strawberry?"},
            },
        }

        err := ollama.Client().Chat(context.Background(), req, func(resp api.ChatResponse) error {
            fmt.Print(resp.Message.Content)
            return nil
        })

        if err != nil {
            log.Fatalf("Chat error: %v", err)
        }
    }

    // Get model counts
    modelCounts := farm.ModelCounts(nil)
    fmt.Printf("Available models: %v\n", modelCounts)
}

Note: When an Ollama instance goes offline, OllamaFarm automatically selects the next online Ollama with the highest priority (lowest priority number) within the same group. This ensures continuous operation and optimal resource utilization without manual intervention.

API Reference

Types

Functions

Farm Methods

Ollama Methods

Contributing

Contributions to OllamaFarm are welcome! Please note the following guidelines:

  1. All pull requests must maintain or improve the existing test coverage.
  2. New features or changes must not break any existing APIs.
  3. Write clear, concise commit messages.
  4. Follow Go best practices and style guidelines.

License

This project is licensed under the MIT LICENSE file in the root directory of this repository.