Awesome
OllamaFarm
OllamaFarm is a Go package that manages multiple Ollama instances, providing a convenient way to interact with a farm of Ollama servers. It offers features like automatic offline detection and failover, model availability tracking, and server selection based on criteria such as model.
Installation
To install OllamaFarm, use the following command:
go get github.com/presbrey/ollamafarm
Usage
Here's an example of how to use OllamaFarm with multiple Ollamas in the same group and different priorities:
package main
import (
"context"
"fmt"
"log"
"github.com/presbrey/ollamafarm"
"github.com/ollama/ollama/api"
)
func main() {
farm := ollamafarm.New()
// Register Ollama servers in the same group with different priorities
farm.RegisterURL("http://ollama1:11434", &ollamafarm.Properties{Group: "4090", Priority: 1})
farm.RegisterURL("http://ollama2:11434", &ollamafarm.Properties{Group: "4090", Priority: 2})
farm.RegisterURL("http://ollama3:11434", &ollamafarm.Properties{Group: "3090", Priority: 1})
// Select an Ollama instance
ollama := farm.First(&ollamafarm.Where{Model: "llama3.1:8b-instruct-fp16"})
if ollama != nil {
// Perform a Chat call
req := &api.ChatRequest{
Model: "llama3.1:8b-instruct-fp16",
Messages: []api.Message{
{Role: "user", Content: "How many letter R are in the word Strawberry?"},
},
}
err := ollama.Client().Chat(context.Background(), req, func(resp api.ChatResponse) error {
fmt.Print(resp.Message.Content)
return nil
})
if err != nil {
log.Fatalf("Chat error: %v", err)
}
}
// Get model counts
modelCounts := farm.ModelCounts(nil)
fmt.Printf("Available models: %v\n", modelCounts)
}
Note: When an Ollama instance goes offline, OllamaFarm automatically selects the next online Ollama with the highest priority (lowest priority number) within the same group. This ensures continuous operation and optimal resource utilization without manual intervention.
API Reference
Types
Farm
: The main struct that manages multiple Ollama instances.Ollama
: Represents an individual Ollama server.Options
: Defines the options for a Farm. All fields are optional.type Options struct { Client *http.Client Heartbeat time.Duration ModelsTTL time.Duration }
Properties
: Defines the properties of an Ollama client. All fields are optional.type Properties struct { Group string Offline bool Priority int }
Where
: Defines the selection criteria for Ollama clients.type Where struct { Group string Model string Offline bool }
Functions
New() *Farm
: Creates a new Farm instance with default options.NewWithOptions(options *Options) *Farm
: Creates a new Farm instance with the given options.
Farm Methods
RegisterClient(id string, client *api.Client, properties *Properties)
: Adds a new Ollama to the Farm if it doesn't already exist.RegisterURL(baseURL string, properties *Properties) error
: Adds a new Ollama to the Farm using the baseURL as the ID.First(where *Where) *Ollama
: Returns the first Ollama that matches the given criteria.Select(where *Where) []*Ollama
: Returns a list of Ollamas that match the given criteria, sorted by ascending Priority.ModelCounts(where *Where) map[string]uint
: Returns a count of all models available across all registered Ollamas.
Ollama Methods
Client() *api.Client
: Returns the Ollama client.Farm() *Farm
: Returns the Farm that this Ollama belongs to.Group() string
: Returns the Ollama's group.Online() bool
: Returns whether the Ollama is online.Priority() int
: Returns the Ollama's priority.
Contributing
Contributions to OllamaFarm are welcome! Please note the following guidelines:
- All pull requests must maintain or improve the existing test coverage.
- New features or changes must not break any existing APIs.
- Write clear, concise commit messages.
- Follow Go best practices and style guidelines.
License
This project is licensed under the MIT LICENSE file in the root directory of this repository.