Home

Awesome

gokv

Go Reference Build status Go Report Card codecov GitHub Releases Mentioned in Awesome Go

Simple key-value store abstraction and implementations for Go

Contents

  1. Features
    1. Simple interface
    2. Implementations
    3. Value types
    4. Marshal formats
    5. Roadmap
  2. Usage
    1. Examples
  3. Project status
  4. Motivation
  5. Design decisions
  6. Related projects
  7. License

Features

Simple interface

Note: The interface is not final yet! See Project status for details.

type Store interface {
    Set(k string, v any) error
    Get(k string, v any) (found bool, err error)
    Delete(k string) error
    Close() error
}

There are detailed descriptions of the methods in the docs and in the code. You should read them if you plan to write your own gokv.Store implementation or if you create a Go package with a method that takes a gokv.Store as parameter, so you know exactly what happens in the background.

Implementations

Some of the following databases aren't specifically engineered for storing key-value pairs, but if someone's running them already for other purposes and doesn't want to set up one of the proper key-value stores due to administrative overhead etc., they can of course be used as well. In those cases let's focus on a few of the most popular though. This mostly goes for the SQL, NoSQL and NewSQL categories.

Feel free to suggest more stores by creating an issue or even add an actual implementation - PRs Welcome.

For differences between the implementations, see Choosing an implementation.
For the Godoc of specific implementations, see https://pkg.go.dev/github.com/philippgille/gokv#section-directories.

Again:
For differences between the implementations, see Choosing an implementation.
For the Godoc of specific implementations, see https://pkg.go.dev/github.com/philippgille/gokv#section-directories.

Value types

Most Go packages for key-value stores just accept a []byte as value, which requires developers for example to marshal (and later unmarshal) their structs. gokv is meant to be simple and make developers' lifes easier, so it accepts any type (with using any/interface{} as parameter), including structs, and automatically (un-)marshals the value.

The kind of (un-)marshalling is left to the implementation. All implementations in this repository currently support JSON and gob by using the encoding subpackage in this repository, which wraps the core functionality of the standard library's encoding/json and encoding/gob packages. See Marshal formats for details.

For unexported struct fields to be (un-)marshalled to/from JSON/gob, the respective custom (un-)marshalling methods need to be implemented as methods of the struct (e.g. MarshalJSON() ([]byte, error) for custom marshalling into JSON). See Marshaler and Unmarshaler for JSON, and GobEncoder and GobDecoder for gob.

To improve performance you can also implement the custom (un-)marshalling methods so that no reflection is used by the encoding/json / encoding/gob packages. This is not a disadvantage of using a generic key-value store package, it's the same as if you would use a concrete key-value store package which only accepts []byte, requiring you to (un-)marshal your structs.

Marshal formats

This repository contains the subpackage encoding, which is an abstraction and wrapper for the core functionality of packages like encoding/json and encoding/gob. The currently supported marshal formats are:

More formats will be supported in the future (e.g. XML).

The stores use this encoding package to marshal and unmarshal the values when storing / retrieving them. The default format is JSON, but all gokv.Store implementations in this repository also support gob as alternative, configurable via their Options.

The marshal format is up to the implementations though, so package creators using the gokv.Store interface as parameter of a function should not make any assumptions about this. If they require any specific format they should inform the package user about this in the GoDoc of the function taking the store interface as parameter.

Differences between the formats:

Roadmap

Usage

First, download the module you want to work with:

Then you can import and use it.

Every implementation has its own Options struct, but all implementations have a NewStore() / NewClient() function that returns an object of a sctruct that implements the gokv.Store interface. Let's take the implementation for Redis as example, which is the most popular distributed key-value store.

package main

import (
    "fmt"

    "github.com/philippgille/gokv"
    "github.com/philippgille/gokv/redis"
)

type foo struct {
    Bar string
}

func main() {
    options := redis.DefaultOptions // Address: "localhost:6379", Password: "", DB: 0

    // Create client
    client, err := redis.NewClient(options)
    if err != nil {
        panic(err)
    }
    defer client.Close()

    // Store, retrieve, print and delete a value
    interactWithStore(client)
}

// interactWithStore stores, retrieves, prints and deletes a value.
// It's completely independent of the store implementation.
func interactWithStore(store gokv.Store) {
    // Store value
    val := foo{
        Bar: "baz",
    }
    err := store.Set("foo123", val)
    if err != nil {
        panic(err)
    }

    // Retrieve value
    retrievedVal := new(foo)
    found, err := store.Get("foo123", retrievedVal)
    if err != nil {
        panic(err)
    }
    if !found {
        panic("Value not found")
    }

    fmt.Printf("foo: %+v", *retrievedVal) // Prints `foo: {Bar:baz}`

    // Delete value
    err = store.Delete("foo123")
    if err != nil {
        panic(err)
    }
}

As described in the comments, that code does the following:

  1. Create a client for Redis
    • Some implementations' stores/clients don't require to be closed, but when working with the interface (for example as function parameter) you must call Close() because you don't know which implementation is passed. Even if you work with a specific implementation you should always call Close(), so you can easily change the implementation without the risk of forgetting to add the call.
  2. Call interactWithStore(), which requires a gokv.Store as parameter. This method then:
    1. Stores an object of type foo in the Redis server running on localhost:6379 with the key foo123
    2. Retrieves the value for the key foo123
      • The check if the value was found isn't needed in this example but is included for demonstration purposes
    3. Prints the value. It prints foo: {Bar:baz}, which is exactly what was stored before.
    4. Deletes the value

Now let's say you don't want to use Redis but Consul instead. You just have to make three simple changes:

  1. Replace the import of "github.com/philippgille/gokv/redis" by "github.com/philippgille/gokv/consul"
  2. Replace redis.DefaultOptions by consul.DefaultOptions
  3. Replace redis.NewClient(options) by consul.NewClient(options)

Everything else works the same way. interactWithStore() is completely unaffected.

Examples

See the examples directory for more code examples.

Project status

Note: gokv's API is not stable yet and is under active development. Upcoming releases are likely to contain breaking changes as long as the version is v0.x.y. This project adheres to Semantic Versioning and all notable changes to this project are documented in CHANGELOG.md.

Planned interface methods until v1.0.0:

The interface might even change until v1.0.0. For example one consideration is to change Get(string, any) (bool, error) to Get(string, any) error (no boolean return value anymore), with the error being something like gokv.ErrNotFound // "Key-value pair not found" to fulfill the additional role of indicating that the key-value pair wasn't found. But at the moment we prefer the current method signature.

Also, more interfaces might be added. For example so that there's a SimpleStore and an AdvancedStore, with the first one containing only the basic methods and the latter one with advanced features such as key-value pair lifetimes (deletion of key-value pairs after a given time), notification of value changes via Go channels etc. But currently the focus is simplicity, see Design decisions.

Motivation

When creating a package you want the package to be usable by as many developers as possible. Let's look at a specific example: You want to create a paywall middleware for the Gin web framework. You need some database to store state. You can't use a Go map, because its data is not persisted across web service restarts. You can't use an embedded DB like bbolt, BadgerDB or SQLite, because that would restrict the web service to one instance, but nowadays every web service is designed with high horizontal scalability in mind. If you use Redis, MongoDB or PostgreSQL though, you would force the package user (the developer who creates the actual web service with Gin and your middleware) to run and administrate the server, even if she might never have used it before and doesn't know how to configure them for high performance and security.

Any decision for a specific database would limit the package's usability.

One solution would be a custom interface where you would leave the implementation to the package user. But that would require the developer to dive into the details of the Go package of the chosen key-value store. And if the developer wants to switch the store, or maybe use one for local testing and another for production, she would need to write multiple implementations.

gokv is the solution for these problems. Package creators use the gokv.Store interface as parameter and can call its methods within their code, leaving the decision which actual store to use to the package user. Package users pick one of the implementations, for example github.com/philippgille/gokv/redis for Redis and pass the redis.Client created by redis.NewClient(...) as parameter. Package users can also develop their own implementations if they need to.

gokv doesn't just have to be used to satisfy some gokv.Store parameter. It can of course also be used by application / web service developers who just don't want to dive into the sometimes complicated usage of some key-value store packages.

Initially it was developed as storage package within the project ln-paywall to provide the users of ln-paywall with multiple storage options, but at some point it made sense to turn it into a repository of its own.

Before doing so I examined existing Go packages with a similar purpose (see Related projects), but none of them fit my needs. They either had too few implementations, or they didn't automatically marshal / unmarshal passed structs, or the interface had too many methods, making the project seem too complex to maintain and extend, proven by some that were abandoned or forked (splitting the community with it).

Design decisions

Related projects

Others:

License

gokv is licensed under the Mozilla Public License Version 2.0.

Dependencies might be licensed under other licenses.