Home

Awesome

Guble Messaging Server

Guble is a simple user-facing messaging and data replication server written in Go.

Codacy Badge Release Docker Build Status Go Report Card codebeat badge Coverage Status GoDoc Awesome-Go

Overview

Guble is in an early state (release 0.4). It is already working well and is very useful, but the protocol, API and storage formats may still change (until reaching 0.7). If you intend to use guble, please get in contact with us.

The goal of guble is to be a simple and fast message bus for user interaction and replication of data between multiple devices:

Working Features (0.4)

Throughput

Measured on an old notebook with i5-2520M, dual core and SSD. Message payload was 'Hello Word'. Load driver and server were set up on the same machine, so 50% of the cpu was allocated to the load driver.

During the tests, the memory consumption of the server was around ~25 MB.

Table of Contents

Roadmap

This is the current (and fast changing) roadmap and todo list:

Roadmap Release 0.5

Roadmap Release 0.6

Roadmap Release 0.7

Guble Docker Image

We are providing Docker images of the server and client for your convenience.

Start the Guble Server

There is an automated Docker build for the master at the Docker Hub. To start the server with Docker simply type:

docker run -p 8080:8080 smancke/guble

To see available configuration options:

docker run smancke/guble --help

All options can be supplied on the commandline or by a corresponding environment variable with the prefix GUBLE_. So to let guble be more verbose, you can either use:

docker run smancke/guble --log=info

or

docker run -e GUBLE_LOG=info smancke/guble

The Docker image has a volume mount point at /var/lib/guble, so if you want to bind-mount the persistent storage from your host you should use:

docker run -p 8080:8080 -v /host/storage/path:/var/lib/guble smancke/guble

Connecting with the Guble Client

The Docker image includes the guble commandline client guble-cli. You can execute it within a running guble container and connect to the server:

docker run -d --name guble smancke/guble
docker exec -it guble /usr/local/bin/guble-cli

Visit the guble-cli documentation for more details.

Build and Run

Since Go makes it very easy to build from source, you can compile guble using a single command. A prerequisite is having an installed Go environment and an empty directory:

sudo apt-get install golang
mkdir guble && cd guble
export GOPATH=`pwd`

Build and Start the Server

Build and start guble with the following commands (assuming that directory /var/lib/guble is already created with read-write rights for the current user):

go get github.com/smancke/guble
bin/guble --log=info

Configuration

CLI OptionEnv VariableValuesDefaultDescription
--envGUBLE_ENVdevelopment | integration | preproduction | productiondevelopmentName of the environment on which the application is running. Used mainly for logging
--health-endpointGUBLE_HEALTH_ENDPOINTresource/path/to/healthendpoint/admin/healthcheckThe health endpoint to be used by the HTTP server.Can be disabled by setting the value to ""
--httpGUBLE_HTTP_LISTENformat: [host]:portThe address to for the HTTP server to listen on
--kvsGUBLE_KVSmemory | file | postgresfileThe storage backend for the key-value store to use
--logGUBLE_LOGpanic | fatal | error | warn | info | debugerrorThe log level in which the process logs
--metrics-endpointGUBLE_METRICS_ENDPOINTresource/path/to/metricsendpoint/admin/metricsThe metrics endpoint to be used by the HTTP server.Can be disabled by setting the value to ""
--msGUBLE_MSmemory | filefileThe message storage backend
--profileGUBLE_PROFILEcpu | mem | blockThe profiler to be used
--storage-pathGUBLE_STORAGE_PATHpath/to/storage/var/lib/gubleThe path for storing messages and key-value data like subscriptions if defined.The path must exists!

APNS

CLI OptionEnv VariableValuesDefaultDescription
--apnsGUBLE_APNStrue | falsefalseEnable the APNS module in general as well as the connector to the development endpoint
--apns-productionGUBLE_APNS_PRODUCTIONtrue | falsefalseEnables the connector to the apns production endpoint, requires the apns option to be set
--apns-cert-fileGUBLE_APNS_CERT_FILEpath/to/cert/fileThe APNS certificate file name, use this as an alternative to the certificate bytes option
--apns-cert-bytesGUBLE_APNS_CERT_BYTEScert-bytes-as-hex-stringThe APNS certificate bytes, use this as an alternative to the certificate file option
--apns-cert-passwordGUBLE_APNS_CERT_PASSWORDpasswordThe APNS certificate password
--apns-app-topicGUBLE_APNS_APP_TOPICtopicThe APNS topic (as used by the mobile application)
--apns-prefixGUBLE_APNS_PREFIXprefix/apns/The APNS prefix / endpoint
--apns-workersGUBLE_APNS_WORKERSnumber of workersNumber of CPUsThe number of workers handling traffic with APNS (default: number of CPUs)

SMS

CLI OptionEnv VariableValuesDefaultDescription
smsGUBLE_SMStrue | falsefalseEnable the SMS gateway
sms_api_keyGUBLE_SMS_API_KEYapi keyThe Nexmo API Key for Sending sms
sms_api_secretGUBLE_SMS_API_SECRETapi secretThe Nexmo API Secret for Sending sms
sms_topicGUBLE_SMS_TOPICtopic/smsThe topic for sms route
sms_workersGUBLE_SMS_WORKERSnumber of workersNumber of CPUsThe number of workers handling traffic with Nexmo sms endpoint

FCM

CLI OptionEnv VariableValuesDefaultDescription
`--fcmGUBLE_FCM`true | falsefalseEnable the Google Firebase Cloud Messaging connector
--fcm-api-keyGUBLE_FCM_API_KEYapi keyThe Google API Key for Google Firebase Cloud Messaging
--fcm-workersGUBLE_FCM_WORKERSnumber of workersNumber of CPUsThe number of workers handling traffic with Firebase Cloud Messaging
--fcm-endpointGUBLE_FCM_ENDPOINTformat: url-schemahttps://fcm.googleapis.com/fcm/sendThe Google Firebase Cloud Messaging endpoint
--fcm-prefixGUBLE_FCM_PREFIXprefix/fcm/The FCM prefix / endpoint

Postgres

CLI OptionEnv VariableValuesDefaultDescription
--pg-hostGUBLE_PG_HOSThostnamelocalhostThe PostgreSQL hostname
--pg-portGUBLE_PG_PORTport5432The PostgreSQL port
--pg-userGUBLE_PG_USERusergubleThe PostgreSQL user
--pg-passwordGUBLE_PG_PASSWORDpasswordgubleThe PostgreSQL password
--pg-dbnameGUBLE_PG_DBNAMEdatabasegubleThe PostgreSQL database name

Run All Tests

go get -t github.com/smancke/guble/...
go test github.com/smancke/guble/...

Clients

The following clients are available:

Protocol Reference

REST API

Currently there is a minimalistic REST API, just for publishing messages.

POST /api/message/<topic>

URL parameters:

Headers

You can set fields in the header JSON of the message by providing the corresponding HTTP headers with the prefix X-Guble-.

Curl example with the resulting message:

curl -X POST -H "x-Guble-Key: Value" --data Hello 'http://127.0.0.1:8080/api/message/foo?userId=marvin&messageId=42'

Results in:

16,/foo,marvin,VoAdxGO3DBEn8vv8,42,1451236804
{"Key":"Value"}
Hello

WebSocket Protocol

The communication with the guble server is done by ordinary WebSockets, using a binary encoding.

Message Format

All payload messages sent from the server to the client are using the following format:

<path:string>,<sequenceId:int64>,<publisherUserId:string>,<publisherApplicationId:string>,<publisherMessageId:string>,<messagePublishingTime:unix-timestamp>\n
[<application headers json>]\n
<body>

example 1:
/foo/bar,42,user01,phone1,id123,1420110000
{"Content-Type": "text/plain", "Correlation-Id": "7sdks723ksgqn"}
Hello World

example 2:
/foo/bar,42,user01,54sdcj8sd7,id123,1420110000

anyByteData

Client Commands

The client can send the following commands.

Send

Publish a message to a topic:

> <path> [<publisherMessageId>]\n
[<header>\n]..
\n
<body>

example:
> /foo

Hello World

Subscribe/Receive

Receive messages from a path (e.g. a topic or subtopic). This command can be used to subscribe for incoming messages on a topic, as well as for replaying the message history.

+ <path> [<startId>[,<maxCount>]]

Note: Currently, the fetching of stored messages does not recognize subtopics.

Examples:

+ /foo         # Subscribe to all future messages matching /foo
+ /foo/bar     # Subscribe to all future messages matching /foo/bar

+ /foo 0       # Receive all message from the topic and subscribe for further incoming messages.

+ /foo 42      # Receive all message with message ids >= 42
               # from the topic and subscribe for further incoming messages.

+ /foo 0 20    # Receive the first (oldest) 20 messages within the topic and stop.
               # (If the topic has less messages, it will stop after receiving all existing ones.)

+ /foo -20     # Receive the last (newest) 20 messages from the topic and then
               # subscribe for further incoming messages.

+ /foo -20 20  # Receive the last (newest) 20 messages within the topic and stop.
               # (If the topic has less messages, it will stop after receiving all existing ones.)

Unsubscribe/Cancel

Cancel further receiving of messages from a path (e.g. a topic or subtopic).

- <path>

example:
- /foo
- /foo/bar

Server Status Messages

The server sends status messages to the client. All positive status messages start with >. Status messages reporting an error start with !. Status messages are in the following format.

'#'<msgType> <Explanation text>\n
<json data>

Connection Message

#ok-connected You are connected to the server.\n
{"ApplicationId": "the app id", "UserId": "the user id", "Time": "the server time as unix timestamp "}

Example:

#connected You are connected to the server.
{"ApplicationId": "phone1", "UserId": "user01", "Time": "1420110000"}

Send Success Notification

This notification confirms, that the messaging system has successfully received the message and now starts transmitting it to the subscribers:

#send <publisherMessageId>
{"sequenceId": "sequence id", "path": "/foo", "publisherMessageId": "publishers message id", "messagePublishingTime": "unix-timestamp"}

Receive Success Notification

Depending on the type of + (receive) command, up to three different notification messages will be sent back. Be aware, that a server may send more receive notifications that you would have expected in first place, e.g. when:

  1. When the fetch operation starts:

    #fetch-start <path> <count>
    • path: the topic path
    • count: the number of messages that will be returned

  2. When the fetch operation is done:

    #fetch-done <path>
    • path: the topic path

  3. When the subscription to new messages was taken:

    #subscribed-to <path>
    • path: the topic path

Unsubscribe Success Notification

An unsubscribe/cancel operation is confirmed by the following notification:

#canceled <path>

Send Error Notification

This message indicates, that the message could not be delivered.

!error-send <publisherMessageId> <error text>
{"sequenceId": "sequence id", "path": "/foo", "publisherMessageId": "publishers message id", "messagePublishingTime": "unix-timestamp"}

Bad Request

This notification has the same meaning as the http 400 Bad Request.

!error-bad-request unknown command 'sdcsd'

Internal Server Error

This notification has the same meaning as the http 500 Internal Server Error.

!error-server-internal this computing node has problems

Topics

Messages can be hierarchically routed by topics, so they are represented by a path, separated by /. The server takes care, that a message only gets delivered once, even if it is matched by multiple subscription paths.

Subtopics

The path delimiter gives the semantic of subtopics. With this, a subscription to a parent topic (e.g. /foo) also results in receiving all messages of the subtopics (e.g. /foo/bar).