Home

Awesome

HFXEventStash

HFXEventStash is an event store implementation for NodeJS using gRPC and Logstash.

npm install --save hfxeventstash

How it works

HFXEventStash uses gRPC and Logstash to provide an asynchronous event store.

The exposed gRPC server has an endpoint that receives a stream of events and puts on Logstash's stdin to be stored in any storage system.

You can specify proto definitions for HFXEventStash, so you can send raw protobufs to server to be parsed into JSON to send to Logstash.

We provide a full implementation of Elasticsearch as an event store, but you're free to choose any output that Logstash can handle.

Configuration

HFXEventStash is configured through some environment variables and directories.

Environment Variables:

Directories:

Healthcheck

To do a healthcheck on HFXEventStash you just need to run node build/healthcheck.js, this process will return 0 if the server is healthy or 1 otherwise.

Stopping the server

HFXEventStash gRPC server accepts the SIGINT and SIGTERM signals to gracefuly shutdown everything, it'll close the server and wait for Logstash to exit (sendin a SIGTERM) if this whole process takes more than SHUTDOWN_TIMEOUT it'll be auto-killed. Be careful with the value of SHUTDOWN_TIMEOUT if you're using in memory queue storage on Logstash, it can cause data loss.

The event structure

This is the request protobuf definition of HFXEventStash server:

message StoreEventRequest {
  enum EventFormat {
    PROTOBUF = 0;
    JSON = 1;
  }
  // This enum defines the format of the "data" property
  EventFormat format = 1;
  // A buffer with data, can be a protobuf message or a JSON string buffer
  bytes data = 2;
  // This can be:
  //   1 - Just a label when the format is JSON 
  //   2 - The protobuf message's type to be parsed by HFXEventStash
  string kind = 3;
  // The ID of event, if you don't specify a value an UUIDv4 will be generated
  string id = 4;
}

And the response is:

message StoreEventResponse {
  bool success = 1;
}

If you're using Elasticsearch

We provide an standard and recommended configurations to store events in Elasticsearch, you can customize it to fit into your needs.

Tutorial for Kubernetes server

TODO

Tutorial for Docker server

  1. Setup your logstash.conf
  2. Start the container: docker run --rm -it -p 42043:42043 -v /path/to/your/logstash.conf:/usr/src/hfxeventstash/lib/logstash.conf gamaops/hfxeventstash:stable
  3. If you want to run a healthcheck, enter in the container and execute: node build/healthcheck.js
    1. After executing the healthcheck you can see the process exit code using the following command: echo $?

Tutorial for clients

Take a look at the examples folder.

import makeClient from 'hfxeventstash';

const client = makeClient({
  uri: '127.0.0.1:42043' // The gRPC HFXEventStash server address
});

const addressbook = {};

const call = client.eventStash.storeEvent((error, response) => {
  console.log('Error:', error);
  console.log('Response:', response);
});

call.write({
  format: 'JSON',
  kind: 'contact.AddressBook',
  data: Buffer.from(JSON.stringify(addressbook)),
});

call.end();

Roadmap

Related Projects