Home

Awesome

Health Checks for microservices

npm version GitHub (release) node (tag) Build Status Commitizen friendly codecov semantic-release Waffle.io - Issues in progress Known Vulnerabilities Greenkeeper badge

A Node.js implementation of the Health Checks API provided by Hootsuite.

<!-- TOC --> <!-- /TOC -->

Installation

npm install --save healthchecks-api

Functionality

Enables an application/service combined Health Check view when used in the ecosystem of microservices. This includes a service/application self check and following dependency checks:

The dependencies can be:

NOTE

The critical/non-critical dependency atribute is an additional (optional) semantic of this module only.

Health Checks API does not specify such.

Classifying a particular dependency as non-critical (critical: false attribute of a dependency configuration) results in reporting it being in a WARN state at the dependency owner level, when the dependency is reported being in either WARN or CRIT state at its own level.

Example configuration for non-critical dependency:

checks:
  - name: service-2
    critical: false
    check: http
    url: http://service-2:3002

By default all dependencies are classified as critical.

Another dependency division is:

NOTE

The traversable dependency capability is resolved by this module in a runtime.

Health status reports

The health is reported in following states:

The overall health state of the subject service/application is an aggregation of its own state and its dependencies state. Aggregation is done with a respect to following (the order matters!):

Usage

The module works as a middleware, exposing the Health Checks API routes via chosen http server framework routing system.

Service details (about endpoint)

The Health Checks API about endpoint is supposed to describe the underlying service using this module. The module takes particular service description attributes either from the Configuration or mapping them from the service's package.json as a fallback. When particular attribute is missing both in the service config and in package.json a default value is taken, when provided.

Here is the table with particular fields, their mapping to config attributes and fallback mapping to package.json and optional defaults:

Attribute nameConfig attribute namepackage.json fallback - attribute mappingStatic or dynamic fallback (defaults)
idnamename-
namenamename-
descriptiondescriptiondescription-
versionversionversion'x.x.x'
hosthost-require('os').hostname()
protocolprotocol-'http'
projectHomeprojectHomehomepage-
projectRepoprojectReporepository.url'unknown'
ownersownersauthor + contributors-
logsLinkslogsLinks--
statsLinksstatsLinks--
dependencieschecks--

NOTE

The final value is resolved with a fallback from left to right, as presented in above table.

Configuration

The module configuration is a single yaml file and represents the subject service/application context. The default path for the config file is ./conf/dependencies.yml.

An example config:

version: "3.1"

name: demo-app
description: Nice demo application :)

checks:
  - check: self
    memwatch: memwatch-next

  - name: mongo
    url: mongodb://mongo/test
    type: internal
    interval: 3000
    check: mongo

  - name: service-1
    url: http://service-1:3001
    type: external
    interval: 1000
    check: http

  - name: service-2
    url: http://service-2:3002
    type: external
    critical: false
    interval: 1000
    check: http

NOTE

Alternatively the configuration can be passed directly to the module initialization as an options.service.config attribute object value:

const healthCheck = require('healthchecks-api');
const express = require('express');
const app = express();
await healthCheck(app,
       {
           adapter: 'express',
           service: {
               config: {
                  name: 'demo-app',
                  description: 'Nice demo application :)',
                  statsLinks: [ 'https://my-stats/demo-app' ],
                  logsLinks: [ 'https://my-logs/demo-app/info', 'https://my-logs/demo-app/debug' ],
                  checks: [
                      {
                          name: 'mongo',
                          url: 'mongodb://mongo/test',
                          type: 'internal',
                          interval: 3000,
                          check: 'mongo',
                      },
                      {
                          name: 'service-1',
                          url: 'http://service-1:3001',
                          interval: 1000,
                          check: 'http',
                      }
                   ]
               },
           },
       })

Initialization

The library initialization depends on chosen http server framework, but in any case this will be about 2 lines of code. See the examples below.

Example - Express.js powered application

See: Express.js framework.

const healthCheck = require('healthchecks-api');

const startServer = async () => {
    const app = express();
    // some initialization steps

    await healthCheck(app);
    // rest of initialization steps
}

startServer();

Check types

Following check types are supported:

self check

The service/application check for its own health. It checks the CPU and Memory consumption [%] and verifies them against given alarm levels:

Example config:

checks:
  - check: self
    memwatch: memwatch-next
    secondsToKeepMemoryLeakMsg: 60
    metrics:
      memoryUsage:
        warn: 90
        crit: 95
      cpuUsage:
        warn: 50
        crit: 80

Memory leak detection

Additionally this check can listen and react to a memory leak event leveraging the memwatch-next library or one of its ancestors/descendants. Due to provide this functionality set the memwatch property to the name of the library in NPM, as shown in the example config above.

The linked library must provide the leak event as in the example below:

const memwatch = require('memwatch-next');
memwatch.on('leak', function(info) { ... });

http check

The health check for a linked HTTP service, usually an API provider.

Example config:

checks:
  - check: http
    name: service-2
    url: http://service-2:3002
    method: get # default
    type: external # default
    interval: 3000 # default [milliseconds]
    critical: true # default

Checks whether the service responds at given url. Determines whether it is traversable (will work only when given method is get) and resolves aggregated service state when so.

mongo check

Checks the availability of the Mongo DB instance at given url.

Example config:

checks:
  - check: mongo
    name: mongo
    url: mongodb://mongo/test
    type: internal
    interval: 3000

redis check

Checks the availability of the Redis instance at given url.

Example config:

checks:
  - check: redis
    name: redis
    url: redis://redis
    type: internal

elasticsearch check

Checks the availability of the Elasticsearch instance at given url.

Example config:

checks:
  - check: elasticsearch
    name: elasticsearch
    url: elasticsearch:9200
    type: internal

mysql check

Checks the availability of the MySQL instance at given url.

Example config:

checks:
 - name: mysql
    url: mysql
    type: internal
    interval: 3000
    check: mysql
    user: root
    password: example
    database: mysql

NOTE

The url config property maps to the host property of the mysql connection options.

Development

Contribution welcome for:

PRs with any improvements and issue reporting welcome as well!

Framework adapters

The module is designed to operate as a middleware in various http based Node.js frameworks. Currently supported frameworks are:

A particular framework implementation is an adapter exposing a single method.

Here's the example for the Express.js framework:

/**
 * Adds a particular Health Check API route to the `http` server framework.
 * The route is one of these difined by the {@link https://hootsuite.github.io/health-checks-api/|Health Checks API} specification.
 * The implementation should call given `route.handler` due to receive a response data.
 * The route handler takes combined request parameters and the service descriptor (usually `package.json` as an object) as the parameters
 * and returns the object with the response data (`status`, `headers`, `contentType` and `body`).
 * 
 * @async
 * @param {Object} service      - The service/application descriptor (usually a package.json);
 * @param {Object} server       - The Express.js application or Route.
 * @param {Object} route        - The Health Check API route descriptor object.
 * @param {string} route.path   - The Health Check API route path.
 * @param {Function} route.path - The Health Check API route handler function.
 * @returns {Promise}
 */
const express = async (service, server, route) => {
    // 1. Expose given route in the `http` server application, here an example for the `Express.js`:
    return server.get(path.join('/status', route.path), async (req, res, next) => {
        try {
            // 2. Combine the `Express.js` route parameters:
            const params = Object.assign({}, req.params, req.query, req.headers);
            // 3. Call given `route.handler` passing combined parameters and given service descriptor:
            const result = await route.handler(params, service);
            // 4. Decorate the `Express.js` response:
            res.status(result.status);
            res.set('Content-Type', result.contentType);
            res.set(result.headers);
            // 5. Return the response body according to given `contentType`:
            switch (result.contentType) {
                case constants.MIME_APPLICATION_JSON:
                    res.json(result.body);
                    break;
                default:
                    res.send(result.body);
            }
        } catch (err) {
            // 6. Deal with the Error according to `Express.js` framework rules.
            next(err);
        }
    });
};
module.exports = express;

An adapter can be declared as follows:

const healthChecks = require('healthchecks-api');

(async () => {
    // The default is 'express' adapter, when not declared:
    await healthChecks(myServer);

    // An internally supported adapter:
    await healthChecks(myServer, {
        adapter: 'express',
    });

    // A module from an npm registry - must export a proper function.
    await healthChecks(myServer, {
        adapter: 'my-adapter-module',
    });

    // An adapter function declared directly:
    await healthChecks(myServer, {
        adapter: async (service, server, route) => {
            // your adapter implementation details
        }
    });
})();

Developing new check types

Create a custom check type class

A custom check class must extend the Check class and implement the asynchronous start() method. The method is supposed to perform the actual check. When the check is performed in intervals (pull model) then the class should use derived this.interval [ms] property, which can be set in the yaml configuration (default is 3000 ms).

const healthChecks = require('healthchecks-api');
const Check = healthChecks.Check;

class MyCheck extends Check {
    constructor(config) {
        super(config);
        // this.config contains the properties from the `yaml` check config part.
        if (this.config.myProperty) {
            // ...
        }
        // class initialization code
    }

    async start() {
        // actual check code to be executed in `this.interval` [ms] intervals.
        // ...
        // set up the resulting state:
        this.ok(); // | this.warn(description, details); | this.crit(description, details);
    }
}
// This is optional - by default the new check type will be the class name in lowercase.
// One can change that by following line.
MyCheck.type = 'mycheck'; // the default

NOTE: The check name to be used in yaml configs is by default the class name in lowercase.

See the particular checks implementations for reference - ./lib/checks/*.

Add the check class to the module check type list

Additional check types (classes) are to be declared in runtime, before starting creating the health checks routes. Here's the example:

const healthCheck = require('healthchecks-api');
const myCheck = require('./lib/my-check.js');

const startServer = async () => {
    const app = express();
    // some initialization steps

    await healthCheck.addChecks(myCheck);

    await healthCheck(app);
    // rest of initialization stepa
}

startServer();

Use the new check type in your yaml configurations:

version: "3.1"

name: demo-app
description: Nice demo application :)

checks:
  - check: mycheck
    # properties are accessible in the class instance property `config` - eg. `this.config.myProperty`.
    myProperty: value

Exporting multiple check classes in a custom module

Check classes can be bundled into a module and optionally published in private or public NPM registry for reusability. The module must export allowable value for the this module's addCkecks method. The addChecks method can take a value of following parameter types as an argument:

Testing

Unit tests

codecov

Run unit tests locally:

npm test

Integration test

The integration test is a Docker Compose based setup. The setup shows the health-check-api functionality reported by Microservice Graph Explorer - the dashboard-like application which presents service dependencies and their health status and allows to traverse dependencies which expose Health Checks API.

One can observe changing health status of the demo-app application by:

The set-up

NOTE

The service-2 is classified as non-critical for the demo-app so it will be reported as WARN at the demo-app dashboard even if it gets the CRIT state.

Running the test

cd ./test/integration
make up

This will build and start the docker-compose services and open the Microservice Graph Explorer in the default browser at http://localhost:9000/#/status/http/demo-app:3000.

Starting and stopping services

make stop SERVICE=service-2
make stop SERVICE=mongo

make start SERVICE=service-2
make start SERVICE=mongo

Emulating high load and memory leak

Following example commands use the localhost:$PORT urls. Port mapping to services:

One can use the Apache Benchmark tool for emulating a high load to an http service, eg:

ab -n 10000 -c 20 http://localhost:3001/heavy

To emulate the memory leak execute following:

curl http://localhost:3000/make-leak

Tearing down the set-up

make down