Home

Awesome

Pensieve

Pensieve is a NPM package to be used in Node.js runtime applications running on EC2/ECS integrated with a StatsD agent. It provides custom metrics with useful context related to your application out of the box.

Table of contents

Use cases

To see how to integrate Pensieve with other libraries, check the examples folder.

Terminology

Check out the Terminology section for explanations related to the terms used in this documentation and library.

Usage

To use Pensieve in your application, add it to your required npm libraries:

$ # cd projects/content-portability
$ npm install @dazn/pensieve

Instantiate Pensieve

import { Pensieve } from "@dazn/pensieve"

// Information & context related to your application
// Will be used in the attributes of custom metrics
const pensieveInstance = new Pensieve({
    appName: "content-portability",
    component: "api",
    env: "prod",
    awsRegion: "eu-central-1",
    appVersion: "v0.0.1",
    host: os.hostname(),
    serviceType: "ecs",
    commitHash: "abcdefg12"
})

Pensieve can "auto-fill" information related to your ECS service, such as AWS availability zone and ECS task ID. Checkout the section ECS metadata for an explanation on how to do it.

Instantiate the StatsD client


import { StatsD } from "hot-shots"

const statsDClient = new StatsD({
    host: process.env.STATSD_HOST || 'statsd.host.com',
    port: Number(process.env.STATSD_PORT) || 8125,
    maxBufferSize: 8192,
    bufferFlushInterval: 1000
})

Note: while not strictly required, it is highly recommended you add the above environmental variables to your ECS container definition.

The advantage is that if multiple StatsD agents are running in the same host, switching from one agent to another (for example Datadog to New Relic) does not require any Node.js code changes but only Terraform changes.

Instantiate your metrics service

import { StatsDService } from "@dazn/pensieve"

// The service you can use to create your own custom metrics
const metricsService = new StatsDService(pensieveInstance, statsDClient)

Done!

Integration with Express.js

  1. Usage
  2. Example
  3. Diagram

Usage in Express.js applications

To monitor inbound requests, add this middleware to each route:

import { inboundRequest } from "@dazn/pensieve/lib/middleware"

// Versioned API's
app.get("/v1/items", inboundRequest(metricsService, { path: "get_items", version: "v1" }), routeFn)
app.get("/v2/items", inboundRequest(metricsService, { path: "get_items", version: "v2" }), routeFn)

// Unversioned API's
app.get("/items", inboundRequest(metricsService, { path: "get_items" }), routeFn)

To monitor outbound requests, submit the outbound request after the action is performed (metricsService initialized as described here. ):

let statusCode;
const startAt = process.hrtime();
try {
    const response: AxiosResponse = await axiosInstance.get( "https://www.google.nl" );
    statusCode = response.status;
} catch (err) {
    console.log(err);
} finally {
    // Submit the outbound request
    metricsService.submitOutboundRequestMetric({
        httpResponseCode: statusCode,
        target: "google",
        httpMethod: "GET",
        routePath: "google_index_get",
        startAt
    });
}

Example of Express.js integrations

To see how to integrate Pensieve with an Express.js app, check the examples folder.

Express.js integration diagram

REST API's integration

<sup>High level diagram of Pensieve integrated in an Express.js app running on ECS</sup>

Submit your own custom metrics

You can also submit your own custom metrics if you want!

metricsService initialized as described here.

// Supported types: 
// IncrementMetric: increments a stat by 1
// GaugeMetric: gauge a stat by a specified amount
// HistogramMetric: send data for histogram stat (DataDog and Telegraf only)
// TimingMetric: sends a timing command with the specified milliseconds (typical for latencies)
import { IncrementMetric } from "@dazn/pensieve/lib/types"

const myOwnIncrementMetric: IncrementMetric = {
  kind: "increment",
  key: "somethingHappened",
  attributes: {
    foo: "bar" // Your own extra custom attributes here
  }
}

metricsService.submit(myOwnIncrementMetric)

ECS metadata

Pensieve also allows you to gather useful context related to the ECS task where the application is running, when applicable.
For this functionality to work, make sure that your ECS container agent has the setting ECS_ENABLE_CONTAINER_METADATA=true enabled.
Follow the AWS documentation to check how to enable it for your own cluster.

const pensieveInstance = new Pensieve(
  {
      appName: "content-portability",
      component: "api",
      env: "prod",
      appVersion: "v0.0.1",
      host: os.hostname(),
      serviceType: "ECS",
      commitHash: "abcdefg12"
  },
  true  // Enable ECS metadata context, disabled by default
)

// ECS metadata available for you:
loggerInstance.info({
  message: "ECS metadata information for you!",
  private: { ECSMetadataContext: pensieveInstance.getECSMetadataContext() }
})

/**
* Log output:
* {
*  message: 'ECS metadata information for you!',
*  private: {
*    ECSMetadataContext: {
*      cluster: 'test-cluster',
*      ecsTaskId: '2b88376d-aba3-4950-9ddf-bcb0f388a40c',
*      containerName: 'test-container-name',
*      containerInstanceArn: 'arn:aws:ecs:region:acc:container-instance/test-cluster/1f73d099-b914-411c-a9ff-81633b7741dd',
*      ecsServiceName: 'simple-app-service-name',
*      containerId: 'aec2557997f4eed9b280c2efd7afccdced',
*      dockerContainerName: '/ecs-console-example-app-1-e4e8e495e8baa5de1a00',
*      imageId: 'sha256:2ae34abc2ed0a22e280d17e13f',
*      availabilityZone: 'us-east-1b'
*    }
*  }
* }        
**/

Metric attributes

Attributes provide context to each metric. All the metrics generated by Pensieve have a set of defined attributes, that can be later be used in New Relic or Datadog to filter the queries necessary to create widgets in dashboards.

Standard attributes

HTTP request metric (inbound and outbound)

Changelog

Please see the CHANGELOG file for more information about what has changed recently.

Roadmap

Contributors