Home

Awesome

node-kafka-streams

Build Status npm version

// suggested Node.js version: v12.16.1
npm install --save kafka-streams
const {KafkaStreams} = require("kafka-streams");

const config = require("./config.json");
const factory = new KafkaStreams(config);

const kstream = factory.getKStream("input-topic");
const ktable = factory.getKTable(/* .. */);

kstream.merge(ktable).filter(/* .. */).map(/* .. */).reduce(/* .. */).to("output-topic");

CHANGES: The latest version brings a lot of changes, please check here before updating.

API Overview

You might also like

README Overview

Prerequisites

A note on native mode

If you are using the native mode (config: { noptions: {} }). You will have to manually install node-rdkafka alongside kafka-streams. (This requires a Node.js version between 9 and 12 and will not work with Node.js >= 13, last tested with 12.16.1)

On Mac OS High Sierra / Mojave: CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib yarn add --frozen-lockfile node-rdkafka@2.7.4

Otherwise: yarn add --frozen-lockfile node-rdkafka@2.7.4

(Please also note: Doing this with npm does not work, it will remove your deps, npm i -g yarn)

Aim of this Library

Description

kafka-streams :octopus: equivalent for nodejs :sparkles::turtle::rocket::sparkles: build on super fast :fire: observables using most.js :metal:

ships with sinek :pray: for backpressure

comes with js and native Kafka client, for more performance and SSL, SASL and Kerberos features

the lib also comes with a few window operations that are more similar to Apache Flink, yet they still feel natural in this api :squirrel:

overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..

async (Promises) and sync stream operators e.g. stream$.map() or stream$.asyncMap()

super easy API :goberserk:

the lib is based on sinek, which is based on kafka-node's ConsumerGroups

Port Progress Overview

Operator Implementations

Additional Operators

Stream Action Implementations

Join Operations

Operation description

KStream Status

KTable Status

KTable <-> KStream Status

Window Operations

KStream

More

Can I use this library yet?

Yes.

Are we ready for production yet?

Probably, yes. :smile:

Even More

Forks or Stars give motivation :bowtie: