Awesome
node-kafka-streams
// suggested Node.js version: v12.16.1
npm install --save kafka-streams
const {KafkaStreams} = require("kafka-streams");
const config = require("./config.json");
const factory = new KafkaStreams(config);
const kstream = factory.getKStream("input-topic");
const ktable = factory.getKTable(/* .. */);
kstream.merge(ktable).filter(/* .. */).map(/* .. */).reduce(/* .. */).to("output-topic");
CHANGES: The latest version brings a lot of changes, please check here before updating.
API Overview
- Quick Start
- Message 'to' and 'from' Apache Kafka
- API Info
- Documentation
- Operator descriptions
- Examples
- Native Client | SSL, SASL, Kerberos
You might also like
README Overview
- node-kafka-streams
- More
Prerequisites
- Kafka broker should be version
>= 0.11.x
- Node.js should be version
>= 8.x.x
A note on native mode
If you are using the native mode (config: { noptions: {} }
).
You will have to manually install node-rdkafka
alongside kafka-streams.
(This requires a Node.js version between 9 and 12 and will not work with Node.js >= 13, last tested with 12.16.1)
On Mac OS High Sierra / Mojave:
CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib yarn add --frozen-lockfile node-rdkafka@2.7.4
Otherwise:
yarn add --frozen-lockfile node-rdkafka@2.7.4
(Please also note: Doing this with npm does not work, it will remove your deps, npm i -g yarn
)
Aim of this Library
- this is not a 1:1 port of the official JAVA kafka-streams
- the goal of this project is to give at least the same options to a nodejs developer that kafka-streams provides for JVM developers
- stream-state processing, table representation, joins, aggregate etc. I am aiming for the easiest api access possible checkout the word count example
Description
kafka-streams :octopus: equivalent for nodejs :sparkles::turtle::rocket::sparkles: build on super fast :fire: observables using most.js :metal:
ships with sinek :pray: for backpressure
comes with js and native Kafka client, for more performance and SSL, SASL and Kerberos features
the lib also comes with a few window
operations that are more similar to Apache Flink,
yet they still feel natural in this api :squirrel:
overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..
async (Promises) and sync stream operators e.g. stream$.map()
or stream$.asyncMap()
super easy API :goberserk:
the lib is based on sinek
, which is based on kafka-node's ConsumerGroups
Port Progress Overview
- core structure
- KStream base - stream as a changelog
- KTable base - stream as a database
- KStream & KTable cloning
- complex stream join structure
- advanced joins see
- windows (for joins) see
- flink like window operations
- word-count example
- more examples
- local-storage for etl actions
- local-storage factory (one per action)
- KStorage example for any DB that supports atomic actions
- backing-up local-storage via kafka
- kafka client implementation
- KTable replay to Kafka (produce)
- stream for topic message production only
- sinek implementation
- backpressure mode for KafkaClient
- auto-json payloads (read-map/write-map)
- auto producer partition and keyed-message handling
- documentation
- API description
- higher join & combine examples
- embed native client
librdkafka
for more performance - SSL
- SASL
- Kerberos
Operator Implementations
- map
- asyncMap
- constant
- scan
- timestamp
- tap
- filter
- skipRepeats
- skipRepeatsWith
- slice
- take
- skip
- takeWhile
- skipWhile
- until
- since
- reduce
- chainReduce
- forEach (observe)
- chainForEach
- drain
- _zip
- _merge
- _join
- _combine
- _sample
- throttle
- debounce
- delay
- multicast
- A description of the operators can be found here
- Missing an operator? Feel free to open an issue :cop:
Additional Operators
- mapStringToArray
- mapArrayToKV
- mapStringToKV
- mapParse
- mapStringify
- atThroughput
- mapWrapKafkaPayload
- mapToFormat
- mapFromFormat
- Want more? Feel free to open an issue :cop:
Stream Action Implementations
- countByKey
- sumByKey
- min
- max
- Want more? Feel free to open an issue :cop:
Join Operations
KStream Status
- merge
- outerJoin
- innerJoin
- leftJoin
- branch
KTable Status
- merge
- outerJoin
- innerJoin
- leftJoin
KTable <-> KStream Status
- merge
- outerJoin
- innerJoin
- leftJoin
Window Operations
KStream
- window
- advanced window
- rolling window
More
Can I use this library yet?
Yes.
Are we ready for production yet?
Probably, yes. :smile:
Even More
Forks or Stars give motivation :bowtie: