Home

Awesome

node-kafka-connect

Build Status

Coverage Status

What can I do with this?

The framework can be used to build connectors, that transfer data to and from Apache Kafka and Databases, very easily. If you are looking for already implemented connectors for you favorite datastore, take a look at the Available Connector Implementations below.

Info

A note on native mode

If you are using the native mode (config: { noptions: {} }). You will have to manually install node-rdkafka alongside kafka-connect. (This requires a Node.js version between 9 and 12 and will not work with Node.js >= 13, last tested with 12.16.1)

On Mac OS High Sierra / Mojave: CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib yarn add --frozen-lockfile node-rdkafka@2.7.4

Otherwise: yarn add --frozen-lockfile node-rdkafka@2.7.4

(Please also note: Doing this with npm does not work, it will remove your deps, npm i -g yarn)

Available Connector Implementations

Creating custom Connectors

yarn add kafka-connect
const source = new TestSourceConfig(config, 
    TestSourceConnector, 
    TestSourceTask, 
    [TestConverter]);
    
source.run().then();
const sink = new TestSinkConfig(config,
    TestSinkConnector, 
    TestSinkTask, 
    [TestConverter]);
 
sink.run().then();

Docs

Debugging

FAQ