Awesome
awesome-kafka
Everything about Apache Kafka
Share your Kafka ❤️ through pull requests :)
What can you expect to see here?
Development
Client libraries
- Go
- Go
- Go
- Go Kafka streaming
- Node
- Node
- Node
- Node Kafka streams
- Node librdkafka
- Php
- Php
- Perl
- Python (asyncio)
- Python (kafka-python)
- Python (pykafka)
- Ruby
- Phobos - Ruby App framework
- Karafka - Ruby and Rails framework
- Dart
- JRuby
- Elixir
- Erlang
- Rust
- LinkedIn wrapper library
- kasper - a lightweight library for processing Kafka topics
- and others
Kafka Streams Libraries
KSQL
Connectors
- JDBC
- Elasticsearch
- HDFS
- Beast - Kafka to BigQuery Sink
- BigQuery
- SQL
- GCloud pubsub
- BigTable
- Google Cloud Storage (GCS)
- HBase
- FileSystem
- Streaming files from a local filesystem
- kafka-connect-protobuf-converter
- IBM MQ Source
- IBM MQ Sink
- Azure IoT Hub
- Pulsar
- HTTP
Producers
Consumers
- Spark
- Storm
- Mapreduce
- Divolte
- Azure Data Explorer/Kusto
- Firehose - Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.
- Raccoon - Raccoon is a high-throughput, low-latency service to collect events in real-time from your web, mobile apps, and services using multiple network protocols and publish to Kafka.
Transformations
- Dagger - Dagger is an easy-to-use, SQL framework to process built on top of Apache Flink for stateful processing of Kafka data.
- Single Message Transformations (SMT)
Testing
- Mocked Streams - Topology testing library for Kafka Streams (no broker required)
- Unit-testing for embedded kafka services
- Kafka JUnit
- Kafka Unit
- Kafka for JUnit
- Karate and Kafka
- embedded-kafka - in-memory Kafka instance to run your tests against
- Kafka Containers
- Mockintosh - through asynchronous Kafka actors
Operations
CI/CD
Operational Utilities
- Cruise control
- DoctorKafka and its successor Orion
- Brucke - Inter-cluster bridge of kafka topics
- Kafka-Kit
- Secor - service for persisting Kafka logs
- kafkabalancer
- Yelp Kafka-Utils
- Cerner Common Kafka utilities
- connectctl - manage kafka connect connectors easily
Monitoring
- Kafka Offset Exporter
- Fluent plugin
- Burrow - Kafka Consumer Lag Checking
- Remora
- Kafka Offset Monitoring
- kafka-eagle
- Kafka-prometheus
- Consumer lag monitoring
Performance tools
Security
Audit
Mirroring
- MirrorMaker
- Message Handler For MirrorMaker to change topic names
- uReplicator
- MirrorTool for Kafka Connect
- Mirus
Backup
Tools
- Karapace - Your Kafka essentials in one tool
- REST-proxy
- Kafka-Pixy - gRPC/REST Proxy
- Kafka Proxy providing SASL/PLAIN authentication and SSL
- Reactive/Akka API
- kafkat
- kt
- kafka-connect-tools
- Node Kafka connect
- Kafka-monitor
- Kafka Auto completion
- Kafka Cheet Sheet
- Kafka Write Protocol encoder/decoder
- Winton Kafka Streams
- Spring-kafka
- Maven quick start for Kafka Connect connectors
- Trifecta
- Kafka Specs
- Strimzi Kafka CLI
Metadata Management
- Go library for Kafka metadata in Zookeeper
- Meteor - Meteor is a scalable, easy-to-use, extensible metadata collection framework from the different cloud providers and on-prem sources and publish to Kafka.
Schema Management
- Stencil - Stencil is a Protobuf schema registry for Kafka that enables you to create, maintain and consume protobuf messages and APIs dynamically, efficiently, and reliably.
- Schema registry
- Node Schema registry
- Schema Registry Transfer SMT
- Expedia Stream Registry
UI
- kafka-manager
- Kafka HQ
- Kafdrop 3.x – web UI for Kafka. For versions older than 0.10.0 use Kafdrop 2.x.
- Kafka Webview
- Kafka Topics UI
- Kafka Connect UI
- Kafka Schema UI
- Kafkawize - self service topic management portal
- Kadmin - Kafka Producer/Consumer UI
- Kowl
- Kafka Streams Topology Sketch Diagram Visualization
- Web based GUI
Deployment
Kubernetes
- Kafka k8s operator
- Strimzi Operator for deploying and running Apache Kafka on Kubernetes
- konsumerator - Kafka Consumer Operator