Awesome
<div align="center"> <img src="/assets/avatar-transparent.png?raw=true" width="86"> </div> <h1 align="center">TypeStream</h1> <br /> <div align="center"> <a href="https://github.com/typestreamio/typestream/blob/main/LICENSE"> <img src="https://img.shields.io/github/license/typestreamio/typestream" /> </a> <a href="https://discord.gg/Ha9sJWXb"> <img src="https://img.shields.io/badge/Chat-on%20Discord-blue" alt="Discord invite" /> </a> </div> <p align="center"> <a href="#why-typestream">Why TypeStream?</a> · <a href="#getting-started">Getting started</a> · <a href="#how-to-contribute">How to contribute</a> · <a href="#code-of-conduct">Code of conduct</a> · <a href="#license">License</a> </p> <h3 align="center">TypeStream is an abstraction layer on top of Kafka that allows you to write and run <i>typed</i> data pipelines with a minimal, familiar syntax.
</h3 >Why TypeStream?
Building streaming data pipelines on top of Kafka comes with some fixed costs. You have to write an app, test it, then deploy and manage it in production. Even for the simplest pipelines, this can be a lot of work.
With TypeStream you can write powerful, typed data pipelines the way you'd write a simple UNIX pipeline in your terminal. For example, imagine you'd like to filter a "books" topic. With TypeStream, it's a one liner:
$ typestream
> grep /dev/kafka/local/topics/books the > /dev/kafka/local/topics/books_with_the
TypeStream will take care of type-checking your pipeline and then run it for you. Here's how it looks like in action:
Another common use case that requires a lot of boilerplate is to enrich a topic with data from an external source. For example, you might have a topic with an ip address field and you may want to enrich it with country information. With TypeStream, you can do it (again!) in a single line:
$ typestream
> cat /dev/kafka/local/topics/page_views | enrich { view -> http "https://api.country.is/#{$view.ip_address}" | cut .country } > /dev/kafka/local/topics/page_views_with_country
Here's how enriching looks like in action:
As you can see from the previous command, in the spirit of UNIX, we used cut to
extract the country field from the response. Here's the kick, you can use cut
(and many other Unix commands) on streams as well:
$ typestream
> cat /dev/kafka/local/topics/books | cut .title > /dev/kafka/local/topics/book_titles
Here's how cutting looks like in action:
Another problem that TypeStream solves is preventing you from writing faulty pipelines by type-checking them before executing them. Here's type checking in action:
Deploying a pipeline to production is as simple as running:
typestream run 'cat /dev/kafka/local/topics/books | cut .title > /dev/kafka/local/topics/book_titles'
See it in action here:
If you'd like to learn more about TypeStream, check out the official documentation.
Getting started
If you use Homebrew:
brew install typestreamio/tap/typestream
$ typestream --version
if you see something like this:
typestream version 2023.08.31+3 42f7762daac1872416bebab7a34d0b79a838d40a (2023-09-02 09:20:52)
then you're good to go! You can now run:
typestream local start
typestream local seed
to start a local TypeStream server and seed it with some sample data. Now you're ready to start writing your own pipelines!
Check out our documentation to learn more about TypeStream.
How to contribute
We love every form of contribution! Good entry points to the project are:
- Our contributing guidelines document.
- Issues with the tag gardening.
- Issues with the tag good first patch.
If you're not sure where to start, open a new issue or hop on to our discord server and we'll gladly help you get started.
Code of Conduct
You are expected to follow our code of conduct when interacting with the project via issues, pull requests, or in any other form. Many thanks to the awesome contributor covenant initiative!