Home

Awesome

Test Actions Status

Fluent Bit Plugin for Amazon Kinesis Data Streams

NOTE: A new higher performance Fluent Bit Kinesis Plugin has been released. Check out our official guidance.

A Fluent Bit output plugin for Amazon Kinesis Data Streams.

Security disclosures

If you think you’ve found a potential security issue, please do not post it in the Issues. Instead, please follow the instructions here or email AWS security directly at aws-security@amazon.com.

Usage

Run make to build ./bin/kinesis.so. Then use with Fluent Bit:

./fluent-bit -e ./kinesis.so -i cpu \
-o kinesis \
-p "region=us-west-2" \
-p "stream=test-stream"

For building Windows binaries, we need to install mingw-64w for cross-compilation. The same can be done using-

sudo apt-get install -y gcc-multilib gcc-mingw-w64

After this step, run make windows-release. Then use with Fluent Bit on Windows:

./fluent-bit.exe -e ./kinesis.dll -i dummy `
-o kinesis `
-p "region=us-west-2" `
-p "stream=test-stream"

Plugin Options

Permissions

The plugin requires kinesis:PutRecords permissions.

Credentials

This plugin uses the AWS SDK Go, and uses its default credential provider chain. If you are using the plugin on Amazon EC2 or Amazon ECS or Amazon EKS, the plugin will use your EC2 instance role or ECS Task role permissions or EKS IAM Roles for Service Accounts for pods. The plugin can also retrieve credentials from a shared credentials file, or from the standard AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN environment variables.

Environment Variables

Fluent Bit Versions

This plugin has been tested with Fluent Bit 1.2.0+. It may not work with older Fluent Bit versions. We recommend using the latest version of Fluent Bit as it will contain the newest features and bug fixes.

Example Fluent Bit Config File

[INPUT]
    Name        forward
    Listen      0.0.0.0
    Port        24224

[OUTPUT]
    Name            kinesis
    Match           *
    region          us-west-2
    stream          my-kinesis-stream-name
    partition_key   container_id
    append_newline  true
    replace_dots    _

AWS for Fluent Bit

We distribute a container image with Fluent Bit and this plugin.

GitHub

github.com/aws/aws-for-fluent-bit

Amazon ECR Public Gallery

aws-for-fluent-bit

Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>

For example, you can pull the image with latest version by:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

If you see errors for image pull limits, try log into public ECR with your AWS credentials:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

You can check the Amazon ECR Public official doc for more details.

Docker Hub

amazon/aws-for-fluent-bit

Amazon ECR

You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:

aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/

For more see our docs.

KPL aggregation

KPL aggregation can be enabled by setting the aggregation parameter to true (default is false). With aggregation enabled each Record in the PutRecords request can contain multiple serialized records in the KCL protobuf structure. This batch of records will only count as a single record towards the Kinesis records per second limit (currently 1000 records/sec per shard).

The advantages of enabling KPL aggregation are:

The disadvantages are:

KPL Aggregated Record Reference: https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md

Tuning for aggregation

When using aggregation the buffers and flush time may need to be tuned. For low volume use cases a longer flush time maybe preferable to take full advantage of the aggregation cost savings.

More specifically, increasing the flush value will ensure the most records are aggregated taking full advantage of the cost savings.

[SERVICE]
     Flush 20

Example Fluent Bit Aggregation Config File

[SERVICE]
     Flush 20

[INPUT]
    Name        forward
    Listen      0.0.0.0
    Port        24224

[OUTPUT]
    Name            kinesis
    Match           *
    region          us-west-2
    stream          my-kinesis-stream-name
    aggregation     true
    append_newline  true

ZLIB Compression

Enabling zlib compression will compress each record individually reducing the network bandwidth required to send logs. Using this feature in conjunction with aggregation can greatly reduce the number of Kinesis shards required.

Compression Advantages:

Compression Disadvantages:

Example config:

[SERVICE]
     Flush 20

[INPUT]
    Name        forward
    Listen      0.0.0.0
    Port        24224

[OUTPUT]
    Name            kinesis
    Match           *
    region          us-west-2
    stream          my-kinesis-stream-name
    compression     zlib
    append_newline  true

New Higher Performance Core Fluent Bit Plugin

We have released a new higher performance Kinesis Streams plugin named kinesis_streams.

That plugin has many of the features of this older, lower performance and less efficient plugin. Please compare this document with its documentation for an up to date feature set comparison between the two plugins.

Do you plan to deprecate this older plugin?

This plugin will continue to be supported. However, we are pausing development on it and will focus on the high performance version instead.

Which plugin should I use?

If the features of the higher performance plugin are sufficient for your use cases, please use it. It can achieve higher throughput and will consume less CPU and memory.

As time goes on we expect new features to be added to the C plugin only, however, this is determined on a case by case basis. There is some feature gap between the two plugins. Please consult the C plugin documentation and this document for the features offered by each plugin.

How can I migrate to the higher performance plugin?

For many users, you can simply replace the plugin name kinesis with the new name kinesis_streams.

Do you accept contributions to both plugins?

Yes. The high performance plugin is written in C, and this plugin is written in Golang. We understand that Go is an easier language for amateur contributors to write code in- that is one of the primary reasons we are continuing to maintain this repo.

However, if you can write code in C, please consider contributing new features to the higher performance plugin.