Home

Awesome

Amazon Kinesis Client Library for .NET

This package provides an interface to the Amazon Kinesis Client Library (KCL) MultiLangDaemon for the .NET Framework.

Developers can use the KCL to build distributed applications that process streaming data reliably at scale. The KCL takes care of many of the complex tasks associated with distributed computing, such as load balancing across multiple instances, responding to instance failures, checkpointing processed records, and reacting to changes in stream volume.

This package wraps and manages the interaction with the MultiLangDaemon, which is provided as part of the Amazon KCL for Java so that developers can focus on implementing their record processing logic.

A record processor in C# typically looks something like the following:

using Amazon.Kinesis.ClientLibrary;

namespace Sample
{
    public class RecordProcessor : IShardRecordProcessor
    {
        public void Initialize(InitializationInput input)
        {
            //
            // Initialize the record processor
            //
        }

        public void ProcessRecords(ProcessRecordsInput input)
        {
            //
            // Process a batch of records from input.Records, and optionally checkpoint by calling
            // input.Checkpointer.Checkpoint() 
            //
        }

        public void LeaseLost(LeaseLossInput leaseLossInput)
        {
            //
            // Perform any cleanup required.
            // This record processor has lost it's lease so checkpointing is not possible.
            // This is why LeaseLostInput does not provide a Checkpointer property,
            //
        }

        public void ShardEnded(ShardEndedInput shardEndedInput)
        {
            //
            // The record process has processed all records in the shard, and will no longer receive records.
            // It is required that this method call shardEndedInput.Checkpointer.Checkpoint() to inform the KCL
            // that the record processor has acknowledged the completion of the shard.
            //
        }

        public void ShutdownRequested(ShutdownRequestedInput shutdownRequestedInput)
        {
            //
            // This is called when the KCL is being shutdown, and if desired the record processor can checkpoint here
            // by calling shutdownRequestedInput.Checkpointer.Checkpoint(...)
            //
        }
    }

    internal class MainClass
    {
        public static void Main(string[] args)
        {
            KclProcess.Create(new RecordProcessor()).Run();
        }
    }
}

For more information about Amazon Kinesis and the client libraries, see the official documentation as well as the Amazon Kinesis forums.

Getting started

Set up your AWS credentials

Before running a KCL application, make sure that your environment is configured to allow the MultiLangDaemon to access your AWS security credentials.

If you've installed the AWS SDK for .NET, you may have already configured your AWS credentials using the SDK credential store in Microsoft Visual Studio; however, because Amazon KCL for .NET applications never deal with your credentials directly but defer to the MultiLangDaemon, this store is not available to your KCL application.

Instead, you can provide your credentials through environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) or a credential profile in your home directory.

Building and running the sample projects

In addition to source code for the Amazon KCL for .NET itself, this repository contains a sample application, which can serve as a starting point for your KCL application. To try out this sample application, you can download a ZIP of the latest sources, the contents of which can be opened as a solution in Microsoft Visual Studio.

The sample application consists of two projects:

The following defaults are used in the sample application:

Running the data producer

To run the data producer, run the SampleProducer project.

Notes

Running the data processor

Because the Amazon KCL for .NET requires the MultiLangDaemon, which is provided by the Amazon KCL for Java, a bootstrap program has been provided. This program downloads all required dependencies prior to invoking the MultiLangDaemon, which executes the processor as a subprocess.

To run the processor, first build the SampleConsumer project, then run the bootstrap project with the following configuration:

Notes

Cleaning up

This sample application creates a few resources in the default region of your AWS account:

Each of these resources will continue to incur AWS service costs until they are deleted. After you are finished testing the sample application, you can delete these resources through the AWS Management Console.

What you should know about the MultiLangDaemon

The Amazon KCL for .NET uses the Amazon KCL for Java internally. We have implemented a Java-based daemon, called the MultiLangDaemon, which handles all of the heavy lifting. Our approach has the daemon spawn the user-defined record processor program as a sub-process. The MultiLangDaemon communicates with this sub-process over standard input/output using a simple protocol, and therefore the record processor program can be written in any language.

At runtime, there will always be a one-to-one correspondence between a record processor, a child process, and an Amazon Kinesis shard. The MultiLangDaemon ensures that, without any developer intervention.

In this release, we have abstracted these implementation details and exposed an interface that enables you to focus on writing record processing logic in C#. This approach enables the Amazon KCL to be language-agnostic, while providing identical features and similar parallel processing model across all languages.

Release Notes

Release 4.0.0 (November 6, 2024)

Release 3.0.1 (April 24, 2024)

Release 3.0.0 (January 12, 2023)

Release 2.0.0 (February 27, 2019)

See Also

License

This library is licensed under the Apache 2.0 License.