Home

Awesome

aws-lambda-fanout

This function answers a need I have had multiple times, where I want to replicate data from an Amazon Kinesis Stream to another account or another region for processing, or to another environment such as development.

This AWS Lambda function can be used to propagate incoming messages from Amazon Kinesis Streams or Amazon DynamoDB Streams to other services (Amazon SNS, Amazon SQS, Amazon Elasticsearch Service, Amazon Kinesis Streams, Amazon Kinesis Firehose, AWS IoT, AWS Lambda, Amazon ElastiCache for Memcached and Redis), regions or accounts. This function generates metrics that will be published to Amazon CloudWatch Metrics, in the Custom/FanOut namespace.

Architecture

This function can be run in the 'public' AWS Lambda mode (and should be the default for most services), or inside an Amazon Virtual Private Cloud (VPC) if you use Amazon ElastiCache (Redis or memcached).

As the configuration data resides in Amazon DynamoDB, and because the function sends metrics via Amazon CloudWatch, the function must have Internet access. To have Internet access, an AWS Lambda function running inside an Amazon VPC must reside in a private subnet with a route through a NAT Gateway (either a self-managed NAT instance or an Amazon Managed NAT Gateway) for Internet. This limitation exists because AWS Lambda functions in an Amazon VPC don't have a public IP Address.

Glossary

Here is a list of terms used in this documentation:

Mappings

The fanout function maps sourcesto targets.

Sources can currently be:

Targets have a specific type defined, and a destination. Currently the allowed types and destination formats are the following:

Configuration entries

Each target is defined by a set of parameters, stored in an Amazon DynamoDB table. The default name of the table is derived from the function <function-name>Targets. Here are the properties for this table:

For sns, sqs, es, kinesis, firehose, iot and lambda:

For memcached and redis:

Supporting cross-account publication

You can send the records to a target account. This feature leverages the STS:AssumeRole API to allow cross-account access.

To activate this feature, you have to specify the role property in the configuration. This property will contain the ARN of the AWS IAM Role from the target account to be used when publishing the data.

If you send data to an account that you don't own, you should specify the externalId property that is used to further limit the access to sts:AssumeRole.

To activate this feature, you need to configure a policy in the AWS IAM Role used by the fanout function to include the sts:AssumeRole action on your target account. You also need to configure the trust relationship of your target account AWS IAM Role to allow sts:AssumeRole calls from the source account.

This feature is not available for redis and memcached.

Supporting cross-region publication

You can also send records to a target region, by specifying the region property in the configuration. This property will contain the name of the target region.

This feature is not available for redis and memcached.

The CLI

To simplify the deployment, configuration and management of the function, a Bash Command Line Interface (CLI) is available. The command is ./fanout and is an automation of the AWS CLI.

Available commands are:

Some parameters (--exec-role, --table, --source) have a short form or a long form. The short form will call the AWS CLI to search for the element and either create it or raise an error if it does not exist, and the long form (the same parameters with -arn suffix: --exec-role-arn, --table-arn, --source-arn) will just accept the parameter as-is. This speeds up execution time, and allow scenarios where the user running the command does not have enough rights to discover the elements, while the function does.

Optional command line parameters are:

Deploy

The deploy command creates the fanout function and supporting resources (AWS IAM Role, Amazon DynamoDB Table). It expects the following parameters:

Example:

./fanout deploy --function fanout

List

The list command retrieves all the existing mappings for a specific source. It expects the following parameters:

Example:

./fanout list --function fanout --source-type kinesis --source inputStream

Register

The register <type> command creates a new mapping for an existing fanout function. As the fanout function caches the configuration for performance reasons (default time of 1 minute), there may be a delay in the activation of this mapping. Note also that unless --active true is specified new mappings are created inactive for safety reasons. It expects the following parameters:

Example:

./fanout register lambda --function fanout --source-type kinesis --source inputStream --id target1 --destination targetFunction

Update

The update command allows you to modify some parameters of your mappings. As the fanout function caches the configuration for performance reasons (default time of 1 minute), there may be a delay in the application of the modification. It expects the following parameters:

Example:

./fanout update --function fanout --source-type kinesis --source inputStream --id target1 --parallel false

(De)activate

The activate and deactivate commands turn on or off a specific mapping. As the fanout function caches the configuration for performance reasons (default time of 1 minute), there may be a delay in the application of the modification. They expect the following parameters:

Example:

./fanout activate --function fanout --source-type kinesis --source inputStream --id target1 ./fanout deactivate --function fanout --source-type kinesis --source inputStream --id target1

Unregister

The unregister command removes an existing mapping fromt the configuration table. As the fanout function caches the configuration for performance reasons (default time of 1 minute), there may be a delay before the target is effectively removed. It expects the following parameters:

Example:

./fanout unregister --function fanout --source-type kinesis --source inputStream --id target1

Destroy

The destroy command removes the fanout function and its configuration. It expects the following parameters:

Example:

./fanout destroy --function fanout

(Un)hook

The hook and unhook commands register and unregister an event source mapping for the specified source. They expect the following parameters:

Example:

./fanout hook --function fanout --source-type kinesis --source inputStream ./fanout unhook --function fanout --source-type kinesis --source inputStream

Pause|Resume

The pause and resume commands turn on or off a specific source. They will respectly disable and enable the event source mapping from the source to the fanout function. They expect the following parameters:

Example:

./fanout pause --function fanout --source-type kinesis --source inputStream ./fanout resume --function fanout --source-type kinesis --source inputStream

License

Copyright 2011-2013 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at

http://aws.amazon.com/apache2.0/

or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.