Awesome

duckdb-nodejs-layer

Packaging DuckDB for usage in AWS Lambda functions with Node.js, and publishing as public Lambda layers.

Usage

You can use the published layers in your own serverless applications by referencing it as outlined in the different framework's docs:

• Serverless Framework
• SAM
• CDK
• CloudFormation

The layers get automatically published to all currently available (in the moment of publishing) AWS regions.

Example usage

You can have a look at the example repository, which uses this AWS Lambda layer: tobilg/serverless-duckdb. More specifically, the src/functions/query.js should give you a good idea on how it can be used.

The recommendation is to use a bundler such as WebPack for the packaging of your application. This means you can locally use (e.g. for testing) the official duckdb npm package (via npm i --save duckdb), but exclude it in the packaging process, because after being deployed to a Lambda function, the updated DuckDB version from the Lambda layer will be used.

Code example

import DuckDB from 'duckdb';

// Instantiate DuckDB
const duckDB = new DuckDB.Database(':memory:');

// Create connection
const connection = duckDB.connect();

// Promisify query method
const query = (query) => {
  return new Promise((resolve, reject) => {
    connection.all(query, (err, res) => {
      if (err) reject(err);
      resolve(res);
    })
  })
}

// Will show DuckDB version
await query(`PRAGMA version;`);

DuckDB layer

This contains a AWS Lambda layer in different flavors, both x86 and arm64 architectures are supported.

The ARNs follow the following logic:

arn:aws:lambda:$REGION:041475135427:layer:duckdb-nodejs-$ARCHITECTURE:$VERSION

where $ARCHITECTURE can have the following values:

• x86
• arm64

Enabled extensions

The following DuckDB default extensions are enabled and contained in the static build:

• parquet: Adds support for reading and writing parquet files
• httpfs: Adds support for reading and writing files over a HTTP(S) connection
• json: Adds support for JSON operations
• fts: Adds support for Full-Text Search Indexes
• icu: Adds support for time zones and collations using the ICU library

x86 layer ARNs

Layer version to DuckDB version mapping:

The ARNs of the latest x86 version of the DuckDB Node.js Lambda layer are:

arm64 layer ARNs

Layer version to DuckDB version mapping:

The ARNs of the latest arm64 version of the DuckDB Node.js Lambda layer are:

x86_64 layer with arrow, aws and httpfs extensions ARNs

This layer contains the arrow, aws and httpfs extensions. This enables Lambda functions that are in a VPC and have no internet access to download these extensions via HTTP (e.g. from extensions.quacking.cloud) for cost savings.

VPC access is for example necessary if you want your Lambda functions to access S3 Express One Zone. You can have a look at how to deploy such kind of setup in the tobilg/serverless-duckdb repo.

Currently, it's only available in us-east-1.

Layer version to DuckDB version mapping:

The ARNs of the latest x86_64 version of the DuckDB Node.js Lambda layer with extensions are: