Awesome

OneTable

One Table to Rule Them All

DynamoDB OneTable Migration Controller

This repository provides hosting of the OneTable Migration Library for DynamoDB.

You should host the OneTable Migrate library (via this repo) so that it executes in the same AWS region and availablity zone as your DynamoDB instance. This will accelerate migrations by minimizing the I/O transfer time.

Use the SenseDeep DynamoDB Developer Studio for a graphical controller for your migrations which provides a complete DynamoDB developer environment with a powerful data browser, single-table designer, provisioning planner, table metrics and control of database migrations.

Use the OneTable CLI for command line control of migrations.

OneTable Migration Features

Mutate database schema and contents via discrete, reversible migrations.
Migrate upwards, downwards, to specific versions.
Automated, ordered sequencing of migrations in both directions.
Named migrations for database maintenance, auditing and other tasks.
Add and remove seed data in any migration.
Quick reset of DynamoDB databases for development.
Show database status and list applied migrations.
Show outstanding migrations.
Stored history of migrations.
Persist migration history and the current OneTable schema in the table.
Control by the SenseDeep DynamoDB Developer Studio GUI.
No module dependencies other than OneTable.
Works with AWS SDK v2 and v3

Installation

git clone git@github.com:sensedeep/onetable-controller.git

Modify

Modify to add migrations scripts under the tables directory. See below for more details.

Deploy

To promote to the cloud defined by your AWS_PROFILE and AWS_REGION. You can also edit the Makefile to define these variables if not defined in your environment.

make promote

Creating Migrations

The tables directory contains sub-directories for each DynamoDB table you wish to manage. There is one sample called TestTable. Rename this to the name of your table and copy this directory for other tables you may wish to manage.

When creating migrations, you need to modify the following files:

migrations/X.Y.Z.js
migrations/NAMED-MIGRATIONS.js
migrations/reset.js
schemas/schema.X.Y.Z.js

Migrations are versioned using SemVer and reside in the top of the per-table directory. There is a 0.0.1.js sample to get you started. This contains:

import schema from './schemas/schema.0.0.1.js'

export default {
    version: '0.0.1',
    description: 'Test migration 0.0.1',
    schema,

    async up(db, migrate, params) {
        console.log(`Up ${this.description}`)
    },

    async down(db, migrate, params) {
        console.log(`Down ${this.description}`)
    }
}

The db property is a OneTable Table instance. You can use it to interact with the DynamoDB table. See OneTable Migrate for samples.

Each migration should have its own versioned schema that defines the data entities at that migration version level. The schemas live by default in the ./schemas directory.

The reset.js migration is a special dev migration that is used by the SenseDeep and the OneTable CLI to "reset" the database to the latest schema and discard all prior data. It is useful when developing and you need to reset the database to a good known state.

Directories and Files

Controller.js

The Controller.js file is the Lambda function that hosts the OneTable Migration Library.

tables

The tables directory contains directories for each DynamoDB table you wish to manage. It also contains an index.js which is the top-level index of tables.

tables/TestTable

This directory contains the sample migrations for a DynamoDB TestTable under the migrations directory.

This directory also contains a schemas directory for the corresponding OneTable schemas for each migration version.

See the OneTable Migration documentation for details of how to use the migration library.

OneTable CLI

The OneTable CLI can control your migration lambda when operating in proxy mode by setting the arn of your migration Lambda.

Lambda Hosting

The Lambda function Controller.js receives proxied commands from the OneTable CLI or from the SenseDeep Serverless Developer Studio.

SenseDeep and/or the OneTable CLI should be configured with the ARN of this Lambda function.

References

Participate

All feedback, contributions and bug reports are very welcome.

OneTable Migrate Issues

Contact

You can contact me (Michael O'Brien) on Twitter at: @SenseDeepCloud, or email and ready my Blog.

SenseDeep

Please try our Serverless trouble shooter SenseDeep.