Home

Awesome

knit

Knit is a tool for making Erlang release upgrades. It also makes Erlang release packages but is primarily focused on making upgrades. Its similar in spirit to rebar's release generation or relx but focusing much more specifically on upgrades.

If you're already using rebar's release generation then trying knit should be straight forward to try as its written to be compatible with the reltool.config.

Releases and Upgrades

This is the obligatory background on Erlang releases and upgrades. An Erlang release is (generally) a self-contained tarball that includes the entire Erlang VM bundled with the standard library and a copy of all the code required to run an Erlang program. You can read more about releases at LYSE or in Erlang's documentation.

While releases are full of awesome, the fun doesn't really start until you get to working with upgrades. Everyone talks about Erlang's hot code loading and some have even ventured into the land of l/1 and nl/1 for loading individual modules into a live VM. Release upgrades are hot code loading on steroids. They provide a repeatable (mostly) scripted approach to apply code loading primitives onto a running system. It can be a bit daunting at first but is really quite neat once you learn the fundamentals.

A release upgrade comprises of two basic components: a copy of the new compiled application code and some metadata about the new release, most notably of which is called a relup. The relup is simply a linear set of instructions for applying hot code loading primitives. The process of applying an upgrade is, in a nutshell, "unpack a tarball and run the relup".

While there are obviously a lot of details surrounding the various bits of this process the most important part to remember is that its a fundamentally simple and straightforward process. Interested readers are encouraged to check out the LYSE docs on relups as the [Erlang docs on relups][erlang-relups] are less than stellar.

One final major concept to be aware of is that of appups. While the idea of the relup is simple, the details involved in generating it are numerous and intertwined. The general idea for generating a relup is that each Erlang application that changes between releases will have an appup. These appups are then "compiled" by systools and merged into a single relup.

Historically the process of generating these appups has been rather painful even when using the various release tools. Knits entire motivation for existence is to improve the process around handling these appup files while making sure the rest of the release/upgrade generation is as painless as possible.

For more information on appups see this LYSE section, the Erlang docs on appups, or the Erlang appup cookbook. To read more about how knit helps with generating appups see the section below called "How Knit Helps with Appups".

Finally, a quick aside on downgrades. While Erlang supports the notion of downgrades between releases, knit does not yet have support. In general it borrows from rebar's theory that you should just roll forward to a new version if something is broken. Its possible that knit will eventually support downgrades but its not currently on the agenda.

An Outline of Release and Upgrade Generation

Before we get too far its helpful to have a general idea on the steps involved in generating release and upgrade tarballs. In a few steps it looks something like this:

  1. Compile all application code (knit relies on rebar for this step)
  2. Generate a release based on reltool.config
  3. Process overlay instructions from reltool.config
  4. Generate the release tarball
  5. Locate previous release versions to upgrade from
  6. Expand each previous release
  7. For each old release find which Erlang applications have changed
  8. Generate an appup for each changed application
  9. Compile all appups into the relup
  10. Package the upgrade tarball

Most tools are pretty good at handling steps 1-4. Everything after that has historically not been the most well documented or easiest to use. Knit tries to make the rest of those steps as painless as possible. See the section "How Knit Helps with Appups" below for a much more detailed description of these steps as well as knit is actually doing that's new.

A Primer on Upgrades

What do we do with upgrade tarballs once we have them?

A Primer on Relup/Appup Instructions

List the most common instructions and use cases for background.

add_module load_module delete_module upgrade - supervisor and code changes styles

Building Knit

This should be as simple as:

$ git clone https://github.com/cloudant/knit.git
$ cd knit
$ make
$ cp knit /usr/local/bin

The output of the build is a single executable knit which is exactly like the rebar executable (a self-contained executable zip file).

Running Knit

This should be as simple as:

$ cd ~/path/to/my/project/
$ knit

Assuming you have either a reltool.config or rel/reltool.config in your project knit will automatically find it and do its thing. Assuming this is your first time running knit you should expect to see rel/$name/ and a new rel/releases/$name-$vsn.tar.gz file. rel/$name is based on the configured boot release (you can read more on that in the section on reltool.config).

Running Knit Again

Once you create a release, if you edit some code and update your application version you can create another release like such:

$ knit

Fairly complicated stuff. Though if you watch closely you'll notice that knit also builds an upgrade package as well. Assuming this all went off without a hitch you should find that rel/upgrades/$name-$newvsn.tar.gz now exists.

At this point if you were to go and edit more code and update your version once again you could generate a third release tarball and second upgrade tarball like such:

$ knit

The intrepid observer will note that this time knit will only generate a single rel/upgrades/$name-$newnewvsn.tar.gz but that it was also looking at the original rel/releases/$name-$vsn.tar.gz. This is because the single upgrade tarball will apply to both $name-$vsn.tar.gz and $name-$newvsn.tar.gz. Neat.

Knit's Command Line Interface

The knit command is fairly basic. The help output looks like such:

Usage: knit [-h] [-V] [-v <verbose>] [-d] [-q] [-l <log_file>]
            [-r <reltool>] [var ...]

  -h, --help            Show this help message and exit.
  -V, --version         Show version information and exit.
  -v, --verbose         Set verbose output. Can be repeated for increased
                        verbosity.
  -d, --debug           Set debug verbosity logging.
  -q, --quiet           Disable all output to the console.
  -l, --log_file        Log all output to the specified file. Unaffected
                        when using quiet.
  -r, --reltool_config  Path to your reltool.config
  var                   Set variables using the name=value syntax

The only real parameter of immediate interest is -r/--reltool_config. If you have a reltool configuration file named something other than reltool.config or rel/reltool.config you'll need to specify the name with this flag.

Setting Config Values from the Command Line

All of the configuration variables are listed below but its important to note that any of these can be overridden via knit's command line variable syntax. It looks similar to rebar's variables with a bit of extra magic.

A basic example looks something like such:

$ knit name1=value1 name2=value2 ...

The extra sauce here is that you can have your values parsed as Erlang strings which can be helpful in some specific circumstances that you'll know if you need once you get there. For the most part the most common settings will work without this as they're written internally to check for string values.

The format for Erlang statements as values looks like:

$ knit name1=!!value1

Where value1 is a valid Erlang statement. More specifically:

$ knit 'name1=!!{this, is, 1.0, "a tuple"}'

Notice that there's no trialing . to terminate the statement as that's automatically appended. Also note the use of quotes so that the shell passes the expression as a single command line argument.

A Quick Primer on reltool.config

The erlang docs on reltool.config are a bit opaque at first but hopefully this quick primer will explain enough to help the unfamiliar make sense of things.

A reltool.config file is a standard "list of Erlang terms" format that is used for things like *.app files. A simplistic rel/reltool.config looks something like such:

{sys, [
    {lib_dirs, ["../apps", "../deps"]},
    {rel, "dbcore", git, [
        kernel,
        stdlib,
        my_cool_app
    ]},
    {rel, "start_clean", "", [kernel, stdlib]},
    {boot_rel, "dbcore"},
    {profile, embedded},
    {excl_sys_filters, ["^bin/.*", "^erts.*/bin/(dialyzer|typer)"]},
    {excl_archive_filters, [".*"]},

    {app, my_cool_app, [{incl_cond, include}]}
]}.

There are a few major things to note about this right away. First is that the {sys, [option()]} structure. This is important as this term is passed directly to reltool:start_server/1.

The second thing to notice is the format of the release descriptions:

{rel, Name, Version, [Application]}

While we generally talk about a release as a single tarball, under the covers a single tarball can actually include multiple releases. While that seems a bit crazy there is actually a common use case for this as demonstrated by {rel, "start_clean", "", [kernel, stdlib]} entry. This "release" is used by system scripts or remsh when connecting to running nodes.

Given that there are multiple releases we need to tell reltool which one is the default release to boot. This is the aptly named boot_rel option. Knit uses the value for boot_rel as the name for all generated files so its important that you pick something you'd want to see in tarball file names.

The {profile, embedded} setting dictates how many extraneous files are included by default in the release tarball. The possible values for this setting are development, standalone, and embedded which include the most, fewer, and fewest files in the resulting release.

Lastly the {app, my_cool_app, [{incl_cond, include}]} tells reltool to include my_cool_app in the generated release along with all of the dependencies listed in my_cool_app.app.

For now its best to just ignore the excl_sys_filters and excl_archive_filters. When you need to learn the details for these options you'll be far along enough that the man pages make sense.

For an extended description of all of the possible sys options the write up by LYSE is probably the best place to check. There's a section labeled "Releases with Reltools" towards the bottom that's helpful.

Knit Specific Config Options

Along with the sys tuple, knit recognizes a number of other options that can be specified in the same reltool.config file. So there's no confusion a reltool.config with knit configuration values would look like such:

{knit_option_1, knit_value_1}.
{knit_option_2, knit_option_2}.

{sys, [
    {lib_dirs, ["../apps", "../deps"]},
    {rel, "dbcore", git, [
        kernel,
        stdlib,
        my_cool_app
    ]},
    {rel, "start_clean", "", [kernel, stdlib]},
    {boot_rel, "dbcore"},
    {profile, embedded},
    {excl_sys_filters, ["^bin/.*", "^erts.*/bin/(dialyzer|typer)"]},
    {excl_archive_filters, [".*"]},

    {app, my_cool_app, [{incl_cond, include}]}
]}.

It doesn't matter where in the file you place things, its just important to note that they aren't nested inside the sys tuple.

The current list of recognized congfiguration options:

Release Overlays

Overlays are basically just a list of file operations applied after a release is generated (but before it is packaged) so that you can do things like including non-Erlang related files in a release or creating common file trees used in your production environments. For instance, including etc config directories is one common use.

A simple overlay setting would look something like such:

{overlay_vars, "my_vars.config"}.
{overlay, [
    {mkdir, "var/log"},
    {copy, "overlay/bin"},
    {copy, "overlay/etc"},
    {copy, "overlay/share"},
    {copy, "../path/\{\{var_name\}\}/thing", "bin/thing"},
    {template, "overlay/etc/config.ini", "etc/config.ini"}
]}.

The basic idea is that variables are loaded from my_vars.config and used while rendering templates and file paths. The rendering is all handled using Mustache templating. Remember that paths are also rendered which can be handy.

Its also important to note that source paths are relative to the specified reltool.config while destination paths are relative to $target_dir/$name/ which contains the lib, releases, and erts-5.y.z directories.

Overlay Commands

How Knit Helps with Appups

As explained above, knit's entire purpose is to make working with appups bearable and as minimally rage inducing as possible. In general knit does this by attempting to automatically generate upgrade instructions. The basic algorithm is borrowed from rebar's appup generation but knit extends this algorithm considerably so that it can be tweaked by providins small bits of metadata directly in the affected modules.

Historically there have been roughly three approaches I've seen to working with appups:

  1. Write them by hand
  2. Hope that rebar's generate-appups gets something close
  3. Use the output of rebar's generate-appups and then edit by hand

Generally speaking rebar will cover roughly 75% of your general upgrade use cases. The downfall is that anything beyond a trivial upgrade requires the user to edit the generated appup to cover the more complicated cases. And the bit that makes this particularly fun is when your upgrades are generated on a build machine so you have to check in the edited appup which then leads to confusion when that breaks the next release (because you forgot to remove it and it doesn't cover the new release version).

Without any tooling, maintaining appups for all version pairs is mind numbingly boring. Maintaining appups by hand would be almost doable if there were a tool that would allow you to have a hybrid "auto generate appups for some version pairs but let me override this particular version". Originally knit was conceived as exactly this tool but was then extended when I realized it would be even easier to just provide a bit of metadata to hopefully cover 99% of upgrade use cases.

So that's knit in a nutshell: Automatically generate appups while providing an interface for coercing that generation for complicated situations. First we'll talk a bit about how that works and then show a number of examples on common non-trivial use cases.

Knit Module Attributes

The way we pass metadata to knit's appup generation algorithm is through module attributes. This way we store appup related information directly in the affected modules which means that our upgrade logic will be less likely to go stale with source changes.

Supported module attributes:

A Note on Module Dependencies

Module dependencies are quite useful but there's a non-obvious effect of using them buried in systools. Basically, any connected set of module dependencies will be grouped in the relup and upgraded between the same suspend/resume calls. Its possible that this can have unintended side effects during an upgrade.

For instance, a common case might be if you have a gen_server behavior paired with a utility module that you want to upgrade simultaneously. Adding a -knit_depends([my_app_util]) would be a straightforward method to accomplish this. The part to be careful on is if you have a second gen_server that you also want to have depend on my_app_util. If you do this then processes running code from both gen_server modules will be suspended simultaneously.

In general its best to try and make sure that your modules can be upgraded independently but sometimes a simple dependency will save you lots of pain trying to support multiple live module versions.

Motivational Use Cases

=== Show how -vsn works

Not a knit thing, but a useful thing to show.

=== Module dependencies couch_server/couch_lru

Yep

=== Priority instead of dependencies

Sometimes easier

=== Changing a supervisor's child specs (and making them a reality)

Stuff

=== Testing an upgrade? Add apply calls to timer:sleep?

Might work?