Awesome

Overview

fwup is a configurable image-based software update utility for embedded Linux-based systems. It primarily supports software upgrade strategies that update entire root filesystem images at once. This includes strategies like swapping back and forth between A and B partitions, recovery partitions, and various trial update/failback scenarios. All software update information is combined into a ZIP archive that may optionally be cryptographically signed. fwup has minimal dependencies and runtime requirements. Scripts are intentionally limited to make failure scenarios easier to reason about. Distribution of software update archives is not a feature. Users can call out to fwup to run upgrades from external media, stream them from the network, or script them using a tool like Ansible if so desired.

Here's a list of features:

1. Uses standard ZIP archives to make debugging and transmission simple.

2. Simple, but flexible configuration language to enable firmware updates on various platforms and firmware update policies.

3. Streaming firmware update processing to simplify target storage requirements.

4. Multiple firmware update task options per archive so that one archive can upgrade varying target configurations

5. Basic disk partitioning (GPT and MBR) and FAT filesystem manipulation

6. Human and machine readable progress.

7. Initialize or update SDCards on your development system whether you're running Linux, OSX, BSD, or Windows. MMC and SDCards are automatically detected and unmounted. No need to scan logs or manually unmount.

8. Firmware archive digital signature creation and verification

9. Delta update support using VCDIFF. See delta update section for details (BETA FEATURE).

10. Sparse file support to reduce number of bytes that need to be written when initializing large file systems (see section on sparse files)

11. Permissive license (Apache 2.0 License - see end of doc for details)

12. Extensively regression tested! Tests also provide examples of functionality.

Internally, fwup has many optimizations to speed up low level disk writes over what can easily be achieved with dd(1). It orders, Flash erase block aligns, and can skip writing large unused sections to minimize write time. It also verifies writes to catch corruption before the device reboots to the new firmware. The goal is to make updates fast enough for code development and reliable enough for production use.

Installing

The simplest way to install fwup is via a package manager or installer.

On OSX, fwup is in homebrew:

brew install fwup

On Linux, download and install the appropriate package for your platform:

• Debian/Ubuntu AMD64 .deb
• Raspbian armhf .deb
• Alpine Linux - Install official apk
• Arch Linux - See fwup-git package on AUR
• Buildroot - Support is included upstream since the 2016.05 release
• Yocto - See meta-fwup

On Windows, fwup can be installed from chocolatey

choco install fwup

Alternatively, download the fwup executable and place it in your path.

If you're using another platform or prefer to build it yourself, download the latest source code release or clone this repository. Then read one of the following files:

• Linux build instructions
• OSX build instructions
• Windows build instructions
• Raspbian build instructions
• FreeBSD/NetBSD/OpenBSD build instructions

When building from source, please verify that the regression test pass on your system (run make check) before using fwup in production. While the tests usually pass, they have found minor issues in third party libraries in the past that really should be fixed.

NOTE: For space-constrained target devices, use ./configure --enable-minimal-build to trim functionality that's rarely used.

Invoking

If you were given a .fw file and just want to install its contents on an SDCard, here's an example run on Linux:

$ sudo fwup example.fw
Use 14.92 GiB memory card found at /dev/sdc? [y/N] y
100%
Elapsed time: 2.736s

If you're on OSX or Windows, leave off the call to sudo.

IMPORTANT: If you're using an older version of fwup, you'll have to specify more arguments. Run: fwup -a -i example.fw -t complete

Here's a list of other options:

Usage: fwup [OPTION]...

Options:
  -a, --apply   Apply the firmware update
  -c, --create  Create the firmware update
  -d <file> Device file for the memory card
  -D, --detect List attached SDCards or MMC devices and their sizes
  -E, --eject Eject removable media after successfully writing firmware.
  --no-eject Do not eject media after writing firmware
  --enable-trim Enable use of the hardware TRIM command
  --exit-handshake Send a Ctrl+Z on exit and wait for stdin to close (Erlang)
  -f <fwup.conf> Specify the firmware update configuration file
  -F, --framing Apply framing on stdin/stdout
  -g, --gen-keys Generate firmware signing keys (fwup-key.pub and fwup-key.priv, or specify with -o)
  -i <input.fw> Specify the input firmware update file (Use - for stdin)
  -l, --list   List the available tasks in a firmware update
  --max-size <blocks> Max size of the destination in 512-byte blocks (usually automatic)
  -m, --metadata   Print metadata in the firmware update
  --metadata-key <key> Only output the specified key's value when printing metadata
  --minimize-writes Skip write if contents match destination
  --no-minimize-writes Don't try to minimize writes when applying firmware updates (default)
  -n   Report numeric progress
  -o <output.fw> Specify the output file when creating an update (Use - for stdout)
  -p, --public-key-file <keyfile> A public key file for verifying firmware updates (can specify multiple times)
  --private-key <key> A private key for signing firmware updates
  --progress-low <number> When displaying progress, this is the lowest number (normally 0 for 0%)
  --progress-high <number> When displaying progress, this is the highest number (normally 100 for 100%)
  --public-key <key> A public key for verifying firmware updates (can specify multiple times)
  -q, --quiet   Quiet
  -s, --private-key-file <keyfile> A private key file for signing firmware updates
  -S, --sign Sign an existing firmware file (specify -i and -o)
  --sparse-check <path> Check if the OS and file system supports sparse files at path
  --sparse-check-size <bytes> Hole size to check for --sparse-check
  -t, --task <task> Task to apply within the firmware update
  -u, --unmount Unmount all partitions on device first
  -U, --no-unmount Do not try to unmount partitions on device
  --unsafe Allow unsafe commands (consider applying only signed archives)
  -v, --verbose   Verbose
  -V, --verify  Verify an existing firmware file (specify -i)
  --verify-writes Verify writes when applying firmware updates to detect corruption (default for writing to device files)
  --no-verify-writes Do not verify writes when applying firmware updates (default for regular files)
  --version Print out the version
  -y   Accept automatically found memory card when applying a firmware update
  -z   Print the memory card that would be automatically detected and exit
  -1   Fast compression (for create)
  -9   Best compression (default)

Examples:

Create a firmware update archive:

  $ fwup -c -f fwup.conf -o myfirmware.fw

Apply the firmware to an attached SDCard. This would normally be run on the host
where it would auto-detect an SDCard and initialize it using the 'complete' task:

  $ fwup -a -i myfirmware.fw -t complete

Apply the firmware update to /dev/sdc and specify the 'upgrade' task:

  $ fwup -a -d /dev/sdc -i myfirmware.fw -t upgrade

Create an image file from a .fw file for use with dd(1):

  $ fwup -a -d myimage.img -i myfirmware.fw -t complete

Generate a public/private key pair:

  $ fwup -g

Store fwup-key.priv in a safe place and fwup-key.pub on the target. To sign
an existing archive run:

  $ fwup -S -s fwup-key.priv -i myfirmware.fw -o signedfirmware.fw

Example usage

The regression tests contain short examples for usage of various script elements and are likely the most helpful to read due to their small size.

Other examples can be found in the bbb-buildroot-fwup for project for the BeagleBone Black and Raspberry Pi. The Nerves Project has more examples and is better maintained.

My real world use of fwup involves writing the new firmware to a place on the Flash that's not in current use and then 'flipping' over to it at the very end. The examples tend to reflect that. fwup can also be used to overwrite an installation in place assuming you're using an initramfs, but that doesn't give protection against someone pulling power at a bad time. Also, fwup's one pass over the archive feature means that firmware validation is mostly done on the fly, so you'll want to verify the archive first (see the -V option).

Helper scripts

While not the original use of fwup, it can be convenient to convert other files to .fw files. fwup comes with the following shell script helper:

• img2fwup - convert a raw image file to a .fw file

A use case for the img2fwup script is to convert a large SDCard image file to one that is compressed and checksummed by fwup for distribution.

Versioning

fwup uses semver for versioning. For example, if you are using 1.0.0, that means no breaking changes until 2.0.0 or new features until 1.1.0. I highly recommend a conservative approach to upgrading fwup once you have devices in the field. For example, if you have 1.0.0 devices in the field, it's ok to update to 1.1.0, but be very careful about using 1.1.0 features in your fwup.conf files. They will be ignored by 1.0.0 devices and that may or may not be what you want.

Configuration file format

fwup uses the Unix configuration library, libconfuse, so its configuration has some similarities to other programs. The configuration file is used to create firmware archives. During creation, fwup embeds a processed version of the configuration file into the archive that has been stripped of comments, has had all variables resolved, and has some additional useful metadata added. Configuration files are organized into scoped blocks and options are set using a key = value syntax. Additionally, configuration files may include configuration fragments and other files by calling include("filename").

Environment variables

For integration into build systems and other scripts, fwup performs variable substitution inside of the configuration files. Keep in mind that environment variables are resolved on the host during firmware update file creation. Generated firmware files do not contain

Environment variables are referenced as follows:

key = ${ANY_ENVIRONMENT_VARIABLE}

It is possible to provide default values for environment variables using the :- symbol:

key = ${ANY_ENVIRONMENT_VARIABLE:-adefault}

Inside configuration files, it can be useful to define constants that are used throughout the file. Constants are referenced identically to environment variables. Here is an example:

define(MY_CONSTANT, 5)

By default, repeated definitions of the same constant do not change that constant's value. In other words, the first definition wins. Note that the first definition could come from a similarly named environment variable. This makes it possible to override a constant in a build script.

In some cases, having the last definition win is preferable for constants that never ever should be overridden by the environment or by earlier calls to define. For this behavior, use define!:

define!(MY_CONSTANT, "Can't override this")

Simple math calculations may also be performed using define-eval() and define-eval!(). For example:

define-eval(AN_OFFSET, "${PREVIOUS_OFFSET} + ${PREVIOUS_COUNT}")

These two functions were added in release 0.10.0, but since they are evaluated at firmware creation time, .fw files created using them are compatible with older versions of fwup.

Finally, file-resource will define a variable named FWUP_SIZE_<resource_name> with the size of the resource. For example the following will create a variable named FWUP_SIZE_zImage:

file-resource zImage {
        host-path = "output/images/zImage"
}

execute("echo zImage size is ${FWUP_SIZE_zImage} bytes")

Global scope

At the global scope, the following options are available:

After setting the above options, it is necessary to create scopes for other options. The currently available scopes are:

file-resource

A file-resource specifies a file on the host that should be included in the archive. Each file-resource should be given a unique name so that it can be referred to by other parts of the update configuration. fwup will automatically record the length and BLAKE2b-256 hash of the file in the archive. These fields are used internally to compute progress and verify the contents of the archive. A typical file-resource section looks like this:

file-resource zImage {
        host-path = "output/images/zImage"
}

Resources are usually stored in the data directory of the firmware archive. This is transparent for most users. If you need to make the .fw file work with other software, it is sometimes useful to embed a file into the archive at another location. This can be done by specifying an absolute path resource as follows:

file-resource "/my_custom_metadata" {
        host-path = "path/to/my_custom_metadata_file"
}

Resource concatenation

Sometimes you need to concatenate multiple files together to form one file-resource. While you can sometimes do this using multiple calls to raw_write, that won't work if you don't know the file offsets a priori or the offsets don't fall on block boundaries. Another alternative is to concatenate files as a prep step to fwup. If that's inconvenient, fwup allows multiple paths to be specified in host-path that are separated by semicolons. They will be concatenated in the order they appear.

file-resource kernel_and_rootfs {
        # Concatenate uImage and the rootfs. OpenWRT mtd splitter will
        # separate them back out at runtime.
        host-path = "output/images/uImage;output/images/rootfs.squashfs"
}

File resource validation checks

When creating archives, fwup can perform validation checking on file resources to catch simple errors. These checks can catch common errors like file resources growing too large to fit on the destination or files truncated due to cancelled builds.

Note that these checks are not performed when applying updates, since the actual length (and a hash) is recorded in the archive metadata and used for verification.

The following checks are supported:

Sizes are given in 512 byte blocks (like everything else in fwup).

Here's an example:

file-resource rootfs.img {
        host-path = "output/images/rootfs.squashfs"
        assert-size-lte = ${ROOTFS_A_PART_COUNT}
}

Files from strings

Sometimes it's useful to create short files inside the fwup config file rather than referencing them. This can be accomplished using the contents key in a file-resource. Variable substitution works in the contents string just like any other string in the fwup configuration file.

file-resource short-file.txt {
        contents = "You're looking at a short\n\
file creating by fwup.\n\
When it was made, FOO was ${FOO}.\n"
}

mbr

A mbr section specifies the contents of the Master Boot Record on the destination media. This section contains the partition table that's read by Linux and the bootloaders for finding the file systems that exist on the media. In comparison to a tool like fdisk, fwup only supports simplistic partition setup, but this is sufficient for many devices. Tools such as fdisk can be used to determine the block offsets and sizes of partitions for the configuration file. Offsets and sizes are given in 512 byte blocks. Here's an example mbr definition:

mbr mbr-a {
        bootstrap-code-host-path = "path/to/bootstrap-data" # should be 440 bytes
        signature = 0x01020304

        partition 0 {
                block-offset = ${BOOT_PART_OFFSET}
                block-count = ${BOOT_PART_COUNT}
                type = 0x1 # FAT12
                boot = true
        }
        partition 1 {
                block-offset = ${ROOTFS_A_PART_OFFSET}
                block-count = ${ROOTFS_A_PART_COUNT}
                type = 0x83 # Linux
        }
        partition 2 {
                block-offset = ${ROOTFS_B_PART_OFFSET}
                block-count = ${ROOTFS_B_PART_COUNT}
                type = 0x83 # Linux
        }
        partition 3 {
                block-offset = ${APP_PART_OFFSET}
                block-count = ${APP_PART_COUNT}
                type = 0x83 # Linux
        }
}

OSIP

If you're using an Intel Edison or similar platform, fwup supports generation of the OSIP header in the MBR. This header provides information for where to load the bootloader (e.g.., U-Boot) in memory. The include-osip option controls whether the header is generated. The OSIP and image record (OSII) option names map directly to the header fields with the exception that length, checksum and image count fields are automatically calculated. The following is an example that shows all of the options:

mbr mbr-a {
    include-osip = true
    osip-major = 1
    osip-minor = 0
    osip-num-pointers = 1

    osii 0 {
        os-major = 0
        os-minor = 0
        start-block-offset = ${UBOOT_OFFSET}
        ddr-load-address = 0x01100000
        entry-point = 0x01101000
        image-size-blocks = 0x0000c000
        attribute = 0x0f
    }

    partition 0 {
        block-offset = ${ROOTFS_A_PART_OFFSET}
        block-count = ${ROOTFS_A_PART_COUNT}
        type = 0x83 # Linux
    }
}

Expanding the final partition

Sometimes it's useful to have the final partition fill the remainder of the storage. This is needed if your target's storage size is unknown and you need to use as much of it as possible. The expand option requests that fwup grow the block-count to be as large as possible. When using expand, the block-count is now the minimum partition size. Only the final partition can be expandable. Here's an example:

mbr mbr-a {
        partition 0 {
                block-offset = ${BOOT_PART_OFFSET}
                block-count = ${BOOT_PART_COUNT}
                type = 0x1 # FAT12
                boot = true
        }
        partition 1 {
                block-offset = ${ROOTFS_A_PART_OFFSET}
                block-count = ${ROOTFS_A_PART_COUNT}
                type = 0x83 # Linux
        }
        partition 2 {
                block-offset = ${ROOTFS_B_PART_OFFSET}
                block-count = ${ROOTFS_B_PART_COUNT}
                type = 0x83 # Linux
        }
        partition 3 {
                block-offset = ${APP_PART_OFFSET}
                block-count = ${APP_PART_COUNT}
                type = 0x83 # Linux
                expand = true
        }
}

Extended partitions

MBR supports more than 4 partitions by using a special extended partition type. Normally if you need more than 4 partitions, you'd want to switch to GPT, but your setup may make this more trouble than it's worth. At a high level, to use the extended partition feature, you need to mark partition 3 as the extended partition. Then you can specified partitions 4 and higher. Each of these partitions will have a 512-byte Extended Boot Record or EBR. Fwup stores all EBRs at the beginning of the extended partition, so if you have partitions 4, 5, and 6, that will require 3 EBRs or 1536 bytes. Partition 4 would start at a block after that. Note that there's flexibility in where EBRs are written so you'll see other programs that make different decisions, but it all works so long as the EBRs are inside of the extended partition and don't overlap with any other partition.

Here's an example of an extended partition:

mbr mbr-a {
    partition 0 {
        block-offset = \${AUTOBOOT_PART_OFFSET}
        block-count = \${AUTOBOOT_PART_COUNT}
        type = 0x6 # FAT12
        boot = true
    }
    partition 1 {
        block-offset = \${BOOT_A_PART_OFFSET}
        block-count = \${BOOT_A_PART_COUNT}
        type = 0xc # FAT32
    }
    partition 2 {
        block-offset = \${BOOT_B_PART_OFFSET}
        block-count = \${BOOT_B_PART_COUNT}
        type = 0xc # FAT32
    }
    partition 3 {
        type = 0xf # Extended partition
        block-offset = \${EXTENDED_PART_OFFSET}
    }
    partition 4 {
        block-offset = \${ROOTFS_A_PART_OFFSET}
        block-count = \${ROOTFS_A_PART_COUNT}
        type = 0x83 # Linux
    }
    partition 5 {
        block-offset = \${ROOTFS_B_PART_OFFSET}
        block-count = \${ROOTFS_B_PART_COUNT}
        type = 0x83 # Linux
    }
    partition 6 {
        block-offset = \${APP_PART_OFFSET}
        block-count = \${APP_PART_COUNT}
        type = 0x83 # Linux
    }
}

Fwup calculates the extended partition's size based on the partitions after it, so you only need to specify the starting block offset.

gpt

A gpt section specifies the contents of a GUID Partition Table. It is similar to the mbr section, but it supports more partitions and uses UUIDs. Here's an example gpt definition:

gpt my-gpt {
    # UUID for the entire disk
    guid = b443fbeb-2c93-481b-88b3-0ecb0aeba911

    partition 0 {
        block-offset = ${EFI_PART_OFFSET}
        block-count = ${EFI_PART_COUNT}
        type = c12a7328-f81f-11d2-ba4b-00a0c93ec93b # EFI type UUID
        guid = 5278721d-0089-4768-85df-b8f1b97e6684 # ID for partition 0 (create with uuidgen)
        name = "efi-part.vfat"
    }
    partition 1 {
        block-offset = ${ROOTFS_PART_OFFSET}
        block-count = ${ROOTFS_PART_COUNT}
        type = 44479540-f297-41b2-9af7-d131d5f0458a # Rootfs type UUID
        guid = fcc205c8-2f1c-4dcd-bef4-7b209aa15cca # Another uuidgen'd UUID
        name = "rootfs.ext2"
        flags = 0xc000000000000
        boot = true
    }
}

See the GPT partition entry header specification for what partition fields mean and how to use them. Of the fields, name, flags, and boot are optional and default to an empty string, zero, or false. The flags field holds the integer value of the partition attribute field. It is a 64-bit number. Bit 2, the legacy BIOS bootable flag, can also be set by specifying boot = true. Both boot and flags can be specified in a partition block.

Call gpt_write to tell fwup to write the protective MBR and primary and backup GPT tables.

U-Boot environment

For systems using the U-Boot bootloader, some support is included for modifying U-Boot environment blocks. In order to take advantage of this, you must declare a uboot-environment section at the top level that describes how the environment block:

uboot-environment my_uboot_env {
    block-offset = 2048
    block-count = 16
}

To use the redundant environment block style, add block-offset-redund with the address where the redundant copy is located:

uboot-environment my_uboot_env {
    block-offset = 2048
    block-count = 16
    block-offset-redund = 2064
}

See the functions in the task section for getting and setting U-Boot variables.

NOTE: Currently, I've only implemented support for U-Boot environments that I use. Notably, this doesn't support big endian targets, and writes to raw NAND parts. Please consider contributing back support for these if you use them.

If fwup's U-Boot support does not meet your needs, it is always possible to create environment images using the mkenvimage utility and raw_write them to the proper locations. This is probably more appropriate when setting lots of variables.

task

The task section specifies a firmware update task. These sections are the main part of the firmware update archive since they describe the conditions upon which an update is applied and the steps to apply the update. Each task section must have a unique name, but when searching for a task, the firmware update tool only does a prefix match. This lets you define multiple tasks that can be evaluated based on conditions on the target hardware. The first matching task is the one that gets applied. This can be useful if the upgrade process is different based on the version of firmware currently on the target, the target architecture, etc. The following table lists the supported constraints:

The remainder of the task section is a list of event handlers. Event handlers are organized as scopes. An event handler matches during the application of a firmware update when an event occurs. Events include initialization, completion, errors, and files being decompressed from the archive. Since archives are processed in a streaming manner, the order of events is deterministic based on the order that files were added to the archive. If it is important that one event happen before another, make sure that file-resource sections are specified in the desired order. The following table lists supported events:

The event scopes contain a list of actions. Actions can format file systems, copy files to file systems or write to raw locations on the destination.

Minimizing writes to the destination

Fwup tries to reduce the writes to the destination media using the following mechanisms:

1. An internal block cache summarizes writes to individual blocks. This is outside of OS caching that fwup tries to disable to ensure ordering of writes. (i.e., switching the active filesystem needs to be after the filesystem writes get made)
2. The --minimize-writes option causes fwup to check if a block on the destination has actually changed before writing it.

The motivation for these features is reduce the number of changes to the disk since each change could be interrupted by a system reset or result in a write failure or corruption. While the chance is remote, some users find peace of mind in the reduced write cycles on FLASH media.

Performance-wise, the internal block caching is almost improves update times. Minimizing writes incurs a block read operation before every write. As one would expect, if the writes are actually redundant, it improves firmware update time. For completely different content, it appears that the additional read does not noticeably affect firmware update time on relatively fast embedded devices (600+ MHz 4-core in testing). Slower single core devices have slightly longer firmware update times.

Delta firmware updates (BETA)

The purpose of delta firmware updates is to reduce firmware update file sizes and their associated costs by sending patches to devices. This requires that one know what firmware is running on the device so that an appropriate patch can be made. As with other features, fwup only addresses the firmware update file processing piece, but assists in this process by 1. uniquely identifying firmware via UUIDs and 2. including xdelta3 decompression support.

This feature is currently BETA, since it may change in backwards incompatible ways based on trial deployments. If you're using this, please avoid deploying it to places that are hard to access just in case.

fwup cannot produce delta .fw archives automatically. However, they are easy to produce manually or via a script. The deployments in progress use scripts to create patches for upgrading all possible firmware versions (keyed off UUID). Here's how:

1. Decide which file resource is a good candidate for delta updates (you can pick more than one). Call this rootfs.img.

2. In the fwup.conf, in the on-resource rootfs.img handlers, specify where the currently running rootfs.img contents exist. For example, if you're doing an A/B upgrades, when you upgrade B, point to the A's rootfs for source contents and vice versa for upgrading A. It will look something like this:

   task upgrade.a {
       on-resource rootfs.img {
           delta-source-raw-offset=${ROOTFS_B_PART_OFFSET}
           delta-source-raw-count=${ROOTFS_B_PART_COUNT}
           raw_write(${ROOTFS_A_PART_OFFSET})
       }
   }
   task upgrade.b {
       on-resource rootfs.img {
           delta-source-raw-offset=${ROOTFS_A_PART_OFFSET}
           delta-source-raw-count=${ROOTFS_A_PART_COUNT}
           raw_write(${ROOTFS_B_PART_OFFSET})
       }
   }
   

3. Obtain the firmware file for the software running on the device. Call this original.fw.

4. Create the firmware file for the new software. Call this update.fw. If you normally sign your firmware files, you can sign it now or sign the resulting patch file depending on what works best for your release process.

5. Run unzip on both original.fw and update.fw in different directories. The rootfs.img file can be found under the data directory when unzipped.

6. mkdir -p my_patch/data

7. xdelta3 -A -S -f -s original/data/rootfs.img update/data/rootfs.img my_patch/data/rootfs.img

8. cp update.fw my_patch/delta_update.fw

9. cd my_patch && zip delta_update.fw data/rootfs.img

The delta_update.fw will have the patch for the rootfs.img and should be quite a bit smaller. If not, check that your before and after rootfs.img files don't have a lot of timestamp changes or are already so compressed that the deltas propagate through the entire image.

Even though you're manually modifying the .fw file, fwup stills provide guarantees on the final bits installed on the device. For one, fwup computes Blake 2B hashes on the final contents of the files, so what matters is what comes out of the xdelta3 decompressor and not what is stored in the archive. Most likely though, xdelta3 will detect corruption since it checks Adler32 checksums as it decompresses.

Delta update on-resource source settings

Where to find the source ("before" version) is always specified in on-resource blocks. It can either be source from raw bytes or from files in a FAT-formatted partition. Only one source is supported in each on-resource block.

Sparse files

Sparse files are files with gaps in them that are only represented on the filesystem in metadata. Not all filesystems support sparse files, but in general, Linux has good support. Creating a sparse file is easy: just seek to a location past the end of file and write some data. The gap is "stored" as a hole in the filesystem metadata. Data is read back from the hole as zeros. Data and holes are restricted to start and end on filesystem block boundaries, so small gaps may be filled in with zeros rather than being stored as a hole.

Why is this important? If you're using fwup to write a large EXT2 partition, you'll find that it contains many gaps. It would be better to just write the EXT2 data and metadata without filling in all of the unused space. Sparse file support in fwup lets you do that. Since EXT2 filesystems legitimately contain long runs of zeros that must be written to Flash, fwup queries the filesystem containing the EXT2 data to find the gaps. Other tools like dd(1) only look for runs of zeros so their sparse file support cannot be used to emulate this. You may see warnings about copying sparse files to Flash and it has to do with tools not writing long runs of zeros. The consequence of fwup querying the filesystem for holes is that this feature only works when firmware update archives are created on operating systems and filesystems that support it. Of course, firmware updates can be applied on systems without support for querying holes in files. Those systems also benefit from not having to write as much to Flash devices. If you instead apply a firmware update to a normal file, though, the OS will likely fill in the gaps with zeros and thus offer no improvement.

There is one VERY important caveat with the sparse file handling: Some zeros in files are important and some are not. If runs in zeros in a file are important and they are written to a file as a "hole", fwup will not write them back. This is catastrophic if the zeros represent things like free blocks on a filesystem. Luckily, the file system formatting utilities write the important zeros to the disk and the OS does not scan bytes to see which ones are runs on zeros and automatically create holes. Programs like dd(1) can do this, though, so it is crucial that you do not run files through dd to make then sparser before passing them to fwup.

This feature is off by default. To turn this feature on, set skip-holes on the resource to true:

file-resource rootfs.img {
        host-path = "output/images/rootfs.img"
        skip-holes = true
}

Disk encryption

The raw_write function has limited support for disk encryption that's compatible with the Linux dm-crypt kernel driver. This makes it possible to write filesystem data in a way that's unreadable at rest. Caveats are in order:

1. fwup does not address handling of secret keys and improper handling can easily compromise the benefits of encryption
2. The .fw archive is not encrypted. This mechanism assumes that the secrecy of the archive is protected by other means. Of course, it is possible pre-encrypt the data in the archive, but then you can't have device-specific secret keys.
3. Only the simplest dm-crypt cipher is currently supported ("aes-cbc-plain"). This has known deficiencies. PRs for other modes that can be incorporated under fwup's Apache License would be appreciated

Various tutorials exist on the Internet for creating encrypted filesystems and mounting filesystems using dm-crypt. fwup is much simpler. It takes a block of bytes to write, encrypts it, and writes it to the destination. For example, if you have a SquashFS-formatted filesystem that you want written encrypted to a disk, you'd have this fragment:

on-resource fs.squashfs {
    raw_write(${PARTITION_START}, "cipher=aes-cbc-plain", "secret=\${SECRET_KEY}")
}

In the above example, the SECRET_KEY is expected to come from an environment variable being set on the device when applying the firmware update. You could, of course, hard-code the secret key in the configuration file to test things out. The key is hex-encoded.

Then, on the device, mount the SquashFS partition but use dm-crypt. The process will look something like this:

losetup /dev/loop0 /dev/mmcblk0p2
cryptsetup open --type=plain --cipher=aes-cbc-plain --key-file=key.txt /dev/loop0 my-filesystem
mount /dev/mapper/my-filesystem /mnt

You will likely need to replace many of the arguments above with ones appropriate for your system.

Reproducible builds

It's possible for the system time to be saved in various places when using fwup. This means that an archive with the same contents, but built at different times results in .fw files with different bytes. See reproducible-builds.org for a discussion on this topic.

fwup obeys the SOURCE_DATE_EPOCH environment variable and will force all timestamps to the value of that variable when needed. Set $SOURCE_DATE_EPOCH to the number of seconds since midnight Jan 1, 1970 (run date +%s) to use this feature.

A better way of comparing .fw archives, though, is to use the firmware UUID. The firmware UUID is computed from the contents of the archive rather than the bit-for-bit representation of the .fw file, itself. The firmware UUID is unaffected by timestamps (with or without SOURCE_DATE_EPOCH) or other things like compression algorithm improvements. This is not to say that SOURCE_DATE_EPOCH is not important, but that the UUID is an additional tool for ensuring that firmware updates are reproducible.

Firmware authentication

Firmware archives can be authenticated using a simple public/private key scheme. To get started, create a public/private key pair by invoking fwup -g. The algorithm used is Ed25519. This generates two file: fwup-key.pub and fwup-key.priv. It is critical to keep the signing key, fwup-key.priv secret.

To sign an archive, pass -s fwup-key.priv to fwup when creating the firmware. The other option is to sign the firmware archive after creation with --sign or -S.

To verify that an archive has been signed, pass -p fwup-key.pub on the command line to any of the commands that read the archive. E.g., -a, -l or -m.

It is important to understand how verification works so that the security of the archive isn't compromised. Firmware updates are applied in one pass to avoid needing a lot of memory or disk space. The consequence of this is that verification is done on the fly. The main metadata for the archive is always verified before any operations occur. Cryptographic hashes (using the BLAKE2b-256 algorithm) of each file contained in the archive is stored in the metadata. The hash for each file is computed on the fly, so a compromised file may not be detected until it has been written to Flash. Since this is obviously bad, the strategy for creating firmware updates is to write them to an unused location first and then switch over at the last possible minute. This is desirable to do anyway, since this strategy also provides some protection against the user disconnecting power midway through the firmware update.

Reboot parameters

The Linux reboot syscall takes an argument when using the LINUX_REBOOT_CMD_RESTART2 command. This is not the shutdown timeout. It's a free text string passed to the Linux kernel. It's use is device and driver-specific. The Raspberry Pi uses it for its TRYBOOT feature. Fwup allows configurations to set the reboot parameters via the reboot_param function:

reboot_param("0 tryboot")

Even though fwup doesn't reboot the system, it can influence the parameter passed to the kernel due to the simplistic way this parameter is passed to the init process (PID 1). Systemd, for example, reads the /run/systemd/reboot-param file when init initiates the reboot. Erlinit with Nerves uses the file /run/reboot-param. Fwup will automatically detect Systemd and write the right file. The --reboot-param-path commandline argument can be used to force a path.

Integration with applications

It is expected that many users will want to integrate fwup with their applications. Many operations can be accomplished by just invoking the fwup executable and parsing the text written to stdout. When applying firmware progress updates are delivered based on commandline options:

1. Human readable - This is the default. Progress is updated from the text 0% to 100%.
2. Numeric (-n) - Progress is printed as 0\n to 100\n
3. Quiet (-q) - No progress is printed

While the above works well for scripts and when errors can be seen by the operator, fwup supports a structured use of stdin/stdout as well. Specify the --framing option to any of the commands to use this option.

The framing feature is influenced by the Erlang VM's port API and should be relatively easy to integrate with non-Erlang VM languages. The framing works around deficiencies in the built-in interprocess communication. For example, by enabling framing, a program can stream a firmware update through fwup's stdin without needing to close its stdout to signal end of file. Another feature aided by framing is knowing what text goes together and whether the text is part of an error message or not. Exit status is still an indicator of success or failure, but the controlling application doesn't need to wait for the program to exit to know what happened.

In --framing mode, all communication with fwup is done in packets (rather than byte streams). A packet starts with a 4 byte length field. The length is a big endian (network byte order) unsigned integer. A zero-length packet (i.e., 4 bytes of zeros) signals end of input.

Input and output packets have different formats. For sending input to fwup (like when streaming a .fw file using stdio), the input bytes should be framed into packets however is most convenient. For example, if bytes are received in 4K chunks, then they can be sent to fwup in 4K packets with a zero-length packet at the end. The packets need not be the same size.

All output packets from fwup have a 2 byte type field at the beginning of the packet:

The following types are defined:

A related option is --exit-handshake. This option was specifically implemented for Erlang to support integration with its port process feature. It may be useful for other integrations where it's more convenient to wait for a final character coming from a subprocess rather than watching for an exit. The problem with Erlang is that it's easy for the message that the process exited to beat the final characters coming out stdout. When this option is enabled, fwup expects the calling process to close stdin when it's ready for fwup to exit.

FAQ

How do I include a file in the archive that isn't used by fwup

The scenario is that you need to store some metadata or some other information that is useful to another program. For example, you'd like to include some documentation or other notes inside the firmware update archive that the destination device can pull out and present on a UI. To do this, just add file-resource blocks for each file. These blocks don't need to be referenced by an on-resource block.

How do I include the firmware version in the archive

If you are using git, you can invoke fwup as follows:

GITDESCRIBE=`git describe` fwup -c -f myupdate.conf -o out.fw

Then in myupdate.conf add the line:

meta-version = "${GITDESCRIBE}"

On the target device, you can retrieve the version by using -m. For example:

$ fwup -m -i out.fw
meta-product = "Awesome product"
meta-description = "A description"
meta-version = "v0.0.1"
meta-author = "Me"
meta-platform = "bbb"
meta-architecture = "arm"
meta-uuid="07a34e75-b7ea-5ed8-b5d9-80c10daf4939"

What's something cool that you can do with fwup

Ok, this isn't really a FAQ, but for some reason people think this is cool. Many systems that I work on are network connected with ssh. Sometimes I update them by doing this:

$ cat mysoftware.fw | ssh root@192.168.1.20 \
    'fwup -a -U -d /dev/mmcblk2 -t upgrade && reboot'

The ability to pipe software updates through fwup comes in handy. This has also gotten me out of situations where, for whatever reason, I no longer had enough space to store the update on the device.

How should I implement multiple signing keys

There are several use cases where it's necessary to have multiple signing keys in use at a time. For example, you could want to enforce that all firmware updates are signed in your infrastructure, but not force everyone to go though the official secure path for QA builds. You may also want to only use each key for a limited amount of signings.

Currently, each firmware file can only have one signature. However, the verifier (device) can specify multiple public keys (repeated -p options). While it is possible to call fwup for each key, specifying multiples keys is recommended to run the verification through fwup to support streamed updates and also to simplify this critical code path.

How do I debug

I apply updates to regular files on my laptop (as opposed to eMMC or SDCards) and examine them with a hex editor. A few other routes might be useful too:

1. Unzip the .fw file to inspect the contents. It's a regular ZIP file and the meta.conf file inside it is the stripped down view of what your configuration looks like after variable substitution, etc.
2. Add the error() function to do printf-style debugging.
3. Find an image that works and skip updating some sections. For example, some processors are very picky on the MBR contents and it's easier to get everything else working before tackling partition constraints.

How do I specify the root partition in Linux

There are a few options. Most people can specify root=/dev/mmcblk0p1 or root=/dev/sda1 or something similar on the kernel commandline and everything will work out fine. On systems with multiple drives and an unpredictable boot order, you can specify root=PARTUUID=01234567-01 where the -01 part corresponds to the 1-based partition index and 01234567 is any signature. In your fwup.conf file's MBR block, specify signature = 0x01234567. A third option is to use an initramfs and not worry about any of this.

How do I get the best performance

In general, fwup writes to Flash memory in large blocks so that the update can occur quickly. Obviously, reducing the amount that needs to get written always helps. Beyond that, most optimizations are platform dependent. Linux caches writes so aggressively that writes to Flash memory are nearly as fast as possible. OSX, on the other hand, does very little caching, so doing things like only working with one FAT filesystem at a time can help. In this case, fwup only caches writes to one FAT filesystem at a time, so mixing them will flush caches. OSX is also slow to unmount disks, so keep in mind that performance can only be so fast on some systems.

How do I update /dev/mmcblock0boot0

The special eMMC boot partitions are updatable the same way as the main partition. When I create .fw files for manufacturing, I create two targets, a complete target that updates the main eMMC and a bootloader target that updates mmcblock0boot0. The manufacturing script runs fwup twice: once for the complete target and then again for the bootloader target.

Also, the /dev/mmcblock0boot0 device is forced read-only by the kernel. To unlock it, run:

echo 0 > /sys/block/mmcblk0boot0/force_ro

What's the best way to identify firmware versions

fwup supports several ways:

1. Store the version in meta-version. This is usually the friendliest for end users.
2. Store the git hash in meta-vcs-identifier. This is good for developers.
3. Use the fwup-computed UUID that's available in meta-uuid' and ${FWUP_META_UUID}`.

Of these, the third one is always available since version fwup v1.2.1. The motivation behind it was to unambiguously know whether installed firmware matches the desired firmware. Since it is computed, .fw files generated with previous versions of fwup have UUIDs.

The first two options require the versions to be added to the fwup.conf file. They are usually added using environment variables so that the version numbers are not hardcoded.

How do I get the firmware metadata formatted as JSON

Use jq!

$ fwup -m -i $FW_FILE | jq -n -R 'reduce inputs as $i ({}; . + ($i | (match("([^=]*)=\"(.*)\"") | .captures | {(.[0].string) : .[1].string})))'
{
  "meta-product": "My Awesome Product",
  "meta-version": "0.1.0",
  "meta-author": "All of us",
  "meta-platform": "imx6",
  "meta-architecture": "arm",
  "meta-creation-date": "2018-11-07T14:46:38Z",
  "meta-uuid": "7add3c6d-230c-5bf1-77ec-5f785e91be40"
}

How do I use "raw" NAND Flash

Some "raw" NAND Flash requires a wear leveling layer such as UBI. See the UBI Example fwup.conf for how to integrate fwup with the UBI toolchain.

How do you pronounce fwup

I used to pronounce it "eff-double-you-up", but then coworkers and others started calling it "fwup" (one syllable) and "fwup-dates" when referring to the .fw files. I now use the one syllable version. This has caused issues in the documentation where "an" is used rather than "a". Feel free to send PRs.

Licenses

This utility contains source code with various licenses. The bulk of the code is licensed with the Apache-2.0 license which can be found in the LICENSE file.

All statically-linked 3rd party source code can be found in the src/3rdparty directory. The following sections summarize the included code licenses.

base64.c

Public Domain or Creative Commons CC0. See file for explanatory text.

FatFS

The FAT filesystem code (FatFs) comes from elm-chan.org and has the following license:

FatFs module is a generic FAT file system module for small embedded systems. This is a free software that opened for education, research and commercial developments under license policy of following terms.

Copyright (C) 2015, ChaN, all right reserved.

• The FatFs module is a free software and there is NO WARRANTY.
• No restriction on use. You can use, modify and redistribute it for personal, non-profit or commercial products UNDER YOUR RESPONSIBILITY.
• Redistributions of source code must retain the above copyright notice.

Monocypher

This package has been dual licensed under the 2-clause BSD and CC-0. See LICENCE.md.

semver.c

fwup uses semver.c for checking versions. semver.c is Copyright (c) Tomás Aparicio and distributed under the MIT License. See LICENSE.

strptime.c

On systems without the function strptime(), a version from Google is included that is distributed under the Apache 2.0 license.

Tiny AES

This code was released into the public domain. See unlicense.txt for details.

Xdelta3

Only the xdelta3 decompressor is currently used by fwup so most of the code in the xdelta3 directory is ignored or disabled. Importantly, fwup's use of xdelta3 does not bring in any xdelta3 dependencies. Xdelta3 is covered by the Apache-2.0 license.