Home

Awesome

Serverless Jetpack 🚀 — Formidable, We build the modern web

npm version Actions Status MIT license Maintenance Status

A faster JavaScript packager for Serverless applications.

Overview

The Serverless framework is a fantastic one-stop-shop for taking your code and packing up all the infrastructure around it to deploy it to the cloud. Unfortunately, for many JavaScript applications, some aspects of packaging are slow, hindering deployment speed and developer happiness.

With the serverless-jetpack plugin, many common, slow Serverless packaging scenarios can be dramatically sped up. All with a very easy, seamless integration into your existing Serverless projects.

<!-- START doctoc generated TOC please keep comment here to allow auto update --> <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> <!-- END doctoc generated TOC please keep comment here to allow auto update -->

Usage

The short, short version

First, install the plugin:

$ yarn add --dev serverless-jetpack
$ npm install --save-dev serverless-jetpack

Add to serverless.yml

plugins:
  - serverless-jetpack

... and you're off to faster packaging awesomeness! 🚀

A little more detail...

The plugin supports all normal built-in Serverless framework packaging configurations in serverless.yml like:

package:
  # Any `include`, `exclude` logic is applied to the whole service, the same
  # as built-in serverless packaging.
  # include: ...
  exclude:
    - "*"
    - "**/node_modules/aws-sdk/**" # included on Lambda.
    - "!package.json"

plugins:
  # Add the plugin here.
  - serverless-jetpack

functions:
  base:
    # ...
  another:
    # ...
    package:
      # These work just like built-in serverless packaging - added to the
      # service-level exclude/include fields.
      include:
        - "src/**"
        - "!**/node_modules/aws-sdk/**" # Faster way to exclude
        - "package.json"

Configuration

Most Serverless framework projects should be able to use Jetpack without any extra configuration besides the plugins entry. However, there are some additional options that may be useful in some projects (e.g., lerna monorepos, yarn workspaces)...

Service-level configurations available via custom.jetpack:

The following function and layer-level configurations available via functions.{FN_NAME}.jetpack and layers.{LAYER_NAME}.jetpack:

Here are some example configurations:

Additional roots

# serverless.yml
plugins:
  - serverless-jetpack

functions:
  base:
    # ...
  another:
    # This example monorepo project has:
    # - `packages/another/src`: JS source code to include
    # - `packages/another/package.json`: Declares production dependencies
    # - `packages/another/node_modules`: One location prod deps may be.
    # - `node_modules`: Another location prod deps may be if hoisted.
    # ...
    package:
      individually: true
    jetpack:
      roots:
        # If you want to keep prod deps from servicePath/CWD package.json
        # - "."
        # Different root to infer prod deps from package.json
        - "packages/another"
    include:
      # Ex: Typically you'll also add in sources from a monorepo package.
      - "packages/another/src/**"

Different base root

# serverless.yml
plugins:
  - serverless-jetpack

custom:
  jetpack:
    # Search for hoisted dependencies to one parent above normal.
    base: ".."

package:
  # ...
  include:
    # **NOTE**: The include patterns now change to allow the underlying
    # globbing libraries to reach below the working directory to our base,
    # so patterns should be of the format:
    # - "!{BASE/,}{**/,}NORMAL_PATTERN"
    # - "!{BASE/,}{**/,}node_modules/aws-sdk/**"
    # - "!{BASE/,}{**/,}node_modules/{@*/*,*}/README.md"
    #
    # ... here with a BASE of `..` that means:
    # General
    - "!{../,}{**/,}.DS_Store"
    - "!{../,}{**/,}.vscode/**"
    # Dependencies
    - "!{../,}{**/,}node_modules/aws-sdk/**"
    - "!{../,}{**/,}node_modules/{@*/*,*}/CHANGELOG.md"
    - "!{../,}{**/,}node_modules/{@*/*,*}/README.md"

functions:
  base:
    # ...

With custom pre-includes

# 1. `preInclude` comes first after internal `**` pattern.
custom:
  jetpack:
    preInclude:
      - "!**" # Start with absolutely nothing (typical in monorepo scenario)

# 2. Jetpack then dynamically adds in production dependency glob patterns.

# 3. Then, we apply the normal serverless `include`s.
package:
  individually: true
  include:
    - "!**/node_modules/aws-sdk/**"

plugins:
  - serverless-jetpack

functions:
  base:
    # ...
  another:
    jetpack:
      roots:
        - "packages/another"
      preInclude:
        # Tip: Could then have a service-level `include` negate subfiles.
        - "packages/another/dist/**"
    include:
      - "packages/another/src/**"

Layers

# serverless.yml
plugins:
  - serverless-jetpack

layers:
  vendor:
    # A typical pattern is `NAME/nodejs/node_modules` that expands to
    # `/opt/nodejs/node_modules` which is included in `NODE_PATH` and available
    # to running lambdas. Here, we use `jetpack.roots` to properly exclude
    # `devDependencies` that built-in Serverless wouldn't.
    path: layers/vendor
    jetpack:
      roots:
        # Instruct Jetpack to review and exclude devDependencies originating
        # from this `package.json` directory.
        - "layers/vendor/nodejs"

How Jetpack's faster dependency filtering works

Serverless built-in packaging slows to a crawl in applications that have lots of files from devDependencies. Although the excludeDevDependencies option will ultimately remove these from the target zip bundle, it does so only after the files are read from disk, wasting a lot of disk I/O and time.

The serverless-jetpack plugin removes this bottleneck by performing a fast production dependency on-disk discovery via the inspectdep library before any globbing is done. The discovered production dependencies are then converted into patterns and injected into the otherwise normal Serverless framework packaging heuristics to efficiently avoid all unnecessary disk I/O due to devDependencies in node_modules.

Process-wise, the serverless-jetpack plugin detects when built-in packaging applies and then takes over the packaging process. The plugin then sets appropriate internal Serverless artifact fields to cause Serverless to skip the (slower) built-in packaging.

The nitty gritty of why it's faster

Let's start by looking at how Serverless packages (more or less):

  1. If the excludeDevDependencies option is set, use synchronous globby() for on disk I/O calls to find all the package.json files in node_modules, then infer which are devDependencies. Use this information to enhance the include|exclude configured options.
  2. Glob files from disk using globby with a root ** (all files) and the include pattern, following symlinks, and create a list of files (no directories). This is again disk I/O.
  3. Filter the in-memory list of files using nanomatch via service + function exclude, then include patterns in order to decide what is included in the package zip file.

This is potentially slow if node_modules contains a lot of ultimately removed files, yielding a lot of completely wasted disk I/O time.

Jetpack, by contrast does the following:

  1. Efficiently infer production dependencies from disk without globbing, and without reading any devDependencies.
  2. Glob files from disk with a root ** (all files), !node_modules/** (exclude all by default), node_modules/PROD_DEP_01/**, node_modules/PROD_DEP_02/**, ... (add in specific directories of production dependencies), and then the normal include patterns. This small nuance of limiting the node_modules globbing to just production dependencies gives us an impressive speedup.
  3. Apply service + function exclude, then include patterns in order to decide what is included in the package zip file.

This ends up being way faster in most cases, and particularly when you have very large devDependencies. It is worth pointing out the minor implication that:

Complexities

Other Serverless plugins that set package.artifact

The serverless-jetpack plugin hooks into the Serverless packaging lifecycle by being the last function run in the before:package:createDeploymentArtifacts lifecycle event. This means that if a user configures package.artifact directly in their Serverless configuration or another plugin sets package.artifact before Jetpack runs then Jetpack will skip the unit of packaging (service, function, layer, etc.).

Some notable plugins that do set package.artifact and thus don't need and won't use Jetpack (or vanilla Serverless packaging for that matter):

Minor differences vs. Serverless globbing

Our benchmark correctness tests highlight a number of various files not included by Jetpack that are included by serverless in packaging our benchmark scenarios. Some of these are things like node_modules/.yarn-integrity which Jetpack knowingly ignores because you shouldn't need it. All of the others we've discovered to date are instances in which serverless incorrectly includes devDependencies...

Layers

Jetpack supports layer packaging as close to serverless as it can. However, there are a couple of very wonky things with serverless' approach that you probably want to keep in mind:

Be careful with include configurations and node_modules

Let's start with how include|exclude work for both Serverless built-in packaging and Jetpack:

  1. Disk read phase with globby(). Assemble patterns in order from below and then return a list of files matching the total patterns.

    1. Start at ** (everything).
    2. (Jetpack only) Add in service and function-level jetpack.preInclude patterns.
    3. (Jetpack only) Add in dynamic patterns to include production node_modules.
    4. Add in service and function-level package.include patterns.

  2. File filtering phase with nanomatch(). Once we have a list of files read from disk, we apply patterns in order as follows to decide whether to include them (last positive match wins).

    1. (Jetpack only) Add in service and function-level jetpack.preInclude patterns.
    2. (Jetpack only) Add in dynamic patterns to include production node_modules.
    3. Add in service and function-level package.exclude patterns.
    4. (Serverless only) Add in dynamic patterns to exclude development node_modules
    5. Add in service and function-level package.include patterns.

The practical takeaway here is the it is typically faster to prefer include exclusions like !foo/** than to use exclude patterns like foo/** because the former avoids a lot of unneeded disk I/O.

Let's consider a pattern like this:

include:
  - "node_modules/**"
exclude:
  - # ... a whole bunch of stuff ...

This would likely be just as slow as built-in Serverless packaging because all of node_modules gets read from disk.

Thus, the best practice here when crafting service or function include configurations is: don't include anything extra in node_modules. It's fine to do extra exclusions like:

# Good. Remove dependency provided by lambda from zip
exclude:
  - "**/node_modules/aws-sdk/**"

# Better! Never even read the files from disk during globbing in the first place!
include:
  - "!**/node_modules/aws-sdk/**"

Packaging files Outside CWD

How files are zipped

A potentially serious situation that comes up with adding files to a Serverless package zip file is if any included files are outside of Serverless' servicePath / current working directory. For example, if you have files like:

- src/foo/bar.js
- ../node_modules/lodash/index.js

Any file below CWD is collapsed into starting at CWD and not outside. So, for the above example, we package / later expand:

- src/foo/bar.js                # The same.
- node_modules/lodash/index.js  # Removed `../`!!!

This most often happens with node_modules in monorepos where node_modules roots are scattered across different directories and nested. In particular, if you are using the custom.jetpack.base option this is likely going to come into play. Fortunately, in most cases, it's not that big of a deal. For example:

- node_modules/chalk/index.js
- ../node_modules/lodash/index.js

will collapse when zipped to:

- node_modules/chalk/index.js
- node_modules/lodash/index.js

... but Node.js resolution rules should resolve and load the collapsed package the same as if it were in the original location.

Zipping problems

The real problems occur if there is a path conflict where files collapse to the same location. For example, if we have:

- node_modules/lodash/index.js
- ../node_modules/lodash/index.js

this will append files with the same path in the zip file:

- node_modules/lodash/index.js
- node_modules/lodash/index.js

that when expanded leave only one file actually on disk!

How to detect zipping problems

The first level is detecting potentially collapsed files that conflict. Jetpack does this automatically with log warnings like:

Serverless: [serverless-jetpack] WARNING: Found 1 collapsed dependencies in .serverless/my-function.zip! Please fix, with hints at: https://npm.im/serverless-jetpack#packaging-files-outside-cwd
Serverless: [serverless-jetpack] .serverless/FN_NAME.zip collapsed dependencies:
- lodash (Packages: 2, Files: 108 unique, 216 total): [node_modules/lodash@4.17.11, ../node_modules/lodash@4.17.15]`

In the above example, 2 different versions of lodash were installed and their files were collapsed into the same path space. A total of 216 files will end up collapsed into 108 when expanded on disk in your cloud function. Yikes!

A good practice if you are using tracing mode is to set: jetpack.collapsed.bail = true so that Jetpack will throw an error and kill the serverless program if any collapsed conflicts are detected.

How to solve zipping problems

So how do we fix the problem?

A first starting point is to generate a full report of the packaging step. Instead of running serverless deploy|package <OPTIONS>, try out serverless jetpack package --report <OPTIONS>. This will produce a report at the end of packaging that gives a full list of files. You can then use the logged message above as a starting point to examine the actual files collapsed in the zip file. Then, spend a little time figuring out the dependencies of how things ended up where.

With a better understanding of what the files are and why we can turn to avoiding collapses. Some options:

Tracing mode

ℹī¸ Experimental: Although we have a wide array of tests, tracing mode is still considered experimental as we roll out the feature. You should be sure to test all the execution code paths in your deployed serverless functions and verify your bundled package contents before using in production.

Jetpack speeds up the underlying dependencies filtering approach of serverless packaging while providing completely equivalent bundles. However, this approach has some fundamental limitations:

Thus, we pose the question: What if we packaged only the files we needed at runtime?

Welcome to tracing mode!

Tracing mode is an alternative way to include dependencies in a serverless application. It works by using Acorn to parse out all dependencies in entry point files (require, require.resolve, static import) and then resolves them with resolve according to the Node.js resolution algorithm. This produces a list of the files that will actually be used at runtime and Jetpack includes these instead of traversing production dependencies. The engine for all of this work is a small, dedicated library, trace-deps.

Tracing configuration

The most basic configuration is just to enable custom.jetpack.trace (service-wide) or functions.{FN_NAME}.jetpack.trace (per-function) set to true. By default, tracing mode will trace just the entry point file specified in functions.{FN_NAME}.handler.

plugins:
  - serverless-jetpack

custom:
  jetpack:
    trace: true

The trace field can be a Boolean or object containing further configuration information.

Tracing options

The basic trace Boolean field should hopefully work for most cases. Jetpack provides several additional options for more flexibility:

Service-level configurations available via custom.jetpack.trace:

A way to allow certain packages to have potentially failing dependencies. Specify each object key as a package name and value as an array of dependencies that might be missing on disk. If the sub-dependency is found, then it is included in the bundle (this part distinguishes this option from ignores). If not, it is skipped without error.

The following function-level configurations available via functions.{FN_NAME}.jetpack.trace and layers.{LAYER_NAME}.jetpack.trace:

Let's see the advanced options in action:

plugins:
  - serverless-jetpack

custom:
  jetpack:
    preInclude:
      - "!**"
    trace:
      ignores:
        # Unconditionally skip `aws-sdk` and all dependencies
        # (Because it already is installed in target Lambda)
        - "aws-sdk"
      allowMissing:
        # For just the `ws` package allow certain lazy dependencies to be
        # skipped without error if not found on disk.
        "ws":
          - "bufferutil"
          - "utf-8-validate"
      dynamic:
        # Force errors if have unresolved dynamic imports
        bail: true
        # Resolve encountered dynamic import misses, either by tracing
        # additional files, or ignoring after confirmation of safety.
        resolutions:
          # **Application Source**
          #
          # Specify keys as relative path to application source files starting
          # with a dot.
          "./src/server/config.js":
            # Manually trace all configuration files for bespoke configuration
            # application code. (Note these are relative to the file key!)
            - "../../config/default.js"
            - "../../config/production.js"

          # Ignore dynamic import misses with empty array.
          "./src/something-else.js": []

          # **Dependencies**
          #
          # Specify keys as `PKG_NAME/path/to/file.js`.
          "bunyan/lib/bunyan.js":
            # - node_modules/bunyan/lib/bunyan.js [79:17]: require('dtrace-provider' + '')
            # - node_modules/bunyan/lib/bunyan.js [100:13]: require('mv' + '')
            # - node_modules/bunyan/lib/bunyan.js [106:27]: require('source-map-support' + '')
            #
            # These are all just try/catch-ed permissive require's meant to be
            # excluded in browser. We manually add them in here.
            - "dtrace-provider"
            - "mv"
            - "source-map-support"

          # Ignore: we aren't using themes.
          # - node_modules/colors/lib/colors.js [127:29]: require(theme)
          "colors/lib/colors.js": []

package:
  include:
    - "a/manual/file-i-want.js"

functions:
  # Functions in service package.
  # - `jetpack.trace.ignores` does not apply.
  # - `jetpack.trace.include` **will** include and trace additional files.
  service-packaged-app-1:
    handler: app1.handler

  service-packaged-app-2:
    handler: app2.handler
    jetpack:
      # - `jetpack.trace.allowMissing` additions are merged into service level
      trace:
        # Trace and include: `app2.js` + `extra/**.js` patterns
        include:
          - "extra/**.js"

  # Individually with no trace configuration will be traced from service-level config
  individually-packaged-1:
    handler: ind1.handler
    package:
      individually: true
      # Normal package include|exclude work the same, but are not traced.
      include:
        - "some/stuff/**"
    jetpack:
      trace:
        # When individually, `ignores` from fn are added: `["aws-sdk", "react-ssr-prepass"]`
        ignores:
          - "react-ssr-prepass"
        # When individually, `allowMissing` smart merges like:
        # `{ "ws": ["bufferutil", "utf-8-validate", "another"] }`
        allowMissing:
          "ws":
            - "another"

  # Individually with explicit `false` will not be traced
  individually-packaged-1:
    handler: ind1.handler
    package:
      individually: true
    jetpack:
      trace: false

Tracing caveats

Handling dynamic import misses

Dynamic imports that use variables or runtime execution like require(A_VARIABLE) or import(`template_${VARIABLE}`) cannot be used by Jetpack to infer what the underlying dependency files are for inclusion in the bundle. That means some level of developer work to handle.

Identify

The first step is to be aware and watch for dynamic import misses. Conveniently, Jetpack logs warnings like the following:

Serverless: [serverless-jetpack] WARNING: Found 6 dependency packages with tracing misses in .serverless/FN_NAME.zip! Please see logs and read: https://npm.im/serverless-jetpack#handling-dynamic-import-misses
Serverless: [serverless-jetpack] .serverless/FN_NAME.zip dependency package tracing misses: [* ... */,"colors","bunyan",/* ... */]

and produces combined --report output like:

### Tracing Dynamic Misses (`6` packages): Dependencies

...
- ../node_modules/aws-xray-sdk-core/node_modules/colors/lib/colors.js [127:29]: require(theme)
- ../node_modules/bunyan/lib/bunyan.js [79:17]: require('dtrace-provider' + '')
- ../node_modules/bunyan/lib/bunyan.js [100:13]: require('mv' + '')
- ../node_modules/bunyan/lib/bunyan.js [106:27]: require('source-map-support' + '')
...

which gives you the line + column number of the dynamic dependency in a given source file and snippet of the code in question.

In addition to just logging this information, you can ensure you have no unaccounted for dynamic import misses by setting jetpack.trace.dynamic.bail = true in your applicable service or function-level configuration.

Diagnose

With the --report output in hand, the recommended course is to identify what the impact is of these missed dynamic imports. For example, in node_modules/bunyan/lib/bunyan.js the interesting require('mv' + '') import is within a permissive try/catch block to allow conditional import of the library if found (and prevent browserify from bundling the library). For our Serverless application we could choose to ignore these dynamic imports or manually add in the imported libraries.

For other dependencies, there may well be "hidden" dependencies that you will need to add to your Serverless bundle for runtime correctness. Things like node-config which dynamically imports various configuration files from environment variable information, etc.

Remedy

Once we have logging information and the --report output, we can start remedying dynamic import misses via the Jetpack feature jetpack.trace.dynamic.resolutions. Resolutions are keys to files with dynamic import misses that allow a developer to specify what imports should be included manually or to simply ignore the dynamic import misses.

Keys: Resolutions take a key value to match each file with missing dynamic imports. There are two types of keys that are used:

Values: Values are an array of extra imports to add in from each file as if they were declared in that very file with require("EXTRA_IMPORT") or import "EXTRA_IMPORT". This means the values should either be relative paths within that package (./lib/auth/noop.js) or other package dependencies (lodash or lodash/map.js). * Note: We choose to support "additional imports" and not just file additions like package.include or jetpack.trace.include. The reason is that for package dependency import misses, the packages can be flattened to unpredictable locations in the node_modules trees and doubly so in monorepos. An import will always be resolved to the correct location, and that's why we choose it. At the same time, tools like package.include or jetpack.trace.includeare still available to use!

Some examples:

bunyan: The popular logger library has some optional dependencies that are not meant only for Node.js. To prevent browser bundling tools from including, they use a curious require strategy of require('PKG_NAME' + '') to defeat parsing. For Jetpack, this means we get dynamic misses reports of:

- node_modules/bunyan/lib/bunyan.js [79:17]: require('dtrace-provider' + '')
- node_modules/bunyan/lib/bunyan.js [100:13]: require('mv' + '')
- node_modules/bunyan/lib/bunyan.js [106:27]: require('source-map-support' + '')

Using resolutions we can remedy these by simple adding imports for all three libraries like:

custom:
  jetpack:
    trace:
      dynamic:
        resolutions:
          "bunyan/lib/bunyan.js":
            - "dtrace-provider"
            - "mv"
            - "source-map-support"

express: The popular server framework dynamically imports engines which produces a dynamic misses report of:

- node_modules/express/lib/view.js [81:13]: require(mod)

In a common case, this is a non-issue if you aren't using engines, so we can simply "ignore" the import miss by setting an empty array resolutions value:

custom:
  jetpack:
    trace:
      dynamic:
        resolutions:
          "express/lib/view.js": []

Once we have analyzed all of our misses and added resolutions to either ignore the miss or add other imports, we can then set trace.dynamic.bail = true to make sure that if future dependency upgrades adds new, unhandled dynamic misses we will get a failed build notification so we know that we're always deploying known, good code.

Tracing results

The following is a table of generated packages using vanilla Serverless vs Jetpack with tracing (using yarn benchmark:sizes).

The relevant portions of our measurement chart.

Results:

ScenarioTypeZipsFilesSizevs Base
simplejetpack1200529417-42.78 %
simplebaseline1433925260
complexjetpack215883835544-18.20 %
complexbaseline221204688648

Command Line Interface

Jetpack also provides some CLI options.

serverless jetpack package

Package a function like serverless package does, just with better options.

$ serverless jetpack package -h
Plugin: Jetpack
jetpack package ............... Packages a Serverless service or function
    --function / -f .................... Function name. Packages a single function (see 'deploy function')
    --report / -r ...................... Generate full bundle report

So, to package all service / functions like serverless package does, use:

$ serverless jetpack package # OR
$ serverless package

... as this is basically the same built-in or custom.

The neat addition that Jetpack provides is:

$ serverless jetpack package -f|--function {NAME}

which allows you to package just one named function exactly the same as serverless deploy -f {NAME} does. (Curiously serverless deploy implements the -f {NAME} option but serverless package does not.)

Benchmarks

The following is a simple, "on my machine" benchmark generated with yarn benchmark. It should not be taken to imply any real world timings, but more to express relative differences in speed using the serverless-jetpack versus the built-in baseline Serverless framework packaging logic.

As a quick guide to the results table:

Machine information:

Results:

ScenarioPkgTypeModeTimevs Base
simpleyarnjetpacktrace4878-74.25 %
simpleyarnjetpackdeps3861-79.62 %
simpleyarnbaseline18941
simplenpmjetpacktrace7290-68.34 %
simplenpmjetpackdeps4017-82.55 %
simplenpmbaseline23023
complexyarnjetpacktrace10475-70.93 %
complexyarnjetpackdeps8821-75.52 %
complexyarnbaseline36032
complexnpmjetpacktrace15644-59.13 %
complexnpmjetpackdeps9896-74.15 %
complexnpmbaseline38282

Maintenance status

Active: Formidable is actively working on this project, and we expect to continue for work for the foreseeable future. Bug reports, feature requests and pull requests are welcome.