Awesome

Buildpipe

A Buildkite plugin to dynamically generate pipelines. Especially useful for monorepos where you want to create dependencies between projects.

Example

initial_pipeline.yml

steps:
  - label: ":buildkite:"
    plugins:
      - jwplayer/buildpipe#v0.10.3:
          dynamic_pipeline: dynamic_pipeline.yml

dynamic_pipeline.yml

env:
  GLOBAL_ENV_VAR: test
projects:
 - label: project1
   path: project1/  # changes in this directory will trigger steps for project1
   skip:
     - deploy*  # skip steps with label matching deploy* (e.g. deploy-prd)
     - test
   env:
     PROJECT_ENV_VAR: project1variable
 - label: project2
   skip: test
   path:
      - project2/
      - project1  # you can trigger a project using multiple paths
 - label: project3
   skip:
     - deploy-stg
   path: project3/somedir/  # subpaths can also be triggered
steps:  # the same schema as regular buildkite pipeline steps
  - label: test
    env:
      BUILDPIPE_SCOPE: project  # this variable ensures a test step is generated for each project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH  # BUILDPIPE_PROJECT_PATH is set by buildpipe
      - make test
  - wait
  - label: build
    branches: "master"
    env:
      BUILDPIPE_SCOPE: project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH
      - make build
      - make publish-image
    agents:
      - queue=build
  - wait
  - label: tag
    branches: "master"
    command:
      - make tag-release
  - wait
  - label: deploy-stg
    branches: "master"
    concurrency: 1
    concurrency_group: deploy-stg
    env:
      BUILDPIPE_SCOPE: project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH
      - make deploy-staging
  - wait
  - block: ":rocket: Release!"
    branches: "master"
  - wait
  - label: deploy-prd
    branches: "master"
    concurrency: 1
    concurrency_group: deploy-prd
    env:
      BUILDPIPE_SCOPE: project
    command:
      - cd $$BUILDPIPE_PROJECT_PATH
      - make deploy-prod

The above pipelines specify the following:

• There are three projects to track in the repository.
• The env variable BUILDPIPE_SCOPE: project tells buildpipe to generate a step for each project if that project changed.
• The skip option will skip any step label matching deploy*.
• The env variable BUILDPIPE_PROJECT_PATH is created by buildpipe as the project's path. If multiple paths are specified for a project, it's the first path.
• There is a global environment variable defined (GLOBAL_ENV_VAR). This variable will be added to every step in the pipeline.
• There is also a project scoped environment variable defined (PROJECT_ENV_VAR). That variable will be added to all steps of the project where it was defined (project1).

Full working example

For a full working example, check out Buildkite Monorepo Example.

Configuration

Plugin

Project schema

Other useful things to note:

• Option skip makes use of Unix shell-style wildcards (Look at .gitignore files for inspiration)
• If multiple paths are specified, the environment variable BUILDPIPE_PROJECT_PATH will be the first path.
• Environment variables are available in the pipeline step.

Environment Variables

Since version 0.9.2, there is the option to define environment variables on different scope levels in the pipeline. Buildkite already supports environment variables defined on a step scope, but buildpipe adds the ability to define global and also project specific environment variables.

Project env vars

env:
  THIS_IS_A_GLOBAL_ENV_VAR: global
  GLOBAL_ENV_VAR_2: another_global_value
projects:
  - label: project1
    path: project1/
    env:
      THIS_IS_A_PROJECT_ENV_VAR: project_scoped
      PROJECT_ENV_VAR_2: second_env_var_value
  - label: project2
    ...
steps:
  - label: step1
    ...

Specifying build projects

Since version 0.10.0, you can manually specify which projects you wish to build using the BUILDKITE_PLUGIN_BUILDPIPE_BUILD_PROJECTS environment variable. If this environemnt variable is provided, it will take precedence over any existing git diff. You can specify the build projects in one of two ways:

1. Provide a comma-separated list of build projects (identified by the project label attribute). For example:

BUILDKITE_PLUGIN_BUILDPIPE_BUILD_PROJECTS="project1,project2"

2. Build all projects in a Buildpipe configuration:

BUILDKITE_PLUGIN_BUILDPIPE_BUILD_PROJECTS="*"

diff_ commands

Depending on your merge strategy, you might need to use different diff command.

Buildpipe assumes you are using a merge strategy on the default branch, which is assumed to be master.

The command for the non-default branch (e.g. when you have a PR up) is:

git log --name-only --no-merges --pretty=format: origin..HEAD

The command for the default branch you merge to is currently:

git log -m -1 --name-only --pretty=format: $BUILDKITE_COMMIT

Requirements

Only curl is required to download the binary.

Installing buildpipe

You can also install buildpipe in the agent bootstrap script:

curl -Lf -o /usr/local/bin/buildpipe https://github.com/jwplayer/buildpipe-buildkite-plugin/releases/download/v${BUILDPIPE_VERSION}/buildpipe-linux \
  && chmod +x /usr/local/bin/buildpipe

Your initial pipeline would need to pass the options as environment variables with prefix BUILDKITE_PLUGIN_BUILDPIPE_. For example:

steps:
  - label: "buildpipe"
    command: buildpipe
    env:
      BUILDKITE_PLUGIN_BUILDPIPE_DYNAMIC_PIPELINE: path/to/dynamic_pipeline.yml
      BUILDKITE_PLUGIN_BUILDPIPE_LOG_LEVEL: debug

Troubleshooting

Buildpipe is incorrectly showing project as changed

Buildkite doesn't by default do clean checkouts. To enable clean checkouts set the BUILDKITE_CLEAN_CHECKOUT environment variable. An example is to modify the pre-checkout hook, .buildkite/hooks/pre-checkout:

#!/bin/bash
set -euo pipefail

echo '--- :house_with_garden: Setting up pre-checkout'

export BUILDKITE_CLEAN_CHECKOUT="true"

Testing

make test

License

Apache v2