Home

Awesome

Deployer

This repository contains an OCurrent pipeline for deploying the various other pipelines we use. When a new commit is pushed to the live branch of a source repository, it builds a new Docker image for the project and upgrades the service to that version.

The list of deployed services is located in doc/services.md.

The main configuration is in pipeline.ml. For example, one entry is:

      ocurrent, "docker-base-images", [
        (* Docker base images @ images.ci.ocaml.org *)
        make_docker
          "Dockerfile"
          [
            make_deployment
              ~branch:"live"
              ~target:"ocurrent/base-images:live"
              [`Ocamlorg_images "base-images_builder"];
          ];
      ];

This says that for the https://github.com/ocurrent/docker-base-images repository:

The pipeline also deploys some MirageOS unikernels, e.g.

mirage, "mirage-www", [
  unikernel "Dockerfile" ~target:"hvt" ["EXTRA_FLAGS=--tls=true"] ["master", "www"];
  unikernel "Dockerfile" ~target:"xen" ["EXTRA_FLAGS=--tls=true"] [];     (* (no deployments) *)
];

This builds each branch and PR of https://github.com/mirage/mirage-www for both hvt and xen targets. For the master branch, the hvt unikernel is deployed as the www Albatross service.

See VM-host.md for instructions about setting up a host for unikernels.

There are 3 different deployer pipelines defined in this project, each deploying a related set of services. Each is configured to deploy itself from its own branch, serves a web UI from its own URL, and is registered with its own GitHub Application. These relations are summarized in the following table:

DeployerBranchURLGitHub App
Tarideslive-ci3https://deploy.ci.devhttps://github.com/apps/deploy-ci-dev
Ocaml_orglive-ocaml-orghttps://deploy.ci.ocaml.orghttps://github.com/apps/deploy-ci-ocaml-org
Miragelive-miragehttps://deploy.mirageos.orghttps://github.com/apps/deploy-mirage-io

The deployer's respective GitHub application is responsible for sending the deployer notifications about updates to the repositories being monitored. If the GitHub application is not updated to correspond to the repositories configured for a deployer, it will receive no updates, and consequently nothing will be deployed. See Updating a deployer configuration.

Testing locally

To test changes to the pipeline, use:

dune exec -- ocurrent-deployer-local --confirm=harmless --submission-service submission.cap \
                                     --github-webhook-secret-file github-secret-file \
                                     --flavour tarides -v \
                                     ocurrent/ocaml-ci

You will need a submission.cap to access an OCluster build cluster (you can run one locally fairly easily if needed), along with a github-secret-file containing a valid GitHub secret for securing webhooks.

Replace ocurrent/ocaml-ci with the GitHub repository you want to check, or omit it to check all of them.

Unlike the full pipeline, this:

You can supply --github-app-id and related options if you want to access GitHub via an app (this gives a higher rate limit for queries, allows setting the result status and handling GitHub webhooks).

Updating a deployer configuration

Each deployer is configured with a list of the repositories it monitors. Whenever a repository is added to or removed from a deployer's configuration, the respective GitHub App must also be updated. Once a configuration change has been made and deployed (see Updating services), update the GitHub App as follows:

Suggested workflows

Updating services

To update a deployment that is managed by ocurrent-deployer (which could be ocurrent-deployer itself):

  1. Make a PR on that project's repository targeting its master branch as usual.
  2. Once it has passed CI/review, a project admin will git push origin HEAD:live to deploy it (replacing live with the configured deployment branch, as needed).
  3. If it works, the PR can be merged to master.

Add a new service

  1. Deploy the service(s) manually using docker stack deploy first.
  2. Once that's working, make a PR against the ocurrent-deployer repository adding a rule to keep the services up-to-date. For the PR:
    • Drop the id_rsa.pub key in the ~/.ssh/authorized_keys file on the machine where you want the deployer to deploy the container.
    • Add the machine where you want to have the deployments to the context/meta folder. eg to add awesome.ocaml.org
      docker --config config/docker context create \
        --docker host=ssh://awesome.ocaml.org \
        --description="awesome.ocaml.org" \
        awesome-ocaml-org
      
    • The hash for the folder inside context/meta is generated with docker context create <machine_name>.
    • Add to known_hosts with ssh-keyscan of the host where you are deploying the service. eg
      ssh-keyscan -H awesome.ocaml.org >> config/ssh/known_hosts