Home

Awesome

This is a replacement for, and rewrite of travis-yaml.

Travis CI build config processing

This library is used for parsing, normalizing, and validating Travis CI build configuration (.travis.yml).

It serves these main functions:

The specification (schema) produced by the library is being used for both normalizing and validating build configs, and for generating reference documentation. I.e. these two parts of the code base are clients to the schema.

Applying the specification to a build configuration produces structured messages on the levels info, warn, and error. Such messages include a message code (e.g. unknown_key, deprecated_key etc.) and arguments (e.g. the given key, value) so they can be translated to human readable messages by clients. These can be used in the UI, and links to the documentation, suggesting fixes.

For example:

yaml = 'rvm: 2.3'
config = Travis::Yml.load(yaml)

config.serialize
# {
#   language: 'ruby',
#   os: ['linux'],
#   rvm: ['2.3']
# }

config.msgs
# [
#   [:info, :language, :default, key: :language, default: 'ruby'],
#   [:info, :language, :default, key: :os, default: 'linux'],
# ]

config.msgs.map { |msg| Travis::Yml.msg(msg) }
# [
#   '[info] on root: missing :language, using the default: "ruby"',
#   '[info] on root: missing :os, using the default: "linux"',
# ]

YAML

The library uses a custom YAML parser based on Psych, using its safe YAML parsing features.

Diverging from the YAML specification this library does not:

For the following reasons:

The parser uses the classes Key, Map, and Seq for representing Ruby strings that are hash keys, hashes, and arrays, in order to be able to carry additional meta information. This is then used while merging config parts, and normalizing and validating the resulting config.

Integers and floats are not converted by the YAML parser, but they are casted later by the build config normalization, so they can be casted only if the respective node wants an integer or float.

Types

Borrowing from YAML's terminology the library uses:

It also uses the following scalar types:

Scalars can be enums, as defined by JSON Schema, i.e. they can have a number of known and allowed values. Examples for keys that hold enums are: language, os, dist etc.

A map can be strict or not strict. Maps are strict by default. A strict map disallows keys that are not known.

A secure resembles our build config format for encrypted key/value pairs. Secures also can be strict or not strict, and they are strict by default. A not strict secure would accept a plain string without producing a warning (e.g. on keys like username or email).

Build config specification

The spec is included to the repository as schema.json.

The spec is defined in Schema::Def, and can be produced by:

Travis::Yml.schema

Classes in Schema::Def use classes in Schema::Type to build a tree of nodes that define allowed keys, types, and various options in a readable and succinct way (using a DSL for describing the schema).

Nodes on this tree that match certain well-known patterns are then transformed according to these patterns using classes in Schema::Type::Form. E.g. a sequence of strings always also accepts a single string, which will then automatically be wrapped into a sequence during the config normalization process. Therefore the JSON Schema needs to accept both forms.

The resulting tree is then serialized by classes in Schema::Json to a large Hash which serves as a specification in the form of a JSON Schema.

A good starting point for exploring the schema definition is the root node.

Examples for various nodes on this specification can be found in the tests, e.g. for the git, heroku, or os nodes.

Most nodes can be sufficiently specified by mapping known keys (e.g. language) to types (str, bool, map, seq etc.) with certain options, such as:

In order to keep the JSON payload reasonably small the library uses JSON Schema's mechanism for defining and referencing shared sub schemas. All nodes that have a registered definition class are exported as such a defined sub schema, and then referenced on the respective nodes that use them.

TBD: mention JSON Schema limitations, and how travis-yml interprets particular types (all, any) in a specific way that is not defined by JSON Schema.

Loading the spec

Before the tree representing the schema can be applied to an actual build configuration it will be turned into another object oriented representation optimized for this purpose, so non-parametrized methods can be memoized for better performance.

The method Travis::Yml.expand returns this object oriented tree, using classes in Doc::Schema.

Applying the spec to a build config

This representation of the schema can be applied to a build configuration by:

# given a YAML string
Travis::Yml.load(yaml)

# given a Ruby hash
Travis::Yml.apply(config)

Both methods also accept an optional options hash. Please see here for a list of known options.

When the schema is applied to a build configuration three things happen:

Examples of type specific change strategies:

Section specific change strategies:

Examples of the validations:

Env vars given as strings will be parsed into hashes using the library sh_vars. Conditions will be parsed using the library travis-conditions.

Summary

There are three sets of classes that are used to build trees:

Only the last one, Doc::Value, is re-used at runtime, i.e. only for the given build configs we build new trees. The Doc::Spec representation is kept in memory, is static, and remains unchanged.

For each build config we then apply all relevant changes (Doc::Change) and all relevant validations (Doc::Validate) to each node.

Expanding a build matrix

A given build configuration can be expanded to job matrix configurations by:

Travis::Yml.matrix(config)

E.g. a build config like:

{
  language: 'ruby',
  ruby: ['2.2', '2.3'],
  env: ['FOO=foo', 'BAR=bar']
}

will be expanded to:

[
  { language: 'ruby', ruby: '2.2', env: { FOO: 'foo' } },
  { language: 'ruby', ruby: '2.2', env: { BAR: 'bar' } },
  { language: 'ruby', ruby: '2.3', env: { FOO: 'foo' } },
  { language: 'ruby', ruby: '2.3', env: { BAR: 'bar' } },
]

Web API

This gem also contains a web API for parsing and expanding a build config, which is used by travis-gatekeeper when processing a build request.

To start the server:

$ bundle exec rackup

The API contains three endpoints:

Home

$ curl -X GET /v1 | jq .

{
  "version": "v1"
}

Parse

The body of the request should be raw YAML. The response contains parsing messages and the validated and normalised config.

$ curl -X POST --data-binary @travis.yml /v1/parse | jq .

{
  "version": "v1",
  "messages": [
    {
      "level": "info",
      "key": "language",
      "code": "default",
      "args": {
        "key": "language",
        "default": "ruby"
      }
    }
  ],
  "full_messages": [
    "[info] on language: missing :language, defaulting to: ruby"
  ],
  "config": {
    "language": "ruby",
    "os": [
      "linux"
    ],
    "dist": "trusty",
    "sudo": false
  }
}

Expand

The body of the request should be the JSON config found in the response from /v1/parse. The response will contain the expanded matrix of jobs.

$ curl -X POST --data-binary @config.json /v1/expand | jq .

{
  "version": "v1",
  "matrix": [
    {
      "os": "linux",
      "language": "ruby",
      "dist": "trusty",
      "sudo": false
    }
  ]
}