Awesome

                                              _  __  __             
                                    ___ _ __ (_)/ _|/ _|  _     _   
                                   / __| '_ \| | |_| |_ _| |_ _| |_ 
                                   \__ \ |_) | |  _|  _|_   _|_   _|
                                   |___/ .__/|_|_| |_|   |_|   |_|  
                                       |_|

NOTE: Active development on spiff is currently paused, including Pull Requests. `spiff++` is a fork of spiff that provides a compatible extension to spiff based on the latest version offering a rich set of new features not yet available in spiff. All fixes provided by the original spiff project will be incorporated into spiff++, also. Because there will be no way back to the spiff source base a new independent spiff++ repository has been created to continue development of spiff++.

spiff is a command line tool and declarative in-domain hybrid YAML templating system. While regular templating systems process a template file by substituting the template expressions by values taken from external data sources, in-domain means that the templating engine knows about the syntax and structure of the processed template. It therefore can take the values for the template expressions directly from the document processed, including those parts denoted by the template expressions itself.

For example:

resource:
  name: bosh deployment
  version: 25
  url: (( "http://resource.location/bosh?version=" version ))
  description: (( "This document describes a " name " located at " url ))

Instead of using only external value sources spiff provides a merging mechanism to merge a template with any number of merging stubs to produce a final document.

It is a command line tool and declarative YAML templating system, specially designed for generating deployment manifests (for example BOSH, Kubernetes or Landscaper manifests).

Besides the CLI there is a golang library enabling the usage of the spiff template processing in any GO program (for example Landscaper).

The templating engine offers enabling access to the filesystem based on a configurable virtual filesystem or the process system to execute commands and incorporate the output into the template processing.

Contents:

Installation
Usage
Feature Flags
Libraries
dynaml Templating Language
Structural Auto-Merge
Bringing it all together
Useful to Know
Error Reporting
Using spiff as Go Library

Installation

Official release executable binaries can be downloaded via Github releases for Darwin, Linux and PowerPC machines (and virtual machines).

Some of spiff's dependencies have changed since the last official release, and spiff will not be updated to keep up with these dependencies. THose dependencies are either fixed or copied into the local code base.

Usage

`spiff merge template.yml [template2.yml ...]`

Merge a bunch of template files into one manifest, printing it out.

See 'dynaml templating language' for details of the template file, or examples/ subdir for more complicated examples.

Example:

spiff merge cf-release/templates/cf-deployment.yml my-cloud-stub.yml

It is possible to read one file from standard input by using the file name -. It may be used only once. This allows using spiff as part of a pipeline to just process a single stream or to process a stream based on several templates/stubs.

The template file (first argument) may be a multiple-document stream containing multiple YAML documents separated by a line containing only ---. Each YAML document will be processed independently with the given stub files. The result is the stream of processed documents in the same order. If a document's root node is marked as temporary, the document is omitted from the output stream. For example, this can be used to generate kubernetes manifests to be used by kubectl.

The merge command offers several options:

The option --partial. If this option is given spiff handles incomplete expression evaluation. All errors are ignored and the unresolvable parts of the yaml document are returned as strings.
With the option --json the output will be in JSON format instead of YAML.
The option --path <path> can be used to output a nested path, instead of the the complete processed document.
If the output is a list, the option --split outputs every list element as separate documen. The yaml format uses as usual --- as separator line. The json format outputs a sequence of json documents, one per line.
With --select <field path> it is possible to select a dedicated field of the processed document for the output
With --evaluate <dynaml expression> it is possible to evaluate a given dynaml expression on the processed document for the output. The expression is evaluated before the selection path is applied, which will then work on the evaluation result.
The option --state <path> enables the state support of spiff. If the given file exists it is put on top of the configured stub list for the given file exists it is put on top of the configured stub list for the merge processing. Additionally to the output of the processed document it is filtered for nodes marked with the &state marker. This filtered document is then stored under the denoted file, saving the old state file with the .bak suffix. This can be used together with a manual merging as offered by the state utility library.
With option --bindings <path> a yaml file can be specified, whose content is used to build additional bindings for the processing. The yaml document must consist of a map. Each key is used as additional binding. The bindings document is not processed, the values are used as defined.
With option --tag <tag>:<path> a yaml file can be specified, whose content is used as value for a predefined global tag (see Tags). Tags can be accessed by reference expressions of the form <tag>::<ref>. In contrast to bindings tagged content does not compete with the nodes in the document, it uses another reference namespace.
With option --define <key>=<value> (shorthand-D) additional binding values can be specified on the command line overriding binding values from the binding file. The option may occur multiple times.

If the key contains dots (.), it will be interpreted as path expression to describe fields in deep map values. A dot (and a \ before a dot) can be escaped by \ to keep it in the field name.
The option --preserve-escapes will preserve the escaping for dynaml expressions and list/map merge directives. This option can be used if further processing steps of a processing result with spiff is intended.
The option --preserve-temporary will preserve the fields marked as temporary in the final document.
The option --features=<featurelist> will enable this given features. New features that are incompatible with the old behaviour must be explicitly enabled. Typically those feature do not break the common behavior but introduce a dedicated interpretation for yaml values that were used as regular values before.

The folder libraries offers some useful utility libraries. They can also be used as an example for the power of this templating engine.

`spiff diff manifest.yml other-manifest.yml`

Show structural differences between two deployment manifests. Here streams with multiple documents are supported, also. To indicate no difference the number of documents in both streams must be identical and each document in the first stream must have no difference compared to the document with the same index in the second stream. Found differences are shown for each document separately.

Unlike basic diffing tools and even bosh diff, this command has semantic knowledge of a deployment manifest, and is not just text-based. For example, if two manifests are the same except they have some jobs listed in different orders, spiff diff will detect this, since job order matters in a manifest. On the other hand, if two manifests differ only in the order of their resource pools, for instance, then it will yield and empty diff since resource pool order doesn't actually matter for a deployment.

Also unlike bosh diff, this command doesn't modify either file.

It's tailed for checking differences between one deployment and the next.

Typical flow:

$ spiff merge template.yml [templates...] > deployment.yml
$ bosh download manifest [deployment] current.yml
$ spiff diff deployment.yml current.yml
$ bosh deployment deployment.yml
$ bosh deploy

`spiff convert --json manifest.yml`

The convert sub command can be used to convert input files to json or just to normalize the order of the fields. Available options are --json, --path, --split or --select according to their meanings for the merge sub command.

`spiff encrypt secret.yaml`

The encrypt sub command can be used to encrypt or decrypt data according to the encrypt dynaml function. The password can be given as second argument or it is taken from the environment variable SPIFF_ENCRYPTION_KEY. The last argument can be used to pass the encryption method (see encrypt function)

The data is taken from the specified file. If - is given, it is read from stdin.

If the option -d is given, the data is decrypted, otherwise the data is read as yaml document and the encrypted result is printed.

Feature Flags

New features that are incompatible with the old behaviour must be explicitly enabled. Typically those features do not break the common behavior but introduce a dedicated interpretation for yaml values that were used as regular values before and can therefore break existing use cases.

The following feature flags are currently supported:

Feature	Since	State	Meaning
`interpolation`	1.7.0-beta-1	alpha	dynaml as part of yaml strings
`control`	1.7.0-beta-4	alpha	yaml based control structures

Active feature flags can be queried using the dynaml function features() as list of strings. If this function is called with a string argument, it returns whether the given feature is currenty enabled.

Features can be enabled by command line using the --features option, by the go library using the WithFeatures function or generally by setting the environment variable SPIFF_FEATURES to a feature list. This setting is alwas used as default. By using the Plain() spiff settings from the go library all environment variables are ignored.

A feature can be specified by name or by name prepended with the prefix no to disable it.

Libraries

The libraries folder contains some useful spiff template libraries. These are basically just stubs that are added to the merge file list to offer the utility functions for the merge processing.

dynaml Templating Language

Spiff uses a declarative, logic-free templating language called 'dynaml' (dynamic yaml).

Every dynaml node is guaranteed to resolve to a YAML node. It is not string interpolation. This keeps developers from having to think about how a value will render in the resulting template.

A dynaml node appears in the .yml file as a string denoting an expression surrounded by two parentheses (( <dynaml> )). They can be used as the value of a map or an entry in a list. The expression might span multiple lines. In any case the yaml string value must not end with a newline (for example using |-)

If a parenthesized value should not be interpreted as an dynaml expression and kept as it is in the output, it can be escaped by an exclamation mark directly after the openeing brackets.

For example, ((! .field )) maps to the string value (( .field )) and ((!! .field )) maps to the string value ((! .field )).

The following is a complete list of dynaml expressions:

`(( foo ))`

Look for the nearest 'foo' key (i.e. lexical scoping) in the current template and bring it in.

e.g.:

fizz:
  buzz:
    foo: 1
    bar: (( foo ))
  bar: (( foo ))
foo: 3
bar: (( foo ))

This example will resolve to:

fizz:
  buzz:
    foo: 1
    bar: 1
  bar: 3
foo: 3
bar: 3

The following will not resolve because the key name is the same as the value to be merged in:

foo: 1

hi:
  foo: (( foo ))

`(( foo.bar.[1].baz ))`

Look for the nearest 'foo' key, and from there follow through to .bar.[1].baz.

A path is a sequence of steps separated by dots. A step is either a word for maps, or digits surrounded by brackets for list indexing. The index might be negative (a minus followed by digits). Negative indices are taken from then end of the list (effective index = index + length(list)).

A path that cannot be resolved lead to an evaluation error. If a reference is expected to sometimes not be provided, it should be used in combination with '||' (see below) to guarantee resolution.

Note: The dynaml grammer has been reworked to enable the usual index syntax, now. Instead of foo.bar.[1] it is possible now to use foo.bar[1].

Note: References are always within the template or stub, and order does not matter. You can refer to another dynamic node and presume it's resolved, and the reference node will just eventually resolve once the dependent node resolves.

e.g.:

properties:
  foo: (( something.from.the.stub ))
  something: (( merge ))

This will resolve as long as 'something' is resolveable, and as long as it brings in something like this:

from:
  the:
    stub: foo

If the path starts with a dot (.) the path is always evaluated from the root of the document. If the document root is a list, the first map level is used to resolve the path expression if it starts with .__map. This can be used to avoid the need for using the own list index (like .[1].path), which might change if list entries are added.

List entries consisting of a map with name field can directly be addressed by their name value as path component.

Note: This also works for the absolute paths for list documents.

e.g.:

The age of alice in

list:
 - name: alice
   age: 25

can be referenced by using the path list.alice.age, instead of list[0].age.

By default a field with name name is used as key field. If another field should be used as key field, it can be marked in one list entry as key by prefixing the field name with the keyword key:. This keyword is removed from by the processing and will not be part of the final processing result.

e.g.:

list:
 - key:person: alice
   age: 25

alice: (( list.alice ))

will be resolved to

list:
 - person: alice
   age: 25

alice:
  person: alice
  age: 25

This new key field will also be observed during the merging of lists.

If the selected key field starts with a !, the key feature is disabled. The exclamation mark is removed from the effective field name, also.

If the values for the key field are not unqiue, it is disables, also.

`(( foo.[bar].baz ))`

Look for the nearest 'foo' key, and from there follow through to the field(s) described by the expression bar and then to .baz.

The index may be an integer constant (without spaces) as described in the last section. But it might also be an arbitrary dynaml expression (even an integer, but with spaces). If the expression evaluates to a string, it lookups the dedicated field. If the expression evaluates to an integer, the array element with this index is addressed. The dot (.) in front of the index operator is optional.

e.g.:

properties:
  name: alice
  foo: (( values.[name].bar ))
  values:
    alice:
      bar: 42

This will resolve foo to the value 42. The dynamic index may also be at the end of the expression (without .bar).

Basically this is the simplier way to express something like eval("values." name ".bar")

If the expression evaluates to a list, the list elements (strings or integers) are used as path elements to access deeper fields.

e.g.:

properties:
  name:
   - foo
   - bar
  foo: (( values.[name] ))
  values:
    foo:
      bar: 42

resolves foo again to the value 42.

Note: The index operator is usable on the root element (.[index]), also.

It is possible, to specify multiple comma separated indicies to successive lists (foo[0][1] is equivalent to `foo[0,1]). In such case the indices may not be again lists.

`(( list.[1..3] ))`

The slice expression can be used to extract a dedicated sub list from a list expression. The range start .. end extracts a list of the length end-start+1 with the elements from index start to end. If the start index is negative the slice is taken from the end of the list from length+start to length+end. If the end index is lower than the start index, the result is an empty array.

e.g.:

list:
  - a
  - b
  - c
foo: (( list.[1..length(list) - 1] ))

The start or end index might be omitted. It is then selected according to the actual size of the list. Therefore list.[1..length(list)] is equivalent to list.[1..].

evaluates foo to the list [b,c].

`(( 1.2e4 ))`

Number literatls are supported for integers and floating point values.

`(( "foo" ))`

String literal. All json string encodings are supported (for exmple \n, \" or \uxxxx).

`(( [ 1, 2, 3 ] ))`

List literal. The list elements might again be expressions. There is a special list literal [1 .. -1], that can be used to resolve an increasing or descreasing number range to a list.

e.g.:

list: (( [ 1 .. -1 ] ))

yields

list:
  - 1
  - 0
  - -1

`(( { "alice" = 25 } ))`

The map literal can be used to describe maps as part of a dynaml expression. Both, the key and the value, might again be expressions, whereby the key expression must evaluate to a string. This way it is possible to create maps with non-static keys. The assignment operator = has been chosen instead of the regular colon : character used in yaml, because this would result in conflicts with the yaml syntax.

A map literal might consist of any number of field assignments separated by a comma ,.

e.g.:

name: peter
age: 23
map: (( { "alice" = {}, name = age } ))

yields

name: peter
age: 23
map:
  alice: {}
  peter: 23

Another way to compose lists based on expressions are the functions makemap and list_to_map.

`(( ( "alice" = 25 ) alice ))`

Any expression may be preluded by any number of explicit scope literals. A scope literal describes a map whose values are available for relative reference resolution of the expression (static scope). It creates an additional local binding for given names.

A scope literal might consist of any number of field assignments separated by a comma ,. The key as well as the value are given by expressions, whereas the key expression must evaluate to a string. All expressions are evaluated in the next outer scope, this means later settings in a scope cannot use earlier settings in the same scope literal.

e.g.:

scoped: (( ( "alice" = 25, "bob" = 26 ) alice + bob ))

yields

scoped: 51

A field name might also be denoted by a symbol ($name).

`(( foo bar ))`

Concatenation expression used to concatenate a sequence of dynaml expressions.

`(( "foo" bar ))`

Concatenation (where bar is another dynaml expr). Any sequences of simple values (string, integer and boolean) can be concatenated, given by any dynaml expression.

e.g.:

domain: example.com
uri: (( "https://" domain ))

In this example uri will resolve to the value "https://example.com".

`(( [1,2] bar ))`

Concatenation of lists as expression (where bar is another dynaml expr). Any sequences of lists can be concatenated, given by any dynaml expression.

e.g.:

other_ips: [ 10.0.0.2, 10.0.0.3 ]
static_ips: (( ["10.0.1.2","10.0.1.3"] other_ips ))

In this example static_ips will resolve to the value [ 10.0.1.2, 10.0.1.3, 10.0.0.2, 10.0.0.3 ] .

If the second expression evaluates to a value other than a list (integer, boolean, string or map), the value is appended to the first list.

e.g.:

foo: 3
bar: (( [1] 2 foo "alice" ))

yields the list [ 1, 2, 3, "alice" ] for bar.

`(( map1 map2 ))`

Concatenation of maps as expression. Any sequences of maps can be concatenated, given by any dynaml expression. Thereby entries will be merged. Entries with the same key are overwritten from left to right.

e.g.:

foo:
  alice: 24
  bob: 25

bar:
  bob: 26
  paul: 27

concat: (( foo bar ))

yields

foo:
  alice: 24
  bob: 25

bar:
  bob: 26
  paul: 27

concat:
  alice: 24
  bob: 26
  paul: 27

`(( auto ))`

Context-sensitive automatic value calculation.

In a resource pool's 'size' attribute, this means calculate based on the total instances of all jobs that declare themselves to be in the current resource pool.

e.g.:

resource_pools:
  - name: mypool
    size: (( auto ))

jobs:
  - name: myjob
    resource_pool: mypool
    instances: 2
  - name: myotherjob
    resource_pool: mypool
    instances: 3
  - name: yetanotherjob
    resource_pool: otherpool
    instances: 3

In this case the resource pool size will resolve to '5'.

`(( merge ))`

Bring the current path in from the stub files that are being merged in.

e.g.:

foo:
  bar:
    baz: (( merge ))

Will try to bring in foo.bar.baz from the first stub, or the second, etc., returning the value from the last stub that provides it.

If the corresponding value is not defined, it will return nil. This then has the same semantics as reference expressions; a nil merge is an unresolved template. See ||.

`<<: (( merge ))`

Merging of maps or lists with the content of the same element found in some stub.

** Attention ** This form of merge has a compatibility propblem. In versions before 1.0.8, this expression was never parsed, only the existence of the key <<: was relevant. Therefore there are often usages of <<: (( merge )) where <<: (( merge || nil )) is meant. The first variant would require content in at least one stub (as always for the merge operator). Now this expression is evaluated correctly, but this would break existing manifest template sets, which use the first variant, but mean the second. Therfore this case is explicitly handled to describe an optional merge. If really a required merge is meant an additional explicit qualifier has to

Note: Instead of using a <<: insert field to place merge expressions it is possible now to use <<<:, also, which allows to use regular yaml parsers for spiff-like yaml documents. <<: is kept for backward compatibility. be used ((( merge required ))).

If the merge key should not be interpreted as regular key instead of a merge directive, it can be escaped by an excalamtion mark (!).

For example, a map key <<<! will result in a string key <<< and <<<!! will result in a string key <<<!

Merging maps

values.yml

foo:
  a: 1
  b: 2

template.yml

foo:
  <<: (( merge ))
  b: 3
  c: 4

spiff merge template.yml values.yml yields:

foo:
  a: 1
  b: 2
  c: 4

Merging lists

values.yml

foo:
  - 1
  - 2

template.yml

foo:
  - 3
  - <<: (( merge ))
  - 4

spiff merge template.yml values.yml yields:

foo:
  - 3
  - 1
  - 2
  - 4

`- <<: (( merge on key ))`

spiff is able to merge lists of maps with a key field. Those lists are handled like maps with the value of the key field as key. By default the key name is used. But with the selector on an arbitrary key name can be specified for a list-merge expression.

e.g.:

list:
  - <<: (( merge on key ))
  - key: alice
    age: 25
  - key: bob
    age: 24

merged with

list:
  - key: alice
    age: 20
  - key: peter
    age: 13

yields

list:
  - key: peter
    age: 13
  - key: alice
    age: 20
  - key: bob
    age: 24

If no insertion of new entries is desired (as requested by the insertion merge expression), but only overriding of existent entries, one existing key field can be prefixed with the tag key: to indicate a non-standard key name, for example - key:key: alice.

`<<: (( merge replace ))`

Replaces the complete content of an element by the content found in some stub instead of doing a deep merge for the existing content.

Merging maps

values.yml

foo:
  a: 1
  b: 2

template.yml

foo:
  <<: (( merge replace ))
  b: 3
  c: 4

spiff merge template.yml values.yml yields:

foo:
  a: 1
  b: 2

Merging lists

values.yml

foo:
  - 1
  - 2

template.yml

foo:
  - <<: (( merge replace ))
  - 3
  - 4

spiff merge template.yml values.yml yields:

foo:
  - 1
  - 2

`<<: (( foo ))`

Merging of maps and lists found in the same template or stub.

Merging maps

foo:
  a: 1
  b: 2

bar:
  <<: (( foo )) # any dynaml expression
  b: 3

yields:

foo:
  a: 1
  b: 2

bar:
  a: 1
  b: 3

This expression just adds new entries to the actual list. It does not merge existing entries with the content described by the merge expression.

Merging lists

bar:
  - 1
  - 2

foo:
  - 3
  - <<: (( bar ))
  - 4

yields:

bar:
  - 1
  - 2

foo:
  - 3
  - 1
  - 2
  - 4

A common use-case for this is merging lists of static ips or ranges into a list of ips. Another possibility is to use a single concatenation expression.

`<<: (( merge foo ))`

Merging of maps or lists with the content of an arbitrary element found in some stub (Redirecting merge). There will be no further (deep) merge with the element of the same name found in some stub. (Deep merge of lists requires maps with field name)

Redirecting merges can be used as direct field value, also. They can be combined with replacing merges like (( merge replace foo )).

Merging maps

values.yml

foo:
  a: 10
  b: 20

bar:
  a: 1
  b: 2

template.yml

foo:
  <<: (( merge bar))
  b: 3
  c: 4

spiff merge template.yml values.yml yields:

foo:
  a: 1
  b: 2
  c: 4

Another way doing a merge with another element in some stub could also be done the traditional way:

values.yml

foo:
  a: 10
  b: 20

bar:
  a: 1
  b: 2

template.yml

bar:
  <<: (( merge ))
  b: 3
  c: 4

foo: (( bar ))

But in this scenario the merge still performs the deep merge with the original element name. Therefore spiff merge template.yml values.yml yields:

bar:
  a: 1
  b: 2
  c: 4
foo:
  a: 10
  b: 20
  c: 4

Merging lists

values.yml

foo:
  - 10
  - 20

bar:
  - 1
  - 2

template.yml

foo:
  - 3
  - <<: (( merge bar ))
  - 4

spiff merge template.yml values.yml yields:

foo:
  - 3
  - 1
  - 2
  - 4

`<<: (( merge none ))`

If the reference of an redirecting merge is set to the constant none, no merge is done at all. This expressions always yields the nil value.

e.g.: for

template.yml

map:
  <<: (( merge none ))
  value: notmerged

values.yml

map:
  value: merged

spiff merge template.yml values.yml yields:

map:
  value: notmerged

This can be used for explicit field merging using the stub function to access dedicated parts of upstream stubs.

e.g.:

template.yml

map:
  <<: (( merge none ))
  value: ((  "alice"  "+" stub(map.value) ))

values.yml

map:
  value: bob

spiff merge template.yml values.yml yields:

test:
  value: alice+bob

This also works for dedicated fields:

template.yml

map:
  value: ((  merge none // "alice"  "+" stub() ))

values.yml

map:
  value: bob

spiff merge template.yml values.yml yields:

test:
  value: alice+bob

`(( a || b ))`

Uses a, or b if a cannot be resolved.

e.g.:

foo:
  bar:
    - name: some
    - name: complicated
    - name: structure

mything:
  complicated_structure: (( merge || foo.bar ))

This will try to merge in mything.complicated_structure, or, if it cannot be merged in, use the default specified in foo.bar.

The operator // additionally checks, whether a can be solved to a valid value (not equal ~).

`(( 1 + 2 * foo ))`

Dynaml expressions can be used to execute arithmetic integer and floating-point calculations. Supported operations are +, -, *, and /. The modulo operator (%) only supports integer operands.

e.g.:

values.yml

foo: 3
bar: (( 1 + 2 * foo ))

spiff merge values.yml yields 7 for bar. This can be combined with concatentions (calculation has higher priority than concatenation in dynaml expressions):

foo: 3
bar: (( foo " times 2 yields " 2 * foo ))

The result is the string 3 times 2 yields 6.

`(( "10.10.10.10" - 11 ))`

Besides arithmetic on integers it is also possible to use addition and subtraction on ip addresses and cidrs.

e.g.:

ip: 10.10.10.10
range: (( ip "-" ip + 247 + 256 * 256 ))

yields

ip: 10.10.10.10
range: 10.10.10.10-10.11.11.1

Subtraction also works on two IP addresses or cidrs to calculate the number of IP addresses between two IP addresses.

e.g.:

diff: (( 10.0.1.0 - 10.0.0.1 + 1 ))

yields the value 256. IP address constants can be directly used in dynaml expressions. They are implicitly converted to strings and back to IP addresses if required by an operation.

Multiplication and division can be used to handle IP range shifts on CIDRs. With division a network can be partioned. The network size is increased to allow at least a dedicated number of subnets below the original CIDR. Multiplication then can be used to get the n-th next subnet of the same size.

e.g.:

subnet: (( "10.1.2.1/24" / 12 ))  # first subnet CIDR for 16 subnets
next: (( "10.1.2.1/24" / 12 * 2)) # 2nd next (3rd) subnet CIDRS

yields

subnet: 10.1.2.0/28
next: 10.1.2.32/28

Additionally there are functions working on IPv4 CIDRs:

cidr: 192.168.0.1/24
range: (( min_ip(cidr) "-" max_ip(cidr) ))
next: (( max_ip(cidr) + 1 ))
num: (( min_ip(cidr) "+" num_ip(cidr) "=" min_ip(cidr) + num_ip(cidr) ))
contains: (( contains_ip(cidr, "192.168.0.2") ))

yields

cidr: 192.168.0.1/24
range: 192.168.0.0-192.168.0.255
next: 192.168.1.0
num: 192.168.0.0+256=192.168.1.0
contains: true

`(( a > 1 ? foo :bar ))`

Dynaml supports the comparison operators <, <=, ==, !=, >= and >. The comparison operators work on integer values. The checks for equality also work on lists and maps. The result is always a boolean value. To negate a condition the unary not opertor (!) can be used.

Additionally there is the ternary conditional operator ?:, that can be used to evaluate expressions depending on a condition. The first operand is used as condition. The expression is evaluated to the second operand, if the condition is true, and to the third one, otherwise.

e.g.:

foo: alice
bar: bob
age: 24
name: (( age > 24 ? foo :bar ))

yields the value bob for the property name.

An expression is considered to be false if it evaluates to

the boolean value false
the integer value 0
an empty string, map or list

Otherwise it is considered to be true

Remark

The use of the symbol : may collide with the yaml syntax, if the complete expression is not a quoted string value.

The operators -or and -and can be used to combine comparison operators to compose more complex conditions.

Remark:

The more traditional operator symbol || (and &&) cannot be used here, because the operator || already exists in dynaml with a different semantic, that does not hold for logical operations. The expression false || true evaluates to false, because it yields the first operand, if it is defined, regardless of its value. To be as compatible as possible this cannot be changed and the bare symbols or and and cannot be be used, because this would invalidate the concatenation of references with such names.

`(( 5 -or 6 ))`

If both sides of an -or or -and operator evaluate to integer values, a bit-wise operation is executed and the result is again an integer. Therefore the expression 5 -or 6 evaluates to 7.

Functions

Dynaml supports a set of predefined functions. A function is generally called like

result: (( functionname(arg, arg, ...) ))

Additional functions may be defined as part of the yaml document using lambda expressions. The function name then is either a grouped expression or the path to the node hosting the lambda expression.

`(( format( "%s %d", alice, 25) ))`

Format a string based on arguments given by dynaml expressions. There is a second flavor of this function: error formats an error message and sets the evaluation to failed.

`(( join( ", ", list) ))`

Join entries of lists or direct values to a single string value using a given separator string. The arguments to join can be dynaml expressions evaluating to lists, whose values again are strings or integers, or string or integer values.

e.g.:

alice: alice
list:
  - foo
  - bar

join: (( join(", ", "bob", list, alice, 10) ))

yields the string value bob, foo, bar, alice, 10 for join.

`(( split( ",", string) ))`

Split a string for a dedicated separator. The result is a list. Instead of a separator string an integer value might be given, which splits the give string into list of length limited strings. The length is counted in runes, not bytes.

e.g.:

list: (( split("," "alice, bob") ))
limited: (( split(4, "1234567890") ))

yields:

list:
  - alice
  - ' bob'
limited:
  - "1234"
  - "5678"
  - "90"

An optional 3rd argument might be specified. It limits the number of returned list entries. The value -1 leads to an unlimited list length.

If a regular expression should be used as separator string, the function split_match can be used.

`(( trim(string) ))`

Trim a string or all elements of a list of strings. There is an optional second string argument. It can be used to specify a set of characters that will be cut. The default cut set consists of a space and a tab character.

e.g.:

list: (( trim(split("," "alice, bob")) ))

yields:

list:
  - alice
  - bob

`(( element(list, index) ))`

Return a dedicated list element given by its index.

e.g.:

list: (( trim(split("," "alice, bob")) ))
elem: (( element(list,1) ))

yields:

list:
  - alice
  - bob
elem: bob

`(( element(map, key) ))`

Return a dedicated map field given by its key.

map:
  alice: 24
  bob: 25
elem: (( element(map,"bob") ))

yields:

map:
  alice: 24
  bob: 25
elem: 25

This function is also able to handle keys containing dots (.).

`(( compact(list) ))`

Filter a list omitting empty entries.

e.g.:

list: (( compact(trim(split("," "alice, , bob"))) ))

yields:

list:
  - alice
  - bob

`(( uniq(list) ))`

Uniq provides a list without dupliates.

e.g.:

list:
- a
- b
- a
- c
- a
- b
- 0
- "0"
uniq: (( uniq(list) ))

yields for field uniq:

uniq:
- a
- b
- c
- 0

`(( contains(list, "foobar") ))`

Checks whether a list contains a dedicated value. Values might also be lists or maps.

e.g.:

list:
  - foo
  - bar
  - foobar
contains: (( contains(list, "foobar") ))

yields:

list:
  - foo
  - bar
  - foobar
contains: true

The function contains also works on strings to look for sub strings or maps to look for a key. In those cases the element must be a string.

e.g.:

contains: (( contains("foobar", "bar") ))

yields true.

`(( basename(path) ))`

The function basename returns the name of the last element of a path. The argument may either be a regular path name or a URL.

e.g.:

pathbase:  (( basename("alice/bob") ))
urlbase:  (( basename("http://foobar/alice/bob?any=parameter") ))

yields:

pathbase:  bob
urlbase:  bob

`(( dirname(path) ))`

The function dirname returns the parent directory of a path. The argument may either be a regular path name or a URL.

e.g.:

pathbase:  (( dirname("alice/bob") ))
urlbase:  (( dirname("http://foobar/alice/bob?any=parameter") ))

yields:

pathbase:  alice
urlbase:  /alice

`(( parseurl("http://github.com") ))`

This function parses a URL and yield a map with all elements of an URL. The fields port, userinfoand password are optional.

e.g.:

url:  (( parseurl("https://user:pass@github.com:443/mandelsoft/spiff?branch=master&tag=v1#anchor") ))

yields:

url:
  scheme: https
  host: github.com
  port: 443
  path: /mandelsoft/spiff
  fragment: anchor
  query: branch=master&tag=v1
  values:
    branch: [ master ]
    tag: [ v1 ]
  userinfo:
    username: user
    password: pass

`(( index(list, "foobar") ))`

Checks whether a list contains a dedicated value and returns the index of the first match. Values might also be lists or maps. If no entry could be found -1 is returned.

e.g.:

list:
  - foo
  - bar
  - foobar
index: (( index(list, "foobar") ))

yields:

list:
  - foo
  - bar
  - foobar
index: 2

The function index also works on strings to look for sub strings.

e.g.:

index: (( index("foobar", "bar") ))

yields 3.

`(( lastindex(list, "foobar") ))`

The function lastindex works like index but the index of the last occurence is returned.

`(( sort(list) ))

The function sort can be used to sort integer or string lists. The sort operation is stable.

e.g.:

list:
  - alice
  - foobar
  - bob

sorted: (( sort(list) ))

yields for sorted

- alice
- bob
- foobar

If other types should be sorted, especially complex types like lists or maps, or a different comparison rule is required, a compare function can be specified as an optional second argument. The compare function must be a lambda expression taking two arguments. The result type must be integeror bool indicating whether a is less then b. If an integer is returned it should be

negative, if a<b
zero, if a==b and
positive if a>b

e.g.:

list:
  - alice
  - foobar
  - bob

sorted: (( sort(list, |a,b|->length(a) < length(b)) ))

yields for sorted

- bob
- alice
- foobar

`(( replace(string, "foo", "bar") ))`

Replace all occurences of a sub string in a string by a replacement string. With an optional fourth integer argument the number of substitutions can be limited (-1 mean unlimited).

e.g.:

string: (( replace("foobar", "o", "u") ))

yields fuubar.

If a regular expression should be used as search string the function replace_match can be used. Here the search string is evaluated as regular expression. It may conatain sub expressions. These matches can be used in the replacement string

e.g.:

string: (( replace_match("foobar", "(o*)b", "b${1}") ))

yields fbooar.

The replacement argument might also be a lambda function. In this case, for every match the function is called to determine the replacement value. The single input argument is a list of actual sub expression matches.

e.g.:

string: (( replace_match("foobar-barfoo", "(o*)b", |m|->upper(m.[1]) "b" ) ))

yields fOObar-barfoo.

`(( substr(string, 1, 2) ))`

Extract a stub string from a string, starting from a given start index up to an optional end index (exclusive). If no end index is given the sub struvt up to the end of the string is extracted. Both indices might be negative. In this case they are taken from the end of the string.

e.g.:

string: "foobar"
end1: (( substr(string,-2) ))
end2: (( substr(string,3) ))
range: (( substr(string,1,-1) ))

evaluates to

string: foobar
end1: ar
end2: bar
range: ooba

`(( match("(f.)(b.)", "xxxfoobar") ))`

Returns the match of a regular expression for a given string value. The match is a list of the matched values for the sub expressions contained in the regular expression. Index 0 refers to the match of the complete regular expression. If the string value does not match an empty list is returned.

e.g.:

matches: (( match("(f.*)*(b.*)", "xxxfoobar") ))

yields:

matches:
- foobar
- foo
- bar

A third argument of type integer may be given to request a multi match of a maximum of n repetitions. If the value is negative all repetions are reported. The result is a list of all matches, each in the format described above.

`(( keys(map) ))`

Determine the sorted list of keys used in a map.

e.g.:

map:
  alice: 25
  bob: 25
keys: (( keys(map) ))

yields:

map:
  alice: 25
  bob: 25
keys:
  - alice
  - bob

`(( length(list) ))`

Determine the length of a list, a map or a string value.

e.g.:

list:
  - alice
  - bob
length: (( length(list) ))

yields:

list:
  - alice
  - bob
length: 2

`(( base64(string) ))`

The function base64 generates a base64 encoding of a given string. base64_decode decodes a base64 encoded string.

e.g.:

base64: (( base64("test") ))
test: (( base64_decode(base64)))

evaluates to

base54: dGVzdA==
test: test

An optional second argument can be used to specify the maximum line length. In this case the result will be multi-line string.

`(( hash(string) ))`

The function hash generates several kinds of hashes for the given string. By default as sha256 hash is generated. An optional second argument specifies the hash type. Possible types are md4, md5, sha1, sha224, sha256, sha384, sha2512, sha512/224or sha512/256.

md5hashes can still be generated by the deprecated finctio md5(string).

e.g.:

data: alice

hash:
  deprecated: (( md5(data) ))
  md4: (( hash(data,"md4") ))
  md5: (( hash(data,"md5") ))
  sha1: (( hash(data,"sha1") ))
  sha224: (( hash(data,"sha224") ))
  sha256: (( hash(data,"sha256") ))
  sha384: (( hash(data,"sha384") ))
  sha512: (( hash(data,"sha512") ))
  sha512_224: (( hash(data,"sha512/224") ))
  sha512_256: (( hash(data,"sha512/256") ))

evaluates to

data: alice
hash:
  deprecated: 6384e2b2184bcbf58eccf10ca7a6563c
  md4: 616c69636531d6cfe0d16ae931b73c59d7e0c089c0
  md5: 6384e2b2184bcbf58eccf10ca7a6563c
  sha1: 522b276a356bdf39013dfabea2cd43e141ecc9e8
  sha224: 38b7e5d5651aaf85694a7a7c6d5db1275af86a6df93a36b8a4a2e771
  sha256: 2bd806c97f0e00af1a1fc3328fa763a9269723c8db8fac4f93af71db186d6e90
  sha384: 96a5353e625adc003a01bdcd9b21b21189bdd9806851829f45b81d3dfc6721ee21f6e0e98c4dd63bc559f66c7a74233a
  sha512: 408b27d3097eea5a46bf2ab6433a7234a33d5e49957b13ec7acc2ca08e1a13c75272c90c8d3385d47ede5420a7a9623aad817d9f8a70bd100a0acea7400daa59
  sha512_224: c3b8cfaa37ae15922adf3d21606e3a9836ba2a9d7838b040b7c96fd7
  sha512_256: ad0a339b08dc090fe3b16eae376f7e162836e8728da9c45466842e19508d7627

`(( bcrypt("password", 10) ))`

The function bcrypt generates a bcrypt password hash for the given string using the specified cost factor (defaulted to 10, if missing).

e.g.:

hash: (( bcrypt("password", 10) ))

evaluates to

hash: $2a$10$b9RKb8NLuHB.tM9haPD3N.qrCsWrZy8iaCD4/.cCFFCRmWO4h.koe

`(( bcrypt_check("password", hash) ))`

The function bcrypt_check validates a password against a given bcrypt hash.

e.g.:

hash: $2a$10$b9RKb8NLuHB.tM9haPD3N.qrCsWrZy8iaCD4/.cCFFCRmWO4h.koe
valid: (( bcrypt_check("password", hash) ))

evaluates to

hash: $2a$10$b9RKb8NLuHB.tM9haPD3N.qrCsWrZy8iaCD4/.cCFFCRmWO4h.koe
valid: true

`(( md5crypt("password") ))`

The function md5crypt generates an Apache MD5 encrypted password hash for the given string.

e.g.:

hash: (( md5crypt("password") ))

evaluates to

hash: $apr1$3Qc1aanY$16Sb5h7U1QrcqwZbDJIYZ0

`(( md5crypt_check("password", hash) ))`

The function md5crypt_check validates a password against a given Apache MD5 encrypted hash.

e.g.:

hash: $2a$10$b9RKb8NLuHB.tM9haPD3N.qrCsWrZy8iaCD4/.cCFFCRmWO4h.koe
valid: (( bcrypt_check("password", hash) ))

evaluates to

hash: $apr1$B77VuUUZ$NkNFhkvXHW8wERSRoi74O1
valid: true

`(( decrypt("secret") ))`

This function can be used to store encrypted secrets in a spiff yaml file. The processed result will then contain the decrypted value. All node types can be encrypted and decrypted, including complete maps and lists.

The password for the decryption can either be given as second argument, or (the preferred way) it can be specified by the environment variable SPIFF_ENCRYPTION_KEY.

An optional last argument may select the encryption method. The only method supported so far is 3DES. Other methods may be added for dedicated spiff versions by using the encryption method registration offered by the spiff library.

A value can be encrypted by using the encrypt("secret") function.

e.g.:

password: this a very secret secret and may never be exposed to unauthorized people
encrypted: (( encrypt("spiff is a cool tool", password) ))
decrypted: (( decrypt(encrypted, password) ))

evaluated to something like

decrypted: spiff is a cool tool
encrypted: d889f9e4cc7ae13effcbc8bb8cd0c38d1fb2197738444f753c48796d7946083e6639e5a1bf8f77648f2a1ddf37023c65ff57d52d0519d1d92cbcf87d3e263cba
password: this a very secret secret and may never be exposed to unauthorized people

`(( rand("[:alnum:]", 10) ))`

The function rand generates random values. The first argument decides what kind of values are requested. With no argument it generates a positive random number in the int64 range.

argument type	result
int	integer value in the range [0,n) for positive n and (n,0] for negative n
bool	boolean value
string	one rune string, where the rune is in the given character range, any combination of character classes or character ranges usable for regexp can be used. If an additional length argument is specified the resulting string will have the given length.

e.g.:

int:   (( rand() ))
int10: (( rand(10) ))
neg10:   (( rand(-10) ))
bool: (( rand(true) ))
string: (( rand("[:alpha:][:digit:]-", 10) ))
upper: (( rand("A-Z", 10) ))
punct: (( rand("[:punct:]", 10) ))
alnum: (( rand("[:alnum:]", 10) ))

evaluates to

int: 8037669378456096839
int10: 7
neg10: -5
bool: true
string: ghhjAYPMlD
upper: LBZQFRSURL
alnum: 0p6KS7EhAj
punct: '&{;,^])"(#'

`(( type(foobar) ))`

The function type yields a string denoting the type of the given expression.

e.g.:

template:
  <<: (( &template ))
  
types:
  - int: (( type(1) ))
  - float: (( type(1.0) ))
  - bool: (( type(true) ))
  - string: (( type("foobar") ))
  - list:   (( type([]) ))
  - map:    (( type({}) ))
  - lambda: (( type(|x|->x) ))
  - template: (( type(.template) ))
  - nil: (( type(~) ))
  - undef: (( type(~~) ))

evaluates types to

types:
- int: int
- float: float
- bool: bool
- string: string
- list: list
- map: map
- lambda: lambda
- template: template

`(( defined(foobar) ))`

The function defined checks whether an expression can successfully be evaluated. It yields the boolean value true, if the expression can be evaluated, and false otherwise.

e.g.:

zero: 0
div_ok: (( defined(1 / zero ) ))
zero_def: (( defined( zero ) ))
null_def: (( defined( null ) ))

evaluates to

zero: 0
div_ok: false
zero_def: true
null_def: false

This function can be used in combination of the conditional operator to evaluate expressions depending on the resolvability of another expression.

`(( valid(foobar) ))`

The function valid checks whether an expression can successfully be evaluated and evaluates to a defined value not equals to nil. It yields the boolean value true, if the expression can be evaluated, and false otherwise.

e.g.:

zero: 0
empty:
map: {}
list: []
div_ok: (( valid(1 / zero ) ))
zero_def: (( valid( zero ) ))
null_def: (( valid( ~ ) ))
empty_def: (( valid( empty ) ))
map_def: (( valid( map ) ))
list_def: (( valid( list ) ))

evaluates to

zero: 0
empty: null
map: {}
list: []
div_ok:   false
zero_def: true
null_def: false
empty_def: false
map_def:  true
list_def: true

`(( require(foobar) ))`

The function require yields an error if the given argument is undefined or nil, otherwise it yields the given value.

e.g.:

foo: ~
bob: (( foo || "default" ))
alice: (( require(foo) || "default" ))

evaluates to

foo: ~
bob: ~
alice: default

`(( stub(foo.bar) ))`

The function stub yields the value of a dedicated field found in the first upstream stub defining it.

e.g.:

template.yml

value: (( stub(foo.bar) ))

merged with stub

stub.yml

foo:
  bar: foobar

evaluates to

value: foobar

The argument passed to this function must either be a reference literal or an expression evaluating to either a string denoting a reference or a string list denoting the list of path elements for the reference. If no argument or an undefined (~~) is given, the actual field path is used.

Please note, that a given sole reference will not be evaluated as expression, if its value should be used, it must be transformed to an expression, for example by denoting (ref) or [] ref for a list expression.

Alternatively the merge operation could be used, for example merge foo.bar. The difference is that stub does not merge, therefore the field will still be merged (with the original path in the document).

`(( tagdef("tag", value) ))`

The function tagdef can be used to define dynamic tags (see Tags). In contrast to the tag marker this function allows to specify the tag name and its intended value by an expression. Therefore, it can be used in composing elements like map or sum to create dynamic tag with calculated values.

An optional third argument can be used to specify the intended scope (local or global). By default a local tag is created. Local tags are visible only at the actual processing level (template or sub), while global tags, once defined, can be used in all further processing levels (stub or template).

Alternatively the tag name can be prefixed with a start (*) to declare a global tag.

The specified tag value will be used as result for the function.

e.g.:

template.yml

value: (( tagdef("tag:alice", 25) ))
alice: (( tag:alice::. ))

evaluates to

value: 25
alice: 25

`(( eval(foo "." bar ) ))`

Evaluate the evaluation result of a string expression again as dynaml expression. This can, for example, be used to realize indirections.

e.g.: the expression in

alice:
  bob: married

foo: alice
bar: bob

status: (( eval( foo "." bar ) ))

calculates the path to a field, which is then evaluated again to yield the value of this composed field:

alice:
  bob: married

foo: alice
bar: bob

status: married

`(( env("HOME" ) ))`

Read the value of an environment variable whose name is given as dynaml expression. If the environment variable is not set the evaluation fails.

In a second flavor the function env accepts multiple arguments and/or list arguments, which are joined to a single list. Every entry in this list is used as name of an environment variable and the result of the function is a map of the given given variables as yaml element. Hereby non-existent environment variables are omitted.

`(( parse(yamlorjson) ))`

Parse a yaml or json string and return the content as yaml value. It can therefore be used for further dynaml evaluation.

e.g.:


json: |
   { "alice": 25 }
result: (( parse( json ).alice ))

yields the value 25 for the field result.

The function parse supports an optional second argument, the parse mode. Here the same modes are possible as for the read function. The default parsing mode is import, the content is just parsed and there is no further evaluation during this step.

`(( asjson(expr) ))`

This function transforms a yaml value given by its argument to a json string. The corresponding function asyaml yields the yaml value as yaml document string.

e.g.:

data:
  alice: 25

mapped:
  json: (( asjson(.data) ))
  yaml: (( asyaml(.data) ))

resolves to

data:
  alice: 25

mapped:
  json: '{"alice":25}'
  yaml: |+
    alice: 25

`(( catch(expr) ))`

This function executes an expression and yields some evaluation info map. It always succeeds, even if the expression fails. The map includes the following fields:

name	type	meaning
`valid`	bool	expression is valid
`error`	string	the error message text of the evaluation
`value`	any	the value of the expression, if evaluation was successful

e.g.:

data:
  fail: (( catch(1 / 0) ))
  valid: (( catch( 5 * 5) ))

resolves to

data:
  fail:
    error: division by zero
    valid: false
  valid:
    error: ""
    valid: true
    value: 25

`(( static_ips(0, 1, 3) ))`

Generate a list of static IPs for a job.

e.g.:

jobs:
  - name: myjob
    instances: 2
    networks:
    - name: mynetwork
      static_ips: (( static_ips(0, 3, 4) ))

This will create 3 IPs from mynetworks subnet, and return two entries, as there are only two instances. The two entries will be the 0th and 3rd offsets from the static IP ranges defined by the network.

For example, given the file bye.yml:

networks: (( merge ))

jobs:
  - name: myjob
    instances: 3
    networks:
    - name: cf1
      static_ips: (( static_ips(0,3,60) ))

and file hi.yml:

networks:
- name: cf1
  subnets:
  - cloud_properties:
      security_groups:
      - cf-0-vpc-c461c7a1
      subnet: subnet-e845bab1
    dns:
    - 10.60.3.2
    gateway: 10.60.3.1
    name: default_unused
    range: 10.60.3.0/24
    reserved:
    - 10.60.3.2 - 10.60.3.9
    static:
    - 10.60.3.10 - 10.60.3.70
  type: manual

spiff merge bye.yml hi.yml

returns

jobs:
- instances: 3
  name: myjob
  networks:
  - name: cf1
    static_ips:
    - 10.60.3.10
    - 10.60.3.13
    - 10.60.3.70
networks:
- name: cf1
  subnets:
  - cloud_properties:
      security_groups:
      - cf-0-vpc-c461c7a1
      subnet: subnet-e845bab1
    dns:
    - 10.60.3.2
    gateway: 10.60.3.1
    name: default_unused
    range: 10.60.3.0/24
    reserved:
    - 10.60.3.2 - 10.60.3.9
    static:
    - 10.60.3.10 - 10.60.3.70
  type: manual

If bye.yml was instead

networks: (( merge ))

jobs:
  - name: myjob
    instances: 2
    networks:
    - name: cf1
      static_ips: (( static_ips(0,3,60) ))

spiff merge bye.yml hi.yml

instead returns

jobs:
- instances: 2
  name: myjob
  networks:
  - name: cf1
    static_ips:
    - 10.60.3.10
    - 10.60.3.13
networks:
- name: cf1
  subnets:
  - cloud_properties:
      security_groups:
      - cf-0-vpc-c461c7a1
      subnet: subnet-e845bab1
    dns:
    - 10.60.3.2
    gateway: 10.60.3.1
    name: default_unused
    range: 10.60.3.0/24
    reserved:
    - 10.60.3.2 - 10.60.3.9
    static:
    - 10.60.3.10 - 10.60.3.70
  type: manual

static_ipsalso accepts list arguments, as long as all transitivly contained elements are either again lists or integer values. This allows to abbreviate the list of IPs as follows:

  static_ips: (( static_ips([1..5]) ))

`(( ipset(ranges, 3, 3,4,5,6) ))`

While the function static_ips for historical reasons relies on the structure of a bosh manifest and works only at dedicated locations in the manifest, the function ipset offers a similar calculation purely based on its arguments. So, the available ip ranges and the required numbers of IPs are passed as arguments.

The first (ranges) argument can be a single range as a simple string or a list of strings. Every string might be

a single IP address
an explicit IP range described by two IP addresses separated by a dash (-)
a CIDR

The second argument specifies the requested number of IP addresses in the result set.

The additional arguments specify the indices of the IPs to choose (starting from 0) in the given ranges. Here again lists of indices might be used.

e.g.:

ranges:
  - 10.0.0.0 - 10.0.0.255
  - 10.0.2.0/24
ipset: (( ipset(ranges,3,[256..260]) ))

resolves ipset to [ 10.0.2.0, 10.0.2.1, 10.0.2.2 ].

If no IP indices are specified (only two arguments), the IPs are chosen starting from the beginning of the first range up to the end of the last given range, without indirection.

`(( list_to_map(list, "key") ))`

A list of map entries with explicit name/key fields will be mapped to a map with the dedicated keys. By default the key field name is used, which can changed by the optional second argument. An explicitly denoted key field in the list will also be taken into account.

e.g.:

list:
  - key:foo: alice
    age: 24
  - foo: bob
    age: 30

map: (( list_to_map(list) ))

will be mapped to

list:
  - foo: alice
    age: 24
  - foo: bob
    age: 30

map:
  alice:
    age: 24
  bob:
    age: 30

In combination with templates and lambda expressions this can be used to generate maps with arbitrarily named key values, although dynaml expressions are not allowed for key values.

`(( makemap(fieldlist) ))`

In this flavor makemap creates a map with entries described by the given field list. The list is expected to contain maps with the entries key and value, describing dedicated map entries.

e.g.:

list:
  - key: alice
    value: 24
  - key: bob
    value: 25
  - key: 5
    value: 25

map: (( makemap(list) ))

yields

list:
  - key: alice
    value: 24
  - key: bob
    value: 25
  - key: 5
    value: 25

map:
  "5": 25
  alice: 24
  bob: 25

If the key value is a boolean or an integer it will be mapped to a string.

`(( makemap(key, value) ))`

In this flavor makemap creates a map with entries described by the given argument pairs. The arguments may be a sequence of key/values pairs (given by separate arguments).

e.g.:

map: (( makemap("peter", 23, "paul", 22) ))

yields

map:
  paul: 22
  peter: 23

In contrast to the previous makemap flavor, this one could also be handled by map literals.

`(( merge(map1, map2) ))`

Beside the keyword merge there is also a function called merge (It must always be followed by an opening bracket). It can be used to merge severals maps taken from the actual document analogous to the stub merge process. If the maps are specified by reference expressions, they cannot contain any dynaml expressions, because they are always evaluated in the context of the actual document before evaluating the arguments.

e.g.:

map1:
  alice: 24
  bob: (( alice ))
map2:
  alice: 26
  peter: 8
result: (( merge(map1,map2) ))

resolves result to

result:
  alice: 26
  bob: 24  # <---- expression evaluated before mergeing

Alternatively map templates can be passed (without evaluation operator!). In this case the dynaml expressions from the template are evaluated while merging the given documents as for regular calls of spiff merge.

e.g.:

map1:
  <<: (( &template ))
  alice: 24
  bob: (( alice ))
map2:
  alice: 26
  peter: 8
result: (( merge(map1,map2) ))

resolves result to

result:
  alice: 26
  bob: 26

A map might also be given by a map expression. Here it is possible to specify dynaml expressions using the usual syntax:

e.g.:

map1:
  alice: 24
  bob: 25

map2:
  alice: 26
  peter: 8

result: (( merge(map1, map2, { "bob"="(( carl ))", "carl"=100 }) ))

resolves result to

result:
  alice: 26
  bob: 100

Instead of multiple arguments a single list argument can be given. The list must contain the maps to be merged.

Nested merges have access to all outer bindings. Relative references are first searched in the actual document. If they are not found there all outer bindings are used to lookup the reference, from inner to outer bindings. Additionally the context (__ctx) offers a field OUTER, which is a list of all outer documents of the nested merges, which can be used to lookup absolute references.

e.g.:

data:
  alice:
    age: 24

template:
  <<: (( &template ))
  bob:  25
  outer1: (( __ctx.OUTER.[0].data )) # absolute access to outer context
  outer2: (( data.alice.age ))       # relative access to outer binding
  sum: (( .bob + .outer2 ))

merged: (( merge(template) ))

resolves merged to

merged:
  bob: 25
  outer1:
    alice:
      age: 24
  outer2: 24
  sum: 49

`(( intersect(list1, list2) ))`

The function intersect intersects multiple lists. A list may contain entries of any type.

e.g.:

list1:
- - a
- - b
- a
- b
- { a: b }
- { b: c }
- 0
- 1
- "0"
- "1"
list2:
- - a
- - c
- a
- c
- { a: b }
- { b: b }
- 0
- 2
- "0"
- "2"
intersect: (( intersect(list1, list2) ))

resolves intersect to

intersect:
- - a
- a
- { a: b }
- 0
- "0"

`(( reverse(list) ))`

The function reverse reverses the order of a list. The list may contain entries of any type.

e.g.:

list:
- - a
- b
- { a: b }
- { b: c }
- 0
- 1
reverse: (( reverse(list) ))

resolves reverse to

reverse:
- 1
- 0
- { b: c }
- { a: b }
- b
- - a

`(( validate(value,"dnsdomain") ))`

The function validate validates an expression using a set of validators. The first argument is the value to validate and all other arguments are validators that must succeed to accept the value. If at least one validator fails an appropriate error message is generated that explains the fail reason.

A validator is denoted by a string or a list containing the validator type as string and its arguments. A validator can be negated with a preceeding ! in its name.

The following validators are available:

Type	Arguments	Meaning
`empty`	none	empty list, map or string
`dnsdomain`	none	dns domain name
`wildcarddnsdomain`	none	wildcard dns domain name
`dnslabel`	none	dns label
`dnsname`	none	dns domain or wildcard domain
`ip`	none	ip address
`cidr`	none	cidr
`publickey`	none	public key in pem format
`privatekey`	none	private key in pem format
`certificate`	none	certificate in pem format
`ca`	none	certificate for CA
`semver`	optional list of constraints	validate semver version against constraints
`type`	list of accepted type keys	at least one type key must match
`valueset`	list argument with values	possible values
`value` or `=`	value	check dedicated value
`gt` or `>`	value	greater than (number/string)
`lt` or `<`	value	less than (number/string)
`ge` or `>=`	value	greater or equal to (number/string)
`le` or `<=`	value	less or equal to (number/string)
`match` or `~=`	regular expression	string value matching regular expression
`list`	optional list of entry validators	is list and entries match given validators
`map`	[[ <key validator>, ] <entry validator> ]	is map and keys and entries match given validators
`mapfield`	<field name> [ , <validator>]	required entry in map
`optionalfield`	<field name> [ , <validator>]	optional entry in map
`and`	list of validators	all validators must succeed
`or`	list of validators	at least one validator must succeed
`not` or `!`	validator	negate the validator argument(s)

If the validation succeeds the value is returned.

e.g.:

dnstarget: (( validate("192.168.42.42", [ "or", "ip", "dnsdomain" ]) ))

evaluates to

dnstarget: 192.168.42.42

If the validation fails an error explaining the failure reason is generated.

e.g.:

dnstarget: (( validate("alice+bob", [ "or", "ip", "dnsdomain" ]) ))

yields the following error:

*condition 1 failed: (is no ip address: alice+bob and is no dns domain: [a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')])

A validator might also be a lambda expression taking at least one argument and returning a boolean value. This way it is possible to provide own validators as part of the yaml document.

e.g.:

val: (( validate( 0, |x|-> x > 1 ) ))

If more than one parameter is declared the additional arguments must be specified as validator arguments. The first argument is always the value to check.

e.g.:

val: (( validate( 0, [|x,m|-> x > m, 5] ) ))

The lambda function may return a list with 1, 2 or 3 elements, also. This can be used to provide appropriate messages.

Index	Meaning
0	the first index always is the match result, it must be evaluatable as boolean
1	if two elements are given, the second index is the message describing the actual result
2	here index 1 decribes the success message and 2 the failure message

e.g.:

val: (( validate( 6, [|x,m|-> [x > m, "is larger than " m, "is less than or equal to " m], 5] ) ))

Just to mention, the validator specification might be given inline as shown in the examples above, but as reference expressions, also. The not, and and or validators accept deeply nested validator specifications.

e.g.:

dnsrecords:
   domain: 1.2.3.4
validator:
  - map
  - - or                              # key validator
    - dnsdomain
    - wildcarddnsdomain
  - ip                                # entry validator

val: (( validate( map, validator)  ))

`(( check(value,"dnsdomain") ))`

The function check can be used to match a yaml structure against a yaml based value checker. Hereby the same check description already described for validate can be used. The result of the call is a boolean value indicating the match result. It does not fail if the check fails.

`(( error("message") ))`

The function error can be used to cause explicit evaluation failures with a dedicated message.

This can be used, for example, to reduce a complex processing error to a meaningful message by appending the error function as default for the potentially failing comples expression.

e.g.:

value: (( <some complex potentially failing expression> || error("this was an error my friend") ))

Another scenario could be omitting a descriptive message for missing required fields by using an error expression as (default) value for a field intended to be defined in an upstream stub.

Math

dynaml support various math functions:

returning integers: ceil, floor, round and roundtoeven

returning floats or integers: abs

returning floats: sin,cos, sinh, cosh, asin, acos, asinh,acosh, sqrt, exp, log, log10,

Conversions

dynaml supports various type conversions between integer, float, bool and string values by appropriate functions.

e.g.:

value: (( integer("5") ))

converts a string to an integer value.

Converting an integer to a string accepts an optional additional integer argument for specifying the base for conversion, for example string(55,2) will result in "110111". The default base is 10. The base must be between 2 and 36.

Accessing External Content

Spiff supports access to content outside of the template and sub files. It is possible to read files, execute commands and pipelines. All those functions exist in two flavors.

A cached flavor executes the operation ones and caches the result for subsequent identical operations. This speeds up the processing, especially for command executions.
If the result evolves over time, it might be useful to always get the latest content. This is the case if the sync function is used, which is intended to synchronize the template processing with a dedicate state (provided by external content). Here the caching operations would not be useful, therefore there is a second uncached flavor. Every function is available with the suffix _uncached (for example read_uncached())

`(( read("file.yml") ))`

Read a file and return its content. There is support for three content types: yaml files,text files and binary files. Reading in binary mode will result in a base64 encoded multi-line string.

If the file suffix is .yml, .yaml or .json, by default the yaml type is used. If the file should be read as text, this type must be explicitly specified. In all other cases the default is text, therefore reading a binary file (for example an archive) urgently requires specifying the binary mode.

An optional second parameter can be used to explicitly specifiy the desired return type: yaml or text. For yaml documents some addtional types are supported: multiyaml, template, templates, import and importmulti.

yaml documents

A yaml document will be parsed and the tree is returned. The elements of the tree can be accessed by regular dynaml expressions.

Additionally the yaml file may again contain dynaml expressions. All included dynaml expressions will be evaluated in the context of the reading expression. This means that the same file included at different places in a yaml document may result in different sub trees, depending on the used dynaml expressions.

If is possible to read a multi-document yaml, also. If the type multiyaml is given, a list node with the yaml document root nodes is returned.

The yaml or json document can also read as template by specifying the type template. Here the result will be a template value, that can be used like regular inline templates. If templates is specified, a multi-document is mapped to a list of templates.

If the read type is set to import, the file content is read as yaml document and the root node is used to substitute the expression. Potential dynaml expressions contained in the document will not be evaluated with the actual binding of the expression together with the read call, but as it would have been part of the original file. Therefore this mode can only be used, if there is no further processing of the read result or the delivered values are unprocessed.

This can be used together with a chained reference (for examle (( read(...).selection ))) to delect a dedicated fragment of the imported document. Then, the evaluatio will be done for the selected portion, only. Expressions and references in the other parts are not evalauted and at all and cannot lead to error.

e.g.:

template.yaml

ages:
  alice: 25

data: (( read("import.yaml", "import").first ))

import.yaml

first:
  age: (( ages.alice ))

second:
  age: (( ages.bob ))

will not fail, because the second section is never evaluated.

This mode should be taken with caution, because it often leads to unexpected results.

The read type importmulti can be used to import multi-document yaml files as a list of nodes.

text documents

A text document will be returned as single string.

binary documents

It is possible to read binary documents, also. The content cannot be used as a string (or yaml document), directly. Therefore the read mode binary has to be specified. The content is returned as a base64 encoded multi-line string value.

`(( exec("command", arg1, arg2) ))`

Execute a command. Arguments can be any dynaml expressions including reference expressions evaluated to lists or maps. Lists or maps are passed as single arguments containing a yaml document with the given fragment.

The result is determined by parsing the standard output of the command. It might be a yaml document or a single multi-line string or integer value. A yaml document should start with the document prefix ---. If the command fails the expression is handled as undefined.

e.g.

arg:
  - a
  - b
list: (( exec( "echo", arg ) ))
string: (( exec( "echo", arg.[0] ) ))

yields

arg:
- a
- b
list:
- a
- b
string: a

Alternatively exec can be called with a single list argument completely describing the command line.

The same command will be executed once, only, even if it is used in multiple expressions.

`(( pipe(data, "command", arg1, arg2) ))`

Execute a command and feed its standard input with dedicated data. The command argument must be a string. Arguments for the command can be any dynaml expressions including reference expressions evaluated to lists or maps. Lists or maps are passed as single arguments containing a yaml document with the given fragment.

The input stream is generated from the given data. If this is a simple type its string representation is used. Otherwise a yaml document is generated from the input data. The result is determined by parsing the standard output of the command. It might be a yaml document or a single multi-line string or integer value. A yaml document should start with the document prefix ---. If the command fails the expression is handled as undefined.

e.g.

data:
  - a
  - b
list: (( pipe( data, "tr", "a", "z") ))

yields

arg:
- a
- b
list:
- z
- b

Alternatively pipe can be called with data and a list argument completely describing the command line.

The same command will be executed once, only, even if it is used in multiple expressions.

`(( write("file.yml", data) ))`

Write a file and return its content. If the result can be parsed as yaml document, the document is returned. An optional 3rd argument can be used to pass the write options. The option arguments might be an integer denoting file permissions (default is 0644) or a comma separated string with options. Supported options are

binary: data is base64 decoded before writing
integer string: file permissions, a leading 0 is indicating an octal value.

`(( tempfile("file.yml", data) ))`

Write a a temporary file and return its path name. An optional 3rd argument can be used to pass write options. It basically behavies like write

Attention: A temporary file only exists during the merge processing. It will be deleted afterwards.

It can be used, for example, to provide a temporary file argument for the exec function.

`(( lookup_file("file.yml", list) ))`

Lookup a file is a list of directories. The result is a list of existing files. With lookup_dir it is possible to lookup a directory, instead.

If no existing files can be found the empty list is returned.

It is possible to pass multiple list or string arguments to compose the search path.

`(( mkdir("dir", 0755) ))`

Create a directory and all its intermediate directories if they do not exist yet.

The permission part is optional (default 0755). The path of the directory might be given by atring like value or as a list of path components.

`(( list_files(".") ))`

List files in a directory. The result is a list of existing files. With list_dirs it is possible to list directories, instead.

`(( archive(files, "tar") ))`

Create an archive of the given type (default is tar) containing the listed files. The result is the base64 encoded archive.

Supported archive types are tar and targz.

files might be a list or map of file entries. In case of a map, the map key is used as default for the file path. A file entry is a map with the following fields:

field	type	meaning
`path`	string	optional for maps, the file path in the archive, defaulted by the map key
`mode`	int or int string	file mode or write options. It basically behavies like the option argument for `write`.
`data`	any	file content, yaml will be marshalled as yaml document. If `mode` indicates binary mode, a string value will be base64 decoded.
`base64`	string	base64 encoded binary data

e.g.:

yaml:
  alice: 26
  bob: 27

files:
  "data/a/test.yaml":
    data: (( yaml ))
  "data/b/README.md":
    data: |+
      ### Test Docu

      **Note**: This is a test

archive: (( archive(files,"targz") ))

content: (( split("\n", exec_uncached("tar", "-tvf", tempfile(archive,"binary"))) ))

yields:

archive: |-
  H4sIAAAAAAAA/+zVsQqDMBAG4Mx5igO3gHqJSQS3go7tUHyBqIEKitDEoW9f
  dLRDh6KlJd/yb8ll+HOd8SY1qbfOJw8zDmQHiIhayjURcZuIQhOeSZlphVwL
  glwsAXvM8mJ23twJ4qfnbB/3I+I4pmboW1uA0LSZmgJETr89VXCUtf9Neq1O
  5blKxm6PO972X/FN/7nKVej/EaIogto6D+XUzpQydpm8ZayA+tY76B0YWHYD
  DV9CEATBf3kGAAD//5NlAmIADAAA
content:
- -rw-r--r-- 0/0              22 2019-03-18 09:01 data/a/test.yaml
- -rw-r--r-- 0/0              41 2019-03-18 09:01 data/b/README.md

files:
  data/a/test.yaml:
    data:
      alice: 26
      bob: 27
  data/b/README.md:
    data: |+
      ### Test Docu

      **Note**: This is a test

yaml:
  alice: 26
  bob: 27

Semantic Versioning Functions

Spiff supports handling of semantic version names. It supports all functionality from the Masterminds Semver Package accepting versions with or without a leading v.

`(( semver("v1.2-beta.1") ))`

Check whether a given string is a semantic version and return its normalized form (without leading v and complete release part with major, minor and and patch version number).

e.g.:

normalized: (( semver("v1.2-beta.1") ))

resolves to

normalized: 1.2.0-beta.1

`(( semverrelease("v1.2.3-beta.1") ))`

Return the release part of a semantic version omitting metadata and prerelease information.

e.g.:

release: (( semverrelease("v1.2.3-beta.1") ))

resolves to

release: v1.2.3

If an additional string argument is given this function replaces the release by the release of the given semantic version preserving metadata and prerelease information.

e.g.:

new: (( semverrelease("1.2.3-beta.1", "1.2.1) ))

resolves to

new: 1.2.1-beta.1

`(( semvermajor("1.2.3-beta.1") ))`

Determine the major version number of the given semantic version. The result is an integer.

e.g.:

major: (( semvermajor("1.2.3-beta.1") ))

resolves to

major: 1

The function semverincmajor can be used to increment the major version number and reset the minor version, patch version and release suffixes.

e.g.:

new: (( semverincmajor("1.2.3-beta.1") ))

resolves to

new: 2.0.0

`(( semverminor("1.2.3-beta.1") ))`

Determine the minor version number of the given semantic version. The result is an integer.

e.g.:

minor: (( semverminor("1.2.3-beta.1") ))

resolves to

minor: 2

The function semverincminor can be used to increment the minor version number and reset the patch version and release suffixes.

e.g.:

new: (( semverincmajor("v1.2.3-beta.1") ))

resolves to

new: v1.3.0

`(( semverpatch("1.2.3-beta.1") ))`

Determine the patch version number of the given semantic version. The result is an integer.

e.g.:

patch: (( semverpatch("1.2.3-beta.1") ))

resolves to

patch: 3

The function semverincpatch can be used to increment the patch version number or reset the release suffixes. If there are rlease suffixes, they are removed and the release info is kept unchanged, otherwise the patch version number is increased.

e.g.:

final: (( semverincpatch("1.2.3-beta.1") ))
new: (( semverincpatch(final) ))

resolves to

final: 1.2.3
new: 1.2.4

`(( semverprerelease("1.2.3-beta.1") ))`

Determine the prerelease of the given semantic version. The result is a string.

e.g.:

prerelease: (( semverprerelease("1.2.3-beta.1") ))

resolves to

prerelease: beta.1

If an additional string argument is given this function sets, replaces or clears (if set to empty string) the prerelease

e.g.:

new: (( semverprerelease("1.2.3-beta.1", "beta.2) ))

resolves to

new: 1.2.3-beta.2

`(( semvermetadata("1.2.3+demo") ))`

Determine the metadata of the given semantic version. The result is a string.

e.g.:

metadata: (( semvermetadata("1.2.3+demo") ))

resolves to

metadata: demo

If an additional string argument is given this function sets, replaces or clears (if set to empty string) the metadata.

e.g.:

new: (( semvermetadata("1.2.3-test", "demo) ))

resolves to

new: 1.2.3+demo

`(( semvercmp("1.2.3", 1.2.3-beta.1") ))`

Compare two semantic versions. A prerelease is always smaller than the final release. The result is an integer with the following values:

result	meaning
-1	first version is before the second version
0	both versions are equal
1	first versuon is after the second one

e.g.:

compare: (( semvercmp("1.2.3", "1.2.3-beta.1") ))

resolves to

compare: 1

`(( semvermatch("1.2.3", "~1.2") ))`

Match the given semantic version against a list of contraints. The result is a boolean. It is possible to specify any number of version constraints. If no constraint is given, the function just checks whether the given string is a semantic version.

e.g.:

match: (( semvermatch("1.2.3", "~1.2") ))

resolves to

match: true

The complete list of possible constraints specification can be found here.

`(( semversort("1.2.3", "1.2.1") ))`

Sort a list of versions in ascending order. A leading v is preserved.

e.g.:

sorted: (( semversort("1.2.3", "1.2.1") ))

resolves to

sorted:
  - 1.2.1
  - 1.2.3

The list of versions to be sorted may also be specified with a single list argument.

X509 Functions

Spiff supports some useful functions to work with X509 certificates and keys. Please refer also to the Useful to Know section to find some tips for providing state.

`(( x509genkey(spec) ))`

This function can be used generate private RSA or ECDSA keys. The result will be a PEM encoded key as multi line string value. If a key size (integer or string) is given as argument, an RSA key will be generated with the given key size (for example 2048). Given one of the string values

"P224"
"P256"
"P384"
"P521"

the function will generate an appropriate ECDSA key.

e.g.:

keys:
  key: (( x509genkey(2048) ))

resolves to something like

key: |+
    -----BEGIN RSA PRIVATE KEY-----
    MIIEpAIBAAKCAQEAwxdZDfzxqz4hlRwTL060pm1J12mkJlXF0VnqpQjpnRTq0rns
    CxMxvSfb4crmWg6BRaI1cEN/zmNcT2sO+RZ4jIOZ2Vi8ujqcbzxqyoBQuMNwdb32
    ...
    oqMC9QKBgQDEVP7FDuJEnCpzqddiXTC+8NsC+1+2/fk+ypj2qXMxcNiNG1Az95YE
    gRXbnghNU7RUajILoimAHPItqeeskd69oB77gig4bWwrzkijFXv0dOjDhQlmKY6c
    pNWsImF7CNhjTP7L27LKk49a+IGutyYLnXmrlarcNYeCQBin1meydA==
    -----END RSA PRIVATE KEY-----

`(( x509publickey(key) ))`

For a given key or certificate in PEM format (for example generated with the x509genkey function) this function extracts the public key and returns it again in PEM format as a multi-line string.

e.g.:

keys:
  key: (( x509genkey(2048) ))
  public: (( x509publickey(key)

resolves to something like

key: |+
    -----BEGIN RSA PRIVATE KEY-----
    MIIEpAIBAAKCAQEAwxdZDfzxqz4hlRwTL060pm1J12mkJlXF0VnqpQjpnRTq0rns
    CxMxvSfb4crmWg6BRaI1cEN/zmNcT2sO+RZ4jIOZ2Vi8ujqcbzxqyoBQuMNwdb32
    ...
    oqMC9QKBgQDEVP7FDuJEnCpzqddiXTC+8NsC+1+2/fk+ypj2qXMxcNiNG1Az95YE
    gRXbnghNU7RUajILoimAHPItqeeskd69oB77gig4bWwrzkijFXv0dOjDhQlmKY6c
    pNWsImF7CNhjTP7L27LKk49a+IGutyYLnXmrlarcNYeCQBin1meydA==
    -----END RSA PRIVATE KEY-----
public: |+
    -----BEGIN RSA PUBLIC KEY-----
    MIIBCgKCAQEAwxdZDfzxqz4hlRwTL060pm1J12mkJlXF0VnqpQjpnRTq0rnsCxMx
    vSfb4crmWg6BRaI1cEN/zmNcT2sO+RZ4jIOZ2Vi8ujqcbzxqyoBQuMNwdb325Bf/
   ...
    VzYqyeQyvvRbNe73BXc5temCaQayzsbghkoWK+Wrc33yLsvpeVQBcB93Xhus+Lt1
    1lxsoIrQf/HBsiu/5Q3M8L6klxeAUcDbYwIDAQAB
    -----END RSA PUBLIC KEY-----

To generate an ssh public key an optional additional format argument can be set to ssh. The result will then be a regular public key format usable for ssh. The default format is pem providing the pem output format shown above.

RSA keys are by default marshalled in PKCS#1 format(RSA PUBLIC KEY) in pem. If the generic PKIX format (PUBLIC KEY) is required the format argument pkix must be given.

Using the format ssh this function can also be used to convert a pem formatted public key into an ssh key,

`(( x509cert(spec) ))`

The function x509cert creates locally signed certificates, either a self signed one or a certificate signed by a given ca. It returns a PEM encoded certificate as a multi-line string value.

The single spec parameter take a map with some optional and non optional fields used to specify the certificate information. It can be an inline map expression or any map reference into the rest of the yaml document.

The following map fields are observed:

Field Name	Type	Required	Meaning
`commonName`	string	optional	Common Name field of the subject
`organization`	string or string list	optional	Organization field of the subject
`country`	string or string list	optional	Country field of the subject
`isCA`	bool	optional	CA option of certificate
`usage`	string or string list	required	usage keys for the certificate (see below)
`validity`	integer	optional	validity interval in hours
`validFrom`	string	optional	start time in the format "Jan 1 01:22:31 2019"
`hosts`	string or string list	optional	List of DNS names or IP addresses
`privateKey`	string	required or publicKey	private key to generate the certificate for
`publicKey`	string	required or privateKey	public key to generate the certificate for
`caCert`	string	optional	certificate to sign with
`caPrivateKey`	string	optional	priavte key for `caCert`

For self-signed certificates, the privateKeyfield must be set. publicKey and the ca fields should be omitted. If the caCertfield is given, the caKey field is required, also. If the privateKeyfield is given together with the caCert, the public key for the certificate is extracted from the private key.

Additional fields are silently ignored.

The following usage keys are supported (case is ignored):

Key	Meaning
`Signature`	x509.KeyUsageDigitalSignature
`Commitment`	x509.KeyUsageContentCommitment
`KeyEncipherment`	x509.KeyUsageKeyEncipherment
`DataEncipherment`	x509.KeyUsageDataEncipherment
`KeyAgreement`	x509.KeyUsageKeyAgreement
`CertSign`	x509.KeyUsageCertSign
`CRLSign`	x509.KeyUsageCRLSign
`EncipherOnly`	x509.KeyUsageEncipherOnly
`DecipherOnly`	x509.KeyUsageDecipherOnly
`Any`	x509.ExtKeyUsageAny
`ServerAuth`	x509.ExtKeyUsageServerAuth
`ClientAuth`	x509.ExtKeyUsageClientAuth
`codesigning`	x509.ExtKeyUsageCodeSigning
`EmailProtection`	x509.ExtKeyUsageEmailProtection
`IPSecEndSystem`	x509.ExtKeyUsageIPSECEndSystem
`IPSecTunnel`	x509.ExtKeyUsageIPSECTunnel
`IPSecUser`	x509.ExtKeyUsageIPSECUser
`TimeStamping`	x509.ExtKeyUsageTimeStamping
`OCSPSigning`	x509.ExtKeyUsageOCSPSigning
`MicrosoftServerGatedCrypto`	x509.ExtKeyUsageMicrosoftServerGatedCrypto
`NetscapeServerGatedCrypto`	x509.ExtKeyUsageNetscapeServerGatedCrypto
`MicrosoftCommercialCodeSigning`	x509.ExtKeyUsageMicrosoftCommercialCodeSigning
`MicrosoftKernelCodeSigning`	x509.ExtKeyUsageMicrosoftKernelCodeSigning

e.g.:

spec:
  <<: (( &local ))
  ca:
    organization: Mandelsoft
    commonName: Uwe Krueger
    privateKey: (( data.cakey ))
    isCA: true
    usage:
      - Signature
      - KeyEncipherment

data:
  cakey: (( x509genkey(2048) ))
  cacert: (( x509cert(spec.ca) ))

generates a self-signed root certificate and resolves to something like

cakey: |+
    -----BEGIN RSA PRIVATE KEY-----
    MIIEpAIBAAKCAQEAwxdZDfzxqz4hlRwTL060pm1J12mkJlXF0VnqpQjpnRTq0rns
    CxMxvSfb4crmWg6BRaI1cEN/zmNcT2sO+RZ4jIOZ2Vi8ujqcbzxqyoBQuMNwdb32
    ...
    oqMC9QKBgQDEVP7FDuJEnCpzqddiXTC+8NsC+1+2/fk+ypj2qXMxcNiNG1Az95YE
    gRXbnghNU7RUajILoimAHPItqeeskd69oB77gig4bWwrzkijFXv0dOjDhQlmKY6c
    pNWsImF7CNhjTP7L27LKk49a+IGutyYLnXmrlarcNYeCQBin1meydA==
    -----END RSA PRIVATE KEY-----
cacert: |+
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgIQb5ex4iGfyCcOa1RvnKSkMDANBgkqhkiG9w0BAQsFADAk
    MQ8wDQYDVQQKEwZTQVAgU0UxETAPBgNVBAMTCGdhcmRlbmVyMB4XDTE4MTIzMTE0
    ...
    pOUBE3Tgim5rnpa9K9RJ/m8IVqlupcONlxQmP3cCXm/lBEREjODPRNhU11DJwDdJ
    5fd+t5SMEit2BvtTNFXLAwz48EKTxsDPdnHgiQKcbIV8NmgUNPHwXaqRMBLqssKl
    Cyvds9xGtAtmZRvYNI0=
    -----END CERTIFICATE-----

`(( x509parsecert(cert) ))`

This function parses a certificate given in PEM format and returns a map of fields:

Field Name	Type	Required	Meaning
`commonName`	string	optional	Common Name field of the subject
`organization`	string list	optional	Organization field of the subject
`country`	string list	optional	Country field of the subject
`isCA`	bool	always	CA option of certificate
`usage`	string list	always	usage keys for the certificate (see below)
`validity`	integer	always	validity interval in hours
`validFrom`	string	always	start time in the format "Jan 1 01:22:31 2019"
`validUntil`	string	always	start time in the format "Jan 1 01:22:31 2019"
`hosts`	string list	optional	List of DNS names or IP addresses
`dnsNames`	string list	optional	List of DNS names
`ipAddresses`	string list	optional	List of IP addresses
`publicKey`	string	always	public key to generate the certificate for

e.g.:

data:
  <<: (( &temporary ))
  spec:
    commonName: test
    organization: org
    validity: 100
    isCA: true
    privateKey: (( gen.key ))
    hosts:
      - localhost
      - 127.0.0.1

    usage:
     - ServerAuth
     - ClientAuth
     - CertSign

  gen:
    key: (( x509genkey() ))
    cert: (( x509cert(spec) ))

cert: (( x509parsecert(data.gen.cert) ))

resolves to

cert:
  commonName: test
  dnsNames:
  - localhost
  hosts:
  - 127.0.0.1
  - localhost
  ipAddresses:
  - 127.0.0.1
  isCA: true
  organization:
  - org
  publickey: |+
    -----BEGIN RSA PUBLIC KEY-----
    MIIBCgKCAQEA+UIZQUTa/j+WlXC394bccBTltV+Ig3+zB1l4T6Vo36pMBmU4JIkJ
    ...
    TCsrEC5ey0cCeFij2FijOJ5kmm4cK8jpkkb6fLeQhFEt1qf+QqgBw3targ3LnZQf
    uE9t5MIR2X9ycCQSDNBxcuafHSwFrVuy7wIDAQAB
    -----END RSA PUBLIC KEY-----
  usage:
  - CertSign
  - ServerAuth
  - ClientAuth
  validFrom: Mar 11 15:34:36 2019
  validUntil: Mar 15 19:34:36 2019
  validity: 99  # yepp, that's right, there has already time passed since the creation

Wireguard Functions

spiff supports some useful functions to work with wireguard keys. Please refer also to the Useful to Know section to find some tips for providing state.

`(( wggenkey() ))`

This function can be used generate private wireguard key. The result will base64 encoded.

e.g.:

keys:
  key: (( wggenkey() ))

resolves to something like

key: WH9xNVJuSuh7sDVIyUAlmxc+woFDJg4QA6tGUVBtGns=

`(( wgpublickey(key) ))`

For a given key (for example generated with the wggenkey function) this function extracts the public key and returns it again in base64 format-

e.g.:

keys:
  key: (( wggenkey() ))
  public: (( wgpublickey(key)

resolves to something like

key: WH9xNVJuSuh7sDVIyUAlmxc+woFDJg4QA6tGUVBtGns=
public: n405KfwLpfByhU9pOu0A/ENwp0njcEmmQQJvfYHHQ2M=

`(( lambda |x|->x ":" port ))`

Lambda expressions can be used to define additional anonymous functions. They can be assigned to yaml nodes as values and referenced with path expressions to call the function with approriate arguments in other dynaml expressions. For the final document they are mapped to string values.

There are two forms of lambda expressions. While

lvalue: (( lambda |x|->x ":" port ))

yields a function taking one argument by directly taking the elements from the dynaml expression,

string: "|x|->x \":\" port"
lvalue: (( lambda string ))

evaluates the result of an expression to a function. The expression must evaluate to a function or string. If the expression is evaluated to a string it parses the function from the string.

Since the evaluation result of a lambda expression is a regular value, it can also be passed as argument to function calls and merged as value along stub processing.

A complete example could look like this:

lvalue: (( lambda |x,y|->x + y ))
mod: (( lambda|x,y,m|->(lambda m)(x, y) + 3 ))
value: (( .mod(1,2, lvalue) ))

yields

lvalue: lambda |x,y|->x + y
mod: lambda|x,y,m|->(lambda m)(x, y) + 3
value: 6

If a complete expression is a lambda expression the keyword lambda can be omitted.

Lambda expressions evaluate to lambda values, that are used as final values in yaml documents processed by spiff.

Note: If the final document still contains lambda values, they are transferred to a textual representation. It is not guaranteed that this representation can correctly be parsed again, if the document is re-processed by spiff. Especially for complex scoped and curried functions this is not possible.

Therefore function nodes should always be temporary or local to be available during processing or merging, but being omitted for the final document.

Positional versus Named Arguments

A typical function call uses positional arguments. Here the given arguments satisfy the declared function parameters in the given order. For lambda values it is also possible to use named arguments in the call expression. Here an argument is assigned to a dedicated parameter as declared by the lambda expression. The order of named arguments can be arbitrarily chosen.

e.g.:

func: (( |a,b,c|->{$a=a, $b=b, $c=c } ))
result: (( .func(c=1, b=2, a=1) ))

It is also posible to combine named with positional arguments. Hereby the positional arguments must follow the named ones.

e.g.:

func: (( |a,b,c|->{$a=a, $b=b, $c=c } ))
result: (( .func(c=1, 1, 2) ))

The same argument MUST NOT be satified by both, a named and a positional argument.

Instead of using the parameter name it is also possible to use the parameter index, instead.

e.g.:

func: (( |a,b,c|->{$a=a, $b=b, $c=c } ))
result: (( .func(3=1, 1) ))

As such, this feature seems to be quite useless, but it shows its power if combined with optional parameters or currying as shown in the next paragraphs.

Scopes and Lambda Expressions

A lambda expression might refer to absolute or relative nodes of the actual yaml document of the call. Relative references are evaluated in the context of the function call. Therefore

lvalue: (( lambda |x,y|->x + y + offset ))
offset: 0
values:
  offset: 3
  value: (( .lvalue(1,2) ))

yields 6 for values.value.

Besides the specified parameters, there is an implicit name (_), that can be used to refer to the function itself. It can be used to define self recursive function. Together with the logical and conditional operators a fibunacci function can be defined:

fibonacci: (( lambda |x|-> x <= 0 ? 0 :x == 1 ? 1 :_(x - 2) + _( x - 1 ) ))
value: (( .fibonacci(5) ))

yields the value 8 for the value property.

By default reference expressions in a lambda expression are evaluated in the static scope of the lambda dedinition followed by the static yaml scope of the caller. Absolute references are always evalated in the document scope of the caller.

The name _ can also be used as an anchor to refer to the static definition scope of the lambda expression in the yaml document that was used to define the lambda function. Those references are always interpreted as relative references related to the this static yaml document scope. There is no denotation for accessing the root element of this definition scope.

Relative names can be used to access the static definition scope given inside the dynaml expression (outer scope literals and parameters of outer lambda parameters)

e.g.:

env:
  func: (( |x|->[ x, scope, _, _.scope ] ))
  scope: definition

call:
   result: (( env.func("arg") ))
   scope: call

yields the result list:

call:
  result:
  - arg
  - call
  - (( lambda|x|->[x, scope, _, _.scope] ))  # the lambda expression as lambda value
  - definition

This also works across multiple stubs. The definition context is the stub the lambda expression is defined in, even if it is used in stubs down the chain. Therefore it is possible to use references in the lambda expression, not visible at the caller location, they carry the static yaml document scope of their definition with them.

Inner lambda expressions remember the local binding of outer lambda expressions. This can be used to return functions based on arguments of the outer function.

e.g.:

mult: (( lambda |x|-> lambda |y|-> x * y ))
mult2: (( .mult(2) ))
value: (( .mult2(3) ))

yields 6 for property value.

Optional Parameters

Trailing parameters may be defaulted in the lambda expression by assigning values in the declaration. Those parameter are then optional, it is not required to specify arguments for those parameters in function calls.

e.g.:

mult: (( lambda |x,y=2|-> x * y ))
value: (( .mult(3) ))

yields 6 for property value.

It is possible to default all parameters of a lambda expression. The function can then be called without arguments. There might be no non-defaulted parameters after a defaulted one.

A call with positional arguments may only omit arguments for optional parameters from right to left. If there should be an explicit argument for the right most parameter, arguments for all parameters must be specified or named arguments must be used. Here the desired optional parameter can explicitly be set prior to the regular positional arguments.

e.g.:

func:  (( |a,b=1,c=2|->{$a=a, $b=b, $c=c } ))
result: (( .func(c=3, 2) ))

evaluates result to

result:
  a: 2
  b: 1
  c: 3

The expression for the default does not need to be a constant value or even expression, it might refer to other nodes in the yaml document. The default expression is always evaluated in the scope of the lambda expression declaration at the time the lambda expression is evaluated.

e.g.:

stub.yaml

default: 2
mult: (( lambda |x,y=default * 2|-> x * y ))

template.yaml

mult: (( merge ))
scope:
  default: 3
  value: (( .mult(3) ))

evaluates valueto 12

Variable Argument Lists

The last parameter in the parameter list of a lambda expression may be a varargs parameter consuming additional argument in a fnction call. This parameter is always a list of values, one entry per additional argument.

A varargs parameter is denoted by a ... following the last parameter name.

e.g.:

func: (( |a,b...|-> [a] b ))
result: (( .func(1,2,3) ))

yields the list [1, 2, 3] for property result.

If no argument is given for the varargs parameter its value is the empty list.

The ... operator can also be used for inline list expansion.

If a vararg parameter should be set by a named argument its value must be a list.

Currying

Using the currying operator (*() a lambda function may be transformed to another function with less parameters by specifying leading argument values.

The result is a new function taking the missing arguments (currying) and using the original function body with a static binding for the specified parameters.

e.g.:

mult: (( lambda |x,y|-> x * y ))
mult2: (( .mult*(2) ))
value: (( .mult2(3) ))

Currying may be combined with defaulted parameters. But the resulting function does not default the leading parameters, it is just a new function with less parameters pinning the specified ones.

If the original function uses a variable argument list, the currying may span any number of the variable argument part, but once at least one such argument is given, the parameter for the variable part is satisfied. It cannot be extended by a function call of the curried function.

e.g.:

func: (( |a,b...|->join(a,b) ))
func1: (( .func*(",","a","b")))
#invalid: (( .func1("c") ))
value: (( .func1() ))

evaluates value to "a,b".

It is also possible to use currying for builtin functions, like join.

e.g.:

stringlist: (( join*(",") ))
value: (( .stringlist("a", "b")  ))

evaluates value to "a,b".

There are several builtin functions acting on unevaluated or unevaluatable arguments, like defined. For these functions currying is not possible.

Using positional arguments currying is only possible from right to left. But currying can also be done for named arguments. Here any parameter combination, regardless of the position in the parameter list, can be preset. The resulting function then has the unsatisfied parameters in their original order. Switching the parameter order is not possible.

e.g.:

func: (( |a,b=1,c=2|->{$a=a, $b=b, $c=c } ))
curry: (( .func(c=3, 2) ))

result: (( .curry(5) ))

evalutes result to

result:
  a: 2
  b: 5
  c: 3

The resulting function keeps the parameter b. Hereby the default value will be kept. Therefore it can just be called without argument (.curry()), which would produce

result:
  a: 2
  b: 1
  c: 3

Attention:

For compatibility reasons currying is also done, if a lambda function without defaulted parameters is called with less arguments than declared parameters.

This behaviour is deprecated and will be removed in the future. It is replaced by the currying operator.

e.g.:

mult: (( lambda |x,y|-> x * y ))
mult2: (( .mult(2) ))
value: (( .mult2(3) ))

evaluates value to 6.

`(( catch[expr|v,e|->v] ))`

This expression evaluates an expression (expr) and then executes a lambda function with the evaluation state of the expression. It always succeeds, even if the expression fails. The lambda function may take one or two arguments, the first is always the evaluated value (or nil in case of an error). The optional second argument gets the error message the evaluation of the expression failed (or nil otherwise)

The result of the function is the result of the whole expression. If the function fails, the complete expression fails.

e.g.:

data:
  fail: (( catch[1 / 0|v,e|->{$value=v, $error=e}] ))
  valid: (( catch[5 * 5|v,e|->{$value=v, $error=e}] ))

resolves to

data:
  fail:
    error: division by zero
    value: null
  valid:
    error: null
    value: 25

`(( sync[expr|v,e|->defined(v.field),v.field|10] ))`

If an expression expr may return different results for different evaluations, it is possible to synchronize the final output with a dedicated condition on the expression value. Such an expression could, for example, be an uncached read, exec or pipe call.

The second element must evaluate to a lambda value, given by either a regular expression or by a lambda literal as shown in the title. It may take one or two arguments, the actual value of the value expression and optionally an error message in case of a failing evaluation. The result of the evaluation of the lamda expression decides whether the state of the evaluation of the value expression is acceptable (true) or not (false).

If the value is accepted, an optional third expression is used to determine the final result of the sync[] expression. It might be given as an expression evaluating to a lambda value, or by a comma separated expression using the same binding as the preceeding lambda literal. If not given, the value of the synched expression is returned.

If the value is not acceptable, the evaluation is repeated until a timeout applies. The timeout in seconds is given by an optional fourth expression (default is 5 min). Either the fourth, or the both, the third and the fourth elements may be omitted.

The lambda values might be given as literal, or by expression, leading to the following flavors:

sync[expr|v,e|->cond,value|10]
sync[expr|v,e|->cond|valuelambda|10]
sync[expr|v,e|->cond|v|->value|10]
sync[expr|condlambda|valuelambda|10]
sync[expr|condlambda|v|->value|10]

with or without the timeout expression.

e.g.:

data:
  alice: 25
result: (( sync[data|v|->defined(v.alice),v.alice] ))

resolves to

data:
  alice: 25
result: 25

This example is quite useless, because the sync expression is a constant. It just demonstrates the usage.

Mappings

Mappings are used to produce a new list from the entries of a list or map, or a new map from entries of a map containing the entries processed by a dynaml expression. The expression is given by a lambda function. There are two basic forms of the mapping function: It can be inlined as in (( map[list|x|->x ":" port] )), or it can be determined by a regular dynaml expression evaluating to a lambda function as in (( map[list|mapping.expression)) (here the mapping is taken from the property mapping.expression, which should hold an approriate lambda function).

The mapping comes in two target flavors: with [] or {} in the syntax. The first flavor always produces a list from the entries of the given source. The second one takes only a map source and produces a filtered or transformed map.

Additionally the mapping uses three basic mapping behaviours:

transforming the values using the keyword map. Here the result of the lambda function is used as new value to replace the original one. Or
filtering using the keywork select. Here the result of the lambda function is used as a boolean to decide whether the entry should be kept (true) or omitted (false).
composing using the keyword sum. Here always the list flavor is used, but the result type and content is completely determined by the parameterization of the statement by successively aggregating one entry after the other into an arbitrary initial value.

Note: The special reference _ is not set for inlined lambda functions as part of the mapping syntax. Therefore the mapping statements (and all other statements using inlined lambda functions as part of their syntax) can be used inside regular lambda functions without hampering the meaning of this special refrence for the surrounding explicit lambda expression.

`(( map[list|elem|->dynaml-expr] ))`

Execute a mapping expression on members of a list to produce a new (mapped) list. The first expression (list) must resolve to a list. The last expression (x ":" port) defines the mapping expression used to map all members of the given list. Inside this expression an arbitrarily declared simple reference name (here x) can be used to access the actually processed list element.

e.g.

port: 4711
hosts:
  - alice
  - bob
mapped: (( map[hosts|x|->x ":" port] ))

yields

port: 4711
hosts:
- alice
- bob
mapped:
- alice:4711
- bob:4711

This expression can be combined with others, for example:

port: 4711
list:
  - alice
  - bob
joined: (( join( ", ", map[list|x|->x ":" port] ) ))

which magically provides a comma separated list of ported hosts:

port: 4711
list:
  - alice
  - bob
joined: alice:4711, bob:4711

`(( map[list|idx,elem|->dynaml-expr] ))`

In this variant, the first argument idx is provided with the index and the second elem with the value for the index.

e.g.

list:
  - name: alice
    age: 25
  - name: bob
    age: 24

ages: (( map[list|i,p|->i + 1 ". " p.name " is " p.age ] ))

yields

list:
  - name: alice
    age: 25
  - name: bob
    age: 24

ages:
- 1. alice is 25
- 2. bob is 24

`(( map[map|key,value|->dynaml-expr] ))`

Mapping of a map to a list using a mapping expression. The expression may have access to the key and/or the value. If two references are declared, both values are passed to the expression, the first one is provided with the key and the second one with the value for the key. If one reference is declared, only the value is provided.

e.g.

ages:
  alice: 25
  bob: 24

keys: (( map[ages|k,v|->k] ))

yields

ages:
  alice: 25
  bob: 24

keys:
- alice
- bob

`(( map{map|elem|->dynaml-expr} ))`

Using {} instead of [] in the mapping syntax, the result is again a map with the old keys and the new entry values. As for a list mapping additionally a key variable can be specified in the variable list.

persons:
  alice: 27
  bob: 26
older: (( map{persons|x|->x + 1} ) ))

just increments the value of all entries by one in the field older:

older:
  alice: 28
  bob: 27

Remark

An alternate way to express the same is to use sum[persons|{}|s,k,v|->s { k = v + 1 }].

`(( map{list|elem|->dynaml-expr} ))`

Using {} instead of [] together with a list in the mapping syntax, the result is again a map with the list elements as key and the mapped entry values. For this all list entries must be strings. As for a list mapping additionally an index variable can be specified in the variable list.

persons:
  - alice
  - bob
length: (( map{persons|x|->length(x)} ) ))

just creates a map mapping the list entries to their length:

length:
  alice: 5
  bob: 3

`(( select[expr|elem|->dynaml-expr] ))`

With select a map or list can be filtered by evaluating a boolean expression for every entry. An entry is selected if the expression evaluates to true equivalent value. (see conditions).

Basically it offers all the mapping flavors available for map[]

e.g.

list:
  - name: alice
    age: 25
  - name: bob
    age: 26


selected: (( select[list|v|->v.age > 25 ] ))

evaluates selected to

selected:
- name: bob
  age: 26

Remark

An alternate way to express the same is to use map[list|v|->v.age > 25 ? v :~].

`(( select{map|elem|->dynaml-expr} ))`

Using {} instead of [] in the mapping syntax, the result is again a map with the old keys filtered by the given expression.

persons:
  alice: 25
  bob: 26
older: (( select{persons|x|->x > 25} ))

just keeps all entries with a value greater than 25 and omits all others:

selected:
  bob: 26

This flavor only works on maps.

Remark

An alternate way to express the same is to use sum[persons|{}|s,k,v|->v > 25 ? s {k = v} :s].

Aggregations

Aggregations are used to produce a single result from the entries of a list or map aggregating the entries by a dynaml expression. The expression is given by a lambda function. There are two basic forms of the aggregation function: It can be inlined as in (( sum[list|0|s,x|->s + x] )), or it can be determined by a regular dynaml expression evaluating to a lambda function as in (( sum[list|0|aggregation.expression)) (here the aggregation function is taken from the property aggregation.expression, which should hold an approriate lambda function).

`(( sum[list|initial|sum,elem|->dynaml-expr] ))`

Execute an aggregation expression on members of a list to produce an aggregation result. The first expression (list) must resolve to a list. The second expression is used as initial value for the aggregation. The last expression (s + x) defines the aggregation expression used to aggregate all members of the given list. Inside this expression an arbitrarily declared simple reference name (here s) can be used to access the intermediate aggregation result and a second reference name (here x) can be used to access the actually processed list element.

e.g.

list:
  - 1
  - 2
sum: (( sum[list|0|s,x|->s + x] ))

yields

list:
  - 1
  - 2
sum: 3

`(( sum[list|initial|sum,idx,elem|->dynaml-expr] ))`

In this variant, the second argument idx is provided with the index and the third elem with the value for the index.

e.g.

list:
  - 1
  - 2
  - 3

prod: (( sum[list|0|s,i,x|->s + i * x ] ))

yields

list:
  - 1
  - 2
  - 3

prod: 8

`(( sum[map|initial|sum,key,value|->dynaml-expr] ))`

Aggregation of the elements of a map to a single result using an aggregation expression. The expression may have access to the key and/or the value. The first argument is always the intermediate aggregation result. If three references are declared, both values are passed to the expression, the second one is provided with the key and the third one with the value for the key. If two references are declared, only the second one is provided with the value of the map entry.

e.g.

ages:
  alice: 25
  bob: 24

sum: (( map[ages|0|s,k,v|->s + v] ))

yields

ages:
  alice: 25
  bob: 24

sum: 49

Projections

Projections work over the elements of a list or map yielding a result list. Hereby every element is mapped by an optional subsequent reference expression. This may contain again projections, dynamic references or lambda calls. Basically this is a simplified form of the more general mapping yielding a list working with a lambda function using only a reference expression based on the elements.

`(( expr.[*].value ))`

All elements of a map or list given by the expression expr are dereferenced with the subsequent reference expression (here .expr). If this expression works on a map the elements are ordered accoring to their key values. If the subsequent reference expression is omitted, the complete value list isreturned. For a list expression this means the identity operation.

e.g.:

list:
  - name: alice
    age: 25
  - name: bob
    age: 26
  - name: peter
    age: 24

names: (( list.[*].name ))

yields for names:

names:
  - alice
  - bob
  - peter

or for maps:

networks:
  ext:
    cidr: 10.8.0.0/16
  zone1:
    cidr: 10.9.0.0/16

cidrs: (( .networks.[*].cidr ))

yields for cidrs:

cidrs:
  - 10.8.0.0/16
  - 10.9.0.0/16

`(( list.[1..2].value ))`

This projection flavor only works for lists. The projection is done for a dedicated slice of the initial list.

e.g.:

list:
  - name: alice
    age: 25
  - name: bob
    age: 26
  - name: peter
    age: 24

names: (( list.[1..2].name ))

yields for names:

names:
  - bob
  - peter

Inline List Expansion

In argument lists or list literals the list expansion operator (...) can be used. It is a postfix operator on any list expression. It substituted the list expression by a sequence of the list members. It can be be used in combination with static list argument denotation.

e.g.:

list:
  - a
  - b
  
result: (( [ 1, list..., 2, list... ]  ))

evaluates result to

result:
  - 1
  - a
  - b
  - 2
  - a
  - b

The following example demonstrates the usage in combination with the varargs operator in functions:

func: (( |a,b...|-> [a] b ))

list:
  - a
  - b

a: (( .func(1,2,3) ))
b: (( .func("x",list..., "z") ))
c: (( [ "x", .func(list...)..., "z" ] ))

evaluates the following results:

a:
- 1
- 2
- 3
b:
- x
- a
- b
- z
c:
- x
- a
- b
- z

Please note, that the list expansion might span multiple arguments (including the varargs parameter) in lambda function calls.

Markers

Nodes of the yaml document can be marked to enable dedicated behaviours for this node. Such markers are part of the dynaml syntax and may be prepended to any dynaml expression. They are denoted by the & character directly followed by a marker name. If the expression is a combination of markers and regular expressions, the expression follows the marker list enclosed in brackets (for example (( &temporary( a + b ) ))).

Note: Instead of using a <<: insert field to place markers it is possible now to use <<<:, also, which allows to use regular yaml parsers for spiff-like yaml documents. <<: is kept for backward compatibility.

`(( &temporary ))`

Maps, lists or simple value nodes can be marked as temporary. Temporary nodes are removed from the final output document, but are available during merging and dynaml evaluation.

e.g.:

temp:
  <<: (( &temporary ))
  foo: bar

value: (( temp.foo ))

yields:

value: bar

Adding - <<: (( &temporary )) to a list can be used to mark a list as temporary.

The temporary marker can be combined with regular dynaml expressions to tag plain fields. Hereby the parenthesised expression is just appended to the marker

e.g.:

data:
  alice: (( &temporary ( "bar" ) ))
  foo: (( alice ))

yields:

data:
  foo: bar

The temporary marker can be combined with the template marker to omit templates from the final output.

`(( &local ))`

The marker &local acts similar to &temporary but local nodes are always removed from a stub directly after resolving dynaml expressions. Such nodes are therefore not available for merging and they are not used for further merging of stubs and finally the template.

`(( &dynamic ))`

This marker can be used to mark a template expression (direct or referenced) to enforce the re-evaluation of the template in the usage context whenever the node is used to override or inject a node value along the processing chain. It can also be used together with &inject or &default.

e.g.:

template.yaml

data: 1

merged with

stub.yaml

id: (( &dynamic &inject &template(__ctx.FILE) ))

will resolve to

id: template.yaml
data: 1

The original template is kept along the merge chain and is evaluated separately in the context of the very stub or template it is used.

Using this marker for nodes not evaluationg to a template value is not possible.

`(( &inject ))`

This marker requests the marked item to be injected into the next stub level, even is the hosting element (list or map) does not requests a merge. This only works if the next level stub already contains the hosting element.

e.g.:

template.yaml

alice:
 foo: 1

stub.yaml

alice:
  bar: (( &inject(2) ))
  nope: not injected
bob:
  <<: (( &inject ))
  foobar: yep

is merged to

alice:
  foo: 1
  bar: 2
bob:
  foobar: yep

`(( &default ))`

Nodes marked as default will be used as default values for downstream stub levels. If no such entry is set there it will behave like &inject and implicitly add this node, but existing settings will not be overwritten.

Maps (or lists) marked as default will be considered as values. The map is used as a whole as default if no such field is defined downstream.

e.g.:

template.yaml

data: { }

stub.yaml

data:
  foobar:
    <<: (( &default ))
    foo: claude
    bar: peter

is merged to

data:
  foobar:
    foo: claude
    bar: peter

Their entries will neither be used for overwriting existing downstream values nor for defaulting non-existng fields of a not defaulted map field.

e.g.:

template.yaml

data:
  foobar:
    bar: bob

stub.yaml

data:
  foobar:
    <<: (( &default ))
    foo: claude
    bar: peter

is merged to

data:
  foobar:
    bar: bob

If sub sequent defaulting is desired, the fields of a default map must again be marked as default.

e.g.:

template.yaml

data:
  foobar:
    bar: bob

stub.yaml

data:
  foobar:
    <<: (( &default ))
    foo: (( &default ("claude") ))
    bar: peter

is merged to

data:
  foobar:
    foo: claude
    bar: bob

Note: The behaviour of list entries marked as default is undefined.

`(( &state ))`

Nodes marked as state are handled during the merge processing as if the marker would not be present. But there will be a special handling for enabled state processing (option --state <path>) at the end of the template processing. Additionally to the regular output a document consisting only of state nodes (plus all nested nodes) will be written to a state file. This file will be used as top-level stub for further merge processings with enabled state support.

This enables to keep state between two merge processings. For regular merging sich nodes are only processed during the first processing. Later processings will keep the state from the first one, because those nodes will be overiden by the state stub added to the end of the sub list.

If those nodes additionally disable merging (for example using (( &state(merge none) ))) dynaml expressions in sub level nodes may perform explicit merging using the function stub() to refer to values provided by already processed stubs (especially the implicitly added state stub). For an example please refer to the state library.

`(( &template ))`

Nodes marked as template will not be evaluated at the place of their occurrence. Instead, they will result in a template value stored as value for the node. They can later be instantiated inside a dynaml expression (see below).

`(( &tag:name ))`

The tag marker can be used to assign a logical name to a node value. This name can then be used in tagged reference expressions to refer to this node value (see below).

A tagged reference has the form <tagname>::<path>. The <path> may denote any sub node of a tagged node. If the value of a complete node (or a simple value node) should be used, the <path> must denote the root path (.).

Templates

A map can be tagged by a dynaml expression to be used as template. Dynaml expressions in a template are not evaluated at its definition location in the document, but can be inserted at other locations using dynaml. At every usage location it is evaluated separately.

`<<: (( &template ))`

The dynaml expression &template can be used to tag a map node as template:

e.g.:

foo:
  bar:
    <<: (( &template ))
    alice: alice
    bob: (( verb " " alice ))

The template will be the value of the node foo.bar. As such it can be overwritten as a whole by settings in a stub during the merge process. Dynaml expressions in the template are not evaluated. A map can have only a single << field. Therefore it is possible to combine the template marker with an expression just by adding the expression in parenthesis.

Adding - <<: (( &template )) to a list it is also possible to define list templates. It is also possible to convert a single expression value into a simple template by adding the template marker to the expression, for example foo: (( &template (expression) ))

The template marker can be combined with the temporary marker to omit templates from the final output.

Note: Instead of using a <<: insert field to place the template marker it is possible now to use <<<:, also, which allows to use regular yaml parsers for spiff-like yaml documents. <<: is kept for backward compatibility.

`(( *foo.bar ))`

The dynaml expression *<reference expression> can be used to evaluate a template somewhere in the yaml document. Dynaml expressions in the template are evaluated in the context of this expression.

e.g.:

foo:
  bar:
    <<: (( &template ))
    alice: alice
    bob: (( verb " " alice ))


use:
  subst: (( *foo.bar ))
  verb: loves

verb: hates

evaluates to

foo:
  bar:
    <<: (( &template ))
    alice: alice
    bob: (( verb " " alice ))

use:
  subst:
    alice: alice
    bob: loves alice
  verb: loves

verb: hates

Scope References

`_`

The special reference _ (self) can be used inside of lambda functions and templates. They refer to the containing element (the lambda function or template).

Additionally it can be used to lookup relative reference expressions starting with the defining document scope of the element skipping intermediate scopes.

e.g.:

node:
  data:
    scope: data
  funcs:
    a: (( |x|->scope ))
    b: (( |x|->_.scope ))
    c: (( |x|->_.data.scope ))
    scope: funcs

call:
  scope: call

  a: (( node.funcs.a(1) ))
  b: (( node.funcs.b(1) ))
  c: (( node.funcs.c(1) ))

evaluates call to

call:
  a: call
  b: funcs
  c: data
  scope: call

`__`

The special reference __ can be used to lookup references as relative references starting with the document node hosting the actually evaluated dynaml expression skipping intermediate scopes.

This can, for example be used to relatively access a lambda value field besides the actual field in a map. The usage of plain function names is reserved for builtin functions and are not used as relative references.

This special reference is also available in expressions in templates and refer to the map node in the template hosting the actually evaluated expression.

e.g.:

templates:
  templ:
    <<: (( &template ))
    self: (( _ ))
    value: (( ($self="value") __.self ))
    result: (( scope ))
    templ: (( _.scope ))

  scope: templates


result:
  inst: (( *templates.templ ))
  scope: result

evaluates result to

result:
  inst:
    result: result
    templ: templates
    
    self:
      <<: (( &template ))
      result: (( scope ))
      self: (( _ ))
      templ: (( _.scope ))
      value: (( ($self="value") __.self ))
    value:
      <<: (( &template ))
      result: (( scope ))
      self: (( _ ))
      templ: (( _.scope ))
      value: (( ($self="value") __.self ))
  scope: result

or with referencing upper nodes:

templates:
  templ:
    <<: (( &template ))
    alice: root
    data:
      foo: (( ($bob="local") __.bob ))
      bar: (( ($alice="local") __.alice ))
      bob: static


result: (( *templates.templ ))

evaluates result to

result:
  alice: root
  data:
    bar: root
    foo: static
    
    bob: static

`___`

The special reference ___ can be used to lookup references in the outer most scope. It can therefore be used to access processing bindings specified for a document processing via command line or API. If no bindings are specified the document root is used.

Calling spiff merge template.yaml --bindings bindings.yaml with a binding of

bindings.yaml

input1: binding1
input2: binding2

and the template

template.yaml

input1: top1
map:
  input: map
  input1: map1
  
  results:
    frommap: (( input1 ))
    fromroot: (( .input1 ))
    frombinding1: (( ___.input1 ))
    frombinding2: (( input2 ))

evaluates map.results to

  results:
    frombinding1: binding1
    frombinding2: binding2
    frommap: map1
    fromroot: top1

`__ctx.OUTER`

The context field OUTER is used for nested merges. It is a list of documents, index 0 is the next outer document, and so on.

Special Literals

`(( {} ))`

Provides an empty map.

`(( [] ))`

Provides an empty list. Basically this is not a dedicated literal, but just a regular list expression without a value.

`(( ~ ))`

Provides the null value.

`(( ~~ ))`

This literal evaluates to an undefined expression. The element (list entry or map field) carrying this value, although defined, will be removed from the document and handled as undefined for further merges and the evaluation of referential expressions.

e.g.:

foo: (( ~~ ))
bob: (( foo || ~~ ))
alice: (( bob || "default"))

evaluates to

alice: default

Access to evaluation context

Inside every dynaml expression a virtual field __ctx is available. It allows access to information about the actual evaluation context. It can be accessed by a relative reference expression.

The following fields are supported:

Field Name	Type	Meaning
`VERSION`	string	current version of spiff
`FILE`	string	name of actually processed template file
`DIR`	string	name of directory of actually processed template file
`RESOLVED_FILE`	string	name of actually processed template file with resolved symbolic links
`RESOLVED_DIR`	string	name of directory of actually processed template file with resolved symbolic links
`PATHNAME`	string	path name of actually processed field
`PATH`	list[string]	path name as component list
`OUTER`	yaml doc	outer documents for nested merges, index 0 is the next outer document
`BINDINGS`	yaml doc	the external bindings for the actual processing (see also ___)

If external bindings are specified they are the last elements in OUTER.

e.g.:

template.yml

foo:
  bar:
    path: (( __ctx.PATH ))
    str: (( __ctx.PATHNAME ))
    file: (( __ctx.FILE ))
    dir: (( __ctx.DIR ))

evaluates to

e.g.:

foo:
  bar:
    dir: .
    file: template.yml
    path:
    - foo
    - bar
    - path
    str: foo.bar.str

Operation Priorities

Dynaml expressions are evaluated obeying certain priority levels. This means operations with a higher priority are evaluated first. For example the expression 1 + 2 * 3 is evaluated in the order 1 + ( 2 * 3 ). Operations with the same priority are evaluated from left to right (in contrast to version 1.0.7). This means the expression 6 - 3 - 2 is evaluated as ( 6 - 3 ) - 2.

The following levels are supported (from low priority to high priority)

||, //
White-space separated sequence as concatenation operation (foo bar)
-or, -and
==, !=, <=, <, >, >=
+, -
*, /, %
Grouping ( ), !, constants, references (foo.bar), merge, auto, lambda, map[], and functions

The complete grammar can be found in dynaml.peg.

String Interpolation

Feature state: alpha

Attention: This is an alpha feature. It must be enabled on the command line with the --interpolation or --features=interpolation option. Also for the spiff library it must explicitly be enabled. By adding the key interpolation to the feature list stored in the environment variable SPIFF_FEATURES this feature will be enabled by default.

Typically a complete value can either be a literal or a dynaml expression. For string literals it is possible to use an interpolation syntax to embed dynaml expressions into strings.

For example

data: test
interpolation: this is a (( data ))

replaces the part between the double brackets by the result of the described expression evaluation. Here the brackets can be escaped by the usual escaping (((!) syntax.

Those string literals will implicitly be converted to complete flat dynaml expressions. The example above will therefore be converted into

(( "this is a " data ))

which is the regular dynaml equivalent. The escaping is very ticky, and may be there are still problems. Quotes inside an embedded dynaml expression can be escaped to enable quotes in string literals.

Incomplete or partial interpolation expressions will be ignored and just used a s string.

Strings inside a dynaml expression are NOT directly interpolated again, thus

data: "test"
interpolated: "this is a (( length(\"(( data ))\") data ))"

will resolve interpolation to this is 10test and not to this is 4test.

But if the final string after the expression evaluation again describes a string interpolation it will be processed, again.

data: test
interpolation: this is a (( "(( data ))" data ))

will resolve interpolation to this is testtest.

The embedded dynaml expression must be concatenatable with strings.

YAML-based Control Structures

Feature state: alpha

In addition to describe conditions and loops with dynaml expressions it is also possible to use elements of the document structure to embed control structures.

Such a YAML-based control structure is always described as a map in YAML/JSON. The syntactical elements are expressed as map fields starting with <<. Additionally, depending on the control structure, regular fields are possible. Control structures finally represent a value for the containing node. They may contain marker expressions (<<), also.

e.g.:

temp:
  <<: (( &temporary ))
  <<if: (( features("control") ))
  <<then:
    alice: 25
    bob: 26

resolves to

final:
  alice: 25
  bob: 26

Tu use this alpha feature the feature flag control must be enabled.

Please be aware: Control structure maps typically are always completely resolved before they are evaluated.

Control Structures in Maps

A control structure itself is always a dedicated map node in a document. It is substituted by a regular value node determined by the execution of the control structure.

The fields of a control structure map are not subject to overwriting by stubs, but the complete structure can be overwritten.

If used as value for a map field the resulting value is just used as effective value for this field.

If a map should be enriched by maps resulting from multiple control structures the special control structure <<merge: can be used. It allows to specify a list of maps which should be merged with the actual control structure map to finally build the result value.

Control Structures in Lists

A control structure can be used as list value, also. In this case there is a dedicated interpretation of the resulting value of the control structure. If it is NOT a list value, for convenience, the value is directly used as list entry and substitutes the control structure map.

If the resulting value is again a list, it is inserted into the containing list at the place of occurrence of the control structure. So, if a list value should be used as dedicated entry in a list, the result of a control structure must be a list with the intended list as entry.

e.g.:

list:
 - <<if: (( features("control") ))
   <<then: alice
 - <<if: (( features("control") ))
   <<then:
   - - peter
 - <<if: (( features("control") ))
   <<then:
   - bob

resolves to

list:
- alice
- - peter
- bob

`<<if:`

The condition structure is defined by the syntax field <<if. It additionally accepts the fields <<then and <<else.

The condition field must provide a boolean value. If it is true the optional <<then field is used to substitute the control structure, otherwise the optional <<else field is used.

If the appropriate case is not specified, the result is the undefined (( ~~ )) value. The containing field is therefore completely omitted from the output.

.e.g.:

x: test1
cond:
  field:
    <<if: (( x == "test" ))
    <<then: alice
    <<else: bob

evaluates cond.field to bob

If the else case is omitted, the cond field would be an empty map (field is omitted, because the contained control structure evaluates to undefined)

A comparable way to do this with regular dynaml could look like this:

cond: (( x == "test" ? "alice" :"bob" ))

A better way more suitable for complex cases would be:

local:
  <<: (( &local))
  then: alice
  else: bob

cond: (( x == "test" ? local.then :local.else ))

`<<switch:`

The switch control structure evaluates the switch value of the <<switch field to a string and uses it to select an appropriate regular field in the control map.

If it is not found the value of the optional field <<default is used. If no default is specified, the control structure evaluates to an error, if no appropriate regular field is available.

The nil value matches the default case. If the switch value is undefined the control evaluates to the undefined value (( ~~ )).

e.g.:

x: alice

value:
  <<switch: (( x ))
  alice: 25
  bob: 26 
  <<default: other

evaluates value to 25.

A comparable way to do this with regular dynaml could look like this:

local:
  <<: (( &local))
  cases:
    alice: 25
    bob: 26
  default: other

value: (( local.cases[x] || local.default ))

`<<type:`

The type control structure evaluates the type of the value of the <<type field and uses it to select an appropriate regular field in the control map.

If it is not found the value of the optional field <<default is used. If no default is specified, the control structure evaluates to an error, if no appropriate regular field is available.

e.g.:

x: alice

value:
  <<type: (( x ))
  string: alice
  <<default: unknown

evaluates value to alice.

A comparable way to do this with regular dynaml could look like this:

local:
  <<: (( &local))
  cases:
    string: alice
  default: unknown

value: (( local.cases[type(x)] || local.default ))

For more complex scenarios not only switching on strings a second syntax can be used. Instead of using fields in the control map as cases, a dedicated field <<cases may contain a list of cases, that are checked sequentially (In this flavor regular fields are not allowed anymore).

Every case is described again by a map containing the fields:

case: the expected value to match the switch value
match: a lambda function taking one argument and yielding a boolean value used to match the given switch value
value: (optional) the resulting value in case of a match. If not defined the result will be the undefined value.

One of case or match must be present.

e.g.:

x: 5
selected:
  <<switch: (( x ))
  <<cases:
    - case:
        alice: 25
      value: alice
    - match: (( |v|->v == 5 ))
      value: bob
  <<default: unknown

resolves to

x: 5
selected: bob

If x would be set to the complex value

x:
  alice: 25

it would resolve to

x:
  alice: 25
selected: alice

`<<for:`

The loop control is able to execute a multi-dimensional loop and produce a list or map based on the value combinations.

The loop ranges are specified by the value of the <<for field. It is possible ol loop over lists or maps. The range specification can be given by either a map or list:

map: the keys of the map are the names of the control variables and the values must be lists or maps specifying the ranges.

The map key might optionally be a comma-separated pair (for example key,value) of variable names. In this case the first name is the name for the index variable and the second one for the value variable.

If multiple ranges are specified iterations are alphabetically ordered by value variable name (first) and index variable name (second) to determine the traversing order.
list: if the control variables are defined by a list, each list element must contain two mandatory and one optional field(s):
- name: the name of the (list or map entry value) control variable
- values: a list to define the value range.
- index : (optional) the name of the variable providing the list index or map key of the loop range (defaulted to index-<name>)
Here the order in the list determine the traversal order.

Traversal is done by recursively iterating follow up ranges for every entry in the actual range. This means the last range is completely iterated for the first values of the first ranges first.

If no index variable is specified for a loop range there is an additional implicit binding for every control variable describing the actual list index or map key of the processed value for this dimension. It is denoted by index-<control variable>

If multiple loop ranges are specified, the ranges may mix iterations over maps and lists.

The iteration result value is determined by the value of the <<do field. It is implicitly handled as template and is evaluated for every set of iteration values.

Lists as Iteration Result

The result of the evaluation using only the <<do value field is a list.

e.g.:

alice:
 - a
 - b
bob:
 - 1
 - 2
 - 3
list:
  <<for: 
     key,alice: (( .alice )) # sorted by using alice as primary sort key
     bob: (( .bob ))
  <<do:
    value: (( alice "-" key "-" bob "-" index-bob ))

evaluates list to

list:
- value: a-0-1-0
- value: a-0-2-1
- value: a-0-3-2
- value: b-1-1-0
- value: b-1-2-1
- value: b-1-3-2

It first iterates over the values for alice. For each such value it then iterates over the values of bob.

A comparable way to do this with regular dynaml could look like this:

list: (( sum[alice|[]|s,key,alice|-> s sum[bob|[]|s,index_bob,bob|->s (alice "-" key "-" bob "-" index_bob)]] ))

A result list may omit entries if the value expression evaluates to the undefined value (~~). The nil value (~) is kept. This way a for control can be used to filter lists.

e.g.:

bob:
- 1
- 2
- 3
filtered:
  <<for: 
     bob: (( .bob ))
  <<do: (( bob == 2 ? ~~ :bob ))

resolves to

bob:
- 1
- 2
- 3
filtered:
- 1
- 3

Maps as Iteration Result

If the result should be a map it is required to additionally specify a key value for every iteration. This is specified by the optional <<mapkey field. Like the <<do field it is implicitly handled as template and re-evaluated for every iteration.

e.g.:

x: suffix

alice:
  - a
  - b
bob:
  - 1
  - 2
  - 3

map: 
  <<for:
     - name: alice
       values: (( .alice ))
     - name: bob
       values:  (( .bob ))
  <<mapkey: (( alice bob ))
  <<do:
    value: (( alice bob x ))

evaluates the field map to

map:
  a1:
    value: a1suffix
  a2:
    value: a2suffix
  a3:
    value: a3suffix
  b1:
    value: b1suffix
  b2:
    value: b2suffix
  b3:
    value: b3suffix

Here the traversal order is irrelevant as long as the generated key values are unique. If several evaluations of the key expression yield the same value the last one will win.

A comparable way to do this with regular dynaml could look like this:

map: (( sum[alice|{}|s,index_alice,alice|-> s sum[bob|{}|s,index_bob,bob|->s {(alice bob)=alice bob x}]] ))

An iteration value is ignored if the key or the value evaluate to the undefined value (( ~~ )). Additionally the key may evaluate to the nil value (( ~ )), also.

e.g.:

bob:
  b1: 1
  b2: 2
  b3: 3
filtered:
  <<for: 
     key,bob: (( .bob ))
  <<mapkey: (( key ))
  <<do: (( bob == 2 ? ~~ :bob ))

bob:
  b1: 1
  b2: 2
  b3: 3
filtered:
  <<for: 
     key,bob: (( .bob ))
  <<mapkey: (( bob == 2 ? ~~ :key ))
  <<do: (( bob ))

resolve to

bob:
  b1: 1
  b2: 2
  b3: 3
filtered:
  b1: 1
  b3: 3

`<<merge:`

With merge it is possible to merge maps given as list value of the <<merge field with regular map fields from the control structure to determine the final map value.

The value for <<merge: may be a single map or a list of maps to join with the directly given fields.

e.g.:

map:
  <<merge: 
    - bob: 26
      charlie: 1
    - charlie: 27
  alice: 25
  charlie: 2

resolves to

map:
  alice: 25
  bob: 26
  charlie: 27

If multiple maps contain the same key, the last value (in order of list) will win.

This might be combined with other control structures, for example to conditionally merge multiple maps:

e.g.:

x: charlie
map:
  <<merge: 
    - <<if: (( x == "charlie" ))
      <<then:
        charlie: 27
    - <<if: (( x == "alice" ))
      <<then:
        alice: 20
  alice: 25
  charlie: 2

resolves to

x: charlie
map:
  alice: 25
  charlie: 27

Structural Auto-Merge

By default spiff performs a deep structural merge of its first argument, the template file, with the given stub files. The merge is processed from right to left, providing an intermediate merged stub for every step. This means, that for every step all expressions must be locally resolvable.

Structural merge means, that besides explicit dynaml merge expressions, values will be overridden by values of equivalent nodes found in right-most stub files. In general, flat value lists are not merged. Only lists of maps can be merged by entries in a stub with a matching index.

There is a special support for the auto-merge of lists containing maps, if the maps contain a name field. Hereby the list is handled like a map with entries according to the value of the list entries' name field. If another key field than name should be used, the key field of one list entry can be tagged with the prefix key: to indicate the indended key name. Such tags will be removed for the processed output.

In general the resolution of matching nodes in stubs is done using the same rules that apply for the reference expressions (( foo.bar.[1].baz )).

For example, given the file template.yml:

foo:
  - name: alice
    bar: template
  - name: bob
    bar: template

plip:
  - id: 1
    plop: template
  - id: 2
    plop: template

bar:
  - foo: template

list:
  - a
  - b

and file stub.yml:

foo:
  - name: bob
    bar: stub

plip:
  - key:id: 1
    plop: stub

bar:
  - foo: stub

list:
  - c
  - d

spiff merge template.yml stub.yml

returns

foo:
- bar: template
  name: alice
- bar: stub
  name: bob

plip:
- id: 1
  plop: stub
- id: 2
  plop: template

bar:
- foo: stub

list:
- a
- b

Be careful that any name: key in the template for the first element of the plip list will defeat the key:id: 1 selector from the stub. When a name field exist in a list element, then this element can only be targeted by this name. When the selector is defeated, the resulting value is the one provided by the template.

Bringing it all together

Merging the following files in the given order

deployment.yml

networks: (( merge ))

cf.yml

utils: (( merge ))
network: (( merge ))
meta: (( merge ))

networks:
  - name: cf1
    <<: (( utils.defNet(network.base.z1,meta.deployment_no,30) ))
  - name: cf2
    <<: (( utils.defNet(network.base.z2,meta.deployment_no,30) ))

infrastructure.yml

network:
  size: 16
  block_size: 256
  base:
    z1: 10.0.0.0
    z2: 10.1.0.0

rules.yml

utils:
  defNet: (( |b,n,s|->(*.utils.network).net ))
  network:
    <<: (( &template ))
    start: (( b + n * .network.block_size ))
    first: (( start + ( n == 0 ? 2 :0 ) ))
    lower: (( n == 0 ? [] :b " - " start - 1 ))
    upper: (( start + .network.block_size " - " max_ip(net.subnets.[0].range) ))
    net:
      subnets:
      - range: (( b "/" .network.size ))
        reserved: (( [] lower upper ))
        static:
          - (( first " - " first + s - 1 ))

instance.yml

meta:
  deployment_no: 1

will yield a network setting for a dedicated deployment

networks:
- name: cf1
  subnets:
  - range: 10.0.0.0/16
    reserved:
    - 10.0.0.0 - 10.0.0.255
    - 10.0.2.0 - 10.0.255.255
    static:
    - 10.0.1.0 - 10.0.1.29
- name: cf2
  subnets:
  - range: 10.1.0.0/16
    reserved:
    - 10.1.0.0 - 10.1.0.255
    - 10.1.2.0 - 10.1.255.255
    static:
    - 10.1.1.0 - 10.1.1.29

Using the same config for another deployment of the same type just requires the replacement of the instance.yml. Using a different instance.yml

meta:
  deployment_no: 0

will yield a network setting for a second deployment providing the appropriate settings for a unique other IP block.

networks:
- name: cf1
  subnets:
  - range: 10.0.0.0/16
    reserved:
    - 10.0.1.0 - 10.0.255.255
    static:
    - 10.0.0.2 - 10.0.0.31
- name: cf2
  subnets:
  - range: 10.1.0.0/16
    reserved:
    - 10.1.1.0 - 10.1.255.255
    static:
    - 10.1.0.2 - 10.1.0.31

If you move to another infrastructure you might want to change the basic IP layout. You can do it just by adapting the infrastructure.yml

network:
  size: 17
  block_size: 128
  base:
    z1: 10.0.0.0
    z2: 10.0.128.0

Without any change to your other settings you'll get

networks:
- name: cf1
  subnets:
  - range: 10.0.0.0/17
    reserved:
    - 10.0.0.128 - 10.0.127.255
    static:
    - 10.0.0.2 - 10.0.0.31
- name: cf2
  subnets:
  - range: 10.0.128.0/17
    reserved:
    - 10.0.128.128 - 10.0.255.255
    static:
    - 10.0.128.2 - 10.0.128.31

Useful to Know

There are several scenarios yielding results that do not seem to be obvious. Here are some typical pitfalls.

The auto merge never adds nodes to existing structures

For example, merging

template.yml
```
foo:
 alice: 25
```
with

stub.yml
```
foo:
 alice: 24
 bob: 26
```
yields
```
foo:
 alice: 24
```
Use <<: (( merge )) to change this behaviour, or explicitly add desired nodes to be merged:

template.yml
```
foo:
 alice: 25
 bob: (( merge ))
```
Simple node values are replaced by values or complete structures coming from stubs, structures are deep merged.

For example, merging

template.yml
```
foo: (( ["alice"] ))
```
with

stub.yml
```
foo:
  - peter
  - paul
```
yields
```
foo:
  - peter
  - paul
```
But the template
```
 foo: [ (( "alice" )) ]
```
is merged without any change.
Expressions are subject to be overridden as a whole

A consequence of the behaviour described above is that nodes described by an expession are basically overridden by a complete merged structure, instead of doing a deep merge with the structues resulting from the expression evaluation.

For example, merging

template.yml
```
men:
  - bob: 24
women:
  - alice: 25

people: (( women men ))
```
with

stub.yml
```
people:
  - alice: 13
```
yields
```
men:
  - bob: 24
women:
  - alice: 25

people:
  - alice: 24
```
To request an auto-merge of the structure resulting from the expression evaluation, the expression has to be preceeded with the modifier prefer ((( prefer women men ))). This would yield the desired result:
```
men:
  - bob: 24
women:
  - alice: 25

people:
  - alice: 24
  - bob: 24
```
Nested merge expressions use implied redirections

merge expressions implicity use a redirection implied by an outer redirecting merge. In the following example
```
meta:
 <<: (( merge deployments.cf ))
 properties:
 <<: (( merge ))
 alice: 42
```
the merge expression in meta.properties is implicity redirected to the path deployments.cf.properties implied by the outer redirecting merge. Therefore merging with
```
deployments:
 cf:
 properties:
 alice: 24
 bob: 42
```
yields
```
meta:
 properties:
 alice: 24
 bob: 42
```

Functions and mappings can freely be nested

e.g.:

pot: (( lambda |x,y|-> y == 0 ? 1 :(|m|->m * m)(_(x, y / 2)) * ( 1 + ( y % 2 ) * ( x - 1 ) ) ))
seq: (( lambda |b,l|->map[l|x|-> .pot(b,x)] ))
values: (( .seq(2,[ 0..4 ]) ))

yields the list [ 1,2,4,8,16 ] for the property values.

Functions can be used to parameterize templates

The combination of functions with templates can be use to provide functions yielding complex structures. The parameters of a function are part of the scope used to resolve reference expressions in a template used in the function body.

e.g.:

relation:
  template:
    <<: (( &template ))
    bob: (( x " " y ))
  relate: (( |x,y|->*relation.template ))

banda: (( relation.relate("loves","alice") ))

evaluates to

relation:
  relate: lambda|x,y|->*(relation.template)
  template:
    <<: (( &template ))
    bob: (( x " " y ))

  banda:
    bob: loves alice

Scopes can be used to parameterize templates

Scope literals are also considered when instantiating templates. Therefore they can be used to set explicit values for relative reference expressions used in templates.

e.g.:
```
alice: 1
template:
 <<: (( &template ))
 sum: (( alice + bob ))
scoped: (( ( $alice = 25, "bob" = 26 ) *template ))
```
evaluates to
```
alice: 1
template:
 <<: (( &template ))
 sum: (( alice + bob ))
scoped:
 sum: 51
```
Aggregations may yield complex values by using templates

The expression of an aggregation may return complex values by returning inline lists or instantiated templates. The binding of the function will be available (as usual) for the evaluation of the template. In the example below the aggregation provides a map with both the sum and the product of the list entries containing the integers from 1 to 4.

e.g.:
```
sum: (( sum[[1..4]|init|s,e|->*temp] ))

temp:
 <<: (( &template ))
 sum: (( s.sum + e ))
 prd: (( s.prd * e ))
init:
 sum: 0
 prd: 1
```
yields for sum the value
```
sum:
 prd: 24
 sum: 10
```
Taking advantage of the undefined value

At first glance it might look strange to introduce a value for undefined. But it can be really useful as will become apparent with the following examples.
- Whenever a stub syntactically defines a field it overwrites the default in the template during merging. Therefore it would not be possible to define some expression for that field that eventually keeps the default value. Here the undefined value can help:
  
  e.g.: merging
  
  template.yml
```
alice: 24
bob: 25
```
  with
  
  stub.yml
```
alice: (( config.alice * 2 || ~ ))
bob: (( config.bob * 3 || ~~ ))
```
  yields
```
alice: ~
bob: 25
```
- There is a problem accessing upstream values. This is only possible if the local stub contains the definition of the field to use. But then there will always be a value for this field, even if the upstream does not overwrite it.
  
  Here the undefined value can help by providing optional access to upstream values. Optional means, that the field is only defined, if there is an upstream value. Otherwise it is undefined for the expressions in the local stub and potential downstream templates. This is possible because the field is formally defined, and will therefore be merged, only after evaluating the expression if it is not merged it will be removed again.
  
  e.g.: merging
  
  template.yml
```
alice: 24
bob: 25
peter: 26
```
  with
  
  mapping.yml
```
config:
  alice: (( ~~ ))
  bob: (( ~~ ))

alice: (( config.alice || ~~ ))
bob: (( config.bob || ~~ ))
peter: (( config.peter || ~~ ))
```
  and
  
  config.yml
```
config:
  alice: 4711
  peter: 0815
```
  yields
```
alice: 4711  # transferred from config's config value
bob: 25      # kept default value, because not set in config.yml
peter: 26    # kept, because mapping source not available in mapping.yml
```
This can be used to add an intermediate stub, that offers a dedicated configuration interface and contains logic to map this interface to a manifest structure already defining default values.
Templates versus map literals

As described earlier templates can be used inside functions and mappings to easily describe complex data structures based on expressions refering to parameters. Before the introduction of map literals this was the only way to achieve such behaviour. The advantage is the possibility to describe the complex structure as regular part of a yaml document, which allows using the regular yaml formatting facilitating readability.

e.g.:
```
scaling:
 runner_z1: 10
 router_z1: 4

 jobs: (( sum[scaling|[]|s,k,v|->s [ *templates.job ] ] ))

templates:
 job:
 <<: (( &template ))
 name: (( k ))
 instances: (( v ))
```
evaluates to
```
scaling:
 runner_z1: 10
 router_z1: 4

jobs:
 - instances: 4
 name: router_z1
 - instances: 10
 name: runner_z1
 ...
```
With map literals this construct can significantly be simplified
```
scaling:
 runner_z1: 10
 router_z1: 4

jobs: (( sum[scaling|[]|s,k,v|->s [ {"name"=k, "value"=v} ] ] ))
```
Nevertheless the first, template based version might still be useful, if the data structures are more complex, deeper or with complex value expressions. For such a scenario the description of the data structure as template should be preferred. It provides a much better readability, because every field, list entry and value expression can be put into dedicated lines.

But there is still a qualitative difference. While map literals are part of a single expression always evaluated as a whole before map fields are available for referencing, templates are evaluated as regular yaml documents that might contain multiple fields with separate expressions referencing each other.

e.g.:
```
range: (( (|cidr,first,size|->(*templates.addr).range)("10.0.0.0/16",10,255) ))

templates:
 addr:
 <<: (( &template ))
 base: (( min_ip(cidr) ))
 start: (( base + first ))
 end: (( start + size - 1 ))
 range: (( start " - " end ))
```
evaluates range to
```
range: 10.0.0.10 - 10.0.1.8
...
```
Defaulting and Requiring Fields

Traditionally defaulting in spiff is done by a downstream template where the playload data file is used as stub.

Fields with simple values can just be specified with their values. They will be overwritten by stubs using the regular spiff document merging mechanisms.

It is more difficult for maps or lists. If a map is specified in the template only its fields will be merged (see above), but it is never replaced as a whole by settings in the playload definition files. And Lists are never merged.

Therefore maps and lists that should be defaulted as a whole must be specified as initial expressions (referential or inline) in the template file.

e.g.: merging of

template.yaml
```
defaults:
 <<: (( &temporary ))
 person:
 name: alice
 age: bob
config:
 value1: defaultvalue
 value2: defaultvalue
 person: (( defaults.person ))
```
and

payload.yaml
```
 config:
 value2: configured
 othervalue: I want this but don't get it
```
evaluates to
```
config:
 person:
 age: bob
 name: alice
 value1: defaultvalue
 value2: configured
```
In such a scenario the structure of the resulting document is defined by the template. All kinds of variable fields or sub-structures must be forseen by the template by using <<: (( merge )) expressions in maps.

e.g.: changing template to

template.yaml
```
defaults:
 <<: (( &temporary ))
 person:
 name: alice
 age: bob
config:
 <<: (( merge ))
 value1: defaultvalue
 value2: defaultvalue
 person: (( defaults.person ))
```
Known optional fields can be described using the undefined (~~) expression:

template.yaml
```
config:
 optional: (( ~~ ))
```
Such fields will only be part of the final document if they are defined in an upstream stub, otherwise they will be completely removed.

Required fields can be defined with the expression (( merge )). If no stub contains a value for this field, the merge cannot be fullfilled and an error is reported. If a dedicated message should be shown instead, the merge expression can be defaulted with an error function call.

e.g.:

template.yaml
```
config:
 password: (( merge || error("the field password is required") ))
```
will produce the following error if no stub contains a value:
```
error generating manifest: unresolved nodes:
	(( merge || error("the field password is required") ))	in c.yaml	config.password	()	*the field password is required 
```
This can be simplified by reducing the expression to the sole error expression.

Besides this template based defaulting it is also possible to provide defaults by upstream stubs using the &default marker. Here the payload can be a downstream file.

X509 and providing State

When generating keys or certificates with the X509 Functions there will be new keys or certificates for every execution of spiff. But it is also possible to use spiff to maintain key state. A very simple script could look like this:

#!/bin/bash
DIR="$(dirname "$0")/state"
if [ ! -f "$DIR/state.yaml" ]; then
  echo "state:" > "$DIR/state.yaml"
fi
spiff merge "$DIR/template.yaml" "$DIR/state.yaml" > "$DIR/.$$" && mv "$DIR/.$$" "$DIR/state.yaml"

It uses a template file (containing the rules) and a state file with the actual state as stub. The first time it is executed there is an empty state and the rules are not overridden, therefore the keys and certificates are generated. Later on, only additional new fields are calculated, the state fields already containing values just overrule the dynaml expressions for those fields in the template.

If a re-generation is required, the state file can just be deleted.

A template may look like this:

state/template.yaml

spec:
  <<: (( &local ))
  ca:
    organization: Mandelsoft
    commonName: rootca
    privateKey: (( state.cakey ))
    isCA: true
    usage:
      - Signature
      - KeyEncipherment
  peer:
    organization: Mandelsoft
    commonName: etcd
    publicKey: (( state.pub ))
    caCert: (( state.cacert ))
    caPrivateKey: (( state.cakey ))
    validity: 100
    usage:
      - ServerAuth
      - ClientAuth
      - KeyEncipherment
    hosts:
      - etcd.mandelsoft.org

state:
  cakey: (( x509genkey(2048) ))
  capub: (( x509publickey(cakey) ))

  cacert: (( x509cert(spec.ca) ))

  key: (( x509genkey(2048) ))
  pub: (( x509publickey(key) ))
  peer: (( x509cert(spec.peer) ))

The merge then generates a rootca and some TLS certificate signed with this CA.

Generating, Deploying and Accessing Status for Kubernetes Resources

The sync function offers the possibility to synchronize the template processing with external content. This can also be the output of a command execution. Therefore the template processing can not only be used to generate a deployment manifest, but also for applying this to a target system and retrieving deployment status values for the further processing.

A typical scenario of this kind could be a kubernetes setup including a service of type LoadBalancer. Once deployed it gets assigned status information about the IP address or hostname of the assigned load balancer. This information might be required for some other deployment manifest.

A simple template for such a deployment could like this:
```
service:
  apiVersion: v1
  kind: Service
  metadata:
    annotations:
      dns.mandelsoft.org/dnsnames: echo.test.garden.mandelsoft.org
      dns.mandelsoft.org/ttl: "500"
    name: test-service
    namespace: default
  spec:
    ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 8080
    sessionAffinity: None
    type: LoadBalancer

deployment:
   testservice: (( sync[pipe_uncached(service, "kubectl", "apply", "-f", "-", "-o", "yaml")|value|->defined(value.status.loadBalancer.ingress)] ))


otherconfig:
   lb: (( deployment.testservice.status.loadBalancer.ingress ))
```

Crazy Shit: Graph Analaysis with spiff

It is easy to describe a simple graph with knots and edges (for example for a set of components and their dependencies) just by using a map of lists.

<details><summary>graph.yaml</summary>

graph:
  a:
  - b
  - c
  b: []
  c:
  - b
  - a
  d:
  - b
  e:
  - d
  - b

</details>

Now it would be useful to figure out whether there are dependency cycles or to determine ordered transitive dependencies for a component.

Let's say something like this:

<details><summary>closures.yaml</summary>

graph:
utilities:

closures: (( utilities.graph.evaluate(graph) ))
cycles: (( utilities.graph.cycles(closures) ))

</details>

Indeed, this can be done with spiff. The only thing required is a "small utilities stub".

<details><summary>utilities.yaml</summary>

utilities:
  <<: (( &temporary ))
  graph:
    _dep: (( |model,comp,closure|->contains(closure,comp) ? { $deps=[], $err=closure [comp]} :($deps=_._deps(model,comp,closure [comp]))($err=sum[deps|[]|s,e|-> length(s) >= length(e.err) ? s :e.err]) { $deps=_.join(map[deps|e|->e.deps]), $err=err} ))
    _deps: (( |model,comp,closure|->map[model.[comp]|dep|->($deps=_._dep(model,dep,closure)) { $deps=[dep] deps.deps, $err=deps.err }] ))
    join: (( |lists|->sum[lists|[]|s,e|-> s e] ))
    min: (( |list|->sum[list|~|s,e|-> s ? e < s ? e :s :e] ))

    normcycle: (( |cycle|->($min=_.min(cycle)) min ? sum[cycle|cycle|s,e|->s.[0] == min ? s :(s.[1..] [s.[1]])] :cycle  ))
    cycle: (( |list|->list ? ($elem=list.[length(list) - 1]) _.normcycle(sum[list|[]|s,e|->s ? s [e] :e == elem ? [e] :s]) :list ))
    norm: (( |deps|->{ $deps=_.reverse(uniq(_.reverse(deps.deps))), $err=_.cycle(deps.err) } ))
    reverse: (( |list|->sum[list|[]|s,e|->[e] s] ))

    evaluate: (( |model|->sum[model|{}|s,k,v|->s { k=_.norm(_._dep(model,k,[]))}] ))
    cycles: (( |result|->uniq(sum[result|[]|s,k,v|-> v.err ? s [v.err] :s]) ))

</details>

And magically spiff does the work just by calling

spiff merge closure.yaml graph.yaml utilities.yaml

<details><summary>And the result is</summary>

   closures:
     a:
       deps:
       - c
       - b
       - a
       err:
       - a
       - c
       - a
     b:
       deps: []
       err: []
     c:
       deps:
       - a
       - b
       - c
       err:
       - a
       - c
       - a
     d:
       deps:
       - b
       err: []
     e:
       deps:
       - d
       - b
       err: []
   cycles:
   - - a
     - c
     - a
   graph:
     a:
     - b
     - c
     b: []
     c:
     - b
     - a
     d:
     - b
     e:
     - d
     - b

</details>

Error Reporting

The evaluation of dynaml expressions may fail because of several reasons:

it is not parseable
involved references cannot be satisfied
arguments to operations are of the wrong type
operations fail
there are cyclic dependencies among expressions

If a dynaml expression cannot be resolved to a value, it is reported by the spiff merge operation using the following layout:

	(( <failed expression> ))	in <file>	<path to node>	(<referred path>)	<tag><issue>

<details><summary>Example</summary>

	(( min_ip("10") ))	in source.yml	node.a.[0]	()	*CIDR argument required

</details>

Cyclic dependencies are detected by iterative evaluation until the document is unchanged after a step. Nodes involved in a cycle are therefore typically reported just as unresolved node without a specific issue.

The order of the reported unresolved nodes depends on a classification of the problem, denoted by a dedicated tag. The following tags are used (in reporting order):

Tag	Meaning
`*`	error in local dynaml expression
`@`	dependent or involved in cyclic dependencies
`-`	subsequent error because of refering to a yaml node with an error

Problems occuring during inline template processing are reported as nested problems. The classification is propagated to the outer node.

If a problem occurs in nested lamba calls the call stack together with the lamba function and is local binding is listed.

<details><summary>Example</summary>

	(( 2 + .func(2) ))	in local/err.yaml	value	()	*evaluation of lambda expression failed: lambda|x|->x > 0 ? _(x - 1) : *(template): {x: 2}
		... evaluation of lambda expression failed: lambda|x|->x > 0 ? _(x - 1) : *(template): {x: 1}
		... evaluation of lambda expression failed: lambda|x|->x > 0 ? _(x - 1) : *(template): {x: 0}
		... resolution of template 'template' failed
			(( z ))	in local/err.yaml	val	()*'z' not found

</details>

In case of parsing errors in dynaml expressions, the error location is shown now. If it is a multi line expression the line a character/symbol number in that line is show, otherwise the line numer is omitted.

<details><summary>Example</summary>

	((
	  2 ++ .func(2)
	))	in local/err.yaml	faulty	()	*parse error near line 2 symbol 2 - line 2 symbol 3: " "

</details>

Using spiff as Go Library

Spiff provides a Go package (spiffing) that can be used to include spiff templates in Go programs.

An example program could look like this:

import (
	"fmt"
	"math"
	"os"

	"github.com/mandelsoft/spiff/dynaml"
	"github.com/mandelsoft/spiff/spiffing"
)

func func_pow(arguments []interface{}, binding dynaml.Binding) (interface{}, dynaml.EvaluationInfo, bool) {
	info := dynaml.DefaultInfo()

	if len(arguments) != 2 {
		return info.Error("pow takes 2 arguments")
	}

	a, b, err := dynaml.NumberOperands(arguments[0], arguments[1])

	if err != nil {
		return info.Error("%s", err)
	}
	_, i := a.(int64)
	if i {
		r := math.Pow(float64(a.(int64)), float64(b.(int64)))
		if float64(int64(r)) == r {
			return int64(r), info, true
		}
		return r, info, true
	} else {
		return math.Pow(a.(float64), b.(float64)), info, true
	}
}

var state = `
state: {}
`
var stub = `
unused: (( input ))
ages:
  alice: (( pow(2,5) ))
  bob: (( alice + 1 ))
`

var template = `
state:
  <<<: (( &state ))
  random: (( rand("[:alnum:]", 10) )) 
ages: (( &temporary ))

example:
  name: (( input ))  # direct reference to additional values 
  sum: (( sum[ages|0|s,k,v|->s + v] ))
  int: (( pow(2,4) ))
  float: 2.1
  pow: (( pow(1.1e1,2.1) ))
`

func Error(err error) {
	if err != nil {
		fmt.Fprintf(os.Stderr, "Error: %s\n", err)
		os.Exit(1)
	}
}

func main() {
	values := map[string]interface{}{}
	values["input"] = "this is an input"

	functions := spiffing.NewFunctions()
	functions.RegisterFunction("pow", func_pow)

	spiff, err := spiffing.New().WithFunctions(functions).WithValues(values)
	Error(err)
	pstate, err := spiff.Unmarshal("state", []byte(state))
	Error(err)
	pstub, err := spiff.Unmarshal("stub", []byte(stub))
	Error(err)
	ptempl, err := spiff.Unmarshal("template", []byte(template))
	Error(err)
	result, err := spiff.Cascade(ptempl, []spiffing.Node{pstub}, pstate)
	Error(err)
	b, err := spiff.Marshal(result)
	Error(err)
	newstate, err := spiff.Marshal(spiff.DetermineState(result))
	Error(err)
	fmt.Printf("==== new state ===\n")
	fmt.Printf("%s\n", string(newstate))
	fmt.Printf("==== result ===\n")
	fmt.Printf("%s\n", string(b))
}

It supports

transforming file data to and from spiffs internal node representation
the processing of stubs and templates with or without state handling
defining an outer binding for injected path names
defining additional spiff functions
enabling/disabling command execution and/or filesystem operations
using a virtual filesystem for file system operations

Awesome

Installation

Usage

spiff merge template.yml [template2.yml ...]

spiff diff manifest.yml other-manifest.yml

spiff convert --json manifest.yml

spiff encrypt secret.yaml

Feature Flags

Libraries

dynaml Templating Language

(( foo ))

(( foo.bar.[1].baz ))

(( foo.[bar].baz ))

(( list.[1..3] ))

(( 1.2e4 ))

(( "foo" ))

(( [ 1, 2, 3 ] ))

(( { "alice" = 25 } ))

(( ( "alice" = 25 ) alice ))

(( foo bar ))

(( "foo" bar ))

(( [1,2] bar ))

(( map1 map2 ))

(( auto ))

(( merge ))

<<: (( merge ))

Merging maps

Merging lists

- <<: (( merge on key ))

<<: (( merge replace ))

Merging maps

Merging lists

<<: (( foo ))

Merging maps

Merging lists

<<: (( merge foo ))

Merging maps

Merging lists

<<: (( merge none ))

(( a || b ))

(( 1 + 2 * foo ))

(( "10.10.10.10" - 11 ))

(( a > 1 ? foo :bar ))

(( 5 -or 6 ))

Functions

(( format( "%s %d", alice, 25) ))

(( join( ", ", list) ))

(( split( ",", string) ))

(( trim(string) ))

(( element(list, index) ))

(( element(map, key) ))

(( compact(list) ))

(( uniq(list) ))

(( contains(list, "foobar") ))

(( basename(path) ))

(( dirname(path) ))

(( parseurl("http://github.com") ))

(( index(list, "foobar") ))

(( lastindex(list, "foobar") ))

`(( sort(list) ))

(( replace(string, "foo", "bar") ))

(( substr(string, 1, 2) ))

(( match("(f.*)(b.*)", "xxxfoobar") ))

(( keys(map) ))

(( length(list) ))

(( base64(string) ))

(( hash(string) ))

(( bcrypt("password", 10) ))

(( bcrypt_check("password", hash) ))

(( md5crypt("password") ))

(( md5crypt_check("password", hash) ))

(( decrypt("secret") ))

(( rand("[:alnum:]", 10) ))

(( type(foobar) ))

(( defined(foobar) ))

(( valid(foobar) ))

(( require(foobar) ))

(( stub(foo.bar) ))

(( tagdef("tag", value) ))

(( eval(foo "." bar ) ))

`spiff merge template.yml [template2.yml ...]`

`spiff diff manifest.yml other-manifest.yml`

`spiff convert --json manifest.yml`

`spiff encrypt secret.yaml`

`(( foo ))`

`(( foo.bar.[1].baz ))`

`(( foo.[bar].baz ))`

`(( list.[1..3] ))`

`(( 1.2e4 ))`

`(( "foo" ))`

`(( [ 1, 2, 3 ] ))`

`(( { "alice" = 25 } ))`

`(( ( "alice" = 25 ) alice ))`

`(( foo bar ))`

`(( "foo" bar ))`

`(( [1,2] bar ))`

`(( map1 map2 ))`

`(( auto ))`

`(( merge ))`

`<<: (( merge ))`

`- <<: (( merge on key ))`

`<<: (( merge replace ))`

`<<: (( foo ))`

`<<: (( merge foo ))`

`<<: (( merge none ))`

`(( a || b ))`

`(( 1 + 2 * foo ))`

`(( "10.10.10.10" - 11 ))`

`(( a > 1 ? foo :bar ))`

`(( 5 -or 6 ))`

`(( format( "%s %d", alice, 25) ))`

`(( join( ", ", list) ))`

`(( split( ",", string) ))`

`(( trim(string) ))`

`(( element(list, index) ))`

`(( element(map, key) ))`

`(( compact(list) ))`

`(( uniq(list) ))`

`(( contains(list, "foobar") ))`

`(( basename(path) ))`

`(( dirname(path) ))`

`(( parseurl("http://github.com") ))`

`(( index(list, "foobar") ))`

`(( lastindex(list, "foobar") ))`

`(( replace(string, "foo", "bar") ))`

`(( substr(string, 1, 2) ))`

`(( match("(f.)(b.)", "xxxfoobar") ))`

`(( keys(map) ))`

`(( length(list) ))`

`(( base64(string) ))`

`(( hash(string) ))`

`(( bcrypt("password", 10) ))`

`(( bcrypt_check("password", hash) ))`

`(( md5crypt("password") ))`

`(( md5crypt_check("password", hash) ))`

`(( decrypt("secret") ))`

`(( rand("[:alnum:]", 10) ))`

`(( type(foobar) ))`

`(( defined(foobar) ))`

`(( valid(foobar) ))`

`(( require(foobar) ))`

`(( stub(foo.bar) ))`

`(( tagdef("tag", value) ))`

`(( eval(foo "." bar ) ))`

`(( env("HOME" ) ))`

`(( parse(yamlorjson) ))`

`(( asjson(expr) ))`

`(( catch(expr) ))`

`(( static_ips(0, 1, 3) ))`

`(( ipset(ranges, 3, 3,4,5,6) ))`

`(( list_to_map(list, "key") ))`

`(( makemap(fieldlist) ))`

`(( makemap(key, value) ))`

`(( merge(map1, map2) ))`

`(( intersect(list1, list2) ))`

`(( reverse(list) ))`

`(( validate(value,"dnsdomain") ))`

`(( check(value,"dnsdomain") ))`

`(( error("message") ))`

`(( read("file.yml") ))`