Home

Awesome

Go Report Card license Release

marionette

marionette is a simple command-line application which is designed to carry out system automation tasks. It was designed to resemble the well-known configuration-management application puppet, albeit in a much simpler and more minimal manner.

marionette contains a small number of built-in modules, providing enough primitives to turn a blank virtual machine into a host running a few services:

In the future it is possible that more modules will be added, but this will require users to file bug-reports requesting them, contribute code, or the author realizing something is necessary.

Installation & Usage

Binaries for several systems are available upon our download page. If you prefer to use something more recent you can install directly from the repository via:

go install github.com/skx/marionette@latest

The main application can then be launched with the path to a set of rules, which it will then try to apply:

marionette [flags] ./rules.txt ./rules2.txt ... ./rulesN.txt

The following flags are supported:

Typically a user would run with -verbose, and a developer might examine the output produced when -debug is specified.

In addition to the general-purpose flags -dp and -dl exist for developers, to dump the output of the parser and lexer, respectively.

Rule Definition

The general form of our rules looks like this:

$MODULE [triggered] {
            name  => "NAME OF RULE",
            arg_1 => "Value 1 ... ",
            arg_2 => [ "array values", "are fine" ],
            arg_3 => "Value 3 .. ",
}

Each rule starts by declaring the type of module which is being invoked, then there is a block containing "key => value" sections. Different modules will accept/expect different keys to configure themselves. (Unknown arguments will generally be ignored.)

A rule may also contain an optional triggered attribute. Rules which contain the triggered modifier are not executed unless explicitly invoked by another rule - think of it as a "handler" if you're used to ansible.

Here is an example rule which executes a shell-command:





# Run a command, unconditionally
shell {
        command => "uptime > /tmp/uptime.txt"
}

Another simple example to illustrate the available syntax might look like the following, which ensures that I have ~/bin/ owned by myself:

directory {
             target  => "/home/${USER}/bin",
             mode    => "0755",
             owner   => "${USER}",
             group   => "${USER}",
}

There are four magical keys which can be supplied to all modules:

NameUsage
requireThis is used for dependency management
notifyThis is used for dependency management
ifThis is used to make a rule conditional
unlessThis is used to make a rule conditional

Dependency Management

There are two keys which can be used to link rules together, to handle dependencies:

Note You only need to give rules names to link them for the purpose of managing dependencies.

Imagine we wanted to create a new directory, and write a file there. We could do that with a pair of rules:

We could wing-it and write the rules in the logical order, but it would be far better to link the two rules explicitly.

There are two ways we could implement this. The simplest way would be this:

shell { command   => "uptime > /tmp/blah/uptime",
        require   => "Create /tmp/blah" }

directory{ name   => "Create /tmp/blah",
           target => "/tmp/blah" }

The alternative would have been to have the directory-creation trigger the shell-execution rule via an explicit notification:





# This command will notify the "Test" rule, if it creates the directory




# because it was not already present.
directory{ target => "/tmp/blah",
           notify => "Test"
}




# Run the command, when triggered/notified.
shell triggered { name         => "Test",
                  command      => "uptime > /tmp/blah/uptime",
}

The difference in these two approaches is how often things run:

You'll note that any rule which is followed by the token triggered will only be executed when it is triggered by name. If there is no notify key referring to that rule it will never be executed.

Conditionals

Rules may be made conditional, via the magical keys if and unless.

The following example runs a command, using apt-get, only if a specific file exists upon the filesystem:

shell { name    => "Upgrade",
        command => "apt-get dist-upgrade --yes --force-yes",
        if      => exists("/usr/bin/apt-get") }

For the reverse, running a rule unless something is true, we can use the unless key:

let arch = `/usr/bin/arch`

file { name   => "Create file",
       target => "/tmp/foo",
       unless => equal( "x86_64", "${arch}" ) }

Here we see that we've used two functions equal and exists, these are both built-in functions which do what you'd expect.

The following list shows all the built-in functions that you may use (but only within the if or unless keys):

More conditional primitives may be added if they appear to be necessary, or if users request them.

Conditionals may also be applied to variable assignments and file inclusion:





# Include a file of rules, on a per-arch basis
include "x86_64.rules" if equal( "${ARCH}","x86_64" )
include "i386.rules"   if equal( "${ARCH}","i386" )




# Setup a ${cmd} to download something, depending on what is present.
let cmd = "curl --output ${dst} ${url}" if on_path("curl")
let cmd = "wget -O ${dst} ${url}"       if on_path("wget")

In addition to these conditional functions the following primitives are built in, and may be freely used:

Examples

You can find a small set of example recipes beneath the examples directory:

Misc. Features

Command Execution

Backticks can be used to execute commands, in variable-assignments and in parameters to rules.

For example we might determine the system architecture like this:

let arch = `/usr/bin/arch`

shell { name    => "Show arch",
        command => "echo We are running on an ${arch} system" }

Here ${arch} expands to the output of the command, as you would expect, with any trailing newline removed.

Note ${ARCH} is available by default, as noted in the pre-declared variables section. This was just an example of command-execution.

Using commands inside parameter values is also supported:

file { name    => "set-todays-date",
       target  => "/tmp/today",
       content => `/usr/bin/date` }

The commands executed with the backticks have any embedded variables expanded before they run, so this works as you'd expect:

let fmt   = "+%Y"

file { name    => "set-todays-date",
       target  => "/tmp/today",
       content => `/bin/date ${fmt}` }

Include Files

You can break large rule-files into pieces, and include them in each other:





# main.in

let prefix="/etc/marionette"

include "foo.in"
include "${prefix}/test.in"

To simplify your recipe writing including other files may be made conditional, just like our rules:





# main.in

include "x86_64.rules" if equal( "${ARCH}","x86_64" )
include "i386.rules"   if equal( "${ARCH}","i386" )

Pre-Declared Variables

The following variables are available by default:

NameValue
${ARCH}The system architecture (as taken from sys.GOARCH).
${HOMEDIR}The home directory of the user running marionette.
${HOSTNAME}The hostname of the local system.
${OS}The operating system name (as taken from sys.GOOS).
${USERNAME}The username of user running marionette.

There are additionally two "magic" variables available which will always have values based upon the current rule-file being processed, whether that is a file specified upon the command-line, or as a result of an include statement:

NameValue
${INCLUDE_DIR}The absolute directory path of the current file being processed.
${INCLUDE_FILE}The absolute path of the current file being processed.

Outputs

Some modules will set "outputs" after they've executed, and those outputs will be documented explicitly in the later list of available modules.

When a module creates an output it will be available for subsequent modules to use, prefixed with the name of the rule which created it.

Here is an example showing the use of the stdout output which the shell module produces:





# Run a command - This will produce "${user.stdout}" and "${user.stderr}"




# variables which can be used later.
shell {
           name => "user",
        command => "/usr/bin/whoami"
}


log {
    message => "STEVE!",
    if      => equal( "${user.stdout}", "skx" )
}
log {
    message => "ROOT!",
    if      => equal( "${user.stdout}", "root" )
}

NOTE: The output stdout is available here as ${user.stdout} - the "user." prefix comes from the name of the rule which invoked the shell. So here we'd be able to process the result of ${kernel-version.output}

shell {
          name => "kernel-version",
          command => "uname -r",
}

Module Types

Our primitives are implemented in 100% pure golang, and are included with our binary, these are now described briefly:

directory

The directory module allows you to create a directory, or change the permissions of one.

Example usage:

directory {  name    => "My home should have a binary directory",
             target  => "/home/steve/bin",
             mode    => "0755",
}

Valid parameters are:

docker

This module allows fetching a container from a remote registry.

docker { image => "alpine:latest" }

The following keys are supported:

NOTE: We don't support private registries, or the use of authentication.

edit

This module allows minor edits to be applied to a file:

edit { name => "Remove my VPN hosts",
       target => "/etc/hosts",
       remove_lines => "\.vpn" }

The following keys are supported:

An example of changing a file might look like this:

edit { target  => "/etc/ssh/sshd_config",
       search  => "^PasswordAuthentication",
       replace => "# PasswordAuthentication",
}

fail

The fail-module is designed to terminate processing, if you find a situation where the local environment doesn't match your requirements. For example:

let path = `which useradd`

fail {
   message => "I can't find a working useradd binary to use",
   if      => empty(path)
}

The only valid parameter is message.

See also log, which will log a message but then continue execution.

file

The file module allows a file to be created, from a local file, or via a remote HTTP-source.

Example usage:

file {  name       => "fetch file",
        target     => "/tmp/steve.txt",
        source_url => "https://steve.fi/",
}

file {  name     => "write file",
        target   => "/tmp/name.txt",
        content  => "My name is Steve",
}

target is a mandatory parameter, and specifies the file to be operated upon.

There are four ways a file can be created:

Other valid parameters are:

Where template is used, the template file is rendered using the text/template Go package

git

Clone a remote repository to a local directory.

Example usage:

git { path => "/tmp/xxx",
      repository => "https://github.com/src-d/go-git",
}

Valid parameters are:

If this module is used to notify another then it will trigger such a notification if either:

group

The group module allows you to add or remove local Unix groups to your system.

Example:

group { group => "sysadmin",
        state => "present" }

http

The http module allows you to make HTTP requests.

Example:

http { url => "https://api.github.com/user",
       method => "PATCH",
       headers => [
         "Authorization: token ${AUTH_TOKEN}",
         "Accept: application/vnd.github.v3+json",
       ],
       body => "{\"name\":\"new name\"}",
       expect => "200" }

Valid parameters are:

The http module is always regarded as having made a change on a successful request.

http Outputs

The following outputs will be set:

link

The link module allows you to create a symbolic link.

Example usage:

link { name => "Symlink test",
       source => "/etc/passwd",  target => "/tmp/password.txt" }

Valid parameters are:

log

The log module allows you to output a message, but continue execution.

Example usage:

log { message => "I'm ${USER} running on ${HOSTNAME}" }

The only valid parameter is message, which may be either a single value, or an array of values.

See also fail, which will log a message but then terminate execution.

package

The package-module allows you to install or remove a package from your system, via the execution of apt-get, dpkg, and yum, as appropriate.

Example usage:





# Install a single package
package { name    => "Install bash",
          package => "bash",
          state   => "installed",
        }




# Uninstall a series of packages
package { package => [ "nano", "vim-tiny", "nvi" ],
          state => "absent" }

Valid parameters are:

sql

The SQL-module allows you to run arbitrary SQL against a database. Two parameters are required driver and dsn, which are used to open the database connection. For example:

sql {
       driver => "sqlite3",
       dsn    => "/tmp/sql.db",

       # Memory based-SQLite could be used like so:
       #  dsn   => "/tmp/foo.db?mode=memory",

       # MySQL connection would look like this:
       #
       #   driver => "mysql",
       #   dsn    => "user:password@(127.0.0.1:3306)/",
}

We support the following drivers:

To specify the query to run you should set one of the following two parameters:

NOTE: You may find you need to append multiStatements=true to your DSN to ensure correct operation when reading SQL from a file.

shell

The shell module allows you to run shell-commands, complete with redirection and pipes.

Example:

shell { name    => "I touch your file.",
        command => "touch /tmp/blah/test.me"
      }

command is the only mandatory parameter. Multiple values can be specified:

shell { name    => "restart the aplication",
        command => [
                     "systemctl stop  foo.service",
                     "systemctl start foo.service",
                   ]
      }

By default commands are executed directly, unless they contain redirection-characters (">", or "<"), or the use of a pipe ("|"). If special characters are used then we instead invoke the command via /bin/bash:

You may specify shell => true to force the use of a shell, despite the lack of redirection/pipe characters:

shell { shell   => true,
        command => "sed .. /etc/file.txt"
      }

shell Outputs

The following outputs will be set:

user

The user module allows you to add or remove local Unix users to your system.

Example:

user { login => "steve",
       state => "present" }

Future Plans

See Also

Github Setup

This repository is configured to run tests upon every commit, and when pull-requests are created/updated. The testing is carried out via .github/run-tests.sh which is used by the github-action-tester action.

Releases are automated in a similar fashion via .github/build, and the github-action-publish-binaries action.

Steve