Home

Awesome

clitest – Command Line Tester

clitest is a portable POSIX shell script that performs automatic testing in Unix command lines.

It's the same concept as in Python's doctest module: you document both the commands and their expected output, using the familiar interactive prompt format, and a specialized tool tests them.

In fact, the doctest official description can also be used for clitest:

Download & install

The full program is just a single shell script file.

Save it, make it executable and move it to a $PATH directory:

curl -sOL https://raw.githubusercontent.com/aureliojargas/clitest/master/clitest
chmod +x clitest
sudo mv clitest /usr/bin

Now check if everything is fine:

clitest --help

Docker image

You can also run clitest in a Docker container (more info in Docker Hub).

docker run --rm -t aureliojargas/clitest --help

Quick Intro

Save the commands and their expected output in a text file:

examples/intro.txt

$ echo "Hello World"
Hello World
$ cd /tmp
$ pwd
/tmp
$ cd "$OLDPWD"
$

Use clitest to run these commands and check their output:

$ clitest examples/intro.txt
#1	echo "Hello World"
#2	cd /tmp
#3	pwd
#4	cd "$OLDPWD"
OK: 4 of 4 tests passed
$

CLI Syntax

There's no syntax to learn.

The test files are identical to the good old command line interface (CLI) you're so familiar:

examples/cut.txt

$ echo "one:two:three:four:five:six" | cut -d : -f 1
one
$ echo "one:two:three:four:five:six" | cut -d : -f 4
four
$ echo "one:two:three:four:five:six" | cut -d : -f 1,4
one:four
$ echo "one:two:three:four:five:six" | cut -d : -f 4,1
one:four
$ echo "one:two:three:four:five:six" | cut -d : -f 1-4
one:two:three:four
$ echo "one:two:three:four:five:six" | cut -d : -f 4-
four:five:six
$

That's it.

Just paste your shell session inside a text file and you have a ready-to-use test suite.

$ clitest examples/cut.txt
#1	echo "one:two:three:four:five:six" | cut -d : -f 1
#2	echo "one:two:three:four:five:six" | cut -d : -f 4
#3	echo "one:two:three:four:five:six" | cut -d : -f 1,4
#4	echo "one:two:three:four:five:six" | cut -d : -f 4,1
#5	echo "one:two:three:four:five:six" | cut -d : -f 1-4
#6	echo "one:two:three:four:five:six" | cut -d : -f 4-
OK: 6 of 6 tests passed
$

There are more examples and instructions in the examples folder. For a real-life collection of hundreds of test files, see funcoeszz test files.

Testable Documentation

Clitest can also extract and run command lines from documentation, such as Markdown files. This very README.md file you are now reading is testable with clitest README.md. All the command lines inside it will be run and checked.

No more malfunctioning shell commands in your READMEs, you can have testable documentation.

Given the following Markdown sample document:

examples/cut.md

The numeric ranges of the Unix command "cut"
============================================

Use single numbers to extract one specific field:

	$ echo "one:two:three:four:five:six" | cut -d : -f 1
	one
	$ echo "one:two:three:four:five:six" | cut -d : -f 4
	four

Use commas to inform more than one field:

	$ echo "one:two:three:four:five:six" | cut -d : -f 1,4
	one:four

Note that inverting the order will *not* invert the output:

	$ echo "one:two:three:four:five:six" | cut -d : -f 4,1
	one:four

Use an hyphen to inform a range of fields, from one to four:

	$ echo "one:two:three:four:five:six" | cut -d : -f 1-4
	one:two:three:four

If you omit the second range number, it matches until the last:

	$ echo "one:two:three:four:five:six" | cut -d : -f 4-
	four:five:six

cut is cool, isn't it?

It is a technical article, not a boring code-only test file. You can read its final (formatted) version here.

You can give this article to clitest, who will identify all the shell command lines inside it, run them and check if the results are the same.

$ clitest --prefix tab examples/cut.md
#1	echo "one:two:three:four:five:six" | cut -d : -f 1
#2	echo "one:two:three:four:five:six" | cut -d : -f 4
#3	echo "one:two:three:four:five:six" | cut -d : -f 1,4
#4	echo "one:two:three:four:five:six" | cut -d : -f 4,1
#5	echo "one:two:three:four:five:six" | cut -d : -f 1-4
#6	echo "one:two:three:four:five:six" | cut -d : -f 4-
OK: 6 of 6 tests passed
$

Note the use of --prefix tab option, to inform clitest that the code blocks are prefixed by a tab in this Markdown file. For files with 4-spaces indented code blocks, use --prefix 4. When using non-indented fenced code blocks (```), such as this README.md, no prefix option is needed.

Examples of testable documentation handled by clitest:

Alternative Syntax: Inline Output

Now a nice extension to the original idea. Using the special marker #=> you can embed the expected command output at the end of the command line.

$ echo "foo"                      #=> foo
$ echo $((10 + 2))                #=> 12

This is the same as doing:

$ echo "foo"
foo
$ echo $((10 + 2))
12
$

Inline outputs are very readable when testing series of commands that result in short texts.

$ echo "abcdef" | cut -c 1        #=> a
$ echo "abcdef" | cut -c 4        #=> d
$ echo "abcdef" | cut -c 1,4      #=> ad
$ echo "abcdef" | cut -c 1-4      #=> abcd

Note: If needed, you can change this marker (i.e., to #→ or ###) at the top of the script or using the --inline-prefix option.

Advanced Tests

When using the #=> marker, you can take advantage of special options to change the default output matching method.

$ head /etc/passwd            #=> --lines 10
$ tac /etc/passwd | tac       #=> --file /etc/passwd
$ cat /etc/passwd             #=> --egrep ^root:
$ echo $((2 + 10))            #=> --regex ^\d+$
$ make test                   #=> --exit 0
$ pwd                         #=> --eval echo $PWD

Options

$ clitest --help
Usage: clitest [options] <file ...>

Options:
  -1, --first                 Stop execution upon first failed test
  -l, --list                  List all the tests (no execution)
  -L, --list-run              List all the tests with OK/FAIL status
  -t, --test RANGE            Run specific tests, by number (1,2,4-7)
  -s, --skip RANGE            Skip specific tests, by number (1,2,4-7)
      --pre-flight COMMAND    Execute command before running the first test
      --post-flight COMMAND   Execute command after running the last test
  -q, --quiet                 Quiet operation, no output shown
  -V, --version               Show program version and exit

Customization options:
  -P, --progress TYPE         Set progress indicator: test, number, dot, none
      --color WHEN            Set when to use colors: auto, always, never
      --diff-options OPTIONS  Set diff command options (default: '-u')
      --inline-prefix PREFIX  Set inline output prefix (default: '#=> ')
      --prefix PREFIX         Set command line prefix (default: '')
      --prompt STRING         Set prompt string (default: '$ ')
$

Exit codes

Fail fast

Use the --first option (or the short version -1) to abort the execution when any test fails.

Useful for Continuous Integration (CI), or when running sequential tests where the next test depends on the correct result of the previous.

Quiet operation

When automating the tests execution, use --quiet to show no output and just check the exit code to make sure all tests have passed. Using --first to fail fast is also a good idea in this case.

if clitest --quiet --first tests.txt
then
    # all tests passed
else
    # one or more tests failed :(
fi

Run specific tests

To rerun a specific problematic test, or to limit the execution to a set of tests, use --test. To ignore one or more tests, use --skip. If needed, you can combine both options to inform a very specific test range. Examples:

clitest --test 1-10    tests.txt   # Run the first 10 tests
clitest --test 1,2,6-8 tests.txt   # Run tests #1, #2, #6, #7 and #8
clitest --skip 11,15   tests.txt   # Run all tests, except #11 and #15
clitest -t 1-10 -s 5   tests.txt   # Run first 10 tests, but skip #5

Pre/post scripts

You can run a preparing script or command before the first test with --pre-flight, for setting env variables and create auxiliary files. At the end of all tests, run a final cleanup script/command with --post-flight to remove temporary files or other transient data.

clitest --pre-flight ./test-init.sh --post-flight 'rm *.tmp' tests.txt

Customization

Use the customization options to extract and test command lines from documents or wiki pages. For example, to test all the command line examples listed inside a Markdown file using the 4-spaces syntax for code blocks:

clitest --prefix 4 README.md

Or maybe you use a different prompt ($PS1) in your documentation?

clitest  --prefix 4 --prompt '[john@localhost ~]$ ' README.md

Nerdiness

Choose the execution shell

The clitest shebang is #!/bin/sh. That's the default shell that will be used to run your test command lines. Depending on the system, that path points to a different shell, such as ash, dash, or bash (running in POSIX mode).

To force your test commands to always run on a specific shell, just call the desired shell before:

clitest tests.txt            # Uses /bin/sh
bash clitest tests.txt       # Uses Bash
ksh clitest tests.txt        # Uses Korn Shell

Portability

This script was carefully coded to be portable between POSIX shells. It's code is validated by checkbashisms and shellcheck.

To make sure it keeps working as expected, after every change clitest is automatically tested in the CI, using the following shells:

Fish shell is not supported (it's not POSIX), but you can use doctest.fish instead.

Portability issues are considered serious bugs, please report them!

Developers: Learn more about portability in POSIX shells:

KISS

A shell script to test shell commands.
No other language or environment involved.

Meta