Home

Awesome

CMake Unit

A unit testing framework for CMake.

Status

Travis CI (Ubuntu)AppVeyor (Windows)CoverageBiicodeLicence
TravisAppVeyorCoverallsBiicodeLicense

Why have a unit-testing framework for CMake

Because CMake is a powerful and battle-tested language for writing build systems for large-scale C++ projects, but its dynamic nature makes it easy to make undetectable errors which later ship as bugs that either you, or the users of your macros, need to work around. We have to put a lot of logic inside out CMake scripts sometimes, like propogation of global state, if conditions for various options and host system configurations and loops over variable argument lists.

It is something you want to get right the first time rather than having to scratch your head about later with the lack of debugging tools for CMake scripts.

Platforms

cmake-unit is written entirely using the CMake language and should work across all platforms where CMake is supported. It had been tested on:

Usage

cmake-unit should be included as a submodule in your project and comes with three files.

CMakeUnitRunner

CMakeUnitRunner contains the main "runner" script for loading and executing test scripts. Include this file in the /CMakeLists.txt.

Tests are defined inline as functions. They are automatically discovered by cmake-unit and the name of the function must be in the format of ${your_namespace}_test_{test_name}. Within each test function are function definitions used to control each "phase" of the test's build. After these functions are called, a call to cmake_unit_configure_test ties all the phases functions together into a single test.

Test functions are subdivided into "phases", each phase having its own script that can be run in place of the default. Usually you will want to override the CONFIGURE or VERIFY phases in order to provide your own project set-up and verification scripts. The build of each project goes through the following phases, in order:

For each phase, a name of a function can be provided which will "override" the default function called for that phase. Some phases are called within different CMake invocations, so you shouldn't assume that state can be shared between the phase functions.

The name of each phase is a keyword argument to cmake_unit_configure_test. Following the phase name, further options can be specified for each phase. Some common options are:

The PRECONFIGURE phase

This is the first phase that is run for the test. It cannot be overridden. It does some initial setup for the test itself, including writing out a special driver script which will be used to invoke this test at CTest time.

Two options are exposed to this unit for this phase, SKIP_GENERATOR_REGEX and SKIP_SYSTEM_REGEX. A list of regular expressions can be provided for each option, which will cause the test not to run when the regex matches CMAKE_SYSTEM and CMAKE_GENERATOR respectively.

The CLEAN phase

This phase is responsible for cleaning the build directory of the test. By default, it calls cmake_unit_invoke_clean, which just removes the test project's CMAKE_BINARY_DIR.

The INVOKE_CONFIGURE phase

This phase is responsible for writing out a stub /CMakeLists.txt and jumping invoking cmake on the resulting project folder. By default it will call cmake_unit_invoke_configure. The written out /CMakeLists.txt will do some setup for the test project, including calling the project command.

cmake_unit_invoke_configure will not configure any languages by default. This is to prevent unnecessary overhead when testing on platforms where configuring language support is quite slow (for instance, Visual Studio and XCode). Instead of overriding the command, usually the only action you will need to take if you need language support is to set the LANGUAGES option (eg, to C CXX).

The CONFIGURE phase

This phase is responsible for actually configuring the project. Any commands run inside this phase are effectively run as though CMake was configuring a project by processing a /CMakeLists.txt, so the full range of commands are available. Usually you will want to override the COMMAND and configure your project as required (or make assertions).

The INVOKE_BUILD phase

This phase is responsible for invoking cmake --build. Usually the COMMAND will not need to be overridden, but if the build can fail or if the project should not be built at all, then ALLOW_FAIL or COMMAND NONE should be specified respectively.

The TARGET option allows you to specify a custom target to build instead of the default one.

The INVOKE_TEST phase

This phase is responsible for invoking ctest. Usually the COMMAND will not need to be overridden, unless you need to invoke ctestin a special way.

The VERIFY phase

This phase is responsible for verifying that the configure, build and test steps went the way you expected. It is executed after the final step of the configure-build-test cycle is completed for this project.

You can inspect the standard output and error of each of these steps. Use the cmake_unit_get_log_for command in order to fetch the path to these log files.

The COVERAGE phase

This phase is responsible for collecting tracefile output and turning it into line-coverage statistics. It is not overridable.

An example of a test

Here is an example of how a test looks in practice:

function (namespace_test_one)

    function (_namespace_configure)

        cmake_unit_create_simple_library (library SHARED FUNCTIONS function)
        cmake_unit_create_simple_executable (executable)
        target_link_libraries (executable library)

        cmake_unit_assert_that (executable is_linked_to library)

    endfunction ()

    function (_namespace_verify)

        cmake_unit_get_log_for (INVOKE_BUILD OUTPUT BUILD_OUTPUT)

        cmake_unit_assert_that ("${BUILD_OUTPUT}"
                                file_contents any_line
                                matches_regex
                                "^.*executable.*$")

    endfunction ()

    cmake_unit_configure_test (INVOKE_CONFIGURE LANGUAGES C CXX
                               CONFIGURE COMMAND _namespace_configure
                               VERIFY COMMAND _namespace_verify)

endfunction ()

The _namespace_configure and _namespace_verify functions are defined within the namespace_test_one function. They are passed to the COMMAND keyword for the CONFIGURE and VERIFY phases on cmake_unit_configure_test.

LANGUAGES C CXX is passed to INVOKE_CONFIGURE. This ensures that compilers are tested and CMake is set up to build and link C and C++ binary code.

Shortcut to skip the build phase

If there's no need to build and test the test project, or to verify it, you can use cmake_unit_configure_config_only_test in place of cmake_unit_configure_test. This will pass INVOKE_BUILD COMMAND NONE and INVOKE_TEST COMMAND NONE to cmake_unit_configure_test along with whatever options you specify.

Discovering tests and running them

cmake_unit_init is what handles the registration and running of each discovered test function. It takes a namespace as the argument to the keyword NAMESPACE. This is the name each test is prefixed with (followed by _test). Any function matching the pattern ^${namespace}test.*$ will be automatically registered. It also takes a list of files considered to be candidates for code coverage as COVERAGE_FILES.

As an example, see the following:

cmake_unit_init (NAMESPACE namespace
                 COVERAGE_FILES "${PROJECT_DIR}/Module.cmake")

CMakeUnit

CMakeUnit contains matchers and a general cmake_unit_assert_that function. You can use them in both the configure and verify stages. If the script hits an assertion failure, it will call message (SEND_ERROR).

Built-in matchers

The following matchers are available at the time of writing this documentation

Writing your own matchers

cmake-unit can be extended with your own matchers. To do this, you will need to write a "callable" function in your project's namespace, for example

function (my_namespace_equal_to_seven)

    list (GET CALLER_ARGN 0 VARIABLE)
    list (GET CALLER_ARGN -1 RESULT_VALUE)

    set (${RESULT_VALUE} "to be equal to 7" PARENT_SCOPE)

    if ("${${VARIABLE}}" EQUAL 7)

        set (${RESULT_VALUE} TRUE PARENT_SCOPE)

    endif ()

endfunction ()

Then you will need to register your project's namespace as a namespace containing matchers

cmake_unit_register_matcher_namespace (my_namespace)

You can start using your matcher like so:

cmake_unit_assert_that (VARIABLE equal_to_seven)

The function my_namespace_equal_to_seven is a callable function abiding by the calling convention set out below. Its first argument will always be the variable-to-be-matched. Depending on what your matcher does, this may be a value or a variable name. The last variable is always the "result variable", which is the name of the variable that you will need to set in the parent scope to indicate the matcher status. By convention, matchers should set this variable to a sentence fragment that would provide a sensible explanation of what happened in the sentence "Expected VARIABLE ..." in case there was a mismatch. Otherwise, the variable should be set to TRUE.

Overridable Phase Functions

Each of these functions is a default for a phase of a cmake-unit test's build cycle. If they are overridden, they can be "chained up" to. CALLER_ARGN will be passed implicitly.

cmake_unit_invoke_clean

Cleans the project BINARY_DIRECTORY (as specified in CALLER_ARGN).

cmake_unit_invoke_configure

Creates a /CMakeLists.txt for this project which does some initial setup and then jumps to the function defined for CONFIGURE. cmake is invoked on a build directory for the folder containing the created /CMakeLists.txt.

cmake_unit_invoke_build

Invokes cmake --build on this project.

cmake_unit_invoke_test

Invokes ctest -C Debug on this project.

Utility Functions

cmake-unit also provides a few utility functions to make writing tests easier.

Strings
cmake_unit_escape_string

Escape all characters from INPUT and store in OUTPUT_VARIABLE

Test data
cmake_unit_get_dirs

Reliably returns the binary and source directories for this test. You should use this instead of CMAKE_CURRENT_SOURCE_DIR and CMAKE_CURRENT_BINARY_DIR where possible, as it will be correct in every phase.

Source File Generation
cmake_unit_create_source_file_before_build

Writes out a source file, for use with add_library, add_executable or source scanners during the configure phase.

If the source is detected as a header based on the NAME property such that it does not have a C or C++ extension, then header guards will be written and function definitions will not be included.

cmake_unit_generate_source_file_during_build

Generates a source file, for use with add_library, add_executable or source scanners during the build phase.

If the source is detected as a header based on the NAME property such that it does not have a C or C++ extension, then header guards will be written and function definitions will not be included.

Binary target generation

These functions can be used to generate binary targets such as simple executables and libraries. There will only be a single source file per executable or library generated.

cmake_unit_create_simple_executable

Creates a simple executable by the name "NAME" which will always have a "main" function.

cmake_unit_create_simple_library

Creates a simple executable by the name "NAME" which will always have a "main" function.

Working with Built Projects
cmake_unit_get_target_location_from_exports

For an exports file EXPORTS and target TARGET, finds the location of a target from an already generated EXPORTS file.

This function should be run in the verify stage in order to determine the location of a binary or library built by CMake. The initial configure step should run export (TARGETS ...) in order to generate this file.

This function should always be used where a binary or library needs to be invoked after build. Different platforms put the completed binaries in different places and also give them a different name. This function will resolve all those issues.

cmake_unit_export_cfg_int_dir

Exports the current CMAKE_CFG_INTDIR variable (known at configure-time) and writes it into the file specified at LOCATION. This file could be read after the build to determine the CMAKE_CFG_INTDIR property

cmake_unit_import_cfg_int_dir

Reads OUTPUT_FILE to import the value of the CMAKE_CFG_INTDIR property and stores the value inside of LOCATION_RETURN. This should be run in the verify phase to get the CMAKE_CFG_INTDIR property for the configure phase generator. Use cmake_unit_export_cfg_int_dir in the configure phase to export the CMAKE_CFG_INTDIR property.

cmake_unit_get_log_for

Gets the LOG_TYPE log for PHASE and stores it in the variable specified in LOG_FILE_RETURN. The returned log is a path to a file. Valid values for the LOG_TYPE parameter are ERROR and OUTPUT.

CMakeTraceToLCov

CMakeTraceToLCov is a script that converts a tracefile generated by using CMAKE_UNIT_LOG_COVERAGE=ON into a Linux Test Project Coverage (LCov) compatible file.

CMakeTraceToLCov should be run in script mode from the toplevel source directory where CMake scripts are to be kept.

There are two cache options which must be set prior to use:

Known Issues

The following issues are known at the time of writing:

Technical Implementation Notes

cmake-unit uses some clever hacks under the hood in order to achieve its "streamlined" test definition syntax.

Calling arbitrary functions

Test auto-discovery, dynamic test loading and custom phase specification is all achieved through the ability to call arbitrary functions. CMake doesn't offer any syntax to do so, but there is a back door using a debugging feature called variable_watch.

Obviously, variable_watch provides its own arguments to the called function, which is not entirely what we want. However, CMake makes it relatively easy to establish a kind of "calling convention" for these called functions. Usage of keyword arguments with CMakeParseArguments is pretty common for most modules. We defined a variable called CALLER_ARGN which functions just like ARGN would in a normal function call. All arguments are passed as keywords.

variable_watch can only be used to register a callback for one variable at a time, so if a function is to be called multiple times, then a register needs to be maintained mapping function names to variable names.

All of this is encapsulated within the cmake_call_function command. This is hosted as a separate block on biicode. It is hoped that eventually this can become part of the core CMake syntax.

Discovering test functions

Test function discovery is by using the "hidden" COMMANDS property of GLOBAL scope. This provides a list of all defined commands at the time of retrieving the property.