Home

Awesome

AUnit

AUnit Tests

A unit testing framework for Arduino platforms inspired by by ArduinoUnit and Google Test. AUnit is almost a drop-in replacement of ArduinoUnit (v2.2) with some advantages. AUnit supports timeouts and test fixtures. It sometimes consumes 50% less flash memory on the AVR platform, and it has been tested to work on the AVR, SAMD21, STM32, ESP8266, ESP32 and Teensy platforms. The assertion error messages were updated in v1.7 to provide seamless integration with Unix tools like vim.

Originally, the AUnit tests were designed to run on the embedded controller itself which allows detection of architecture-specific problems. But the uploading, flashing, and execution process is often slow and flaky, causing the iteration cycle to take too much time. It is often more effective to execute the AUnit tests natively on a host machine (running Linux, MacOS, or FreeBSD) using the EpoxyDuino companion project. Once the unit tests are running on the Linux or MacOS host machine, they can be incorporated into a continuous integration system like Jenkins system or a cloud-based system like GitHub Actions.

Version: 1.7.1 (2023-06-15)

Changelog: CHANGELOG.md

Table of Contents

<a name="Summary"></a>

Summary

AUnit was created to solve 3 problems with ArduinoUnit 2.2:

In contrast:

The AUniter command line tools can compile the unit tests, upload them to Arduino boards attached to the serial ports of the local machine, and validate the output of the AUnit test runner. In addition, the AUniter script can be integrated into a Jenkins continuous integration service running on the local machine, and the unit tests can be monitored automatically.

Unit tests written using AUnit can often be compiled and executed natively on Linux or MacOS using the EpoxyDuino library. The output on the Serial object is redirected to the stdout of the Unix host. This provides another avenue for implementing continuous builds or integration.

<a name="ArduinoUnitCompatible"></a>

ArduinoUnit Compatible Features

For basic unit tests written using ArduinoUnit 2.2, only two changes are required to convert to AUnit:

Most of the core macros are compatible between ArduinoUnit and AUnit:

AUnit also supports exclude and include filters:

Filters can be accessed through Command Line Flags on desktop machines using EpoxyDuino

The various assertion and test status messages can be enabled or disabled using the Verbosity flags on per test basis:

<a name="MissingFeatures"></a>

Missing Features

Here are the features which have not been ported over from ArduinoUnit 2.2:

<a name="AddedFeatures"></a>

Added Features

Here are the features in AUnit which are not available in ArduinoUnit 2.2:

Every feature of AUnit is unit tested using AUnit itself.

<a name="Installation"></a>

Installation

The latest stable release is available in the Arduino IDE Library Manager. Search for "unit test" or "AUnit", select "AUnit", then click the "Install" button.

The development version can be installed by cloning the GitHub repository, checking out the develop branch, then manually copying over the contents to the ./libraries directory used by the Arduino IDE. (The result is a directory named ./libraries/AUnit.) See the Preferences menu for the location of the Arduino sketch directory. The master branch contains the stable release.

Using either installation method, you may need to restart the Arduino IDE to pick up the new library.

<a name="SourceCode"></a>

Source Code

The source files are organized as follows:

<a name="Documentation"></a>

Documentation

<a name="Examples"></a>

Examples

The examples/ directory has a number of examples:

In the tests/ directory, there are unit tests to test the AUnit framework itself:

Perhaps the best way to see AUnit in action is through real life examples. All my libraries use AUnit for testing and for continuous integration through EpoxyDuino. Here are some of my libraries:

<a name="Usage"></a>

Usage

In this section, information about differences between AUnit and ArduinoUnit will appear in a note marked by ArduinoUnit Compatibility.

<a name="HeaderAndNamespace"></a>

Header and Namespace

To prevent name clashes with other libraries and code, all classes in the AUnit library are defined in the aunit namespace. The user will mostly interact with the TestRunner class. It can be referenced with an explicit namespace qualifier (i.e. aunit::TestRunner), or we can use a using directive like this:

#include <AUnit.h>
using aunit::TestRunner;

or we can import the entire aunit namespace:

#include <AUnit.h>
using namespace aunit;

Similar to ArduinoUnit, many of the "functions" in this framework (e.g. test(), testing(), assertXxx()) are defined as #define macros which live in the global namespace, so it is usually not necessary to import the entire aunit namespace.

<a name="VerboseMode"></a>

Verbose Mode

By default, AUnit generates terse assertion messages by leaving out the string arguments of the various assertXxx() macros. If you would like to get the same verbose output as ArduinoUnit, use the following header instead:

#include <AUnitVerbose.h>

The flash memory consumption on an 8-bit AVR may go up by 20-25% for medium to large tests. On Teensy ARM, ESP8266 or ESP32, the increased memory size probably does not matter too much because these microcontrollers have far more flash and static memory.

<a name="DefiningTests"></a>

Defining the Tests

The usage of AUnit is basically identical to ArduinoUnit. The following macros are used to create a test:

The code in { } following these macros becomes the body of a method in a subclass derived from the base class indicated above. The test() and testF() macros place the code body into the TestOnce::once() method. The testing() and testingF() macros place the code body into the TestAgain::again() method.

The test() and testing() macros support 1 or 2 arguments. The one-argument version is inherited from ArduinoUnit. The two-argument version is analogous to the TEST() macro in GoogleTest, where the suiteName can be used to organize multiple tests into a collection of similar tests. The grouping is purely in the naming scheme of the generated code, there is no functional relationship between these tests.

During static initialization, the constructor of the object adds itself to an internal list. The root of that list is given by Test::getRoot(). The TestRunner::run() method traverses the linked list, executing each test case until it passes, fails or is skipped.

Here is a rough outline of an AUnit unit test sketch:

#line 2 AUnitTest.ino

#include <AUnit.h>
using namespace aunit;

test(example) {
  ...
  assertXxx(...)
  ...
}

test(ExampleTest, example) {
  ...
}

testing(looping) {
  ...code...
  if (...) {
    pass();
  } else if (...) {
    failTestNow();
  } else {
    skipTestNow();
  }
}

testing(LoopingTest, looping) {
  ...
}

class CustomTestOnce: public TestOnce {
  protected:
    // optional
    void setup() override {
      TestOnce::setup();
      ...setup code...
    }

    // optional
    void teardown() override {
      ...teardown code...
      TestOnce::teardown();
    }

    void assertBigStuff() {
      ...higher level assertions...
    }
};

testF(CustomTestOnce, example_test) {
  ...
  assertBigStuff();
  ...
}

class CustomTestAgain: public TestAgain {
  protected:
    // optional
    void setup() override {
      TestAgain::setup();
      ...setup code...
    }

    // optional
    void teardown() override {
      ...teardown code...
      TestAgain::teardown();
    }

    void assertBigStuff() {
      ...various assertions...
    }
};

testingF(CustomTestAgain, example_test) {
  ...
  assertBigStuff();
  ...
}

void setup() {
  Serial.begin(115200);
  while (! Serial); // Wait until Serial is ready - Leonardo/Micro

  TestRunner::exclude("*");
  TestRunner::include("looping*");
  TestRunner::include("CustomTestAgain", "example*");
}

void loop() {
  TestRunner::run();
}

ArduinoUnit Compatibility: The basic structure of the unit test is identical to ArduinoUnit. AUnit adds the testF() and testingF() macros, and the two-argument versions of test() and testing() which are not available in ArduinoUnit. The Test class in ArduinoUnit has been replaced with the TestAgain class in AUnit.

<a name="GeneratedClass"></a>

Generated Class and Instance Names

The arguments to the various test*() macros are used to generate the name for the subclasses of TestOnce or TestAgain, and generate the names of the instances of those classes. For reference, here are the rules:

The instance name is available within the test code using the Test::getName() method.

<a name="BinaryAssertions"></a>

Binary Assertions

Inside the test() and testing() macros, the following assertions are available. These are essentially identical to ArduinoUnit:

<a name="AssertionMessageFormat"></a>

Assertion Message Format

When the assertion passes, nothing is printed by default. This can be controlled by the TestRunner::setVerbosity() method. See Controlling Verbosity.

When the assertion fails, an error message of the following format is printed:

SampleTest.ino:10: Assertion failed: (2) == (1)

The format of the assertion failure messages was changed in v1.7 to the following:

{fileName}:{lineNumber}: Assertion failed: {expression}

This format is a widely used in many other programs, for example, the C compiler gcc, the C++ compiler g++, the Python 3 interpreter python3, grep, and the GNU Make program make. In particular, the quickfix feature in the vim text editor can parse this error format and jump directly to the fileName and lineNumber indicated by the error message. See the instructions in EpoxyDuino to see how to run unit tests on a Linux or MacOS machine inside the vim editor so that the editor jumps directly to the files and line numbers where the assertion failure occurred.

<a name="SupportedParameterTypes"></a>

Supported Parameter Types

The 6 core assert macros (assertEqual(), assertNotEqual(), assertLess(), assertMore(), assertLessOrEqual(), assertMoreOrEqual()) support the following 18 combinations for their parameter types:

The assertEqual() and assertNotEqual() support arbitrary pointer types through implicit casts to const void*:

All 9 combinations of the 3 string types (char*, String, and __FlashStringHelper*) are supported.

These macros perform deep comparisons for string types instead of just comparing their pointer values. This is different than the ASSERT_EQ() and ASSERT_NE() macros in Google Test which perform only pointer comparisons. In other words, assertEqual() with string types is equivalent to ASSERT_STREQ() in Google Test.

Also for string types, these macros support nullptr (unlike the underlying strcmp() function from the C-library). The nullptr string is defined to be "smaller" than any non-null string, including the empty string. Two nullptr strings are considered to be equal however.

Additionally, the usual C++ implicit type conversion and function overloading matching algorithms apply to support additional argument types. For example, the following type conversions will occur:

Note that char, signed char, and unsigned char are 3 distinct types in C++, so a (char, char) will match exactly to one of the assertXxx() methods.

ArduinoUnit Compatibility: The names of the macros are identical. However, the type inference logic of two (a, b) arguments in the assertXxx(a, b) is slightly different. ArduinoUnit allows the two parameters to be slightly different types, at the expense of a compiler warning. In AUnit, the warning becomes a compiler error. See the "Parameters Must Match Types" section below.

<a name="ParameterTypesMustMatch"></a>

Parameter Types Must Match

In ArduinoUnit, the assertXxx() macros could be slightly different types, for example:

unsigned int uintValue = 5;
assertEqual(5, uintValue);

If the compiler warnings are enabled in the Preferences box of the IDE, a warning from the compiler is printed:

../ArduinoUnit/src/ArduinoUnitUtility/Compare.h:17:28: warning:
    comparison between signed and unsigned integer expressions [-Wsign-compare]
    return (!(a<b)) && (!(b<a));

In AUnit, the above code produces a compiler error (not a warning) like this:

.../AUnit/src/aunit/Assertion.h:29:66: error: call of overloaded
    'assertion(const char [14], int, int, const char[3], <unresolved overloaded
    function type>, unsigned int&)' is ambiguous
    if (!aunit::assertion(__FILE__,__LINE__,(arg1),opName,op,(arg2)))\
...

The compiler cannot find an appropriate overloaded version of assertEqual().

The solution is to make the parameters the same type:

assertEqual(5U, uintValue);

On the AVR platform, both a (short) and (int) are 16-bit types, so the following will produce a compiler error:

unsigned short ushortValue = 5;
assertEqual(5U, ushortValue);

But on Teensy-ARM, ESP8266, and ESP32, a 16-bit (short) can be promoted to a 32-bit (int) without loss of precision, so the above will compile just fine. For portability, the following should be used on all platforms:

unsigned short ushortValue = 5;
assertEqual((unsigned short) 5, ushortValue);

The integer type promotion rules and function overload matching rules can be difficult to remember (and sometimes difficult to understand). The best way to avoid these compiler errors is to make sure that the assertion parameter types are identical, potentially using explicit casting.

<a name="PointerComparisons"></a>

Pointer Comparisons

Version 1.4 adds pointer comparison to assertEqual() and assertNotEqual(). Arbritary pointers are implicitly cast to a const void* and compared to each other. If the assertion fails, the pointer is converted to an integer type, and the hexadecimal value of the pointer is printed. For example,

test(voidPointer) {
  const int aa[] = {1, 2};
  const long bb[] = {1, 2};

  assertEqual(aa, bb);
}

This test will fail with the following error message:

AUnitTest.ino:338: Assertion failed: (aa=0x3FFFFF38) == (bb=0x3FFFFF30).
Test voidPointer failed.

Comparison against the nullptr will work:

test(nullPointer) {
  const int aa[] = {1, 2};
  assertEqual(aa, nullptr);
}

This will print the following:

AUnitTest.ino:348: Assertion failed: (aa=0x3FFFFF58) == (nullptr=0x0).
Test nullPointer failed.

Comparing a string type (i.e. const char*, or const __FlashStringHelper*) to a nullptr will cause an error due to ambiguous matches on overloaded functions. The solution is to explicitly cast the nullptr to the corresponding string type:

test(stringPointer) {
  const char aa[] = "abc";

  // assertEqual(aa, nullptr); // Causes errors

  assertEqual(aa, (const char*) nullptr); // Works.
}

<a name="CaseInsensitiveStrings"></a>

Case Insensitive String Comparisons

Two macros provide case-insensitive string comparisons (analogous to ASSERT_STRCASEEQ() and ASSERT_STRCASENE() in Google Test):

The supported types for (a, b) are all 9 combinations of Arduino string types:

ArduinoUnit Compatibility: Not available in ArduinoUnit.

<a name="ApproximateComparisons"></a>

Approximate Comparisons

Floating point values are difficult to compare because of internal rounding errors. Google Test provides two types of macros to handle floating points:

Since floating point operations are relatively rare in Arduino programming, AUnit offers only the equivalent of ASSERT_NEAR() function:

Upon failure, the error messages will look something like:

AUnitTest.ino:517: Assertion failed: |(1.00) - (1.10)| > (0.20).
AUnitTest.ino:527: Assertion failed: |(4.00) - (1.10)| <= (0.20).

Unlike Google Test where ASSERT_NEAR() supports only the double type, both assertNear() and assertNotNear() support integral types as well. The full list of supported types is:

Other primitive types (e.g. char or float) will be automatically converted to one of these supported types by the compiler. (64-bit integer types long long and unsigned long long are not supported here.)

Note that the abs() of 2 values of a signed integer type can be larger than the maximum value that can be represented by the given signed type. Since signed integer overflow is an undefined behavior in C and C++11, I cannot predict what the compiler will do in that case. Unsigned types should not have this problem because the distance between two values of an unsigned type should always fit inside the given unsigned type. Technically, a similar problem exists for the floating point types (which are naturally signed), but it is unlikely that you are dealing with floating point values so close to the maximum values.

<a name="BooleanAssertions"></a>

Boolean Assertions

The following boolean asserts are also available:

ArduinoUnit Compatibility: These are identical to ArduinoUnit.

<a name="TestFixtures"></a>

Test Fixtures

When the unit tests become more complex, using test fixtures will allow you to place common data objects and methods into a class that can be shared among multiple test cases. This concept matches very closely to the test fixtures in Google Test.

To create a test fixture:

  1. Derives a new class from either TestOnce (if you want to run the test just once), or TestAgain (if you want to run the test repeatedly).
  2. Add any data objects inside the class.
  3. Optionally add a virtual void setup() {...} method to perform any common initialization code. Be sure to call the parent's setup() method in the first line to chain any setup() methods defined by the parents. There may be multiple parent classes.
  4. Optionally add a virtual void teardown() {...} method to perform any common clean up code. Be sure to call the parent's teardown() method in the last line to chain any teardown() methods defined by the parents. There may be multiple parent classes.
  5. Add any additional shared methods into this new class.

To define your tests, use the testF() macro like this:

class CustomTestOnce: public TestOnce {
  protected:
    // optional
    void setup() override {
      TestOnce::setup();
      ...setup code...
    }

    // optional
    void teardown() override {
      ...teardown code...
      TestOnce::teardown();
    }

    void assertCustomStuff() {
      ...common code...
    }

    int sharedValue;
};

testF(CustomTestOnce, calculate) {
  ...test code here...
}

No constructor for CustomTestOnce needs to be defined.

Underneath the covers, the testF() macro creates a subclass named CustomTestOnce_calculate which inherits from TestOnce class. The test code becomes the body of the CustomTestOnce_calculate::once() method. The name of this test has the class name prepended, so it is CustomTestOnce_calculate, which prevents name collision with other testF() tests with the same name using a different test fixture class.

To define a continuous test, use the testingF() macro like this:

class CustomTestAgain: public TestAgain {
  protected:
    // optional
    void setup() override {
      TestAgain::setup();
      ...setup code...
    }

    // optional
    void teardown() override {
      ...teardown code...
      TestAgain::teardown();
    }

    void assertCustomStuff() {
      ...common code...
    }

    int sharedValue;
};

testingF(CustomTestAgain, calculate) {
  ...test code here...
}

Similarly, the testingF() macro creates a subclass named CustomTestAgain_calculate, and the test code becomes the body of the CustomTestAgain_calculate::again() method.

See examples/fixtures/fixtures.ino to see a working example of the testF() macro.

ArduinoUnit Compatibility: The testF() and testingF() macros, and the teardown() virtual method are available only in AUnit (and Google Test), not ArduinoUnit.

<a name="EarlyReturnDelayedAssertions"></a>

Early Return and Delayed Assertions

AUnit (like ArduinoUnit and Google Test) does not use C++ exceptions. Instead, the various assertXxx() macros perform an early return if the condition evaluates to false. That means that the assertions can only bail out of the current method, not the calling method. If you are using test fixtures, and create a shared custom assert function, e.g. the assertCustomStuff() method above, any assertXxx() statements in shared method will bail out of that method only. The statement after the assertCustomStuff() will continue to execute.

In other words, in the following example, if the assertCustomStuff() fails, then assertMoreStuff() inside testF() will execute:

class CustomTestOnce: public TestOnce {
  protected:
    void assertCustomStuff() {
      assertEqual(sharedValue, 3);

      // This will not execute if the assertEqual() above fails.
      assertLess(...);
    }

    void assertMoreStuff() {
      assertEqual(...);
    }

    int sharedValue;
};

// DON'T DO THIS
testF(CustomTestOnce, dontDoThis) {
  assertCustomStuff();

  // This will execute even if assertCustomStuff() fails.
  assertMoreStuff();
}

// DO THIS INSTEAD
testF(CustomTestOnce, doThis) {
  assertNoFatalFailure(assertCustomStuff());
  assertNoFatalFailure(assertMoreStuff());
}

The solution is to use the assertNoFatalFailure(statement) macro which checks whether the inner statement returned with a fatal assertion. If so, then it returns immediately, preventing execution from continuing to the code that follows. This macro is modeled after the ASSERT_NO_FATAL_FAILURE(statement) macro in Google Test that provides the same functionality.

<a name="MetaAssertions"></a>

Meta Assertions

The following methods from ArduinoUnit have also been implemented:

The checkTestXxx() methods check the status of the test named name and returns a bool. The execution continues even if false.

The assertTestXxx() methods stops the unit test if the status check returns false, and prints assertion messages that look like this:

AUnitTest.ino:366: Assertion passed: Test slow_pass is done.
AUnitTest.ino:372: Assertion passed: Test slow_pass is not failed.
AUnitTest.ino:448: Assertion passed: Test slow_skip is skipped.
AUnitTest.ino:451: Assertion passed: Test slow_skip is not timed out.

(The human readable version of being expired will always be timed out or not timed out on the Serial output.)

The following macros define extern references to test case objects which live in other .cpp files. These are required for the above meta assertions if the test cases are defined in another file:

ArduinoUnit Compatibility: The methods marked by [*] are only available in AUnit. Also, the assertion messages are different. ArduinoUnit reuses the format used by the assertXxx() macros, so prints something like the following:

AUnitTest.ino:439: Assertion passed: (test_slow_skip_instance.state=2) >= (Test::DONE_SKIP=2).

AUnit has a separate message handler to print a customized message for the assertTestXxx() meta assertion macros.

<a name="UnconditionalTermination"></a>

Unconditional Termination

The following macros can be used inside the body of test() or testing() macro to terminate a test unconditionally. Each macro prints a short message, and returns immediately from the test, much like an assertXxx() macro that fails.

The messages look like:

AUnitTest.ino:360: Status passed.
AUnitTest.ino:378: Status failed.
AUnitTest.ino:380: Status skipped.
AUnitTest.ino:391: Status timed out.

The following methods on the Test class also set the status of the test, but these methods do not print any messages (which makes debugging difficult) and they do not terminate the test immediately.

In most cases, the failTestNow(), skipTestNow() and expireTestNow() macros are more useful than the equivalent methods in the Test class. However, in a testing() loop test, the pass() method is probably better than the passTestNow() macro because we usually don't want to see an error message from a passing test.

ArduinoUnit Compatibility: The method(s) marked by [*] are only available in AUnit.

<a name="OverridableMethods"></a>

Overridable Methods

The following methods are defined at the Test base class level:

The TestOnce class defines:

The TestAgain class defines:

ArduinoUnit Compatibility: These are functionally the same as ArduinoUnit except with different class names. Instead of Test use TestAgain. Instead of Test::loop use TestAgain::again(). ArduinoUnit does not support a teardown() method.

<a name="RunningTests"></a>

Running the Tests

We run the test cases in the global loop() method by calling TestRunner::run(). The tests are sorted according to the name of the test given in the argument in the test() or testing() macro.

Each call to the run() method causes one test case to run and be resolved. The next call to run() executes the next test case. This design allows the loop() method to perform a small amount of work and return periodically to allow the system to perform some actions. On some systems, such as the ESP8266, an error is generated if loop() takes too much CPU time.

...
void loop() {
  TestRunner::run();
}

ArduinoUnit Compatibility: This is equivalent to called Test::run() in ArduinoUnit. AUnit sorts the tests in the same way as ArduinoUnit. In ArduinoUnit, each call to Test::run() will process the entire list of currently active test cases. In AUnit, each call to TestRunner::run() performs only a single test case, then returns.

<a name="FilteringTestCases"></a>

Filtering Test Cases

Six filtering methods are available on the TestRunner class:

These methods are called from the global setup() method, for example:

void setup() {
  TestRunner::include("looping*");
  TestRunner::exclude("CustomTestAgain", "*");
  TestRunner::include("CustomTestAgain", "test*");
  TestRunner::include("CustomTestAgain", "test*");
  TestRunner::includesub("net");
  TestRunner::excludesub("net");
  ...
}

Excluded tests bypass their Test::setup() and Test::teardown() methods and terminate immediately. For the purposes of reporting, excluded tests are counted as "skipped".

The 2-argument versions of include() and exclude() correspond to the 2 arguments of testF() and testingF().

The filtering methods are also available as command line flags and arguments (--include, --exclude, --includesub --excludesub) if the test program is compiled using EpoxyDuino under a Unix-like environment. See the EpoxyDuino section below.

Implicit Exclude All: If the first filtering request is an "include" (i.e. include(pattern), include(testClass, pattern), includesub(substring)), all tests are excluded by default initially, instead of being included by default. Otherwise, the first "include" statement would have no effect.

ArduinoUnit Compatibility: The equivalent versions in ArduinoUnit are Test::exclude() and Test::include() The matching algorithm in AUnit is not as powerful as the one in ArduinoUnit. AUnit supports only a single wildcard character * and that character can appear only at the end if it is present. For example, the following are accepted:

AUnit provides 2-argument versions of include() and exclude()

<a name="OutputPrinter"></a>

Output Printer

The default output printer is the Serial instance. This can be changed using the TestRunner::setPrinter() method:

#include <AUnit.h>
using aunit::TestRunner;
...

void setup() {
  Serial1.begin(...);

  TestRunner::setPrinter(&Serial1);
  ...
}

void loop() {
  TestRunner::run();
}

ArduinoUnit Compatibility: This is the equivalent of the Test::out static member variable in ArduinoUnit.

<a name="ControllingVerbosity"></a>

Controlling the Verbosity

The default verbosity of the test results can be controlled using the TestRunner::setVerbosity() method:

#include <AUnit.h>
using aunit::TestRunner;
using aunit::Verbosity;
...
void setup() {
  ...
  TestRunner::setVerbosity(Verbosity::kAll);
  ...
}

Every test is assigned this default verbosity just before its Test::setup() is called. A unit test can choose to modify the verbosity calling one of the following methods:

at the beginning of the test definition, like this:

test(enable_assertion_passed_messages) {
  enableVerbosity(Verbosity::kAssertionPassed);
  ...
}

The values of verbosity are defined by the static constants of the Verbosity utility class:

ArduinoUnit Compatibility: The following ArduinoUnit variables do not exist:

The bit field constants have slightly different names:

<a name="LineNumberMismatch"></a>

Line Number Mismatch

AUnit suffers from the same compiler/preprocessor bug as ArduinoUnit that causes the built-in __LINE__ macro to be off by one. The solution is to add:

#line 2 {file.ino}

as the first line of a unit test sketch.

ArduinoUnit Compatibility: This problem is identical to ArduinoUnit.

<a name="TestFrameworkMessages"></a>

Test Framework Messages

<a name="AssertionMessage"></a>

Assertion Message

The various assertXxx() macros in AUnit print a message upon pass or fail. For example, if the assertion was:

int expected = 3;
int counter = 4;
assertEquals(expected, counter);

The error message (if enabled, which is the default) is:

AUnitTest.ino:134: Assertion failed: (3) == (4).

Asserts with bool values produce customized messages, printing "true" or "false" instead of using the Print class default conversion to int:

assertEquals(true, false);

AUnitTest.ino:134: Assertion failed: (true) == (false).

Similarly, the assertTrue() and assertFalse() macros provide more customized messages:

bool ok = false;
assertTrue(ok);

AUnitTest.ino:134: Assertion failed: (false) is true.

and

bool ok = true;
assertFalse(ok);

AUnitTest.ino:134: Assertion failed: (true) is false.

ArduinoUnit Compatibility: ArduinoUnit captures the arguments of the assertEqual() macro and prints:

AUnitTest.ino:134: Assertion failed: (expected=3) == (counter=4).

Each capture of the parameter string consumes flash memory space. If the unit test has numerous assertXxx() statements, the flash memory cost is expensive. AUnit omits the parameters to reduce flash memory space by about 33%.

The messages for asserts with bool values are customized for better clarity (partially to compensate for the lack of capture of the string of the actual arguments, and are different from ArduinoUnit.

<a name="VerboseModeMessage"></a>

Verbose Mode Message

If you use the verbose header:

#include <AUnitVerbose.h>

the assertion message will contain the string fragments of the arguments passed into the assertXxx() macros, like this:

AUnitTest.ino:134: Assertion failed: (expected=3) == (counter=4).
AUnitTest.ino:134: Assertion failed: (ok=false) is true.

instead of:

AUnitTest.ino:134: Assertion failed: (3) == (4).
AUnitTest.ino:134: Assertion failed: (false) is true.

ArduinoUnit Compatibility: _As of v1.7, the assertion message format is compatible with the vim editor and other Linux/MacOS/Unix tools, and no longer compatible with ArduinoUnit

<a name="TestCaseSummary"></a>

Test Case Summary

As each test case finishes, the TestRunner prints out the summary of the test case like this:

Test bad failed.
Test looping_pass passed.
Test looping_skip skipped.
Test looping_until timed out.

ArduinoUnit Compatibility: These are identifcal to ArduinoUnit, except that the "timed out" status is new to AUnit. See Test Timeout section below.

<a name="TestRunnerSummary"></a>

Test Runner Summary

At the end of the test run, the TestRunner prints out the summary of all test cases, like this:

TestRunner duration: 0.05 seconds.
TestRunner summary: 12 passed, 0 failed, 2 skipped, 1 timed out, out of 15 test(s).

ArduinoUnit Compatibility: The message format is slightly different than ArduinoUnit. I changed "Test summary" to "TestRunner summary" because the former looks identical to the message that could have been printed by a test(summary) test case. AUnit also adds information about tests which timed out. See below.

<a name="TestTimeout"></a>

Test Timeout

ArduinoUnit Compatibility: Only available in AUnit.

From my experience, it seems incredibly easy to write a testing() test case which accidentally runs forever because the code forgets to call an explicit pass(), fail() or skip().

The TestRunner in AUnit applies a timeout value to all the test cases that it runs. The default timeout is 10 seconds. A timeout value of 0 means an infinite timeout, which means that the testing() test case may run forever. The value of the timeout is stored as a uint16_t type, so the maximum timeout is 65535 seconds or a bit over 18 hours.

Currently, the timeout value is global to all test cases. Time out values for individual tests cannot be set independently. If a test does not finish before that time, then the test is marked as timed out (internally implemented by the Test::expire() method) and a message is printed like this:

Test looping_until timed out.

The timeout value can be changed by calling the static TestRunner::setTimeout() method. Here is an example that sets the timeout to 30 seconds instead:

void setup() {
  ...
  TestRunner::setTimeout(30);
  ...
}

ArduinoUnit Compatibility: Only available in AUnit.

<a name="GoogleTestAdapter"></a>

GoogleTest Adapter

It may be possible to run simple unit tests written using Google Test API on an Arduino platform by using the aunit/contrib/gtest.h adapter. This adapter layer provides a number of Google Test macros which map to their equivalent macros in AUnit:

To use the gtest.h adapter, include the following headers:

#include <AUnit.h>
#include <aunit/contrib/gtest.h>

or

#include <AUnitVerbose.h>
#include <aunit/contrib/gtest.h>

<a name="CommandLineTools"></a>

Command Line Tools

Each unit test is an independent *.ino program. You can run it using your Arduino IDE. But there are 2 command line tools that can be used to run them.

<a name="AUniter"></a>

AUniter

The auniter.sh script used to be part of this project, but now lives in its own AUniter (https://github.com/bxparks/AUniter) project. The auniter.sh script is a wrapper around:

Using auniter.sh, you can compile, upload and validate multiple AUnit tests on multiple Arduino boards using a single command.

The script can monitor the serial port and determine if the unit test passed or failed, and it will print out a summary of all unit tests at the end. Full details are given in the AUniter project, but here are some quick examples copied from the AUniter/README.md file:

<a name="EpoxyDuino"></a>

EpoxyDuino

Instead of running the unit tests on the actual microcontrollers themselves, you can compile and execute AUnit unit tests natively on Linux or MacOS machines using the EpoxyDuino (https://github.com/bxparks/EpoxyDuino) project. EpoxyDuino provides a minimal Arduino programming environment that is usually sufficient to compile and run AUnit test units on the Unix host machine. It relies on the native C++ compiler, GNU Make, and Makefile files for each *.ino unit test like this:

APP_NAME := SampleTest
ARDUINO_LIBS := AUnit
include ../../../EpoxyDuino/EpoxyDuino.mk

The unit test is compiled into a binary (SampleTest.out) using the `make command, and the binary can be executed like this:

$ make
$ ./SampleTest.out

Here are a few tips when writing unit tests to run under EpoxyDuino:

Delay(1000)

For real Arduino boards, you get more reliable unit tests if you add a delay(1000) at the start of the program. For EpoxyDuino, this is not necessary, so I recommend calling this only on real Arduino boards, like this:

void setup() {
#if ! defined(EPOXY_DUINO)
  delay(1000); // Wait for stability on some boards, otherwise garage on Serial
#endif

  Serial.begin(115200);
  while (! Serial); // Wait until Serial is ready - Leonardo/Micro

#if defined(EPOXY_DUINO)
  Serial.setLineModeUnix(); // use Unix line terminator instead of DOS
#endif
  ...
}

void loop() {
  aunit::TestRunner::run();
}

Exit() Status Code

On real Arduino boards, the unit test (or any program for that matter) never terminates. The loop() function executes forever. On Linux or MacOS using EpoxyDuino, the test program will terminate at the end through the exit() function. If the tests are successful (i.e. passing or skipped), it will call exit(0). If there are any failing tests (i.e. failed or timed out), it will call exit(1).

<a name="CommandLineFlagsAndArguments"></a>

Command Line Flags and Arguments

(Added in v1.6)

The standard Arduino environment does not provide command line arguments, since a microcontroller does not normally provide a command line environment. However, if the AUnit test program is compiled under EpoxyDuino, the standard Unix command line parameters (argc and argv) become available through the extern int epoxy_argc and extern const char* const* epoxy_argv global variables. These allow the TestRunner class to provide command line flags and arguments as follows:

$ ./test.out --help
Usage: ./test.out [--help] [--include pattern,...] [--exclude pattern,...]
   [--includesub substring,...] [--excludesub substring,...]
   [--] [substring ...]

Example, the following runs all tests with substring "net" or "led" in its name, and skips all others:

$ ./test.out net led

Flags:

Arguments:

The command line flags and arguments are processed after any hardcoded calls to TestRunner::include() and TestRunner::exclude() methods in the global setup() method.

The flags and command line arguments are processed in order of appearance on the command line.

Similar to the hardcoded calls to TestRunner::include() and TestRunner::exclude(), if the first command line flag is an --include or --includesub, then all tests are excluded by default initially. Otherwise, the first include flag would have no effect.

<a name="ContinuousIntegration"></a>

Continuous Integration

There are several ways to incorporate AUnit into a continuous integration system. At the infrastructure level, you can use either a system like Jenkins running on a local machine, or use a cloud-based continuous integration system like GitHub Actions. For each of those infrastructures, you can choose to use the Arduino IDE or CLI build tools, or you can use EpoxyDuino to compile and run against a Linux or MacOS environment.

The option matrix looks like this:

+----------------+-------------------------+------------------------+
|\ CI environment|                         |                        |
| \____________  | Local (e.g. Jenkins)    | Cloud (e.g. GitHub)    |
|              \ |                         |                        |
| Build tool    \|                         |                        |
+----------------+-------------------------+------------------------+
|                | * can execute tests     | * verify compile-only  |
|                |   on microcontroller    | * cannot actually run  |
| Arduino        | * complex setup and     |   tests on             |
| IDE/CLI        | * maintenance           |   microcontroller      |
|                | * can be slow           | * complex installation |
|                | * (not recommended)     |   and setup            |
|                |                         | * (unverified)         |
+----------------+-------------------------+------------------------+
|                | * verify execution on   | * verify execution on  |
|                |   Unix environment      |   Unix environment     |
| EpoxyDuino     | * faster than IDE/CLI   | * simple setup and     |
|                | * complex setup and     |   maintenance          |
|                |   and maintenance       | * very fast            |
|                | * (unnecessary)         | * (recommended)        |
+----------------+-------------------------+------------------------+

<a name="IdePlusCloud"></a>

Arduino IDE/CLI + Cloud

The big advantage of using the Arduino IDE/CLI to run the AUnit unit tests is that you can compile them using the precise compiler and tool chain that will be used against the specific microcontroller that you are interested in verifying. There can be subtle compiler differences (e.g. size of int) or differences in how the Arduino programming environment was implemented (e.g. availability of the FPSTR() macro). If you use a cloud-based CI infrastructure, then you must install the complete Arduino IDE/CLI environment into your cloud test runner. Although I think it's theoretically possible, I have never actually verified that this can be done.

<a name="IdePlusJenkins"></a>

Arduino IDE/CLI + Jenkins

This setup is described in Continuous Integration with Jenkins, and it worked reasonably well for small number of unit tests. The problem is that the Arduino IDE is far too slow when the number of unit tests become non-trivial. And it also takes too much effort to maintain the local Jenkins infrastructure. The Jenkins environment seems to be brittle due to its complexity of all of its moving parts. This is the only environment where you can connect a real Arduino microcontroller to the local machine and have the unit tests run on the actual microcontroller. Perhaps for certain situations, running the unit tests on actual hardware is a requirement. But for most people, I no longer recommend this environment.

<a name="EpoxyDuinoPlusJenkins"></a>

EpoxyDuino + Jenkins

Once the Jenkins environment is up and running, I have verified that it is easy to run the unit tests using EpoxyDuino, since it needs just a C++ compiler and GNU Make. Things will compile and run a lot faster than using the Arduino IDE/CLI. However, this combination suffers from the same problem of maintaining the Jenkins environment. If the unit tests are running in an Unix environment anyway, it seems far easier to just run them in the cloud. So I don't recommend using this setup. Just use a cloud CI provider as described below.

<a name="EpoxyDuinoPlusCloud"></a>

EpoxyDuino + Cloud (Recommended)

A cloud-based continuous integration service like GitHub Actions is easy to setup for EpoxyDuino. Often the C++ compiler and GNU make tools are already installed in the Docker container used by the CI system. The only additional setup is to install EpoxyDuino, AUnit and other dependent Arduino libraries.

Here are some example YAML files for GitHub Actions:

In the various aunit_tests.yml files, the Setup step installs the various dependent libraries using the git clone command, for example:

git clone https://github.com/bxparks/EpoxyDuino
git clone https://github.com/bxparks/AUnit

These commands install the default branch for those repositories, which for most of my libraries will be the develop branch. This makes sense for me because I want the unit tests to run against the latest commits. However, for many others, it is probably better to use the master branch because it contains the stable releases:

git clone --branch master https://github.com/bxparks/EpoxyDuino
git clone --branch master https://github.com/bxparks/AUnit

Using EpoxyDuino with a cloud CI provider (like GitHub Actions) is my recommended configuration for running AUnit tests because it is easy to setup and maintain and the tests run fast.

<a name="Tips"></a>

Tips

Collection of useful tidbits.

<a name="DebuggingFixtures"></a>

Debugging Assertions in Fixtures

When using test fixtures with the testF() and testingF() macros, it's often useful to create helper assertions, such as the assertCustomStuff() below. Debugging such assertion statements can be tricky. I've found that turning on messages for successful assertions (with a enableVerbosity(Verbosity::kAssertionPassed)) statement can be very helpful:

class CustomTestOnce: public TestOnce {
  protected:
    // optional
    void setup() override {
      TestOnce::setup();
      ...setup code...
    }

    // optional
    void teardown() override {
      ...teardown code...
      TestOnce::teardown();
    }

    void assertCustomStuff() {
      assertEqual(...);
      ...
      for (...) {
        ...
        assertEqual(...);
        ...
      }
    }

    int sharedValue;
};

testF(CustomTestOnce, calculate) {
  enableVerbosity(Verbosity::kAssertionPassed);

  ...test code here...
  assertCustomStuff();
}

<a name="ClassHierarchy"></a>

Class Hierarchy

To support test fixtures in a more natural way, the class hierarchy in AUnit is slightly different than ArduinoUnit. In ArduinoUnit we have a two level hierarchy:

  Test ::loop()
    ^
    |
 TestOnce ::once()

In AUnit, the functionality that supports the testing() macro has been migrated to a separate class called TestAgain, like this:

       Test ::loop()
         ^
         |
      Assertion
         ^
         |
     MetaAssertion
        ^  ^
       /    \
      /      \
TestAgain  TestOnce
::again()  ::once()

Normally, deep inheritance hierarchies like this should be avoided. However, placing the Assertion and MetaAssertion classes inside the Test hierarchy allowed those assertion statements to have access to the internal states of the Test instance. This made certain features (like the early return upon delayed failure) slightly easier to implement. For the most part, the end-users can ignore the existence of the Assertion and MetaAssertion classes and think of this as a simple 2-level inheritance tree.

<a name="PrivateHelperMethods"></a>

Testing Private Helper Methods

There is a school of thought which says that unit tests should test only the publically exposed methods of a class or library. I agree mostly with that sentiment, but not rigidly. I think it is sometimes useful to write unit tests for protected or private methods. For example, when creating a chain of small helper methods, which build up to larger publically exposed methods, it is extremely useful to write unit tests for the helper methods in isolation.

Normally those helper methods would be private because they are used only within that class, and we don't want to expose them to the public API. One option is to make them public but add a comment in the function to say that it is exposed only for testing purposes. This does not seem satisfactory because users will tend to ignore such comments if the helper functions are useful.

I think a better way is to keep the helper functions private but make the unit tests a friend class of the target class. The syntax for doing this can be tricky, it took me a number of attempts to get this right, especially if when using namespaces for the target class:

//------------------- Target.h -------------

// Auto-generated test class names.
class Test_helper;
class TargetSuite_helper;
class TargetTest_helper;

namespace mylib {

class Target {
  public:
    void publicMethod() {
      ...
      int a = helper();
      ...
    }

  private:
    // Must have the global scope operator '::'
    friend class ::Test_helper;
    friend class ::TargetSuite_helper;
    friend class ::TargetTest_helper;

    static int helper() {...}
};

}

//------------------- TargetTest.ino -------------

#include <AUnit.h>
#include "Target.h"

using namespace aunit;
using namespace mylib;

test(helper) {
  assertEqual(1, Target::helper(...));
}

test(TargetSuite, helper) {
  assertEqual(1, Target::helper(...));
}

class TargetTest: public TestOnce {
  ...
};

testF(TargetTest, helper) {
  assertEqual(1, Target::helper(...));
}

The tricky part is that Target.h must have forward declarations of the various auto-generated AUnit test classes. And within the Target class itself, the friend declarations need to have a global scope :: specifier before the name of the test class.

<a name="Benchmarks"></a>

Benchmarks

<a name="MemoryBenchmark"></a>

MemoryBenchmark

The MemoryBenchmark directory collects the flash and static RAM usage of the AUnit library on various microcontroller boards, and renders the results as tables embedded in the README.md file. Here are 2 highlights:

Arduino Nano (8-bit)

+---------------------------------------------------------------------+
| Functionality                          |  flash/  ram |       delta |
|----------------------------------------+--------------+-------------|
| Baseline                               |   1586/  185 |     0/    0 |
|----------------------------------------+--------------+-------------|
| AUnit Single Test                      |   4456/  366 |  2870/  181 |
| AUnit Single Test Verbose              |   4500/  366 |  2914/  181 |
+---------------------------------------------------------------------+

ESP8266 (32-bit)

+---------------------------------------------------------------------+
| Functionality                          |  flash/  ram |       delta |
|----------------------------------------+--------------+-------------|
| Baseline                               | 264949/27984 |     0/    0 |
|----------------------------------------+--------------+-------------|
| AUnit Single Test                      | 268021/28148 |  3072/  164 |
| AUnit Single Test Verbose              | 268081/28148 |  3132/  164 |
+---------------------------------------------------------------------+

<a name="CompareArduinoUnit"></a>

Compared to ArduinoUnit 2.2

AUnit consumes as much as 65% less flash memory than ArduinoUnit 2.2 on an AVR platform (e.g. Arduino UNO, Nano), and 30% less flash on the Teensy-ARM platform (e.g. Teensy LC ). (ArduinoUnit 2.3 reduces the flash memory by 30% or so, which means that AUnit can still consume significantly less flash memory.)

Here are the resource consumption (flash and static) numbers from AceButtonTest containing 26 test cases using 331 assertXxx() statements, compiled using AUnit and ArduinoUnit 2.2 on 5 different microcontrollers:

---------------------------+---------+-------------+-------------+
Platform (resource)        |     Max | ArduinoUnit |       AUnit |
---------------------------+---------+-------------+-------------|
Arduino Nano (flash)       |   30720 |       54038 |       18928 |
Arduino Nano (static)      |    2048 |        1061 |         917 |
---------------------------+---------+-------------+-------------|
Teensy LC (flash)          |   63488 |       36196 |       26496 |
Teensy LC (static)         |    8192 |        2980 |        2780 |
---------------------------+---------+-------------+-------------|
Teensy 3.2 (flash)         |  262144 |       51236 |       37920 |
Teensy 3.2 (static)        |   65536 |        5328 |        5236 |
---------------------------+---------+-------------+-------------|
ESP8266 - ESP-12E (flash)  | 1044464 |    does not |      268236 |
ESP8266 - ESP-12E (static) |   81920 |     compile |       33128 |
---------------------------+---------+-------------+-------------|
ESP8266 - ESP-01 (flash)   |  499696 |    does not |      268236 |
ESP8266 - ESP-01 (static)  |   47356 |     compile |       33128 |
---------------------------+---------+-------------+-------------+

Not all unit test sketches will experience a savings of 65% of flash memory with AUnit, but a savings of 30-50% seems to be common.

<a name="SystemRequirements"></a>

System Requirements

<a name="Hardware"></a>

Hardware

Tier 1: Fully supported

These boards are tested on each release:

Tier 2: Should work

These boards should work but I don't test them as often:

Tier 3: May work, but not supported

Tier Blacklisted

The following boards are not supported and are explicitly blacklisted to allow the compiler to print useful error messages instead of hundreds of lines of compiler errors:

<a name="ToolChain"></a>

Tool Chain

This library was validated using:

This library is not compatible with:

(See Issue #56 and Issue #66).

It should work with PlatformIO but I have not tested it extensively.

The library works under Linux or MacOS (using both g++ and clang++ compilers) using the EpoxyDuino (https://github.com/bxparks/EpoxyDuino) emulation layer.

<a name="OperatingSystem"></a>

Operating System

I use Ubuntu 20.04 for the vast majority of my development. I expect that the library will work fine under MacOS and Windows, but I have not tested them.

<a name="License"></a>

License

MIT License

<a name="FeedbackAndSupport"></a>

Feedback and Support

If you have any questions, comments, or feature requests for this library, please use the GitHub Discussions for this project. If you have bug reports, please file a ticket in GitHub Issues. Feature requests should go into Discussions first because they often have alternative solutions which are useful to remain visible, instead of disappearing from the default view of the Issue tracker after the ticket is closed.

Please refrain from emailing me directly unless the content is sensitive. The problem with email is that I cannot reference the email conversation when other people ask similar questions later.

<a name="Authors"></a>

Authors