Awesome

NinjaTurtlesMutation

If you're not familiar with what is mutation testing, this article give a good explanation on the java case.

NinjaTurtlesMutation is a fork of NinjaTurtles. It aims to provide a fast mutation testing tool for codebases using NUnit v2.6.4.

NTM is fast and generates easy to read reports of mutation coverage:

Table of Contents

• Prerequisites
• Installation
• Usage
  • Test a class
  • Test all classes in a namespace
  • Generate a file report
    • Xml
    • Html
• Differences with NinjaTurtles
• Licence
• Authors
• Copyrights

Prerequisites

• Visual Studio 2010+. It's recommended that you have NuGet installed.
• .NET Framework v4.5+

Installation

The followings steps assume that you are using a sh-like terminal

Clone the repository

git clone git@github.com:criteo/NinjaTurtlesMutation.git

Build NinjaTurtlesMutation

To build from command-line (tested with MSBuild 4.6.1 / .NET 4.0.3):

/c/Windows/Microsoft.NET/Framework/v4.0.30319/MSBuild.exe NinjaTurtlesMutation.sln

or you can open NinjaTurtlesMutation.sln and build from Visual Studio

Set the PATH to the executable

export PATH=$PATH:"path/to/your/clone/NinjaTurtlesMutation.Console/bin/Debug"

Usage

To illustrate basic usages, I'll use a dummy solution with two projects. One for the library, one for the tests

You can find this demo project inside Demofiles/PrimeFinderMutationPlayground

| PrimeFinderMutationPlayground
    | PrimeFinderMutationPlayground.IDummy
    | PrimeFinderMutationPlayground.Dummy
    | PrimeFinderMutationPlayground.PrimeFinder
| PrimeFinder.UTest
    | PrimeFinder.UTest.T_PrimeMutationPlayground

The library has one namespace (PrimeFinderMutationPlayground) with three classes. The test library has one namespace (PrimeFinder.UTest) with one class. When built, this solution generates two dlls: PrimeFinderMutationPlayground.dll and PrimeFinder.UTest.dll

Test a class

ntm.exe run -c PrimeFinderMutationPlayground.PrimeFinder path/to/test/assembly/PrimeFinder.UTest.dll

This command start a mutation test on all methods contained in the PrimeFinder class. During his progress, NTM displays the full name of each method tested. This output the following:

NinjaTurtlesMutation - mutation testing for .NET (version 1.0.0.0)
Copyright (C) 2016 Tony Roussel
Copyright (C) 2012 David Musgrove

Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder.IsPrime(Int32)
Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder.IsPrimed(Int32, IDummy)
Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder..ctor

Surviving mutants:
  c:\Users\t.roussel\PrimeFinder\PrimeFinderMutationPlayground\PrimeFinderMutationPlayground\PrimeFinder.cs
      Line 20
          Deleting sequence points from IL.                                 (1)
              000c: deleting Ldstr, Newobj, Throw

[...]

      Line 26
          Deleting sequence points from IL.                                 (1)
              0035: deleting Ldc_I4, Stloc, Br

          Replacing writes to local variables and fields.                   (1)
              0036: write substitution Boolean.CS$1$0000 => Boolean.CS$4$0001




Mutation score: 0,88 (77 / 88)
Mutation testing failed

TL;DR: From bottom to top, you can find mutation test's result (fail/success), the mutation score, details about surviving mutants, and methods that NTM has tested.

At the bottom, NTM displays the mutation score (0.88): which is the ratio mutants killed over mutants generated (77 / 88). The higher the better, capping at 1.

Above the mutation score, surviving mutants are described. The listing is sorted by files, lines and then mutation type. Numbers in parentheses tell you how many mutants were generated by this kind of mutation operator, at this line. For each of those mutants, details of the IL modifications are printed.

And then on top of the surviving mutants list, NTM display all tested methods. Here we can see that 3 methods were found inside PrimeFinder class.

Test all classes in a namespace

ntm.exe run -N PrimeFinderMutationPlayground path/to/test/assembly/PrimeFinder.UTest.dll

This command start a mutation test on all methods, of every classes contained in PrimeFinderMutationPlayground namespace. The output is like the "class mode" described in the previous section, except it cover more methods:

NinjaTurtlesMutation - mutation testing for .NET (version 1.0.0.0)
Copyright (C) 2016 Tony Roussel
Copyright (C) 2012 David Musgrove

Running mutation tests for PrimeFinderMutationPlayground.Dummy..ctor
Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder.IsPrime(Int32)
Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder.IsPrimed(Int32, IDummy)
Running mutation tests for PrimeFinderMutationPlayground.PrimeFinder..ctor

Surviving mutants:
  c:\Users\t.roussel\PrimeFinder\PrimeFinderMutationPlayground\PrimeFinderMutationPlayground\PrimeFinder.cs
      Line 20
          Deleting sequence points from IL.                                 (1)
              000c: deleting Ldstr, Newobj, Throw

[...]

Mutation score: 0,88 (77 / 88)
Mutation testing failed

Generate a file report

You can generate two kind of reports: XML and HTML.

They both contain the path and code of the original source files, a mutated lines listing, and which one survived.

The html file is built from the xml. So some informations can only be found in the xml report. These informations include the start/end lines/columns of the mutations (a mutation can span on several line), and details of IL changes made for each mutants, even the one that has been killed.

I want the xml report !

Use the --output [-o] option, and give a filename with it. It should look like this:

ntm.exe run -o mutation_report.xml -N PrimeFinderMutationPlayground PrimeFinder.UTest.dll

I want the html report !

Like the xml report, use --output [-o] to specify a filename and set the format with the --format [-f] option as follow:

ntm.exe run -o mutation_report.html -f HTML -N PrimeFinderMutationPlayground PrimeFinder.UTest.dll

Differences with NinjaTurtles

• Only compatible with tests made with NUnit (2.6.2 to 2.6.4)
• Have a namespace mode to test several classes at once
• Report of surviving mutants in the console output
• Tests run are completely separated from the NTM application. A dispatching process exchange tests queries/results with NTM. The dispatcher also handle the workload between several test runner processes
• A NUnit.Core.RemoteTestRunner instance to run tests (no longer a nunit-console process)
• Benchmark mutant's tests set to detect infinite loop quicker
• Better handling of methods with generic parameters
• The HTML report include details about mutated line that survive
• Use symlink instead of file copy for non-mutated assembly
• Stability improvement

Licence

GPL/LGPL Version 3, 29 June 2007 More details in LICENSE files

Authors

Tony Roussel

David Musgrove - Owner of the NinjaTurtles project

A list of original NinjaTurtles project's owners/contributors can be found in the codeplex page

Copyrights

Copyright (C) 2016 Tony Roussel

Copyright (C) 2012-14 David Musgrove and others