Home

Awesome

Tribble

Build Status

Have a jvm application with an open socket? Need to test your application against un-expected input? Tribble can help generating and testing un-expected input and find out what happens.

Tribble is an easy to use fuzz testing tool for java applications using coverage to guide the process. Heavily based on the wonderful GoFuzz and AFL.

It uses Jacoco to get coverage stats and has a maven plug for running.

Usage

Maven

<repositories>
    <repository>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>bintray</name>
        <url>http://jcenter.bintray.com</url>
    </repository>
</repositories>
<pluginRepositories>
    <pluginRepository>
        <snapshots>
            <enabled>false</enabled>
        </snapshots>
        <id>central</id>
        <name>bintray-plugins</name>
        <url>http://jcenter.bintray.com</url>
    </pluginRepository>
</pluginRepositories>
<dependency>
    <groupId>org.catapult.sa</groupId>
    <artifactId>tribble-core_2.12</artifactId>
    <version>0.8.0</version>
    <classifier>shadow</classifier>
    <scope>test</scope>
</dependency>
<plugin>
    <groupId>org.catapult.sa</groupId>
    <artifactId>tribble-maven-plugin_2.12</artifactId>
    <version>0.8.0</version>
    <configuration>
        <target>org.catapult.sa.testcase.TestCase</target>
    </configuration>
</plugin>

SBT

resolvers += Resolver.jcenterRepo
"org.catapult.sa" %% "tribble-core" % "0.8.0" % "test" classifier "shadow"

Command Line

Example Test Class

import org.catapult.sa.tribble.{FuzzResult, FuzzTest}

class TestCase extends FuzzTest {

  def test(data : Array[Byte]): FuzzResult = {
    Fish.wibble(data)
    FuzzResult.OK
  }
}

General Usage

Stats will be printed to stderr at regular intervals. A folder called failed will be created on startup and populated with both inputs and stack traces that they cause. The file names will be the md5 hash of the input. Inputs which generate new code paths will be saved to the corpus directory with a file name that is an md5 hash of the code coverage stats.

Try and avoid printing to stdout and stderr in your test cases this can slow things down quite a lot. There is a one second time out on each test run.

When creating corpus entries you can append .hex to the file name and have the file hex encoded rather than raw bytes. This can make it a lot easier to include non printing characters. Generated corpus entries and failed inputs will use this if it finds unprintable characters in an input array.

If you need a particular version of scala we provide builds against 2.12. Change the bit on the end of the group name If you need something else edit the gradle.properties file and then run ./gradlew install

Why?

Because we needed one, you probably do too. If you don't think you do you don't have enough tests yet.

What is fuzz testing?

Fuzz testing (or smoke testing) is the process of sending "random" data (we'll come back to this in a bit) into a system and seeing what happens. The process of doing this has been around for a long time. However most of the time the input is generated from known input and has no way of knowing what happened inside the system under test. Systems like AFL and GoFuzz are able to instrument the system and know when they have found new code paths. This enables them to further modify the input data to work their way through the application.

Using systems like this is more effective and efficient that just sending random data into the system. Unfortunately such systems either don't work with Java/the JVM or have to start the application externally. With the JVM this introduces a significant slowdown as the optimisations and compilations from previous runs are lost.

The GoFuzz tool allows the creation of a simple test function which is used as the basis of the test. Tribble used a similar approach and is based heavily on the work of GoFuzz and AFL, although not yet as advanced. This allows the tests to be run with out restarting the JVM, which makes things a lot faster. However it does mean that if you find a JVM crashing bug you will lose the node. We can recover from most out of memory errors. We use Jacoco to generate code coverage stats for a run, which lets us work out the paths through the application we have found.

Why the name Tribble?

Lots of small fuzzy creatures from Star trek. It seemed like a reasonable name when I started this.

"Letting engineers name things is like letting the marketing department build them"

"Advanced" settings

There are some extra settings you may wish to use if needed. They are accessible either from the maven plugin or the command line.

TagParameterDescriptionDefault
target--targetClassFully qualified class name of a class which implements org.catapult.sa.tribble.FuzzTest
corpusPath--corpusPath to the corpus foldercorpus
failedPath--failedPath to the failed folderfailed
threads--threadsNumber of threads to use2
timeout--timeoutTime out for an individual test run in milliseconds1000
count--countnumber of iterations to run. Number greater than zero for a limit-1
verbose--verboseShould verbose mutator stats be printed.false
ignoreClassesList of class prefixes e.g org.apache.hadoop. to ignore from coverage. This is useful if you find some classes are already loaded.
--ignoreClassesComma Separated list of class prefixes to ignore. e.g org.apache.hadoop.
disabledMutatorsList of mutator class names to disable
--disabledMutationsComma separated list of mutators to disable

Extensions

Custom mutators

It is possible to add extra mutators using a plugin like system. Create a class that extends the org.catapult.sa.tribble.mutators.Mutator trait. The list the class in a file called org.catapult.sa.tribble.mutators The default mutation engine will look for files with that name that contain lists of classes to use as mutators.

Extra keywords

If you are fuzzing a system which takes defined keywords it can take a while for the system to come up with them on its own. There is a keyword mutator, which can be extended by adding your own file called org.catapult.sa.tribble.keywords with one entry per line. On start up the keyword mutator will search the class path for files with that name.

Design goals:

  1. Easy to use and setup
  2. Easy to develop and extent
  3. Fast

Where goals conflict the higher design goal should take precedence. Code that is harder to update but faster should be used with extreme caution.

Where next?

See the Issues list.

Contributing

See the CONTRIBUTING.md file and the Code of conduct

Contributors

Thanks

This project owes a huge thank you to the AFL project and go-fuzz for the ideas. JaCoCo for the code coverage tool this is built on top of.