Awesome

FoRTE-Research's Fuzzing Benchmarks

This repository contains a collection of benchmarks and seed inputs to make fuzzing research more readily comparable and reproducable. For instructions on installing and running a given benchmark, refer to the readme file in its respective directory.

We welcome any suggestions for improving this fuzzing benchmark corpus! Our only criteria for additional benchmarks is that they are compatible with AFL-Clang, AFL-QEMU, and AFL-Dyninst tracing (as all 8 below are).

<table> <tr> <td align=center colspan="2"><div>Presented in our paper <a href="https://arxiv.org/abs/1812.11875">Full-speed Fuzzing: Reducing Fuzzing Overhead through Coverage-guided Tracing</a> (to appear in the 2019 IEEE Symposium on Security and Privacy).</td> </tr> <tr> <td>Citing this repository:</td> <td> <code class="rich-diff-level-one">@inproceedings{nagy:fullspeedfuzzing,</code> <code class="rich-diff-level-one">title = {Full-speed Fuzzing: Reducing Fuzzing Overhead through Coverage-guided Tracing},</code> <code class="rich-diff-level-one">author = {Stefan Nagy and Matthew Hicks},</code> <code class="rich-diff-level-one">booktitle = {{IEEE} Symposium on Security and Privacy (Oakland)},</code> <code class="rich-diff-level-one">year = {2019},}</code> </td> </tr> <tr> <td>Developers:</td> <td>Stefan Nagy (<a href="mailto:snagy2@vt.edu">snagy2@vt.edu</a>) and Matthew Hicks (<a href="mailto:mdhicks2@vt.edu">mdhicks2@vt.edu</a>)</td> </tr> <tr> <td>License:</td> <td><a href="/FoRTE-Research/UnTracer-AFL/blob/master/LICENSE">MIT License</a></td> </tr> <tr> <td>Disclaimer:</td> <td>This software is strictly a research prototype.</td> </tr> </table>

BENCHMARK STATISTICS

We utilized Dyninst to compute the following bechmark statistics for the 8 binaries evaluated in our paper. Note that we compiled all with Clang/Clang++.

benchname	libname	type	basic blocks	basic block edges
bsdtar	libarchive	archiv	31379	43390
cert-basic	libksba	crypto	9958	14120
cjson	cjson	web	1447	2038
djpeg	libjpeg	img	4844	6776
pdftohtml	poppler	doc	54596	71945
readelf	binutils	dev	21249	31086
sfconvert	audiofile	audio	5603	7403
tcpdump	tcpdump	net	33743	48791

For our paper, we collected statistics on the 24hr fuzzing testcase corpora for each benchmark. Note that these numbers reflect corpora generated using AFL with QEMU-based tracing.

benchname	libname	type	24hr corpus size	testcases/24hr	100ms timeouts
bsdtar	libarchive	archiv	90.9G	25.6M	4
cert-basic	libksba	crypto	7.5G	10.7M	6
cjson	cjson	web	4.5G	14.5M	221K
djpeg	libjpeg	img	30.1G	21.0M	656
pdftohtml	poppler	doc	0.2G	1.2M	107
readelf	binutils	dev	3.8G	14.9M	7
sfconvert	audiofile	audio	3.7G	10.1M	373K
tcpdump	tcpdump	net	2.7G	27.1M	5

BUILDING BENCHMARKS

Before you attempt to build the benchmarks, there are several dependencies that you must resolve:

sudo apt install build-essential clang libgpg-error-dev libfontconfig1-dev libpcap-dev

We provide the script buildAll.sh to compile all benchmarks from source. Edit the following parameters to reflect the desired C and C++ compilers, and any assembler parameters:

compiler=""
compilerXX=""
passToAS=""

Before building using AFL-modified compilers, make sure to upate PATH and AFL-PATH system variables to point to where you have AFL installed.

COLLECTING BENCHMARKS

We also provide the script collectAll.py to copy all compiled benchmark binaries to the current directory and append them with a use-specific (as specified on the command line) postfix:

Run as follows:

python /path/to/FoRTE-FuzzBench/collectAll.py [binaryPostfix]

All benchmarks will be copied to the current directory and appended the specified postfix (leave blank if none).