Awesome
Summary
Mesos is a tool to gather binary code coverage on all user-land Windows targets without need for source or recompilation. It also provides an automatic mechanism to save a full minidump of a process if it crashes under mesos.
Mesos is technically just a really fast debugger, capable of handling tens of millions of breakpoints. Using this debugger, we apply breakpoints to every single basic block in a program. These breakpoints are removed as they are hit. Thus, mesos converges to 0-cost coverage as gathering coverage only has a cost the first time the basic block is hit.
Why?
This is effectively the successor of my 5+ year old Chrome IPC fuzzer. It doesn't have any fuzz components in it, but it is a high-performance debugger. This debugger can apply millions of breakpoints to gather coverage, and handle thousands of breakpoints per second to modify memory to inject inputs.
This strategy has worked out well for me historically and still is my go-to tooling for fuzzing targets on live systems.
Out of the box it can be used to gather simple code coverage but it's designed to be easily modified to add fast breakpoint handlers to inject inputs. For example, put a breakpoint after NtReadFile() returns and modify the buffer in flight. I used this in Chrome to modify inbound IPC traffic in the browser.
Features
Code coverage
Automatic full minidump saving
IDA Coloring
Quick Usage Guide
Set %PATH% such that idat64.exe is in it:
path %PATH%;"C:\Program Files\IDA 7.2"
Generate mesos (the first time will be slow):
powershell .\offline_meso.ps1 <pid>
python generate_mesos.py process_ida
Gather coverage on target!
cargo build --release
target\release\mesos.exe <pid>
Applying 1.6 million breakpoints? No big deal.
C:\dev\mesos>target\release\mesos.exe 13828
mesos is 64-bit: true
target is 64-bit: true
[ 0.003783] Applied 5629 breakpoints ( 5629 total breakpoints) notepad.exe
[ 0.028071] Applied 61334 breakpoints ( 66963 total breakpoints) ntdll.dll
[ 0.035298] Applied 25289 breakpoints ( 92252 total breakpoints) kernel32.dll
[ 0.058815] Applied 55611 breakpoints ( 147863 total breakpoints) kernelbase.dll
...
[ 0.667417] Applied 11504 breakpoints ( 1466344 total breakpoints) oleacc.dll
[ 0.676151] Applied 19557 breakpoints ( 1485901 total breakpoints) textinputframework.dll
[ 0.705431] Applied 66650 breakpoints ( 1552551 total breakpoints) coreuicomponents.dll
[ 0.717276] Applied 25202 breakpoints ( 1577753 total breakpoints) coremessaging.dll
[ 0.720487] Applied 7557 breakpoints ( 1585310 total breakpoints) ntmarta.dll
[ 0.732045] Applied 28569 breakpoints ( 1613879 total breakpoints) iertutil.dll
API
Currently this tool has a debugger lib you can easily bring in and start using to make custom debuggers. However this API is not finalized yet. I suggest not building anything using it yet as it may change very quickly. Once this message is removed it's probably stable :)
Performance
- We can register (request breakpoints to be at module load) about ~6 million/second
- We can apply them (actually install the breakpoints into the target at about ~3 million/second
- We can clear breakpoints at about 15 million/second
- We can hit and handle about 10k breakpoints/second
Given breakpoints are cleared as they're hit for coverage, that means you can observe 10k new blocks per second. Once you've hit a breakpoint they no longer have a performance cost!
C:\dev\mesos\examples\benchmark>cargo run --release
Finished release [optimized + debuginfo] target(s) in 0.03s
Running `target\release\benchmark.exe`
mesos is 64-bit: true
target is 64-bit: true
Registered 1000000 breakpoints in 0.162230 seconds | 6164072.8 / second
Applied 1000000 breakpoints in 0.321347 seconds | 3111897.0 / second
Cleared 1000000 breakpoints in 0.067024 seconds | 14920028.6 / second
Hit 100000 breakpoints in 10.066440 seconds | 9934.0 / second
Usage
To use mesos there are 3 major steps. First, the modules of a running process are saved. Second, these modules are loaded in IDA which then outputs a list of all basic blocks into the meso format. And finally, mesos is run against a target process to gather coverage!
Creating meso_deps.zip
This step is the first thing we have to do. We create a ZIP file containing all of the modules loaded into a given PID.
This script requires no internet and is designed to be easily dropped onto new VMs so mesos can be generated for your target application. It depends on PowerShell v5.0 or later which is installed by default on Windows 10 and Windows Server 2016.
Run, with <pid> replaced with the process ID you want to gather coverage on:
C:\dev\mesos>powershell .\offline_meso.ps1 8484
Powershell is 64-bit: True
Target is 64-bit: True
C:\dev\mesos>
Optionally you can supply -OutputZip <zipfile> to change the output zip file name
This will create a meso_deps.zip that if you look at contains all of the modules used in the process you ran the script targeting.
Example output:
C:\dev\mesos>powershell .\offline_meso.ps1 8484 -OutputZip testing.zip
Powershell is 64-bit: True Target is 64-bit: True C:\dev\mesos>powershell Expand-Archive testing.zip -DestinationPath example
C:\dev\mesos>powershell Get-ChildItem example -rec -File -Name
cache\c_\program files\common files\microsoft shared\ink\tiptsf.dll
cache\c_\program files\intel\optaneshellextensions\iastorafsserviceapi.dll
cache\c_\program files\widcomm\bluetooth software\btmmhook.dll
cache\c_\program files (x86)\common files\adobe\coresyncextension\coresync_x64.dll
...
Generating meso files
To generate meso files we operate on the meso_deps.zip we created in the last step. It doesn't matter where this zip came from. This allows the zip to have come from a VM that the PowerShell script was run on.
Basic usage is:
python generate_mesos.py process_ida
This will use the meso_deps.zip file as an input, and use IDA to process all executables in the zip file and figure out where their basic blocks are.
This will create a cache folder with a bunch of files in it. These files are named based on the module name, the modules TimeDateStamp in the PE header, and the ImageSize field in the PE header. This is what DLLs are uniqued by in the PDB symbol store, so it should be good enough for us here too.
You'll see there are files with no extension (these are the original binaries), there are files with .meso extensions (the breakpoint lists), and .i64 files (the cached IDA database for the original binary).
Symbol resolution
There is no limitation on what can make these meso files. The quality of the symbol resolution depends on the tool you used to generate and it's ability to resolve symbols. For example with IDA if you have public/private symbols your _NT_SYMBOL_PATH should be configured correctly.
More advanced usage
Check the programs usage for the most recent usage. But there are _whitelist and _blacklist options that allow you to use a list of strings to filter the amount of mesos generated.
This is helpful as coverage outside of your target module is probably not relevant and just introduces overheads and unnecessary processing.
C:\dev\mesos>python generate_mesos.py
Usage:
generate_mesos.py process_ida
Processes all files in the meso_deps.zip file
generate_mesos.py process_ida_whitelist <str 1> <str 2> <str ...>
Processes files only containing one of the strings provided
generate_mesos.py process_ida_blacklist <str 1> <str 2> <str ...>
Processes files all files except for those containing one of the provided strings
Examples:
python generate_mesos.py process_ida_whitelist system32
Only processes files in `system32`
python generate_mesos.py process_ida_blacklist ntdll.dll
Process all files except for `ntdll.dll`
Path requirements for process_ida_*: must have `idat64.exe` in your PATH
Example usage
C:\dev\mesos>python generate_mesos.py process_ida_whitelist system32
Processing cache/c_/windows/system32/advapi32.dll
Processing cache/c_/windows/system32/bcryptprimitives.dll
Processing cache/c_/windows/system32/cfgmgr32.dll
...
Processing cache/c_/windows/system32/user32.dll
Processing cache/c_/windows/system32/uxtheme.dll
Processing cache/c_/windows/system32/win32u.dll
Processing cache/c_/windows/system32/windows.storage.dll
Processing cache/c_/windows/system32/wintypes.dll
Meso usage
Now we're onto the actual debugger. We've created meso files to tell it where to put breakpoints in each module.
First we need to build it with Rust!
cargo build --release
And then we can simply run it with a PID!
target\release\mesos.exe <pid>
Command-line options
Currently there are few options to mesos, run mesos without arguments to get the most recent list.
C:\dev\mesos>target\release\mesos.exe
Usage: mesos.exe <pid> [--freq | --verbose | --print] <explicit meso file 1> <explicit meso file ...>
--freq - Treats all breakpoints as frequency breakpoints
--verbose - Enables verbose prints for debugging
--print - Prints breakpoint info on every single breakpoint
[explicit meso file] - Load a specific meso file regardless of loaded modules
Standard usage: mesos.exe <pid>
Example usage
C:\dev\mesos>target\release\mesos.exe 13828
mesos is 64-bit: true
target is 64-bit: true
[ 0.004033] Applied 5629 breakpoints ( 5629 total breakpoints) notepad.exe
[ 0.029248] Applied 61334 breakpoints ( 66963 total breakpoints) ntdll.dll
[ 0.037032] Applied 25289 breakpoints ( 92252 total breakpoints) kernel32.dll
[ 0.062844] Applied 55611 breakpoints ( 147863 total breakpoints) kernelbase.dll
...
[ 0.739059] Applied 66650 breakpoints ( 1552551 total breakpoints) coreuicomponents.dll
[ 0.750266] Applied 25202 breakpoints ( 1577753 total breakpoints) coremessaging.dll
[ 0.754485] Applied 7557 breakpoints ( 1585310 total breakpoints) ntmarta.dll
[ 0.766119] Applied 28569 breakpoints ( 1613879 total breakpoints) iertutil.dll
...
[ 23.544097] Removed 5968 breakpoints in imm32.dll
[ 23.551529] Syncing code coverage database...
[ 23.675103] Sync complete (169694 total unique coverage entries)
Detached from process 13828
Why not use cargo run?
When running in cargo run the Ctrl+C handler does not work correctly, and does not allow us to detach from the target program cleanly.
Limitations
Since this relies on a tool (IDA) to identify blocks, if the tool incorrectly identifies a block it could result in us inserting a breakpoint over data. Further it's possible to miss coverage if a block is not correctly found.
Why doesn't it do more?
Well. It really just allows fast breakpoints. Feel free to rip it apart and add your own hooks to functions. It could easily be used to fuzz things :)
Why IDA?
I tried a bunch of tools and IDA was the only one that seemed to work well. Binja probably would also work well but I don't have it installed and I'm not familiar with the API. I have a coworker who wrote a plugin for it and that'll probably get pull requested in soon.
The meso files are just simple files, anyone can generate them from any tool
Technical Details
Minidump autogenned filenames
The generated minidump filenames are designed to give a high-level of glance value at crashes. It includes things like the exception type, faulting address, and rough classification of the bug.
Currently if it's an access violation we apply the following classification:
- Determine the access type (read, write, execute)
- For reads the filename contains: "read"
- For writes the filename contains: "WRITE"
- For execute the filename contains: "DEP"
- Determine if it's a non-canonical 64-bit address
- For non-canonical addresses the filename contains: NONCANON
- Otherwise determine if it's a NULL dereference (within 32 KiB +- of NULL)
- Will put "null" in the filename
- Otherwise it's considered a non-null deref and "HIGH" appears in the filename
It's intended that more severe things are in all caps to give higher glance value of prioritizing which crash dumps to look into more.
Example minidump filename for chrome:
crash_c0000005_chrome_child.dll+0x2c915c0_WRITE_null.dmp
Meso file format
Coming soon (once it's stable)