Home

Awesome

visualize_logs

A Python library and command line tools to provide log visualization.

Gallery

When you view these plots you will need JavaScript turned on. The plots are interactive and you can select borders around the pieces you would like to zoom into. You can double click to zoom out. You can also hover over nodes and more information will be displayed. The plot controls will be in the upper right hand corner of the plot.

The plot will look different depending on your browser (Chrome, Firefox, etc...) and the size of your browser. I typically use Chrome on a Mac with a very large size to see everything I want to see. The smaller your browser is, the more crunched it will be. If you change your browser size, be sure to click 'Reload'.

Cuckoo JSON Reports

Kovter Sample 1

SHA256: 15c237f6b74af2588b07912bf18e2734594251787871c9638104e4bf5de46589

This sample was identified in the following blog post.

Kovter Sample 2

SHA256: bffe7ccbcf69e7c787ff10d1dc7dbf6044bffcb13b95d851f4a735917b3a6fdf

This sample was identified in the following blog post.

Ransomware

SHA256: 9b462800f1bef019d7ec00098682d3ea7fc60e6721555f616399228e4e3ad122

wwwlgoogle dot com Adware

SHA256: e64910e3549a6c6e01be814b40e0f1fca02db45d5d19e2882a90914cef1c799e

This sample came from wwwlgoogle dot com.

ProcMon CSV Logs

The "focused" views were generated by selecting just the PIDs I wanted to show with ProcMon before saving the data to a CSV.

Kovter Sample 1

SHA256: 15c237f6b74af2588b07912bf18e2734594251787871c9638104e4bf5de46589

This sample was identified in the following blog post.

Kovter Sample 2

SHA256: bffe7ccbcf69e7c787ff10d1dc7dbf6044bffcb13b95d851f4a735917b3a6fdf

This sample was identified in the following blog post.

Ransomware

SHA256: 9b462800f1bef019d7ec00098682d3ea7fc60e6721555f616399228e4e3ad122

wwwlgoogle dot com Adware

SHA256: e64910e3549a6c6e01be814b40e0f1fca02db45d5d19e2882a90914cef1c799e

This sample came from wwwlgoogle dot com.

Log Type Support

This package currently supports the following types of logs:

More logs types are coming...

Requirements

Python v3

Install Python v3. I like to use virtualenv with my Python installs.

This program was written with Python 3 on a Mac and Windows 7. It should work with Python 2 and other OS's, but it has not been tested extensively. Please file an issue if you have problems running it somewhere. I use Windows less than I use a Mac, so your Windows mileage may vary.

Graphviz

Graphviz must be installed and available in your path (dot, neato, etc..)

To install Graphviz correctly on a Mac, you will probably want to run the following command:

brew install graphviz --with-gts

Installation

# pip install visualize_logs

... or you can clone and ...

# python setup.py install

Usage

This package can be used as a library. Use the information in the documentation section below to use it this way. This package also contains command line tools which are outlined below.

plotcuckoojson

If you have a cuckoo-modified JSON report, you can use this tool to plot the results.

# plotcuckoojson -h
usage: plotcuckoojson [-h] [-f HTMLFile] [-t TITLE] [-na] [-fa] [-fc] [-fm]
                      [-fp] [-fd] [-fw] [-fr] [-ra] [-rc] [-rd] [-rw] [-rr]
                      [-ignpaths IgnPathsFile.txt]
                      [-inclpaths InclPathsFile.txt] [-gp GRAPHVIZPROG]
                      CuckooJSONReportFile

Application to graph cuckoo-modified JSON reports

positional arguments:
  CuckooJSONReportFile  cuckoo-modified JSON report file

optional arguments:
  -h, --help            show this help message and exit
  -f HTMLFile, --file HTMLFile
                        Create the html report. Default name is
                        cuckoojson.html
  -t TITLE, --title TITLE
                        The title for the plot
  -na, --nonetwork      Turn off all network activity
  -fa, --nofiles        Turn off all file activity
  -fc, --nofilecreates  Turn off file create activity
  -fm, --nofilemoves    Turn off file move activity
  -fp, --nofilecopies   Turn off file copy activity
  -fd, --nofiledeletes  Turn off file delete activity
  -fw, --nofilewrites   Turn off file write activity
  -fr, --nofilereads    Turn off file read activity
  -ra, --noregistry     Turn off all registry activity
  -rc, --noregcreates   Turn off registry create activity
  -rd, --noregdeletes   Turn off registry delete activity
  -rw, --noregwrites    Turn off registry write activity
  -rr, --noregreads     Turn off registry read activity
  -ignpaths IgnPathsFile.txt, --ignorepathsfile IgnPathsFile.txt
                        File containing regular expressions to ignore for
                        files and registry. One RE per line.
  -inclpaths InclPathsFile.txt, --includepathsfile InclPathsFile.txt
                        File containing regular expressions to include for
                        files and registry. Overrides ignores. One RE per
                        line.
  -gp GRAPHVIZPROG, --graphvizprog GRAPHVIZPROG
                        The graphviz layout program to use. Valid options are
                        dot, neato, twopi, circo, fdp, sfdp, patchwork and
                        osage. Research the graphviz website for more
                        information on these types of layouts. IF YOU SUPPLY
                        AN INVALID VALUE THIS PROGRAM WILL NOT WORK! Default:
                        sfdp

You can run it like this:

# plotcuckoojson 1_report.json 
Reading log: 1_report.json
Plotting log: 1_report.json

... and then your plot appears in your web browser! It is also saved to cuckoojson.html.

plotprocmoncsv

The best use case is if you start your ProcMon capture before you run the file you are analyzing. If a process is not started the associated network connections may not be connected to the process in the plot. I also could not get ProcMon to capture TCP data when WinPCAP was installed. You may not want to install WinPCAP if you are interested in TCP data.

This this library feels like it is taking a long time, it is likely that you are trying to import a lot of ProcMon data. You can always filter your data and save it as a CSV showing just the events you want to graph.

# plotprocmoncsv -h
usage: plotprocmoncsv [-h] [-f HTMLFile] [-pa] [-pf] [-pu] [-pt] [-pr] [-pfw]
                      [-pfr] [-pfd] [-pfn] [-ptcp] [-pus] [-pur] [-prr] [-prw]
                      [-prd] [-sa] [-sp] [-st] [-su] [-sf] [-sh] [-sr]
                      [-ignpaths IgnPathsFile.txt]
                      [-inclpaths InclPathsFile.txt]
                      ProcMonCSVFile

Application to graph ProcMon CSV files

positional arguments:
  ProcMonCSVFile        ProcMon CSV file

optional arguments:
  -h, --help            show this help message and exit
  -f HTMLFile, --file HTMLFile
                        Create the log file. Default name is procmoncsv.html
  -pa, --plotall        Plot all aspects
  -pf, --plotfile       Plot all file aspects
  -pu, --plotudp        Plot all UDP aspects
  -pt, --plottcp        Plot all TCP aspects
  -pr, --plotreg        Plot all Registry aspects
  -pfw, --plotfilewrites
                        Plot file writes
  -pfr, --plotfilereads
                        Plot file reads
  -pfd, --plotfiledeletes
                        Plot file deletes
  -pfn, --plotfilerenames
                        Plot file renames
  -ptcp, --plottcpconnects
                        Plot TCP connects
  -pus, --plotudpsends  Plot UDP sends
  -pur, --plotudprecvs  Plot UDP receives
  -prr, --plotregreads  Plot Registry reads
  -prw, --plotregwrites
                        Plot Registry writes
  -prd, --plotregdeletes
                        Plot Registry deletes
  -sa, --showalllabels  Show all labels
  -sp, --showproclabels
                        Show process labels
  -st, --showtcplabels  Show TCP labels
  -su, --showudplabels  Show UDP labels
  -sf, --showfilelabels
                        Show file labels
  -sh, --showhostlabels
                        Show host labels
  -sr, --showreglabels  Show Registry labels
  -ignpaths IgnPathsFile.txt, --ignorepathsfile IgnPathsFile.txt
                        File containing regular expressions to ignore in the
                        Path column. One RE per line.
  -inclpaths InclPathsFile.txt, --includepathsfile InclPathsFile.txt
                        File containing regular expressions to include in the
                        Path column. Overrides ignores. One RE per line.

You can run it like this:

# plotprocmoncsv -pa -sp -st -sh wwwlgoogle.CSV 
Reading log: wwwlgoogle.CSV
Plotting log: wwwlgoogle.CSV

... and then your plot appears in your web browser! It is also saved to procmoncsv.html.

Sample data

Cuckoo-Modified Sample Data

You can find some sample JSON reports from cuckoo-modified in the cuckoo-modified-json directory.

There are four traces:

ProcMon CSV Sample Data

You can find some sample CSV from ProcMon in the [ProcMon/Sample Data] (ProcMon/Sample Data/) directory. There are eight traces:

Documentation

The library documentation can be found at: https://keithjjones.github.io/visualize_logs.github.io/

Resources

Similar Projects

These projects are very similar to this one and are worth trying if you are unfamiliar with them. They were the inspiration behind this project. This project was meant to compliment these tools, not replace them. This project is just a different way to get to the same goal, with the goal being to support numerous log types in one Python package in the future.

License:

This application is covered by the Creative Commons BY-SA license.

Contributing:

If you would like to contribute you can fork this repository, make your changes, and then send me a pull request to my "dev" branch.