Awesome

 _________   _    _   ______  _____  ______
| | | | | \ | |  | | | |  \ \  | |  | |  \ \     /.)
| | | | | | | |  | | | |  | |  | |  | |  | |    /)\|
|_| |_| |_| \_|__|_| |_|  |_| _|_|_ |_|  |_|   // /
                                              /'" "

Online Hash Checker for Virustotal and Other Services
Florian Roth

What is Munin?

Munin is a online hash checker utility that retrieves valuable information from various online sources

The current version of Munin queries the following services:

Screenshot

Default Mode - Read Hashes from File

Munin Screenshot

Usage

usage: munin.py [-h] [-f path] [--vh search-string]
                [--vhrule search-string] [-o output] [--vtwaitquota]
                [--vtminav min-matches] [--limit hash-limit]
                [--vhmaxage days] [-c cache-db] [-i ini-file]
                [-s sample-folder] [--comment] [-p vt-comment-prefix]
                [--download] [-d download_path] [--nocache] [--nocsv]
                [--verifycert] [--sort] [--web] [-w port] [--cli]
                [--rescan] [--debug]

Online Hash Checker

optional arguments:
  -h, --help            show this help message and exit
  -f path               File to process (hash line by line OR csv with hash
                        in each line - auto-detects position and comment)
  --vh search-string    Query Valhalla for hashes by keyword, tags, YARA
                        rule name, Mitre ATT&CK software (e.g. S0154),
                        technique (e.g. T1023) or threat group (e.g. G0049)
  --vhrule search-string
                        Query Valhalla for hashes via rules by keyword,
                        tags, YARA rule name, Mitre ATT&CK software (e.g.
                        S0154), technique (e.g. T1023) or threat group
                        (e.g. G0049)
  -o output             Output file for results (CSV)
  --vtwaitquota         Do not continue if VT quota is exceeded but wait
                        for the next day
  --vtminav min-matches
                        Minimum number of AV matches to query hash info
                        from VT"
  --limit hash-limit    Exit after handling this much new hashes in batch
                        mode (cache ignored).
  --vhmaxage days       Maximum age of sample on Valhalla to process
  -c cache-db           Name of the cache database file (default: vt-hash-
                        db.json)
  -i ini-file           Name of the ini file that holds the API keys
  -s sample-folder      Folder with samples to process
  --comment             Posts a comment for the analysed hash which
                        contains the comment from the log line
  -p vt-comment-prefix  Virustotal comment prefix
  --download            Enables Sample Download from Hybrid Analysis.
                        SHA256 of sample needed.
  -d download_path      Output Path for Sample Download from Hybrid
                        Analysis. Folder must exist
  --nocache             Do not use cache database file
  --nocsv               Do not write a CSV with the results
  --verifycert          Verify SSL/TLS certificates
  --sort                Sort the input lines
  --web                 Run Munin as web service
  -w port               Web service port
  --cli                 Run Munin in command line interface mode
  --rescan              Trigger a rescan of each analyzed file
  --debug               Debug output

Features

Retrieves valuable information from Virustotal via API (JSON response) and other information via permalink (HTML parsing)
Retrieves extra information from a list of platforms
Keeps a history (cache) to query the services only once for a hash that may appear multiple times in the text file
Cached objects are stored in JSON
Creates CSV file with the findings for easy post-processing and reporting
Appends results to a previous CSV if available

Displays

Hash and comment (comment is the rest of the line of which the hash has been extracted)
AV vendor matches based on a user defined list
Filenames used in the wild
PE information like the description, the original file name and the copyright statement
Signer of a signed portable executable
Result based on Virustotal ratio
First and last submission
Tags for certain indicators: Harmless, Signed, Expired, Revoked, MSSoftware

Extra Checks

Queries Malshare.com for sample uploads
Queries Hybrid-Analysis.com for reports
Queries multiple MISP instances for available events
Queries Any.run sandbox for reports
Queries CAPE sandbox for reports
Queries URLhaus for reports
Queries Malshare for available samples
Queries Valhalla for YARA rule matches
Imphash duplicates in current batch > allows you to spot overlaps in import table hashes
PE signature duplicate checks

Operation Modes

Default - by providing an input file (-f) with hashes or sample directory (-s)
Query - to search hashes from Valhalla by keyword, tags, ATT&CK technique (e.g. T1023), ATT&CK threat group (e.g. G0049) or rule name (-q)
Command Line Interface - using the --cli parameter
Web Service Mode - using the --web parameter

Getting started

Download / clone the repo
Install required packages: pip3 install -r requirements.txt (on macOS add --user)
Set the API keys for the different services in your custom ini file cp munin.ini my.ini (see section Get the API Keys for help)
Use the demo file for a first run: python munin.py -i my.ini -f munin-demo.txt

Requirements

Python 3.7 and higher
Internet Connection (Proxy Support; SSL/TLS interception can be a problem)

Typical Command Lines

Process a Virustotal Retrohunt result and sort the lines before checking so that matched signatures are checked in blocks

python3 munin.py -i my.ini -f ~/Downloads/retro_hunt

Process a directory with samples and check their hashes online

python3 munin.py -i my.ini -s ~/malware/case34

Use the command line interface mode (new in v0.14)

python3 munin.py -i my.ini

Get the API Keys

Virustotal

Create an account here https://www.virustotal.com/#/join-us
Check Profile > My API key for your public API key

MalShare

Malware Bazaar

Hybrid Analysis

Create an account here https://www.hybrid-analysis.com/signup
After login, check Profile > API key

MISP

Log into your MISP
Go to your profile "My Profile"
The value of Authkey is used as API key
Note that the .ini file uses both a list for the MISP instances and for the respective API keys

Valhalla

Currently for customers or invited researchers only
https://valhalla.nextron-systems.com/

Hashlookup

Hashlookup CIRCL's instance is provided free of charge and served as a best-effort basis.

Command Line Interface Mode

Start munin with --cli and follow the instruction.

E.g.

python3 munin.py -i my.ini --cli

Paste content with hash values in it and then press CTRL+D to finalize the input. The last line needs a line break at its end.

In the default, it will create a CSV file with the current date in the file name.

Munin CLI

Web Service Mode

Start munin with --web and optional select a port -w port.

E.g.

python3 munin.py -i my.ini --web -w 8080

The web service waits for strings in the following URL scheme.

http://server:port/<string>

The string can be any string without line breaks, e.g.

Emotet:1585ad28f7d1e0ca696e6c6c2f1d008a
ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa;IOC1
dc9b5e8aa6ec86db8af0a7aa897ca61db3e5f3d2e0942e319074db1aaccfdc83

The result will look like this:

{
    "comment": "Emotet",
    "commenter": "-",
    "comments": "0",
    "copyright": "Copyright (C) America Online, Inc. 1999 - 2004",
    "description": "Utilities",
    "expired": false,
    "filenames": "sourcedev.exe, MISCUTIL, x8ykNnr_9WofXq7Nh_xuEzSPW.exe, jwuKBLWN681ztj6Zks.exe",
    "filetype": "Win32 EXE",
    "first_submitted": "2019-01-19 13:46:21 UTC ( 2 months, 2 weeks ago )",
    "firstsubmission": "2019-01-19 13:46:21 UTC ( 2 months, 2 weeks ago )",
    "harmless": false,
    "hash": "1585ad28f7d1e0ca696e6c6c2f1d008a",
    "hybrid_available": false,
    "hybrid_compromised": "-",
    "hybrid_date": "-",
    "hybrid_score": "-",
    "imphash": "2820d9bdc397f88a8a1e957e1a824482",
    "last_submitted": "2019-02-27 09:44:03",
    "malshare_available": false,
    "md5": "1585ad28f7d1e0ca696e6c6c2f1d008a",
    "misp_available": true,
    "misp_events": "",
    "misp_info": [],
    "mssoft": false,
    "origname": "-",
    "positives": 48,
    "rating": "malicious",
    "res_color": "\u001b[41m",
    "result": "48 / 64",
    "revoked": false,
    "sha1": "4561d0ad575d5f02fb06e062a37de15861c3bd89",
    "sha256": "35e304d10d53834e3e41035d12122773c9a4d183a24e03f980ad3e6b2ecde7fa",
    "signed": false,
    "signer": "-",
    "total": 64,
    "urlhaus_available": true,
    "vendor_results": {
        "CrowdStrike": "win/malicious_confidence_100% (W)",
        "ESET-NOD32": "a variant of Win32/Kryptik.GOUY",
        "F-Secure": "Trojan.TR/AD.Emotet.pdiuu",
        "GData": "Trojan.GenericKD.40960256",
        "Kaspersky": "HEUR:Trojan.Win32.Generic",
        "McAfee": "Emotet-FLL!1585AD28F7D1",
        "Microsoft": "Trojan:Win32/Emotet.DN",
        "Sophos": "Mal/Emotet-Q",
        "Symantec": "Trojan.Gen.2",
        "TrendMicro": "-"
    },
    "virus": "Microsoft: Trojan:Win32/Emotet.DN / Kaspersky: HEUR:Trojan.Win32.Generic / McAfee: Emotet-FLL!1585AD28F7D1 / CrowdStrike: win/malicious_confidence_100% (W) / ESET-NOD32: a variant of Win32/Kryptik.GOUY / Symantec: Trojan.Gen.2 / F-Secure: Trojan.TR/AD.Emotet.pdiuu / Sophos: Mal/Emotet-Q / GData: Trojan.GenericKD.40960256",
    "virusbay_available": false,
    "vt_positives": 48,
    "vt_queried": false,
    "vt_total": 64,
    "vt_verbose_msg": "Scan finished, information embedded"
}

The queries to Virustotal need to be throttled. Therefore the web service applies a cool down time, that is minimized by subtracting the time it took to process all other platforms from the wait time of 15 seconds.

cooldown_time = vt_wait_time - process_time

During the cooldown, requests will return this response:

{"status": "VT cooldown active"}

The cool down is not relevant when requesting hashes that are already in the lookup cache.

Munin Hosts

The Munin host and IP checker script (munin-host.py) retrieves more information on IP addresses and host/domain names in IOC lists.

Usage

    usage: munin-host.py [-h] [-f path] [-o output] [-m max-items] [-c cache-db]
                        [-i ini-file] [--nocache] [--nocsv] [--recursive]
                        [--download] [-d download_path] [--dups] [--noresolve]
                        [--ping] [--debug]

    Virustotal Online Checker (IP/Domain)

    optional arguments:
      -h, --help        show this help message and exit
      -f path           File to process (hash line by line OR csv with hash in
                        each line - auto-detects position and comment)
      -o output         Output file for results (CSV)
      -m max-items      Maximum number of items (urls, hosts, samples) to show
      -c cache-db       Name of the cache database file (default: vt-hosts-
                        db.json)
      -i ini-file       Name of the ini file that holds the API keys
      --nocache         Do not use the load the cache db (vt-check-cache.pkl)
      --nocsv           Do not write a CSV with the results
      --recursive       Process the resolved IPs as well
      --download        Try to download the URLs (directories with host/ip names)
      -d download_path  Store the downloads to the given directory
      --dups            Do not skip duplicate hashes
      --noresolve       Do not perform DNS resolve test on found domain names
      --ping            Perform ping check on IPs (speeds up process if many
                        public but internally routed IPs appear in text file)
      --debug           Debug output

Screenshot

Munin Hosts_Screenshot

Examples

Parse the demo file, extract IPs and hosts, don't just check the domains that are still resolvable and download samples directly from the remote systems.

python3 munin-host.py -i your-key.ini -f ./munin-hosts-demo.txt --noresolve --download

Warning

Using munin-host.py in an IDS monitored network will cause numerous alerts as munin-host.py performs DNS lookups for malicious domains and has the option to download malicious samples.

Issues

pycurl on macOS

The script munin-host.py requires the module pycurl. It's sometimes tricky to make it work on macOS as it requires an openssl to be installed, which is then used in the build process.

If error's occur try the following (some environments will require pip3)

pip uninstall pycurl
brew update
brew reinstall openssl
export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
export PYCURL_SSL_LIBRARY=openssl
pip install pycurl --global-option="--with-openssl"

Hugin for Virustotal Retrohunts

The Hugin script (hugin.py) retrieves and displays information to all samples returned in a retrohunt. The big advantage is that you don't have to wait 15 seconds between each sample request but pull the full JSON result file via v3 of the Virustotal API. This way you get your results immediately. The disadvantage is that other services like Any.run, Hybrid-Analysis, MISP or Valhalla don't get queried with Hugin.

Usage

usage: hugin.py [-h] [-r retrohunt-name] [-i ini-file]
                [--csv-path CSV_PATH] [--debug] [--no-comments]

Retrohunt Checker

optional arguments:
  -h, --help           show this help message and exit
  -r retrohunt-name    Name for the queried retrohunt
  -i ini-file           Name of the ini file that holds the VT API key
  --csv-path CSV_PATH  Write a CSV with the results
  --debug              Debug output
  --no-comments        Skip VirusTotal comments

Examples

Parse a retrohunt and export a CSV file with the results.

python3 hugin.py -i config-with-your-key.ini -r retrohunt-123456789