Home

Awesome

MalwareMultiScan

Tests API UI Scanners

Self-hosted VirusTotal / OPSWAT MetaDefender wannabe API for scanning URLs and files by multiple antivirus solutions.

MalwareMultiScan UI

IMPORTANT: version 1.5 introduces breaking changes in containers configuration and docker-compose.yaml layout. Please see releases page and changelog of docker-compose.yaml and README.md for the additional details.

Introduction

I faced a need to scan user-uploaded files in one of my work projects in an automated mode to ensure they don't contain any malware. Using VirusTotal was not an option because of a) legal restrictions and data residency limitations b) scanning by hash-sums would not be sufficient because the majority of files are generated / modified by users.

After googling, I stumbled upon a fantastic maliceio/malice project. Unfortunately, it looks abandoned, and most plugins do not work for the moment. In addition to that, I had an intention to use the .NET stack to align with the internal infrastructure.

In the end, it's nothing but the set of Docker containers running the agent. That agent downloads the remote file to the temp folder, then launches the vendor command-line scanning utility with proper arguments, and parses the output with a regular expression to extract a detected malware name.

Installation & Usage

IMPORTANT: MalwareMultiScan is not intended as a publicly-facing API / UI. It has (intentionally) no authorization, authentication, rate-limiting, or logging. Therefore, it should be used only as an internal / private API or behind the restrictive API gateway.

Whole solution can be started with docker-compose up executed in a root folder of repository.

It can be also deployed to the Docker Swarm cluster by using the command docker stack deploy malware-multi-scan --compose-file docker-compose.yaml.

After the start the Demo Web UI will become available under http://localhost:8888.

See components chapter below and the docker-compose.yaml file.

Configuration

Configuration of API and Scanners is performed by passing the environment variables. Descriptions and default values are provided below.

MalwareMultiScan.Api

MalwareMultiScan.Scanner

MalwareMultiScan.Ui

API Endpoints

Callback URL

Both /api/queue/url and /api/queue/file also accept an optional callbackUrl parameter with the http(s) URL in it. This URL will be requested by the POST method with JSON serialized ScanResultMessage in a body on every update from scan backends. Query string will contain id parameter that corresponds to the id of the scan result and backend parameter with the id of backend which completed the scan.

I.e. when you define callbackUrl=http://localhost:1234/scan-results, the POST request will be made to http://localhost:1234/scan-results?id=123&backend=dummy with a body

{
  "Status": 1,
  "Duration": 5,
  "Threats": ["Malware.Dummy.Result"]
}

Supported Scan Engines

NameDockerfileEnabledComments
ClamAVClamav.Dockerfile:white_check_mark:
ComodoComodo.Dockerfile:white_large_square:
DrWebDrWeb.Dockerfile:white_large_square:Pass license key to the DRWEB_KEY build arg.
DummyDockerfile:white_check_mark:Scan backend made for testing. Returns Malware.Dummy.Result threat for every scan after 5 seconds.
KESKES.Dockerfile:white_large_square:Pass license key to the KES_KEY build arg. KES 11 does not work in Docker.
McAfeeMcAfee.Dockerfile:white_large_square:
SophosSophos.Dockerfile:white_large_square:
DefenderWindowsDefender.Dockerfile:white_check_mark:

More scan backends can be added in the future. Some of the popular ones do not have command line scanning utility, Linux version, or don't start in Docker container. Feel free to raise an issue if you know any in addition to the list above.

Components

Workflow

  1. On startup all Scanners register themselves in Consul with a service name equal to scanner and the BackendId metadata field equal to the value of BACKEND_ID environment variable. They also register a TTL check and listen for Hangfire background job in a queue named under the BackendId metadata field.

  2. Third-party client triggers /api/queue/url or /api/queue/file of the MalwareMultiScan.Api.

  3. MalwareMultiScan.Api sends a query to Consul and receives the list of alive scan backends with the service name scanner.

  4. MalwareMultiScan.Api schedules a Hangfire background job in a queue named under the BackendId metadata field.

  5. Scanners picks up a job from queue, starts the scan and sends result back to the default queue of Hangfire.

  6. MalwareMultiScan.Api picks a job from the default` queue of Hangfire and updates the state of the scan.

  7. If callback URL was specified during the step #2, MalwareMultiScan.Api triggers a HTTP POST request to the specified URL. See Callback URL for details.

Prerequisites

Parts

Plans

See issues for the list of planned features, bug-fixes, and improvements.