Home

Awesome

YAIDS

Yara-Based IDS

Yara as an Intrusion Detection System / Yet Another Intrusion Detection System

An Intrusion Detection System (IDS), utilizing Yara and multi-threading

Build Code Analysis Tests

License Latest Version

Website Repo


Table of Contents


Features

YAIDS is a Multi-Threaded Intrusion Detection System using Yara.

Installation

Prerequisites

Ubuntu (Quick Install)

  1. Review and verify the quick-install-ubuntu.sh file.
  2. Run: curl "https://raw.githubusercontent.com/wrayjustin/yaids/main/quick-install-ubuntu.sh" | bash

Other Distributions (Basic Install)

  1. Install Development Tools (git, gcc, glibc, etc.)
  2. Install libpcap (including development headers)
  3. Install libyara (including development headers)
  4. Checkout the project: git clone https://github.com/wrayjustin/yaids.git
  5. Run: ./build.sh

Manual/Development Installation

  1. Install Development Tools (git, gcc, glibc, etc.)
  2. Install libpcap (including development headers)
  3. Install libyara (including development headers)
  4. Checkout the project: git clone https://github.com/wrayjustin/yaids.git
  5. Run: ./init.sh, ./configure, make, and make install

Manual (Without AutoTools)

  1. Install Development Tools (git, gcc, glibc, etc.)
  2. Install libpcap (including development headers)
  3. Install libyara (including development headers)
  4. Compile (from the src directory): gcc <GCC OPTIONS> yaids.c yaidsconf.c yaidsio.c yaidspcap.c yaidsyara.c yaidsthread.c -o yaids -I. -I.. -I../include/ -lpcap -lyara -lpthread -lm <ADDITIONAL LIBRARIES>

Versioning

YAIDS uses a three value version number: X.Y.Z. The version number is a modified form of SemVer (Semantic Versioning).

Given the close dependency on Yara, the first version value maps to the compatible Major Yara release. While libpcap is another major dependency of yaids, libpcap maintains binary compatibility across releases.

The YAIDS version fields: YARA_MAJOR.MAJOR.MINOR (For example, 4.0.0)

Releases that increment the YARA_MAJOR field are highly likely to be incompatible with differing versions of Yara. Releases that increment the MAJOR field may be incompatible with previous yaids releases. Whereas MINOR releases should be forwards and backwards compatible within the same YARA_MAJOR.MAJOR release family.

Note: At present, YAIDS is provided via rolling-release; that is, there is currently only a single line of releases. However, that may potentially change on the release of the next major Yara version.

Usage

YAIDS can run on either an interface (live capture) or stored PCAP files. You can use any Yara supported rules.

Examples

When running on a live interface, uee the following basic options: yaids -i <interface> -y <path/to/compiled/yara/rules>.

To process an exiisting PCAP file, use the following basic options: yaids -r <path/to/pcap/file> -y <path/to/compiled/yara/rules>.

Options

You can obtain more information from the help message by running: yaids --help or man yaids

YAIDS -- 4.0.1
        Yara as an Intrusion Detection System / Yet Another Intrusion Detection System
        An Intrusion Detection System (IDS), utilizing Yara and multi-threading
        COPYRIGHT (C) Justin M. Wray | Licensed: BSD 3-Clause
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Usage:
        yaids
                -i    <DEVICE>            Interface: The libpcap Interface Device name (-i or -r is required)
                -r    <FILE>              Read: PCAP Input File (-i or -r is required)
                -y    <FILE>              Rules: Yara Rules File, compiled (required)
                -n    <NAME>              Prefix Name: Output files, prefix (optional, defaults to epoch)
                -w    <PATH>              Output Path: Output path (optional, defaults to current working directory)
                -f    <FILE>              BPF: BPF (PCAP Filter) File (optional, no traffic filtering)
                -t    <INT>               Threads: Number of Yara Threads (optional, defaults to 2 + 4 x CPU cores)
                -l    <INT>               Timelimit: Number of seconds to run, before exiting (optional, defaults to 0 - run until finished or interrupted)
                -s                        Silent Mode: Write output files, but don't output alerts to STDOUT (optional, incompatible with Output Only Mode)
                -o                        Output Only Mode: Output alerts to STDOUT only, does not write to files (optional, incompatible with Silent Mode)
                -x                        Flush Mode: Prevents buffering of file output (optional)
                -a                        Fast Alert Mode: Skip parsing packet headers for alerts (optional)
                -d                        Debug Mode: Verbose output (optional)
                -v                        Version: Show Version
                -h                        Help: Show Help (this list)

Output

By default, YAIDS processed output to both stdout and to log files. The default stdout provides alerts (rule matches), and two logs are generated - the alerts and PCAP of the triggered packets.

The alert output contains the following fields:

An example alert:

2020-12-20 01:01:01 - Metasploit_Download [$s1:0x42:3] {GET /metasploitframework-latest.msi} [171:171/171] (ETH/IP/TCP) 10:00:00:00:00:B3 > 10:00:00:00:00:F7 - 198.51.100.12:41309 > 203.0.113.37:80

Rules

YAIDS supports any Yara-compatible rule, including enabled and adequately configured modules.

For convenience, YAIDS also provides a "drop-in" replacement for yara (the Yara Rules Compiler). In addition to some feature enhancements, yaidsc sets "external" variables for various packet offsets.

You can learn more about writing Yara rules from the Yara documentation: Writing Yara Rules.

yaidsc

yaidsc is a drop-in replacement for yarac using an identical syntax. If needed, you can also pass additional options to yarac. Note that yaidsc is not a binary replacement for yarac; it is a wrapper script. The two major feature improvments are:

Usage:
        yaidsc <INPUT [NAMESPACE]FILES|DIRECTORIES...> <OUTPUT FILE> 

 * If the input is a directory, all '.yar' files will be compiled.
 * You can compile multiple files or directories, one ruleset.
 * Namespaces can be provided, but only for individual files (not directories).

Packet Headers and Offsets

To provide simple network-related rule creation, without the need to perform complex header parsing and calculations, yaids includes a padded string-based header before the PCAP data. This additional data is only included during the yara scanning (not in the PCAP output, etc.). However, this does mean that the Packet Offsets are modified, specifically by 255 bytes. As a result, the raw packet data (including the unprocessed header) begins at offset 256.

Offsets

The processed (string-based) header uses the following format:

ValueOffset LocationField Size
Payload Offset01 (Byte)
Original Packet (Length; This is regardless of the Capture Length)15 (Str)
Frame Protocol614 (Str)
Network Protocol2014 (Str)
Transport Protocol3414 (Str)
Frane Source Address4817 (Str)
Frame Destination Address6517 (Str)
Network Source Address8246 (Str)
Transport Source (Port)1285 (Str)
Network Destination Address:13346 (Str)
Transport Destination (Port)1795 (Str)

Again, the packet data begins at byte 256.

Offset Examples

One of the most powerful values of the processed header is the Payload Offset. This offset value can be used in a yara condition, to match payload-specific data. Keep in mind, you need to jump an additional 255 bytes to the begining of he packet data, in additon to payload offset. For example:

rule example_01 {
    meta:
        author = "YAIDS.io"
        description = "YAIDS Example (HTTP GET Requests)"
        more_info = "https://yaids.io"
    strings:
        $string1 = "GET"
    condition:
        $string1 at (int8(0) + 255)
}

The remaining header offsets provide the ability to perform conditional matching (filtering) on the listed packet attributes (addresses, ports, etc.). For example:

rule example_02 {
    meta:
        author = "YAIDS.io"
        description = "YAIDS Example (HTTP GET Requests)"
        more_info = "https://yaids.io"
    strings:
        $string1 = "GET"
        $frameProtocol = "ETH"
        $netProtocol = "IP"
        $transportProtocol = "TCP"
        $frameSource = "00:16:47:9D:F2:C2"
        $frameDest = "00:0C:29:89:D0:73"
        $networkSource = "192.168.21.101"
        $transportSource = "80"
        $networkDest = "192.168.202.90"
        $transportDest = "33783"
    condition:
        $frameProtocol at 6 and
        $netProtocol at 20 and
        $transportProtocol at 34 and
        $frameSource at 48 and
        $frameDest at 65 and
        $networkSource at 82 and
        $transportSource at 128 and
        $networkDest at 133 and
        $transportDest at 179 and
        $string1 at (int8(0) + 255)
}
Yara External Variables

To simplify using the packet header values, yaidsc (the rule compliler) will automatically provide external variables. When using yaidsc to compile your rules, there's no need to memorize the offset locations.

External Variables:

ValueVariable
Original Packet (Length; This is regardless of the Capture Length)headerStart
Frame ProtocolframeProtocol
Network ProtocolnetProtocol
Transport ProtocoltransportProtocol
Frane Source AddressframeSource
Frame Destination AddressframeDest
Network Source AdderessnetworkSource
Transport Source (Port)transportSource
Network Destination Address:networkDest
Transport Destination (Port)transportDest

Compare the previous two example to this example using the external variables:

rule example_03 {
    meta:
        author = "YAIDS.io"
        description = "YAIDS Example (HTTP GET Requests)"
        more_info = "https://yaids.io"
    strings:
        $string1 = "GET"
        $frameProtocol = "ETH"
        $netProtocol = "IP"
        $transportProtocol = "TCP"
        $frameSource = "00:16:47:9D:F2:C2"
        $frameDest = "00:0C:29:89:D0:73"
        $networkSource = "192.168.21.101"
        $transportSource = "80"
        $networkDest = "192.168.202.90"
        $transportDest = "33783"
    condition:
        $frameProtocol at frameProtocol and
        $netProtocol at netProtocol and
        $transportProtocol at transportProtocol and
        $frameSource at frameSource and
        $frameDest at frameDest and
        $networkSource at networkSource and
        $transportSource at transportSource and
        $networkDest at networkDest and
        $transportDest at transportDest and
        $string1 at (int8(0) + 255)
}

Supported Protocols

YAIDS supports all libpcap-compatible data (network or otherwise).

However, the header parsing does have some limitations. Only the following protocols will be parsed.

Supported Layer-2 / Frame Protocols
Protocol NameYAIDS Value
EthernetETH
Supported Layer-3 / Network Protocols
Protocol NameYAIDS Value
IPv4IP
IPv6IP6
8021AD8021AD
8021Q8021Q
8021QINQ8021QINQ
AARPAARP
ARPARP
ATALKATALK
DECDNSDECDNS
DECDTSDECDTS
DNDN
IPXIPX
LANBRIDGELANBRIDGE
LATLAT
LOOPBACKLOOPBACK
MOPDLMOPDL
MOPRCMOPRC
MPLSMPLS
MPLS_MULTIMPLS_MULTI
NSNS
PPPOEDPPPOED
PPPOESPPPOES
PPPPPP
PUPPUP
REVARPREVARP
SCASCA
SPRITESPRITE
TEBTEB
TRAILTRAIL
VEXPVEXP
VPRODVPROD
Supported Layer-4 / Transport Protocols
Protocol NameYAIDS Value
TCPTCP
UDPUDP
ICMPICMP

Remember, you can write rules for any traffic type, but if the protocols are not listed above, yaids will not automatically parse the headers and therefore will not populate the External Variables

In cases where the protocol is unsupported, the default value will be "UNKN", which will be used in the alert output and the External Variables.

Technical Details

The processing flow of YAIDS is straight-forward; data comes in, is processed, and then goes out: INPUT->PROCESSING->OUTPUT.

The complexity comes from multi-threading. Although network streams (from a device or file) are serial, the packets' processing can be computationally expensive and long-lived. Put simply, packets traverse the network faster than an IDS can examine.

YAIDS resolves this disadvantage by parallelizing the processing of packets. Thus, the flow is more complex:

To obtain a better understanding of this processing flow, review the flowchart.

Priority

For live captures, the INPUT is prioritized to prevent missing/dropping packets. For offline processing (reading of a PCAP file), the processing is prioritized to optimize the speed.

Priority is achieved through the utilization of a triple mutex:

Data Queues

Data queues are created via a series of structs with a pointer to the subsequent struct. This is combined with a separate "queue" struct containing pointers to the first and last data structs.

+-----------------+     +-----------------+     +-----------------+     +-----------------+
|                 |     |                 |     |                 |     |                 |
|  DATA           |     |  DATA           |     |  DATA           |     |  DATA           |
|                 |     |                 |     |                 |     |                 |
|          Next*-------->          Next*-------->          Next*-------->          Next*-------->NULL
+------^----------+     +-----------------+     +-----------------+     +------^----------+
       |                                                                       |
       |                                                                       |
+------|-----------------------------------------------------------------------|----------+
| First*                                                                   Last*          |
|                                                                                         |
|                                          QUEUE                                          |
|                                                                                         |
+-----------------------------------------------------------------------------------------+

Flowchart

                                    *-----------------*
                                    |                 |
                                    |     Packets     |
                                    |                 |
                                    |                 |
                                    *--------v--------*
                                             |
                                             |
                                             |
                                             |
                                    #--------v--------#
                                    |                 |
                                    |     Input       |
                                    |     Thread      |
                                    |                 |
                                    #--------v--------#
                                             |
                                             |
                                             |
                                             |
                                    +--------V--------+
                                    |                 |
                                    |     Input       |
                                    |     Queue       |
                                    |                 |
                                    +--v--v-----v--v--+
         +-----------------------------+  |     |  +------------------------------
         |                                |     |                                |
         |                       +--------+     +--------+                       }
         |                       |                       |                       |
#--------V--------#     #--------V--------#     #--------V--------#     #--------V--------#
|                 |     |                 |     |                 |     |                 |
|    Yara         |     |    Yara         |     |    Yara         |     |    Yara         |
|    Thread 1     |     |    Thread 2     |     |    Thread 3     |     |    Thread ...   |
|                 |     |                 |     |                 |     |                 |
#--------v--------#     #--------v--------#     #--------v--------#     #--------v--------#
         |                       |                       |                       |
         |                       +--------+     +--------+                       |
         |                                |     |                                |
         +-----------------------------+  |     |  +-----------------------------|
                                    +--V--V-----V--V--+
                                    |                 |
                                    |     Output      |
                                    |     Queue       |
                                    |                 |
                                    +--------v--------+
                                             |
                                             |
                                             |
                                             |
                                    #--------V--------#
                                    |                 |
                                    |     Output      |
                                    |     Thread      |
                                    |                 |
                                    #---v---------v---#
                                        |         |
                                        |         |
                                 +------+         +------+
                                 |                       |
                        *--------V--------*     *--------V--------*
                        |                 |     |                 |
                        |      PCAP       |     |      Alert      |
                        |      File       |     |      File       |
                        |                 |     |                 |
                        *-----------------*     *-----------------*

Code Testing

YAIDS is tested using both automated and manual processes. The testing conducted covers functional testing, security testing, and code quality analysis.

Some of the tools utilized for testing include:

In addition to the third-party tools listed above, yaids inclues a test suite.

Test Suite

The test suite includes PCAP files and Yara Rules, and verifies that multiple modes of yaids return the correct alerts and output.

The following modes are tested:

The following test cases are included in the test suite:

Test NameTest CountDescription
test_A6HTTP Traffic/Rules
test_B3FTP Traffic/Rules
test_C1UDP Traffic/Rules
test_D9Traffic Attributes (Source, Destination, etc.)
test_E1Combined Ruleset
test_F3BPF Test

All tests are run three times to ensure consistent results. Additionally, the tests are run using both standard PCAP and PCAPNG files.

NOTE: The PCAP files included for testing purposes are sources from NETRESEC.

Automated Testing Workflows

There are three sets of tests conducted automatically on the yaids/main branch. The statuses of these tests are displayed at the top of the documentation. Furthermore, you can view the historical results for these workflows via GitHub Actions.

License

COPYRIGHT (C) 2020 Justin M. Wray - wray.justin@gmail.com / https://www.justinwray.com
ALL RIGHTS RESERVED

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software without
specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.