Awesome
YAIDS
Yara-Based IDS
Yara as an Intrusion Detection System / Yet Another Intrusion Detection System
An Intrusion Detection System (IDS), utilizing Yara and multi-threading
Table of Contents
Features
YAIDS is a Multi-Threaded Intrusion Detection System using Yara.
- Multi-Threaded
- Network/Stream IDS
- Performant (written in C)
- Yara Pattern Matching Engine
- Supports any PCAP compatible data stream (Network, USB, Bluetooth, etc.)
- Supports BPF (traffic filtering)
- Supports all valid Yara rules (including modules)
- Includes External Variables to build traffic/packet attribute conditions
- Multiple logging modes (alerts, PCAP data, console, file, etc.)
Installation
Prerequisites
- Development Toolkit (git, gcc, glibc, etc.)
- libpcap
- libyara (Version 4.X.X)
- POSIX Platform (Linux, etc. -- YAIDS has been tested on Ubuntu 18.04 and 20.04)
Ubuntu (Quick Install)
- Review and verify the
quick-install-ubuntu.sh
file. - Run:
curl "https://raw.githubusercontent.com/wrayjustin/yaids/main/quick-install-ubuntu.sh" | bash
Other Distributions (Basic Install)
- Install Development Tools (git, gcc, glibc, etc.)
- Install
libpcap
(including development headers) - Install
libyara
(including development headers) - Checkout the project:
git clone https://github.com/wrayjustin/yaids.git
- Run:
./build.sh
Manual/Development Installation
- Install Development Tools (git, gcc, glibc, etc.)
- Install
libpcap
(including development headers) - Install
libyara
(including development headers) - Checkout the project:
git clone https://github.com/wrayjustin/yaids.git
- Run:
./init.sh
,./configure
,make
, andmake install
Manual (Without AutoTools)
- Install Development Tools (git, gcc, glibc, etc.)
- Install
libpcap
(including development headers) - Install
libyara
(including development headers) - Compile (from the
src
directory):gcc <GCC OPTIONS> yaids.c yaidsconf.c yaidsio.c yaidspcap.c yaidsyara.c yaidsthread.c -o yaids -I. -I.. -I../include/ -lpcap -lyara -lpthread -lm <ADDITIONAL LIBRARIES>
Versioning
YAIDS uses a three value version number: X.Y.Z. The version number is a modified form of SemVer
(Semantic Versioning).
Given the close dependency on Yara
, the first version value maps to the compatible Major Yara release. While libpcap
is another major dependency of yaids
, libpcap
maintains binary compatibility across releases.
The YAIDS version fields: YARA_MAJOR.MAJOR.MINOR (For example, 4.0.0)
- YARA_MAJOR - The compatible Yara (Major) version
- MAJOR - Incremented on significant changes
- MINOR - Incremented on minor changes or patches
Releases that increment the YARA_MAJOR field are highly likely to be incompatible with differing versions of Yara.
Releases that increment the MAJOR field may be incompatible with previous yaids
releases. Whereas MINOR releases
should be forwards and backwards compatible within the same YARA_MAJOR.MAJOR release family.
Note: At present, YAIDS is provided via rolling-release; that is, there is currently only a single line of releases. However, that may potentially change on the release of the next major Yara version.
Usage
YAIDS can run on either an interface (live capture) or stored PCAP files. You can use any Yara supported rules.
Examples
When running on a live interface, uee the following basic options: yaids -i <interface> -y <path/to/compiled/yara/rules>
.
To process an exiisting PCAP file, use the following basic options: yaids -r <path/to/pcap/file> -y <path/to/compiled/yara/rules>
.
Options
You can obtain more information from the help message by running: yaids --help
or man yaids
YAIDS -- 4.0.1
Yara as an Intrusion Detection System / Yet Another Intrusion Detection System
An Intrusion Detection System (IDS), utilizing Yara and multi-threading
COPYRIGHT (C) Justin M. Wray | Licensed: BSD 3-Clause
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Usage:
yaids
-i <DEVICE> Interface: The libpcap Interface Device name (-i or -r is required)
-r <FILE> Read: PCAP Input File (-i or -r is required)
-y <FILE> Rules: Yara Rules File, compiled (required)
-n <NAME> Prefix Name: Output files, prefix (optional, defaults to epoch)
-w <PATH> Output Path: Output path (optional, defaults to current working directory)
-f <FILE> BPF: BPF (PCAP Filter) File (optional, no traffic filtering)
-t <INT> Threads: Number of Yara Threads (optional, defaults to 2 + 4 x CPU cores)
-l <INT> Timelimit: Number of seconds to run, before exiting (optional, defaults to 0 - run until finished or interrupted)
-s Silent Mode: Write output files, but don't output alerts to STDOUT (optional, incompatible with Output Only Mode)
-o Output Only Mode: Output alerts to STDOUT only, does not write to files (optional, incompatible with Silent Mode)
-x Flush Mode: Prevents buffering of file output (optional)
-a Fast Alert Mode: Skip parsing packet headers for alerts (optional)
-d Debug Mode: Verbose output (optional)
-v Version: Show Version
-h Help: Show Help (this list)
Output
By default, YAIDS processed output to both stdout
and to log files. The default stdout
provides alerts
(rule matches), and two logs are generated - the alerts and PCAP of the triggered packets.
The alert output contains the following fields:
- Timestamp (YYYY-MM-DD HH:MM:SS)
- Rule Name
- Matching Rule Condition and Trigger Location
- Matching Packet Data
- Header Length: Length Yara Processed / PCAP Capture Length / Original Packet Length
- Protocols: Link-Layer / Network / Transport
- Layer-2 Source > Layer-2 Destination Addresses
- Layer-3 Source > Layer-3 Destination Addresses and Layer-4 Ports (Address:Port)
An example alert:
2020-12-20 01:01:01 - Metasploit_Download [$s1:0x42:3] {GET /metasploitframework-latest.msi} [171:171/171] (ETH/IP/TCP) 10:00:00:00:00:B3 > 10:00:00:00:00:F7 - 198.51.100.12:41309 > 203.0.113.37:80
Rules
YAIDS supports any Yara-compatible rule, including enabled and adequately configured modules.
For convenience, YAIDS also provides a "drop-in" replacement for yara
(the Yara Rules Compiler).
In addition to some feature enhancements, yaidsc
sets "external" variables for various packet offsets.
You can learn more about writing Yara rules from the Yara documentation: Writing Yara Rules.
yaidsc
yaidsc
is a drop-in replacement for yarac
using an identical syntax. If needed, you can also pass
additional options to yarac
. Note that yaidsc
is not a binary replacement for yarac
; it is a
wrapper script. The two major feature improvments are:
- Multi-File Support, including Directories (which will compile all
*.yar
files in the directories provided) - External Variables for Packet Fields
Usage:
yaidsc <INPUT [NAMESPACE]FILES|DIRECTORIES...> <OUTPUT FILE>
* If the input is a directory, all '.yar' files will be compiled.
* You can compile multiple files or directories, one ruleset.
* Namespaces can be provided, but only for individual files (not directories).
Packet Headers and Offsets
To provide simple network-related rule creation, without the need to perform complex header parsing and
calculations, yaids
includes a padded string-based header before the PCAP data. This additional data
is only included during the yara
scanning (not in the PCAP output, etc.). However, this does mean that
the Packet Offsets are modified, specifically by 255
bytes. As a result, the raw packet data (including
the unprocessed header) begins at offset 256
.
Offsets
The processed (string-based) header uses the following format:
Value | Offset Location | Field Size |
---|---|---|
Payload Offset | 0 | 1 (Byte) |
Original Packet (Length; This is regardless of the Capture Length) | 1 | 5 (Str) |
Frame Protocol | 6 | 14 (Str) |
Network Protocol | 20 | 14 (Str) |
Transport Protocol | 34 | 14 (Str) |
Frane Source Address | 48 | 17 (Str) |
Frame Destination Address | 65 | 17 (Str) |
Network Source Address | 82 | 46 (Str) |
Transport Source (Port) | 128 | 5 (Str) |
Network Destination Address: | 133 | 46 (Str) |
Transport Destination (Port) | 179 | 5 (Str) |
Again, the packet data begins at byte 256
.
Offset Examples
One of the most powerful values of the processed header is the Payload Offset
.
This offset value can be used in a yara
condition
, to match payload-specific data.
Keep in mind, you need to jump an additional 255
bytes to the begining of he packet data,
in additon to payload offset. For example:
rule example_01 {
meta:
author = "YAIDS.io"
description = "YAIDS Example (HTTP GET Requests)"
more_info = "https://yaids.io"
strings:
$string1 = "GET"
condition:
$string1 at (int8(0) + 255)
}
The remaining header offsets provide the ability to perform conditional matching (filtering) on the listed packet attributes (addresses, ports, etc.). For example:
rule example_02 {
meta:
author = "YAIDS.io"
description = "YAIDS Example (HTTP GET Requests)"
more_info = "https://yaids.io"
strings:
$string1 = "GET"
$frameProtocol = "ETH"
$netProtocol = "IP"
$transportProtocol = "TCP"
$frameSource = "00:16:47:9D:F2:C2"
$frameDest = "00:0C:29:89:D0:73"
$networkSource = "192.168.21.101"
$transportSource = "80"
$networkDest = "192.168.202.90"
$transportDest = "33783"
condition:
$frameProtocol at 6 and
$netProtocol at 20 and
$transportProtocol at 34 and
$frameSource at 48 and
$frameDest at 65 and
$networkSource at 82 and
$transportSource at 128 and
$networkDest at 133 and
$transportDest at 179 and
$string1 at (int8(0) + 255)
}
Yara External Variables
To simplify using the packet header values, yaidsc
(the rule compliler) will automatically
provide external variables
. When using yaidsc
to compile your rules, there's no need to memorize
the offset locations.
External Variables:
Value | Variable |
---|---|
Original Packet (Length; This is regardless of the Capture Length) | headerStart |
Frame Protocol | frameProtocol |
Network Protocol | netProtocol |
Transport Protocol | transportProtocol |
Frane Source Address | frameSource |
Frame Destination Address | frameDest |
Network Source Adderess | networkSource |
Transport Source (Port) | transportSource |
Network Destination Address: | networkDest |
Transport Destination (Port) | transportDest |
Compare the previous two example to this example using the external variables:
rule example_03 {
meta:
author = "YAIDS.io"
description = "YAIDS Example (HTTP GET Requests)"
more_info = "https://yaids.io"
strings:
$string1 = "GET"
$frameProtocol = "ETH"
$netProtocol = "IP"
$transportProtocol = "TCP"
$frameSource = "00:16:47:9D:F2:C2"
$frameDest = "00:0C:29:89:D0:73"
$networkSource = "192.168.21.101"
$transportSource = "80"
$networkDest = "192.168.202.90"
$transportDest = "33783"
condition:
$frameProtocol at frameProtocol and
$netProtocol at netProtocol and
$transportProtocol at transportProtocol and
$frameSource at frameSource and
$frameDest at frameDest and
$networkSource at networkSource and
$transportSource at transportSource and
$networkDest at networkDest and
$transportDest at transportDest and
$string1 at (int8(0) + 255)
}
Supported Protocols
YAIDS supports all libpcap-compatible data (network or otherwise).
However, the header parsing does have some limitations. Only the following protocols will be parsed.
Supported Layer-2 / Frame Protocols
Protocol Name | YAIDS Value |
---|---|
Ethernet | ETH |
Supported Layer-3 / Network Protocols
Protocol Name | YAIDS Value |
---|---|
IPv4 | IP |
IPv6 | IP6 |
8021AD | 8021AD |
8021Q | 8021Q |
8021QINQ | 8021QINQ |
AARP | AARP |
ARP | ARP |
ATALK | ATALK |
DECDNS | DECDNS |
DECDTS | DECDTS |
DN | DN |
IPX | IPX |
LANBRIDGE | LANBRIDGE |
LAT | LAT |
LOOPBACK | LOOPBACK |
MOPDL | MOPDL |
MOPRC | MOPRC |
MPLS | MPLS |
MPLS_MULTI | MPLS_MULTI |
NS | NS |
PPPOED | PPPOED |
PPPOES | PPPOES |
PPP | PPP |
PUP | PUP |
REVARP | REVARP |
SCA | SCA |
SPRITE | SPRITE |
TEB | TEB |
TRAIL | TRAIL |
VEXP | VEXP |
VPROD | VPROD |
Supported Layer-4 / Transport Protocols
Protocol Name | YAIDS Value |
---|---|
TCP | TCP |
UDP | UDP |
ICMP | ICMP |
Remember, you can write rules for any traffic type, but if the protocols are not listed above,
yaids
will not automatically parse the headers and therefore will not populate the External Variables
In cases where the protocol is unsupported, the default value will be "UNKN", which will be used in the
alert output and the External Variables
.
Technical Details
The processing flow of YAIDS is straight-forward; data comes in, is processed, and then goes out: INPUT->PROCESSING->OUTPUT.
The complexity comes from multi-threading. Although network streams (from a device or file) are serial, the packets' processing can be computationally expensive and long-lived. Put simply, packets traverse the network faster than an IDS can examine.
YAIDS resolves this disadvantage by parallelizing the processing of packets. Thus, the flow is more complex:
- Input: INPUT->INPUT QUEUE
- Processing: IINPUT QUEUE->PROCESSING->OUTPUT QUEUE
- Output: OUTPUT QUEUE->OUTPUT
To obtain a better understanding of this processing flow, review the flowchart.
Priority
For live captures, the INPUT is prioritized to prevent missing/dropping packets. For offline processing (reading of a PCAP file), the processing is prioritized to optimize the speed.
Priority is achieved through the utilization of a triple mutex
:
- Read/Low Priority
- Next
- Write/High Priority
Data Queues
Data queues are created via a series of structs
with a pointer to the subsequent struct
.
This is combined with a separate "queue" struct
containing pointers to the first and last data structs
.
+-----------------+ +-----------------+ +-----------------+ +-----------------+
| | | | | | | |
| DATA | | DATA | | DATA | | DATA |
| | | | | | | |
| Next*--------> Next*--------> Next*--------> Next*-------->NULL
+------^----------+ +-----------------+ +-----------------+ +------^----------+
| |
| |
+------|-----------------------------------------------------------------------|----------+
| First* Last* |
| |
| QUEUE |
| |
+-----------------------------------------------------------------------------------------+
Flowchart
*-----------------*
| |
| Packets |
| |
| |
*--------v--------*
|
|
|
|
#--------v--------#
| |
| Input |
| Thread |
| |
#--------v--------#
|
|
|
|
+--------V--------+
| |
| Input |
| Queue |
| |
+--v--v-----v--v--+
+-----------------------------+ | | +------------------------------
| | | |
| +--------+ +--------+ }
| | | |
#--------V--------# #--------V--------# #--------V--------# #--------V--------#
| | | | | | | |
| Yara | | Yara | | Yara | | Yara |
| Thread 1 | | Thread 2 | | Thread 3 | | Thread ... |
| | | | | | | |
#--------v--------# #--------v--------# #--------v--------# #--------v--------#
| | | |
| +--------+ +--------+ |
| | | |
+-----------------------------+ | | +-----------------------------|
+--V--V-----V--V--+
| |
| Output |
| Queue |
| |
+--------v--------+
|
|
|
|
#--------V--------#
| |
| Output |
| Thread |
| |
#---v---------v---#
| |
| |
+------+ +------+
| |
*--------V--------* *--------V--------*
| | | |
| PCAP | | Alert |
| File | | File |
| | | |
*-----------------* *-----------------*
Code Testing
YAIDS is tested using both automated and manual processes. The testing conducted covers functional testing, security testing, and code quality analysis.
Some of the tools utilized for testing include:
In addition to the third-party tools listed above, yaids
inclues a test suite.
Test Suite
The test suite includes PCAP files and Yara Rules, and verifies that multiple modes of yaids
return
the correct alerts and output.
The following modes are tested:
- Output Mode
- Silent Mode
- Default Mode / Re-Processing of the Default Mode output
The following test cases are included in the test suite:
Test Name | Test Count | Description |
---|---|---|
test_A | 6 | HTTP Traffic/Rules |
test_B | 3 | FTP Traffic/Rules |
test_C | 1 | UDP Traffic/Rules |
test_D | 9 | Traffic Attributes (Source, Destination, etc.) |
test_E | 1 | Combined Ruleset |
test_F | 3 | BPF Test |
All tests are run three times to ensure consistent results. Additionally, the tests are run using both standard PCAP and PCAPNG files.
NOTE: The PCAP files included for testing purposes are sources from NETRESEC.
Automated Testing Workflows
There are three sets of tests conducted automatically on the yaids/main branch. The statuses of these tests are displayed at the top of the documentation. Furthermore, you can view the historical results for these workflows via GitHub Actions.
License
COPYRIGHT (C) 2020 Justin M. Wray - wray.justin@gmail.com / https://www.justinwray.com
ALL RIGHTS RESERVED
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its contributors
may be used to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.