Home

Awesome

<p align="center"> <a href="https://www.matano.dev"><img src="assets/cover_wide.png" width=600></a> </p> <p align="center"> <!-- <a href="#"><img src="https://img.shields.io/badge/Deploys%20to-AWS-%23FF9900.svg?style=for-the-badge&logo=amazon-aws&logoColor=white&labelColor=232F3E"/></a> <a href="#"><img src="https://img.shields.io/badge/rust-%233A3B3C.svg?style=for-the-badge&logo=rust&labelColor=B1513E&logoColor=white"/></a> <br/> --> <a href="https://discord.gg/YSYfHMbfZQ" target="_blank"><img src="https://img.shields.io/badge/rust-%233A3B3C.svg?label=built with&logo=rust&logoColor=ffffff&color=B1513E&labelColor=0d1117"/></a> <a href="#"><img src="https://img.shields.io/badge/deploys%20to-aws-%23FF9900.svg?logo=amazon-aws&logoColor=white&labelColor=232F3E"/></a> <a href="https://discord.gg/YSYfHMbfZQ" target="_blank"><img src="https://img.shields.io/discord/996484553290022973.svg?label=join us&logo=discord&logoColor=ffffff&color=7389D8&labelColor=6A7EC2"/></a> <a href="https://twitter.com/intent/follow?screen_name=matanolabs" target="_blank"><img src="https://img.shields.io/twitter/follow/matanolabs?style=social" alt="Twitter Follow"/></a> <a href="/LICENSE" target="_blank"><img src="https://img.shields.io/github/license/matanolabs/matano?style=flat"/></a> <a href="https://bestpractices.coreinfrastructure.org/projects/6478"><img src="https://bestpractices.coreinfrastructure.org/projects/6478/badge"></a> </p>

Open source security data lake for AWS

Matano Open Source Security data lake is an open source cloud-native security data lake, built for security teams on AWS.

[!NOTE] Matano offers a commercial managed Cloud SIEM for a complete enterprise Security Operations platform. Learn more.

<div> <h3 align="center"> <a href="https://www.matano.dev/docs">Docs</a> <span> | </span> <a href="https://www.matano.dev">Website</a> <span> | </span> <a href="https://discord.gg/YSYfHMbfZQ">Community</a> </h3> </div>

Features

<div align="center"> <br> <img src="assets/matano_athena.png" width="650"> </div> <br>

Architecture

<div align="center"> <br> <img src="assets/diagram.png" width="600"> </div>

👀 Use cases

✨ Integrations

Managed log sources

Alert destinations

Query engines

Quick start

View the complete installation instructions

Installation

Install the matano CLI to deploy Matano into your AWS account, and manage your deployment.

Linux

curl -OL https://github.com/matanolabs/matano/releases/download/nightly/matano-linux-x64.sh
chmod +x matano-linux-x64.sh
sudo ./matano-linux-x64.sh

macOS

curl -OL https://github.com/matanolabs/matano/releases/download/nightly/matano-macos-x64.sh
chmod +x matano-macos-x64.sh
sudo ./matano-macos-x64.sh

Deployment

Read the complete docs on getting started

To get started, run the matano init command.

<div align="center"> <img src="assets/matano-init.gif" width="600"> </div> <br>

Directory structure

Once initialized, your Matano directory is used to control & manage all resources in your project e.g. log sources, detections, and other configuration. It is structured as follows:

➜  example-matano-dir git:(main) tree
├── detections
│   └── aws_root_credentials
│       ├── detect.py
│       └── detection.yml
├── log_sources
│   ├── cloudtrail
│   │   ├── log_source.yml
│   │   └── tables
│   │       └── default.yml
│   └── zeek
│       ├── log_source.yml
│       └── tables
│           └── dns.yml
├── matano.config.yml
└── matano.context.json

When onboarding a new log source or authoring a detection, run matano deploy from anywhere in your project to deploy the changes to your account.

🔧 Log Transformation & Data Normalization

Read the complete docs on configuring custom log sources

Vector Remap Language (VRL), allows you to easily onboard custom log sources and encourages you to normalize fields according to the Elastic Common Schema (ECS) to enable enhanced pivoting and bulk search for IOCs across your security data lake.

Users can define custom VRL programs to parse and transform unstructured logs as they are being ingested through one of the supported mechanisms for a log source (e.g. S3, SQS).

VRL is an expression-oriented language designed for transforming observability data (e.g. logs) in a safe and performant manner. It features a simple syntax and a rich set of built-in functions tailored specifically to observability use cases.

Example: parsing JSON

Let's have a look at a simple example. Imagine that you're working with HTTP log events that look like this:

{
  "line": "{\"status\":200,\"srcIpAddress\":\"1.1.1.1\",\"message\":\"SUCCESS\",\"username\":\"ub40fan4life\"}"
}

You want to apply these changes to each event:

Adding this VRL program to your log source as a transform step would accomplish all of that:

log_source.yml
transform: |
  . = object!(parse_json!(string!(.json.line)))
  .source.ip = del(.srcIpAddress)
  del(.username)
  .message = downcase(string!(.message))

schema:
  ecs_field_names:
    - source.ip
    - http.status

The resulting event 🎉:

{
  "message": "success",
  "status": 200,
  "source": {
    "ip": "1.1.1.1"
  }
}

📝 Writing Detections

Read the complete docs on detections

Use detections to define rules that can alert on threats in your security logs. A detection is a Python program that is invoked with data from a log source in realtime and can create an alert.

Examples

Detect failed attempts to export AWS EC2 instance in AWS CloudTrail logs.

def detect(record):
  return (
    record.deepget("event.action") == "CreateInstanceExportTask"
    and record.deepget("event.provider") == "ec2.amazonaws.com"
    and record.deepget("event.outcome") == "failure"
  )

Detect Brute Force Logins by IP across all configured log sources (e.g. Okta, AWS, GWorkspace)

detect.py
def detect(r):
    return (
        "authentication" in r.deepget("event.category", [])
        and r.deepget("event.outcome") == "failure"
    )


def title(r):
    return f"Multiple failed logins from {r.deepget('user.full_name')} - {r.deepget('source.ip')}"


def dedupe(r):
    return r.deepget("source.ip")
detection.yml
---
tables:
  - aws_cloudtrail
  - okta_system
  - o365_audit
alert:
  severity: medium
  threshold: 5
  deduplication_window_minutes: 15
  destinations:
    - slack_my_team

Detect Successful Login from never before seen IP for User

from detection import remotecache

# a cache of user -> ip[]
user_to_ips = remotecache("user_ip")

def detect(record):
    if (
      record.deepget("event.action") == "ConsoleLogin" and
      record.deepget("event.outcome") == "success"
    ):
        # A unique key on the user name
        user = record.deepget("user.name")

        existing_ips = user_to_ips[user] or []
        updated_ips = user_to_ips.add_to_string_set(
          user,
          record.deepget("source.ip")
        )

        # Alert on new IPs
        new_ips = set(updated_ips) - set(existing_ips)
        if existing_ips and new_ips:
            return True

🚨 Alerting

Read the complete docs on alerting

Alerts table

All alerts are automatically stored in a Matano table named matano_alerts. The alerts and rule matches are normalized to ECS and contain context about the original event that triggered the rule match, along with the alert and rule data.

Example Queries

Summarize alerts in the last week that are activated (exceeded the threshold)

select
  matano.alert.id as alert_id,
  matano.alert.rule.name as rule_name,
  max(matano.alert.title) as title,
  count(*) as match_count,
  min(matano.alert.first_matched_at) as first_matched_at,
  max(ts) as last_matched_at,
  array_distinct(flatten(array_agg(related.ip))) as related_ip,
  array_distinct(flatten(array_agg(related.user))) as related_user,
  array_distinct(flatten(array_agg(related.hosts))) as related_hosts,
  array_distinct(flatten(array_agg(related.hash))) as related_hash
from
  matano_alerts
where
  matano.alert.first_matched_at > (current_timestamp - interval '7' day)
  and matano.alert.activated = true
group by
  matano.alert.rule.name,
  matano.alert.id
order by
  last_matched_at desc

Delivering alerts

You can deliver alerts to external systems. You can use the alerting SNS topic to deliver alerts to Email, Slack, and other services.

<div align="center"> <br> <img src="assets/matano_slack_alert.png" width="600"> <br> <i>A medium severity alert delivered to Slack</i> </div>

❤️ Community support

For general help on usage, please refer to the official documentation. For additional help, feel free to use one of these channels to ask a question:

👷 Contributors

Thanks go to these wonderful people (emoji key):

<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --> <!-- prettier-ignore-start --> <!-- markdownlint-disable --> <table> <tbody> <tr> <td align="center" valign="top" width="14.28%"><a href="https://github.com/shaeqahmed"><img src="https://avatars.githubusercontent.com/u/13088492?v=4?s=100" width="100px;" alt="Shaeq Ahmed"/><br /><sub><b>Shaeq Ahmed</b></sub></a><br /><a href="#maintenance-shaeqahmed" title="Maintenance">🚧</a></td> <td align="center" valign="top" width="14.28%"><a href="https://www.matano.dev/"><img src="https://avatars.githubusercontent.com/u/9027301?v=4?s=100" width="100px;" alt="Samrose"/><br /><sub><b>Samrose</b></sub></a><br /><a href="#maintenance-Samrose-Ahmed" title="Maintenance">🚧</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/kai-ten"><img src="https://avatars.githubusercontent.com/u/11355908?v=4?s=100" width="100px;" alt="Kai Herrera"/><br /><sub><b>Kai Herrera</b></sub></a><br /><a href="https://github.com/matanolabs/matano/commits?author=kai-ten" title="Code">💻</a> <a href="#ideas-kai-ten" title="Ideas, Planning, & Feedback">🤔</a> <a href="#infra-kai-ten" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/rams3sh"><img src="https://avatars.githubusercontent.com/u/5143597?v=4?s=100" width="100px;" alt="Ram"/><br /><sub><b>Ram</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Arams3sh" title="Bug reports">🐛</a> <a href="#ideas-rams3sh" title="Ideas, Planning, & Feedback">🤔</a> <a href="#userTesting-rams3sh" title="User Testing">📓</a></td> <td align="center" valign="top" width="14.28%"><a href="http://zbmowrey.com/"><img src="https://avatars.githubusercontent.com/u/14931610?v=4?s=100" width="100px;" alt="Zach Mowrey"/><br /><sub><b>Zach Mowrey</b></sub></a><br /><a href="#ideas-zbmowrey" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/matanolabs/matano/issues?q=author%3Azbmowrey" title="Bug reports">🐛</a> <a href="#userTesting-zbmowrey" title="User Testing">📓</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/marcin-kwasnicki"><img src="https://avatars.githubusercontent.com/u/91739800?v=4?s=100" width="100px;" alt="marcin-kwasnicki"/><br /><sub><b>marcin-kwasnicki</b></sub></a><br /><a href="#userTesting-marcin-kwasnicki" title="User Testing">📓</a> <a href="https://github.com/matanolabs/matano/issues?q=author%3Amarcin-kwasnicki" title="Bug reports">🐛</a> <a href="#ideas-marcin-kwasnicki" title="Ideas, Planning, & Feedback">🤔</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/gdrapp"><img src="https://avatars.githubusercontent.com/u/346463?v=4?s=100" width="100px;" alt="Greg Rapp"/><br /><sub><b>Greg Rapp</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Agdrapp" title="Bug reports">🐛</a> <a href="#ideas-gdrapp" title="Ideas, Planning, & Feedback">🤔</a></td> </tr> <tr> <td align="center" valign="top" width="14.28%"><a href="https://github.com/niheconomoum"><img src="https://avatars.githubusercontent.com/u/22075648?v=4?s=100" width="100px;" alt="Matthew X. Economou"/><br /><sub><b>Matthew X. Economou</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Aniheconomoum" title="Bug reports">🐛</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/jarretraim"><img src="https://avatars.githubusercontent.com/u/981154?v=4?s=100" width="100px;" alt="Jarret Raim"/><br /><sub><b>Jarret Raim</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Ajarretraim" title="Bug reports">🐛</a></td> <td align="center" valign="top" width="14.28%"><a href="https://mdfranz.dev/"><img src="https://avatars.githubusercontent.com/u/47213?v=4?s=100" width="100px;" alt="Matt Franz"/><br /><sub><b>Matt Franz</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Amdfranz" title="Bug reports">🐛</a></td> <td align="center" valign="top" width="14.28%"><a href="https://www.linkedin.com/in/francescofaenzi/"><img src="https://avatars.githubusercontent.com/u/45026063?v=4?s=100" width="100px;" alt="Francesco Faenzi"/><br /><sub><b>Francesco Faenzi</b></sub></a><br /><a href="#ideas-FrancescoFaenzi" title="Ideas, Planning, & Feedback">🤔</a></td> <td align="center" valign="top" width="14.28%"><a href="https://nishant.daspatnaik.com/"><img src="https://avatars.githubusercontent.com/u/1339669?v=4?s=100" width="100px;" alt="Nishant Das Patnaik"/><br /><sub><b>Nishant Das Patnaik</b></sub></a><br /><a href="#ideas-dpnishant" title="Ideas, Planning, & Feedback">🤔</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/timoguin"><img src="https://avatars.githubusercontent.com/u/671968?v=4?s=100" width="100px;" alt="Tim O'Guin"/><br /><sub><b>Tim O'Guin</b></sub></a><br /><a href="#ideas-timoguin" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/matanolabs/matano/issues?q=author%3Atimoguin" title="Bug reports">🐛</a> <a href="https://github.com/matanolabs/matano/commits?author=timoguin" title="Code">💻</a></td> <td align="center" valign="top" width="14.28%"><a href="https://github.com/francescor"><img src="https://avatars.githubusercontent.com/u/424577?v=4?s=100" width="100px;" alt="Francesco R."/><br /><sub><b>Francesco R.</b></sub></a><br /><a href="https://github.com/matanolabs/matano/issues?q=author%3Afrancescor" title="Bug reports">🐛</a></td> </tr> <tr> <td align="center" valign="top" width="14.28%"><a href="http://grue.io"><img src="https://avatars.githubusercontent.com/u/555914?v=4?s=100" width="100px;" alt="Joshua Sorenson"/><br /><sub><b>Joshua Sorenson</b></sub></a><br /><a href="https://github.com/matanolabs/matano/commits?author=grue" title="Code">💻</a> <a href="https://github.com/matanolabs/matano/commits?author=grue" title="Documentation">📖</a></td> <td align="center" valign="top" width="14.28%"><a href="http://www.nevermind.co.nz"><img src="https://avatars.githubusercontent.com/u/171317?v=4?s=100" width="100px;" alt="Chris Smith"/><br /><sub><b>Chris Smith</b></sub></a><br /><a href="https://github.com/matanolabs/matano/commits?author=chrismsnz" title="Code">💻</a></td> </tr> </tbody> </table> <!-- markdownlint-restore --> <!-- prettier-ignore-end --> <!-- ALL-CONTRIBUTORS-LIST:END --> <!-- prettier-ignore-start --> <!-- markdownlint-disable --> <!-- markdownlint-restore --> <!-- prettier-ignore-end --> <!-- ALL-CONTRIBUTORS-LIST:END -->

This project follows the all-contributors specification. Contributions of any kind are welcome!

License

<img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=03c989f6-90f5-4982-b002-a48635f10b5d"/>