Awesome

Specter 👻

Specter is a CLI tool that lets you simply and quickly search and monitor your Databricks audit logs.

Are you ready to haunt your (lake)house?

Key Features

• Search Logs: Filter and search through audit logs based on user, service, action, event time, and source IP.
• Monitor Queries: Run pre-defined monitoring queries to detect potentially suspicious activity.

Under the Hood

• Unity Catalog: Built-in audit log tables.
• Delta Sharing: Open data sharing protocol.
• DuckDB + Go-DuckDB: Fast in-memory SQL analysis.
• Cobra: Go CLI framework.

Installation

• Create a new table from system.access.audit.
  • NOTE: If you're in the private preview, you could try Delta Sharing a materialized view for fresher data.
  • NOTE: These audit tables may contain a lot of data depending on your Databricks usage. It's recommended to review the size and filter down based on your org's requirements.

CREATE OR REPLACE TABLE logging
AS select * from system.access.audit 
where event_time >= '2024-01-01T00:00:00'

• Create a Delta Share for the table.

  • Set-up Delta Sharing
  • Add a recipient
  • Create a share
  • Add IP of compute executing Specter to IP ACL

• Download config.share file from activation link.

  • NOTE: This should be considered extremely sensitive!

• Download Go (1.21+ required).

go version

• Clone the repository and download dependencies.

git clone https://github.com/jdbraun/specter.git
cd specter
go mod tidy
mkdir config

• Add the config.share file to /config.
• Run the following commands.

go build -o specter
./specter interactive

Usage

Search

The search command allows you to filter audit logs based on various criteria like user, service, and action.

All available service and action names can be found in the documentation.

specter search --user user@email.com --service clusters --action delete --date 2024-10-09 --export results

• --user: Filter by user email (e.g., user@email.com).
• --service: Filter by service name (e.g., clusters).
• --action: Filter by action name (e.g., delete).
• --date: Filter by event date (e.g., 2024-10-09).
• --export: Export the results to a CSV file (e.g., permissionsCheck).

Monitor

The monitor command runs pre-defined SQL queries that can help you detect potential security issues.

specter monitor --list
specter monitor --run 1

• --list: Lists all the pre-defined monitoring queries.
• --run: Runs a specific monitoring query by its number or all if set to 0.

Monitoring Queries

• repeated_unauthorized_uc_data_requests - Detect repeated unauthorized UC data requests
• destructive_activities_last_90_days - Monitor destructive activities over the last 90 days
• changes_to_workspace_configuration - Detect changes to workspace configuration in the last 24 hours

specter monitor --list 

Monitoring queries sourced from the repository for the blog post: Improve Lakehouse Security Monitoring using System Tables in Databricks Unity Catalog.

Examples

Search Results:

Monitor List Results:

Monitor Results:

Limitations

• Specter has only been validated on MacOS and Databricks on AWS.
• Data needs to be incrementally refreshed at source until materialized view Delta Sharing is out of private preview.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Contributions

Feel free to contribute by submitting issues or pull requests.