Awesome

pySigma Backend Cookiecutter Template

Cookiecutter template for a pySigma backend.

Features

• Backend and processing pipeline classes
• pyproject.toml template
• Testing with pytest
• CI testing with GitHub Actions
• Automatic test deployment on test PyPI on Git version tagging.
• Automatic deployment to PyPI on GitHub release.

Usage

Install Cookiecutter if not yet done with:

python3 -m pip install --user cookiecutter

This cookiecutter uses Poetry for installation of current dependencies. Install it as described here.

Generate a pySigma backend project from this template with:

cookiecutter https://github.com/SigmaHQ/cookiecutter-pySigma-backend.git

Developing a Backend

1. Define the tokens in the backend class.
2. Implement required Sigma rule transformations (e.g. field mappings, value transformations) as processing pipeline
3. Implement further code required to implement output in the backend targets query language.
4. Implement tests to ensure correctness and quality.
5. Remove pipelines directory if backend doesn't defines any pipelines.

Install dependencies as required with poetry add <package>.

Coverage Badge

While creation of a repository from the Cookiecutter template, you're asked for a Gist identifier for the coverage badge:

coverage_badge [True]:
coverage_gist [GitHub Gist identifier containing coverage badge JSON expected by shields.io.]:

This identifier has only be provided if the coverage badge should be embedded in the README of your backend project and coverage_badge is set to True.

To get this identifier navigate to the GitHub Gist main page, create a Gist with some dummy content (it is generated by the GitHub Actions job) and copy&paste the identifier after your user name to the terminal that runs Cookiecutter:

https://gist.github.com/<your user name>/<the required identifier (hex id)>

Publishing a Backend

Generally there are two options:

• Host the new backend within your own account and publish the backend package to PyPI on your own.
• Ask the SigmaHQ project maintainers to host the backend within the SigmaHQ organization and take care of the PyPI release.

Integration into Sigma CLI

A new backend has to be integrated into Sigma CLI. The technical integration is described in the Sigma CLI README. To be integrated in Sigma CLI releases the backend has to meet some requirements:

• Custom backend code should be well tested with automatic CI tests.
• The backend must use the intended interfaces for initialization, processing and returning results as well as error states. It should not output anything directly or terminate the running program on errors.
• The developer of the backend takes over its ownership and actively maintains it (bug fixes etc.). Unmaintained or erroneous backends will be removed after a while.

These requirements should ensure a certain quality of the Sigma toolchain.

Self-Publishing

PyPI

If you don't have a PyPI account or want to release the backend with a new account, register new accounts on the:

• Test instance - this is not necessary to publish a package but strongly recommended.
• Productive instance

As releasing packages is a sensitive thing, especially with security-related software, immediate configuration of multifactor authentication of the PyPI accounts is strongly recommended.

Finally create an authentication token.

Repository Configuration

With the template configuration you need to configure the following secrets in your repository:

• GIST_SECRET (optional): the secret of the GitHub Gist containing the coverage badge information.
• TEST_PYPI_API_TOKEN (optional but recommended): PyPI token for deployment to test instance.
• PYPI_API_TOKEN: PyPI token for deployment to productive instance.

Release

1. Conduct a test release by tagging the commit with a version tag in the format vx.y.z, e.g. v0.1.0. This version has to be in sync with the version parameter of pyproject.toml.
2. If the test deployment finishes successfully, create a GitHub release from this tag to publish it to the productive PyPI.

After the first release the authentication token should be regenerated with a scope restricted to the backend package.