Home

Awesome

StreamFlow

CI Tests

The StreamFlow framework is a container-native Workflow Management System (WMS) written in Python 3. It has been designed around two main principles:

Use StreamFlow

PyPI

The StreamFlow module is available on PyPI, so you can install it using pip.

pip install streamflow

Please note that StreamFlow requires python >= 3.8. Then you can execute it directly from the CLI

streamflow run /path/to/streamflow.yml

Docker

StreamFlow Docker images are available on Docker Hub. In order to run a workflow inside the StreamFlow image

The script below gives an example of StreamFlow execution in a Docker container

docker run -d \
    --mount type=bind,source="$(pwd)"/my-project,target=/streamflow/project \
    --mount type=bind,source="$(pwd)"/results,target=/streamflow/results \
    --mount type=bind,source="$(pwd)"/tmp,target=/tmp/streamflow \
    alphaunito/streamflow \
    streamflow run /streamflow/project/streamflow.yml

Kubernetes

It is also possible to execute the StreamFlow container as a Job in Kubernetes. In this case, StreamFlow is able to deploy Helm charts directly on the parent cluster through the ServiceAccount credentials. In order to do that, the inCluster option must be set to true for each involved module on the streamflow.yml file

deployments:
  helm-deployment:
    type: helm
    config:
      inCluster: true
      ...

A Helm template of a StreamFlow Job can be found in the helm/chart folder.

Please note that, in case RBAC is active on the Kubernetes cluster, a proper RoleBinding must be attached to the ServiceAccount object, in order to give StreamFlow the permissions to manage deployments of pods and executions of tasks.

CWL Compatibility

StreamFlow relies on the Common Workflow Language (CWL) standard to design workflow models. CWL conformance badges for StreamFlow are reported below.

CWL v1.0

Classes

Required features

Optional features

CWL v1.1

Classes

Required features

Optional features

CWL v1.2

Classes

Required features

Optional features

Contribute to StreamFlow

See contribution.

StreamFlow Team

Iacopo Colonnelli iacopo.colonnelli@unito.it (creator and maintainer)
Barbara Cantalupo barbara.cantalupo@unito.it (maintainer)
Marco Aldinucci aldinuc@di.unito.it (maintainer)

Gaetano Saitta gaetano.saitta@edu.unito.it (contributor)
Alberto Mulone alberto.mulone@edu.unito.it (contributor)