Awesome
StreamFlow
The StreamFlow framework is a container-native Workflow Management System (WMS) written in Python 3. It has been designed around two main principles:
- Allow the execution of tasks in multi-container environments, in order to support concurrent execution of multiple communicating tasks in a multi-agent ecosystem.
- Relax the requirement of a single shared data space, in order to allow for hybrid workflow executions on top of multi-cloud or hybrid cloud/HPC infrastructures.
Use StreamFlow
PyPI
The StreamFlow module is available on PyPI, so you can install it using pip.
pip install streamflow
Please note that StreamFlow requires python >= 3.9
. Then you can execute it directly from the CLI
streamflow run /path/to/streamflow.yml
Docker
StreamFlow Docker images are available on Docker Hub. In order to run a workflow inside the StreamFlow image
- A StreamFlow project, containing a
streamflow.yml
file and all the other relevant dependencies (e.g. a CWL description of the workflow steps and a Helm description of the execution environment) needs to be mounted as a volume inside the container, for example in the/streamflow/project
folder - Workflow outputs, if any, will be stored in the
/streamflow/results
folder. Therefore, it is necessary to mount such location as a volume in order to persist the results - StreamFlow will save all its temporary files inside the
/tmp/streamflow
location. For debugging purposes, or in order to improve I/O performances in case of huge files, it could be useful to mount also such location as a volume - The path of the
streamflow.yml
file inside the container (e.g./streamflow/project/streamflow.yml
) must be passed as an argument to the Docker container
The script below gives an example of StreamFlow execution in a Docker container
docker run -d \
--mount type=bind,source="$(pwd)"/my-project,target=/streamflow/project \
--mount type=bind,source="$(pwd)"/results,target=/streamflow/results \
--mount type=bind,source="$(pwd)"/tmp,target=/tmp/streamflow \
alphaunito/streamflow \
streamflow run /streamflow/project/streamflow.yml
Kubernetes
It is also possible to execute the StreamFlow container as a Job
in Kubernetes.
In this case, StreamFlow is able to deploy Helm
charts directly on the parent cluster through the
ServiceAccount
credentials. In order to do that, the inCluster
option must be set to true
for each
involved module on the streamflow.yml
file
deployments:
helm-deployment:
type: helm
config:
inCluster: true
...
A Helm
template of a StreamFlow Job
can be found in the helm/chart
folder.
Please note that, in case RBAC is active on the
Kubernetes cluster, a proper RoleBinding
must be attached to the ServiceAccount
object, in order to give
StreamFlow the permissions to manage deployments of pods and executions of tasks.
CWL Compatibility
StreamFlow relies on the Common Workflow Language (CWL) standard to design workflow models. CWL conformance badges for StreamFlow are reported below.
CWL v1.0
Classes
Required features
Optional features
CWL v1.1
Classes
Required features
Optional features
CWL v1.2
Classes
Required features
Optional features
Contribute to StreamFlow
See contribution.
StreamFlow Team
Iacopo Colonnelli iacopo.colonnelli@unito.it (creator and maintainer)
Barbara Cantalupo barbara.cantalupo@unito.it (maintainer)
Marco Aldinucci aldinuc@di.unito.it (maintainer)
Gaetano Saitta gaetano.saitta@edu.unito.it (contributor)
Alberto Mulone alberto.mulone@edu.unito.it (contributor)