Awesome

Kedro Kubeflow Plugin

About

The main purpose of this plugin is to enable running kedro pipeline on Kubeflow Pipelines. It supports translation from Kedro pipeline DSL to kfp (pipelines SDK) and deployment to a running kubeflow cluster with some convenient commands.

The plugin can be used together with kedro-docker to simplify preparation of docker image for pipeline execution.

Documentation

For detailed documentation refer to https://kedro-kubeflow.readthedocs.io/

Usage guide

Usage: kedro kubeflow [OPTIONS] COMMAND [ARGS]...
 
   Interact with Kubeflow Pipelines
 
 Options:
   -h, --help  Show this message and exit.
 
 Commands:
   compile          Translates Kedro pipeline into YAML file with Kubeflow pipeline definition
   init             Initializes configuration for the plugin
   list-pipelines   List deployed pipeline definitions
   run-once         Deploy pipeline as a single run within given experiment.
   schedule         Schedules recurring execution of latest version of the pipeline
   ui               Open Kubeflow Pipelines UI in new browser tab
   upload-pipeline  Uploads pipeline to Kubeflow server

Configuration file

kedro init generates configuration file for the plugin, but users may want to adjust it to match the run environment requirements: https://kedro-kubeflow.readthedocs.io/en/latest/source/02_installation/02_configuration.html