Awesome
dask-remote
Aims to collect tools for deploying a persistent dask
cluster, in particular on Kubernetes.
dask_remote.runner
ClusterProcess
provides a way to run anyCluster
in a python process, thus allowing easy way to build CLIs and other non-interactive cluster deployments. The process can manage e.g. aLocalCluster
, aKubeCluster
, or aDeploymentCluster
.ClusterProcessProxy
provides a process and thread-safe for eachClusterProcess
to allow access to methods and attributes such asscale
ApiProcess
provides a way to expose the proxy methods via a RESTful API built onFastAPI
, as well as a way to run a simpleuvicorn
server exposing this API in a separate process.
With a Cluster and the API server both running, we can e.g. scale the cluster over REST:
$ curl -X POST http://localhost:8000/scale/42
dask_remote.client
ApiCluster
provides a way to interact with a remote cluster via a REST API from your code or notebook. It provides the expected methods and an interactive Widget.
Cluster widget | Client widget |
---|---|
![]() | ![]() |
dask_remote.deployment
Provides a DeploymentCluster
class for managing scaling via a Kubernetes Deployment of worker Pods.