KubernetesΒΆ
Kubernetes is a popular system for deploying distributed applications on clusters, particularly in the cloud. You can use Kubernetes to launch Dask workers in the following two ways:
Helm: You can launch a Dask scheduler, several workers, and an optional Jupyter Notebook server on a Kubernetes easily using Helm.
helm repo update # get latest helm charts helm install stable/dask # deploy standard dask chart
This is a good choice if you want to do the following:
- Run a managed Dask cluster for a long period of time
- Also deploy a Jupyter server from which to run code,
- Share the same Dask cluster between many automated services
- Try out Dask for the first time on a cloud-based system like Amazon, Google, or Microsoft Azure (see also our Cloud documentation)
See Dask and Helm documentation for more information.
Native: You can quickly deploy Dask workers on Kubernetes from within a Python script or interactive session using Dask-Kubernetes.
from dask_kubernetes import KubeCluster cluster = KubeCluster.from_yaml('worker-template.yaml') cluster.scale(20) # add 20 workers cluster.adapt() # or create and destroy workers dynamically based on workload from dask.distributed import Client client = Client(cluster)
This is a good choice if you want to do the following:
- Dynamically create a personal and ephemeral deployment for interactive use
- Allow many individuals the ability to launch their own custom dask deployments, rather than depend on a centralized system
- Quickly adapt Dask cluster size to the current workload
See Dask-Kubernetes documentation for more information.
You may also want to see the documentation on using Dask with Docker containers to help you manage your software environments on Kubernetes.