Deploying Dagster on Helm#

Overview#

Kubernetes is a container orchestration system for automating deployment, scaling, and management of containerized applications. Dagster uses Kubernetes in combination with Helm, a package manager for Kubernetes applications. Using Helm, users specify the configuration of required Kubernetes resources to deploy Dagster through a values file or command-line overrides. References to values.yaml in the following sections refer to Dagster's values.yaml.

Dagster publishes a fully-featured Helm chart to manage installing and running a production-grade Kubernetes deployment of Dagster. For each Dagster component in the chart, Dagster publishes a corresponding Docker image on DockerHub.

Prerequsites#

kubectl should be configured with your desired Kubernetes cluster. You should understand the basics of Helm, and Helm 3 should be installed. If you are creating your own user code images, Docker should be installed as well.

Versioning#

The Dagster Helm chart is versioned with the same version numbers as the Dagster Python library, and ideally should only be used together when the version numbers match.

In the following tutorial, we install the most recent version of the Dagster Helm chart. To use an older version of the Chart, a --version flag can be passed to helm upgrade. If you are using a chart version before 0.11.13, you will also need to update the tags of the Dagster provided images to match the Chart version. After 0.11.13, this will automatically be done for you.

Deployment Architecture#

Components#

Daemon#

The daemon periodically checks the Runs table in PostgreSQL for Pipeline Runs in that are ready to be launched. The daemon also runs the dagster-native scheduler, which has exactly-once guarantees.

The Daemon launches the run via the K8sRunLauncher, creating a Run Worker Job with the image specified in the User Code Deployment.

Dagit#

The Dagit webserver communicates with the User Code Deployments via gRPC to fetch information needed to populate the Dagit UI. Dagit does not load or execute user-written code to ensure robustness, and will remain available even when user code contains errors. Dagit frequently checks whether the User Code Deployment has been updated; and if so, the new information is fetched.

Dagit can be horizontally scaled by setting the dagit.replicaCount field in the values.yaml.

By default, it is configured with a K8sRunLauncher, which creates a new Kubernetes Job per pipeline run.

Database#

The user can connect an external database (i.e. using a cloud provider's managed database service, like RDS) or run PostgreSQL on Kubernetes. This database stores Pipeline Runs, Events, Schedules, etc and powers much of the real-time and historical data visible in Dagit. In order to maintain a referenceable history of events, we recommend connecting an external database for most use cases.

Run Worker#

The Run Worker is responsible for executing the solids in topological order. The Run Worker uses the same image as the User Code Deployment at the time the run was requested. The Run Worker uses ephemeral compute, and completes once the run is finished. Events that occur during the run are written to the database, and are displayed in Dagit.

The Run Worker jobs and pods are not automatically deleted so that users are able to inspect results. It is up to the user to delete old jobs and pods after noting their status.

Executor#

The Executor determines how the run worker will execute each step of a pipeline. Different executors offer different levels of isolation and concurrency. Common choices are in_process_executor (all steps run serially in a single process in the single pod), multiprocess_executor (multiple processes in the single pod), and kubernetes-job (each step runs in a separate pod). For the latter, you can choose between the celery_k8s_job_executor, and the k8s_job_executor which removes the Celery dependency. Generally, increasing isolation incurs some additional overhead per step (e.g. starting up a new Kubernetes job vs starting a new process within a pod). Different executors can be configured per-run in the execution block.

User Code Deployment#

A User Code Deployment runs a gRPC server and responds to Dagit's requests for information (such as: "List all of the pipelines in each repository" or "What is the dependency structure of pipeline X?"). The user-provided image for the User Code Deployment must contain a repository definition and all of the packages needed to execute within the repository.

Users can have multiple User Code Deployments. A common pattern is for each User Code Deployment to correspond to a different repository.

This component can be updated independently from other Dagster components, including Dagit. As a result, updates to repositories can occur without causing downtime to any other repository or to Dagit. After updating, if there is an error with any repository, an error is surfaced for that repository within Dagit; all other repositories and Dagit will still operate normally.

Walkthrough#

Build Docker image for User Code#

Skip this step if using Dagster's example User Code image dagster/user-code-example.

Create a Docker image containing the Dagster repository and any dependencies needed to execute the objects in the repository. For reference, here is an example Dockerfile and the corresponding User Code directory.

Push Docker image to registry#

Skip this step if using Dagster's example User Code image.

Publish the image to a registry that is accessible from the Kubernetes cluster, such as AWS ECR or DockerHub.

Add the Dagster Helm chart repository#

The Dagster chart repository contains the versioned charts for all Dagster releases. Add the remote url under the namespace dagster to install the Dagster charts.

helm repo add dagster https://dagster-io.github.io/helm

Configure your User Deployment#

Update the dagster-user-deployments.deployments section of the Dagster chart's values.yaml to include your deployment.

The following snippet works for Dagster's example User Code image.

dagster-user-deployments:
  enabled: true
  deployments:
    - name: "k8s-example-user-code-1"
    image:
      repository: "docker.io/dagster/user-code-example"
      tag: latest
      pullPolicy: Always
    dagsterApiGrpcArgs:
      - "-f"
      - "/example_project/example_repo/repo.py"
    port: 3030

The dagsterApiGrpcArgs field expects a list of arguments for dagster api grpc which is run upon Deployment creation and starts the gRPC server. To find the applicable arguments, read here.

Install the Dagster Helm chart#

Install the Helm chart and create a release. Below, we've named our release dagster. We use helm upgrade --install to create the release if it does not exist; otherwise, the existing dagster release will be modified:

helm upgrade --install dagster dagster/dagster -f /path/to/values.yaml

Helm will launch several pods including PostgreSQL. You can check the status of the installation with kubectl. If everything worked correctly, you should see output like the following:

$ kubectl get pods
NAME                                              READY   STATUS    RESTARTS   AGE
dagster-dagit-645b7d59f8-6lwxh                    1/1     Running   0          11m
dagster-k8s-example-user-code-1-88764b4f4-ds7tn   1/1     Running   0          9m24s
dagster-postgresql-0                              1/1     Running   0          17m

Run a pipeline in your deployment#

After Helm has successfully installed all the required kubernetes resources, start port forwarding to the Dagit pod via:

export DAGIT_POD_NAME=$(kubectl get pods --namespace default \
  -l "app.kubernetes.io/name=dagster,app.kubernetes.io/instance=dagster,component=dagit" \
  -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $DAGIT_POD_NAME 8080:80

Visit http://127.0.0.1:8080, navigate to the playground, select the default preset, and click Launch Execution.

You can introspect the jobs that were launched with kubectl:

$ kubectl get jobs
NAME                                               COMPLETIONS   DURATION   AGE
dagster-run-c8f4e3c2-0915-4317-a168-bf8c86810fb2   1/1           4s         6s

Within Dagit, you can watch pipeline progress live update and succeed!

Conclusion#

We deployed Dagster, configured with the default K8sRunLauncher, onto a Kubernetes cluster using Helm.