Module 7 closed with the runtime substrate: Kubernetes for most data teams in 2026, with the operator pattern doing the heavy lifting for Spark, Flink, and the rest. Module 8 starts one layer up. Now that the workloads have a place to run, what triggers them, in what order, with what dependencies, and what watches them when they fail? That is the orchestrator’s job, and the choice of orchestrator is one of the more consequential decisions a data platform makes.
The choice has gotten richer in the last few years. For most of the 2010s “orchestrator” effectively meant Airflow, with cron and bespoke scripts as the small-scale alternative. By 2026 there are four serious contenders, each with a defensible niche: Airflow, Prefect, Dagster, and Argo Workflows. They are not interchangeable. They make different assumptions about what a pipeline is, what state the orchestrator owns, and what the user’s mental model should look like. Picking among them is a real architectural decision, not a matter of taste.
This lesson walks through what an orchestrator does, then through each of the four, then through the framing that helps a team pick: task-oriented vs asset-oriented (which lesson 58 expands), and managed vs self-hosted (which is usually the decision that gets made first in practice).
What an orchestrator actually does
Strip the marketing and an orchestrator is responsible for a small, stable list of jobs.
Schedule jobs. Run this pipeline at 02:00 every day. Run that one when a new file lands in S3. Run this third one when an upstream pipeline completes successfully.
Track state. This task is queued. That one is running. The other one finished at 02:14 with exit code 0. A history of which tasks ran, when, and how they ended is the orchestrator’s persistent log of what your data system did.
Manage dependencies. Task B depends on task A. Do not start B until A succeeds. If A fails, do not start B. If A is delayed, B waits.
Retry on failure. Transient failures (a network blip, a flaky API) should retry automatically, with backoff, up to a configured limit. Real failures should surface to a human.
Alert on issues. A failed task, a task that ran too long, a SLA missed: the orchestrator emits a signal that a human picks up. Slack, PagerDuty, email, whatever the team uses.
Give you a UI. When something is wrong at 03:00 you need a screen that shows the last twenty-four hours of runs, the failed tasks, the logs, and the dependency graph, ideally without reading source code.
Every orchestrator does these things. The differences are in how they expose them, what other things they bring along, and what assumptions they bake into the model.
Airflow: the original
Airflow came out of Airbnb in 2014, joined the Apache incubator in 2016, and graduated to top-level project status in 2019. Maxime Beauchemin, the original author, wrote it in response to the painful experience of running batch pipelines via cron and home-grown scripts at a company that was scaling fast. The design choice that defined Airflow was Python-first DAGs: pipelines are Python files that import operators and wire them together, and the scheduler reads those files to figure out what to run.
That choice cuts both ways.
The good side: an enormous ecosystem of operators ships with Airflow or as community packages. BigQuery, Snowflake, Redshift, S3, GCS, Spark, dbt, Slack, every major cloud’s compute and storage services, every major data tool. If you want to orchestrate something, the operator probably exists. The 2020s also brought Kubernetes-native deployment via the official Helm chart and the KubernetesExecutor (lesson 55), which let each task run in its own pod with its own resource budget and its own container image.
The not-so-good side: DAGs as Python files mix orchestration with logic. The scheduler parses every DAG file periodically, which means accidentally importing a heavy library at the top of a DAG file makes the scheduler slow. Backfills (rerunning a date range with the latest code) were painful before Airflow 2.x and only fully reasonable from 2.3 onward. The 1.x to 2.x migration in 2020 was real engineering work for shops that had built up significant Airflow infrastructure.
Airflow’s installation is heavy by modern standards. The default architecture has a webserver, a scheduler, a metadata database (Postgres or MySQL), a result backend, and a pool of workers (Celery, Kubernetes, or local). Running this yourself is several days of work to do well and an ongoing on-call burden. Most shops in 2026 either use a managed Airflow (Astronomer, AWS MWAA, GCP Cloud Composer) or have a platform team that maintains it as a shared service.
When Airflow wins. Big organisations that already standardised on it. Shops that need the operator ecosystem. Teams whose pipelines are predominantly time-scheduled batch jobs with long DAGs. Anywhere with enough scale that “yet another tool” is a harder sell than “the existing tool, used well”.
Prefect: the developer-experience answer
Prefect started in 2018 as Jeremiah Lowin’s response to the frictions he had hit running Airflow at scale. The 1.x line was a from-scratch reimagining; the 2.x line (2022) was a rewrite that simplified the model further; the 3.x line (2024) tightened the API again. The headline feature is the developer experience: flows are decorated Python functions, dependencies are inferred from how you call them, and the API feels closer to writing normal Python than authoring a DAG.
from prefect import flow, task
@task
def extract():
return load_from_source()
@task
def transform(rows):
return clean(rows)
@flow
def daily_etl():
rows = extract()
cleaned = transform(rows)
write_to_warehouse(cleaned)That is a working Prefect flow. The dependency between extract and transform is implicit in the data flow. There is no DAG file to register, no separate scheduler config to write. Dynamic flows (where the shape of the DAG depends on runtime data) are first-class, which Airflow supports awkwardly through the dynamic task mapping API but Prefect handles natively.
When Prefect wins. Smaller teams that value developer experience. Pipelines whose shape is dynamic (a fan-out over an unknown number of files, a per-customer flow whose count varies by day). Teams that want a managed orchestrator with a generous free tier and a low-friction onboarding.
The trade-off is a smaller ecosystem than Airflow. Prefect has integrations for the major data tools, but the long tail of niche operators that Airflow accumulated over a decade is not there. Most teams do not hit this in practice; it shows up at the margins.
Dagster: the asset-oriented answer
Dagster, also from 2019, took a different swing. The core insight: orchestrators that think in tasks are missing what data engineers actually care about, which is the assets the pipeline produces. A task is a means to an end. The end is the customer table, the daily revenue mart, the trained model artefact. So make the asset the unit of orchestration.
In Dagster, you declare assets:
from dagster import asset
@asset
def raw_events():
return load_from_kafka()
@asset
def sessionized_events(raw_events):
return sessionize(raw_events)
@asset
def customer_lifetime_value(sessionized_events):
return compute_clv(sessionized_events)The dependencies between assets are read from the function signatures. Dagster builds the asset graph automatically. Lineage is not a thing you add; it is the model. The orchestration (which task runs when) is derived from the asset graph, not declared separately.
This shift, which lesson 58 explores in depth, has consequences. Lineage comes for free. dbt models are first-class assets, not opaque tasks the orchestrator runs. Typed inputs and outputs let Dagster validate at runtime that an asset actually produced what it claimed to. The UI shows asset freshness (“last materialised two hours ago”) instead of just task status (“succeeded at 02:14”).
When Dagster wins. Teams that care about lineage, data quality, and software-engineering discipline applied to data. Heavy dbt users who want their orchestrator and their transformation tool to share a vocabulary. New platforms with no orchestrator legacy where the asset-oriented model can be adopted from day one.
The trade-offs are conceptual buy-in (the team has to learn to think in assets, not tasks) and a smaller ecosystem than Airflow’s, though the gap has narrowed. Migrating an existing Airflow shop to Dagster is a real project, not a weekend.
Argo Workflows: the Kubernetes-native answer
Argo Workflows is the odd one in the list. It was not built primarily for data; it was built for Kubernetes. Each step in an Argo workflow is a pod. The workflow itself is a Kubernetes custom resource defined in YAML. The orchestrator is a controller that watches workflow resources and reconciles them, the same pattern as the Spark Operator and the rest of the Kubernetes ecosystem (lesson 55).
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: daily-etl
spec:
entrypoint: pipeline
templates:
- name: pipeline
dag:
tasks:
- name: extract
template: extract-step
- name: transform
template: transform-step
dependencies: [extract]Argo is popular for ML pipelines (Kubeflow Pipelines is built on Argo) and for general workflow orchestration in shops that have committed hard to Kubernetes. It scales to thousands of concurrent pods. It is fast, simple, and aligns well with the rest of a cloud-native stack.
When Argo wins. ML-heavy teams using Kubeflow. Platform teams that want their orchestrator to feel like the rest of Kubernetes. Workloads that are dominated by short, parallel container runs rather than long-lived batch jobs with rich data semantics.
The trade-off is YAML and weaker support for data-specific concerns. Argo does not know what a Snowflake query is. It does not have native dbt integration. Lineage is whatever you bolt on. For pipelines that are mostly “run this Python in a container, then run that one”, Argo is great. For pipelines that are mostly “run this Snowflake transformation, then this dbt model, then publish to BigQuery”, an orchestrator with first-class data integrations is usually a better fit.
Managed vs self-hosted
The orchestrator choice gets entangled, in practice, with the managed-vs-self-hosted choice. Most shops in 2026 land on managed, and the reasoning is consistent.
Self-hosting an orchestrator means running its metadata database (usually Postgres), its scheduler, its workers, its UI, its retry logic, its log storage. For Airflow that is half a dozen components with their own scaling and upgrade stories. For Prefect, Dagster, Argo, slightly less but still real. Anyone who has run Airflow upgrades from 1.10 to 2.x and then 2.x to 2.7 knows the operational tax.
The managed offerings absorb that tax in exchange for money: Astronomer (Airflow), Prefect Cloud, Dagster Cloud, Databricks Workflows (their own orchestrator, integrates with Databricks jobs), AWS MWAA (managed Airflow), GCP Cloud Composer (managed Airflow), Azure Data Factory (Microsoft’s orchestrator). The pricing varies, but for most teams the math is “we pay tens of thousands a year, we save one engineer-month a year on cluster ops, and we get a vendor to call when the scheduler dies on a Sunday”.
The exceptions are organisations large enough to staff a dedicated platform team that runs orchestration as a shared service, and organisations with regulatory constraints that make a managed SaaS hard. Both are real, both are common at scale, and both involve trading money for control.
The decision tree
Pulling it together, the rough decision tree most teams walk through:
flowchart TD
Q1{Pipeline complexity?}
Q1 -->|Few jobs, simple deps| CRON[cron + scripts]
Q1 -->|Real DAGs, real ops| Q2{Asset model fits?}
Q2 -->|Yes, lineage matters| Q3a{Managed?}
Q2 -->|No, task-oriented OK| Q3b{Existing tool?}
Q3a -->|Yes| DC[Dagster Cloud]
Q3a -->|No, want to host| DAGSTER[Self-host Dagster]
Q3b -->|Yes, Airflow| Q4{Managed?}
Q3b -->|No, fresh start| Q5{Style?}
Q4 -->|Yes| AST[Astronomer/MWAA/Composer]
Q4 -->|No| AF[Self-host Airflow]
Q5 -->|Pythonic, dynamic| PRE[Prefect]
Q5 -->|K8s-native, simple| ARGO[Argo Workflows]Diagram to create: a polished version of the decision tree above, with the four leaf orchestrators visually distinguished. The point is that the choice splits cleanly on three axes (asset vs task, managed vs self-hosted, ecosystem fit) and that all four leaves are defensible answers for some team.
The same simple ETL expressed in Airflow vs Dagster sketches the difference. Airflow:
with DAG("daily_etl", schedule="@daily") as dag:
e = PythonOperator(task_id="extract", python_callable=extract)
t = PythonOperator(task_id="transform", python_callable=transform)
l = PythonOperator(task_id="load", python_callable=load)
e >> t >> lDagster:
@asset
def raw_data():
return extract()
@asset
def cleaned_data(raw_data):
return transform(raw_data)
@asset
def warehouse_table(cleaned_data):
return load(cleaned_data)Same pipeline, different mental models. Airflow is “run these tasks in order on this schedule”. Dagster is “these are the assets you produce; figure out how to keep them fresh”. The next lesson is about why that second framing changes how teams think at scale.
What this lesson set up
Module 8’s job is to take the running architecture and make it operable: orchestration (this lesson), asset-oriented framing (next lesson), observability (lesson 59), reliability practices (the rest of the module). The orchestrator is the spine. The choice you make here propagates into every other decision: how you instrument lineage, how you handle retries, how you define an SLO for a data product. Picking deliberately is worth the time.
For most mid-to-large data shops in 2026, the practical answer is one of: managed Airflow if you have an Airflow legacy, Dagster Cloud if you are starting fresh and care about data products, Prefect Cloud if you want the developer experience and have dynamic workflows, Argo if your platform is Kubernetes-first and your workloads are container-shaped. All four are good. The one that fits your team’s mental model and your existing infrastructure is almost always the right one.
Citations and further reading
- Apache Airflow documentation,
https://airflow.apache.org/docs/(retrieved 2026-05-01). The official reference for the original orchestrator, its operator catalogue, and the Kubernetes integrations. - Prefect documentation,
https://docs.prefect.io/(retrieved 2026-05-01). The 3.x docs cover flows, tasks, deployments, and the hybrid execution model. - Dagster documentation,
https://docs.dagster.io/(retrieved 2026-05-01). The asset-oriented model is the heart of the docs; the dbt and Snowflake integrations are useful starting points. - Argo Workflows documentation,
https://argo-workflows.readthedocs.io/(retrieved 2026-05-01). The Kubernetes-native workflow engine, with the YAML and operator-pattern references. - Maxime Beauchemin, “The Rise of the Data Engineer” and the original Airflow announcements on the Medium engineering blog (2017 onward). Useful context on why Airflow exists in the shape it does.