🔀Workflow Orchestration

Three philosophies

Pick the right axis and the whole comparison snaps into focus. That axis is: what is the unit of work, who guarantees it survives failure, and how do you author it? Each tool answers differently, and almost everything else — languages, scaling, pricing, lock-in — is downstream of the answer.

• Temporal — the durable-execution engine. The unit is a workflow function that calls activities. Durability is owned by the engine: it records an event history and rebuilds in-memory state by replaying the log after any crash. You author in code (Go, Java, Python, TypeScript, .NET, PHP). The pitch: "write code as if failure doesn't exist."
• Airflow — the DAG scheduler. The unit is a task inside a directed acyclic graph. There is no replay and no resumed call stack: durability is the scheduler re-running tasks whose state lives in a metadata database, on the assumption that tasks are idempotent. You author in Python that declares the graph; the bodies do the work. It is built for batch pipelines on a schedule.
• Step Functions — the managed state machine. The unit is a state in a finite-state machine. Durability is owned by the managed AWS service, which persists every state transition for you. You author declaratively in Workflow Studio or raw ASL JSON; the real logic lives in the AWS services each state invokes.

The comparison matrix

The exhaustive side-by-side. Use the controls to filter rows down to one dimension or highlight a single tool's column. (Both persist across reloads; Copy as Markdown always exports the full matrix.)

Execution & durability

"What survives a crash?" is the question that separates these three most sharply. They give three genuinely different answers, and confusing them is the most common mistake in this space.

• Temporal replays a log. Every step a workflow takes is appended to a durable event history. When a worker dies mid-execution, a fresh worker re-runs the function from the top, but each already-recorded step returns its stored result instantly instead of re-executing. When replay catches up to the last event, the in-memory state — locals, call stack, pending awaits — is exactly as it was, and execution continues. The function literally resumes mid-line. (This is the subject of the companion page.)
• Airflow re-runs tasks. There is no replayed call stack. The scheduler tracks each task's state (queued, running, success, failed) in the metadata database. If a task or worker dies, the scheduler simply re-queues that task from the start, up to its retries count. Durability is therefore an emergent property of idempotent tasks plus the metadata DB, at task granularity — not a resumed program.
• Step Functions persists transitions. The managed service records the result of every state transition. If a downstream service fails, the state machine retries or catches per its ASL rules; the execution itself is never "lost" because AWS holds its state. Standard workflows give exactly-once execution of each state; Express trades that for throughput and is at-least-once.

Same workflow, three ways

One toy use case, written in each tool: fetch an order, total it, then branch — notify a manager for big orders, otherwise send a receipt. The shape of each program is the philosophy made concrete.

NO_RETRY = RetryPolicy(maximum_attempts=1)   # Temporal retries ∞ by default; opt out

@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order_id: str) -> str:
        order = await workflow.execute_activity(
            fetch_order, order_id,
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=RetryPolicy(maximum_attempts=5),   # opt in: retry the fetch
        )
        total = await workflow.execute_activity(
            compute_total, order, retry_policy=NO_RETRY)
        if total > 1000:                          # ordinary branching, in code
            await workflow.execute_activity(
                notify_manager, order_id, retry_policy=NO_RETRY)
        else:
            await workflow.execute_activity(
                send_receipt, order_id, retry_policy=NO_RETRY)
        return "done"

@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False)
def order_pipeline(order_id: str = "A-42"):     # run input, via DAG params
    @task(retries=5)                            # auto-retried
    def fetch_order(order_id: str) -> dict:
        return api.get_order(order_id)

    @task
    def compute_total(order: dict) -> float:
        return sum(i["price"] for i in order["items"])

    @task.branch                            # pick the next task by id
    def route(total: float) -> str:
        return "notify_manager" if total > 1000 else "send_receipt"

    @task
    def notify_manager(order_id: str): ...
    @task
    def send_receipt(order_id: str): ...

    total = compute_total(fetch_order(order_id))
    route(total) >> [notify_manager(order_id), send_receipt(order_id)]

order_pipeline()

{
  "StartAt": "FetchOrder",
  "States": {
    "FetchOrder": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "fetchOrder",
                       "Payload": { "order_id.$": "$.order_id" } },
      "Retry": [ { "ErrorEquals": ["States.TaskFailed"],
                  "MaxAttempts": 5, "BackoffRate": 2.0 } ],
      "Next": "ComputeTotal"
    },
    "ComputeTotal": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Next": "Route"
    },
    "Route": {
      "Type": "Choice",
      "Choices": [ { "Variable": "$.total",
                    "NumericGreaterThan": 1000, "Next": "NotifyManager" } ],
      "Default": "SendReceipt"
    },
    "NotifyManager": { "Type": "Task", "Resource": "arn:aws:states:::sns:publish", "End": true },
    "SendReceipt":   { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "End": true }
  }
}

Read top to bottom: Temporal's branch is a Python if; Airflow's is a @task.branch that returns the id of the next task in the graph; Step Functions' is a declarative Choice state. None of the real work happens in the Step Functions document — it happens in the Lambda functions and SNS topics it points at.

Scheduling & triggering

How does a run begin? This is one of the most practical differences, and it reflects each tool's heritage.

• Airflow is time-first. Its native instinct is the cadence: a DAG declares a schedule (a cron string or a timetable) and the scheduler creates a run per interval. Modern Airflow adds data-aware scheduling — a DAG can run when its upstream Assets update — and external event triggers, but the batch-interval model is its center of gravity.
• Step Functions is event/API-first. An execution starts on an explicit StartExecution call — directly, from API Gateway, or, most commonly, from EventBridge rules and schedules. It is built to react to events flowing through AWS, with cron available via an EventBridge schedule.
• Temporal is code/API-first. A workflow starts when a client calls start_workflow() from anywhere in your system — typically in response to a user action or an event. For recurring runs, Temporal Schedules provide cron-like and calendar triggering as a first-class server feature.

Rule of thumb: if you think in "every night at 2am, process yesterday's data," that is Airflow's home turf. If you think in "when this event happens, run this process," Step Functions and Temporal start a run per event rather than per interval.

State, data & retries

Passing data between steps

Because Temporal workflows are ordinary functions, intermediate state is just local variables and activity return values held in memory (and reconstructed on replay). Airflow tasks are separate processes, so values move through XCom, persisted in the metadata DB and meant for small payloads — bulk data should travel through external storage (S3, a warehouse) with only references in XCom. Step Functions threads a JSON document through the states, reshaped per state with JSONPath or JSONata, with a hard 256 KB payload limit between states (again: keep large data in S3 and pass pointers).

Retries & failure

All three express retries declaratively, per step — but their defaults are opposite, which is easy to miss. Temporal retries every activity automatically by default (unlimited attempts, exponential backoff), so you attach a retry policy to cap it — or, as in the code above, a maximum_attempts=1 policy to switch it off; it also models all-or-nothing flows as sagas with explicit compensation in code. Airflow and Step Functions are the reverse — a step does not retry unless you opt in: Airflow via retries/retry_delay (default 0) plus trigger rules and on_failure_callback hooks; Step Functions via Retry and Catch blocks per state, routing caught errors to a fallback. So matching them up takes extra code on the Temporal side (the NO_RETRY policy) and zero on the others — the opt-out-vs-opt-in difference made literal.

Architecture

What you actually operate is wildly different across the three — from "a database plus a few stateless services" to "literally nothing."

Airflow: several cooperating components

Airflow self-hosted is a small distributed system around a central metadata database. The scheduler decides what to run and hands tasks to an executor (Local, Celery, or Kubernetes); the DAG processor parses your Python DAG files; workers execute tasks; the triggerer runs deferred/async waits; and a webserver / API server serves the UI and REST API. Everyone coordinates through the metadata DB.

Step Functions: a black box

There is no architecture to run. Step Functions is a fully managed AWS service: you register a state machine and call StartExecution. AWS owns the scheduler, the durable store, the retries, and the scaling. Your "components" are the AWS services each Task state invokes — Lambda, SQS, SNS, DynamoDB, ECS, and, via SDK integrations, 200+ services and thousands of API actions.

Temporal: a service plus your workers

Temporal sits in between: a Temporal Service (its Frontend, History, and Matching roles, backed by a database such as Cassandra, MySQL, or PostgreSQL) plus your own stateless workers that host the workflow and activity code and long-poll for tasks. You run the cluster (or buy Temporal Cloud) and you run the workers; the durable state lives in the service. The internals get a full treatment on the durable-execution page.

Deployment & managed offerings

This is where lock-in is decided. Two of the three are open source and portable; one is a single cloud.

For Airflow, note the distinction the marketing pages blur: MWAA and Cloud Composer are managed deployments of the real Apache Airflow — not forks — so a DAG can in principle move between them and a self-hosted cluster. Astronomer is the main commercial vendor (and steward of much of the project), but it is third-party rather than a hyperscaler offering. Step Functions has no self-hosted form at all: choosing it is choosing AWS.

Scale & limits

All three scale to serious production load, but they hit different ceilings.

• Temporal scales by adding stateless workers and by sharding workflow histories across the cluster; high-scale users run millions of concurrent executions. The practical limit per workflow is event-history size — long-running loops use Continue-As-New to start a fresh history and avoid unbounded growth.
• Airflow scales workers horizontally (Celery or Kubernetes) and throttles with pools and parallelism settings. Its historical bottleneck is the scheduler and metadata DB under very high task volume — a constraint teams operating Airflow at scale have written about candidly (Shopify's account of running thousands of DAGs is a good primary read). It is tuned for throughput of batch work, not low-latency per-event execution.
• Step Functions autoscales transparently. The numbers are AWS account quotas (some soft/adjustable): Standard supports thousands of state transitions per second; Express is built for very high event rates (on the order of 100,000 executions/second). The hard caps that bite are the duration limits — Standard 1 year, Express 5 minutes — and the 256 KB payload between states.

Observability & versioning

Watching a run

Each ships a first-party UI tuned to its model. Airflow's React UI is pipeline-ops centric: the Grid and Graph views show task status across runs, with logs per task instance. Step Functions' console renders the state machine graph and animates each execution's path, with Standard executions keeping full history in-console and Express sending logs to CloudWatch. Temporal's Web UI is execution-centric: it shows the complete event history of any single workflow, which is the natural debugging unit for durable execution.

Changing a running workflow

Long-lived workflows must survive their own code changing underneath them. Temporal takes this most seriously because replay makes it hard: workflow versioning (patching) and Worker Versioning let new code coexist with in-flight executions (detailed on the durable-execution page). Airflow provides DAG versioning so a run completes on the DAG version it started with. Step Functions supports versions and aliases of a state machine, so you can publish an immutable version and shift traffic deliberately.

Cost models

Three different shapes of bill, which matter as much as features at scale.

• Temporal — the OSS server is free; you pay for the infrastructure you run it on (and the operational effort). Temporal Cloud instead bills per action (workflow and activity operations) plus storage, so cost tracks usage rather than fixed servers.
• Airflow — OSS is free plus infrastructure. Managed Airflow is priced per environment: MWAA and Cloud Composer charge for the running environment (size/uptime) regardless of how busy your DAGs are, so an idle Airflow still costs money.
• Step Functions — pure pay-per-use, and the two workflow types bill on different axes:
  • Standard: $0.000025 per state transition, with 4,000 free transitions/month. Predictable, audit-grade, can run a year.
  • Express: $1.00 per million requests plus $0.00001667 per GB-second of duration. Cheap at very high volume and short durations.

When to use which

The honest one-liners, then a flow.

• Reach for Temporal when the work is long-lived, stateful application logic that must survive crashes and deploys — sagas with compensation, multi-day human-in-the-loop, microservice orchestration — expressed as ordinary code in your language and portable across clouds.
• Reach for Airflow when the work is scheduled batch data pipelines (ETL/ELT/ML) and you want a rich operator ecosystem plus a pipeline-ops UI, and your team thinks in Python and dependency graphs. Not for per-request or durable-app logic.
• Reach for Step Functions when you are all-in on AWS and want to orchestrate AWS services with minimal ops and pay-per-use. Choose Standard for long-running, auditable, exactly-once flows (up to a year); Express for high-volume, short (<5 min), at-least-once event processing.

Common misconceptions

Further reading

Temporal

• Durable Execution — the companion design doc on this site: the full Temporal/Cadence deep-dive — event history, deterministic replay, the durable toolkit, server internals, and versioning.
• temporal.io and docs.temporal.io — the primary source for the concepts and SDK examples summarized here.

Apache Airflow

• airflow.apache.org — the official site and docs (architecture overview, core concepts, executors); current stable is the 3.x line.
• apache/airflow on GitHub — the source and release notes.
• Apache Airflow — Wikipedia — project history and high-level overview.
• Amazon MWAA and Google Cloud Composer — the first-party managed Airflow services.
• Shopify Engineering — lessons learned running Airflow at scale and "Airflow's problem" — practitioner perspectives on Airflow's scaling and design edges. An introduction to Airflow for the core model.

AWS Step Functions

• aws.amazon.com/step-functions and the Developer Guide — workflow types, ASL, integration patterns, and quotas.
• Step Functions pricing — the exact Standard and Express figures used above.
• Datadog — AWS Step Functions knowledge center — a concise external overview.