♾️Durable Execution

Why durable execution exists

Consider a checkout that has to charge a card, reserve inventory, wait up to three days for a shipping partner to confirm, email a receipt, and refund everything if any step ultimately fails. None of that is hard logic. What's hard is that the process must survive a deploy, a pod eviction, or a datacenter blip at any instant, and resume without double-charging or losing track of where it was.

The traditional way to build this is to shred the procedure into fragments glued together by infrastructure: a row in a database holding a status column, a queue per step, a cron to sweep for stuck rows, retry/back-off wrappers, idempotency keys to dedupe the at-least-once delivery, and a dead-letter queue for the rest. Each piece is reasonable; together they reconstruct, badly, a single thing — a program whose execution survives failure. The business logic ends up scattered across queue topology and database state where no one can read it as a story.

• State lives outside the code. "Where is this order?" is answered by joining a status column against queue depth, not by reading a function top to bottom.
• Partial failure is everywhere. A crash between "charged the card" and "recorded that we charged the card" is a bug you must design around at every single step.
• Waiting is expensive. Blocking three days for a callback means either holding a process open or inventing yet another timer-and-resume dance.
• Retries duplicate work. At-least-once queues mean every consumer must be idempotent, or you charge twice.

Durable execution inverts the arrangement. You write the procedure as one ordinary function — charge, reserve, await the confirmation, email, and a compensation path on failure — and a workflow engine makes that function itself durable. Cadence's original framing from Uber is the cleanest statement of the ambition: stop describing orchestration in a DSL or YAML and instead let developers "write workflows as freely as writing programs," on the thesis that "any program is a workflow." Temporal phrases the result as a promise to the developer: your code "execute[s] effectively once and to completion, regardless of whether it takes seconds or years."

The programming model

The model has four nouns. Learn these and the rest is detail.

• Workflow. Your durable function. It expresses the orchestration — the order of steps, the waits, the branching, the error handling — and the engine preserves its state across failures. A Workflow Definition is the code; a Workflow Execution is one running instance of it.
• Activity. A plain function that does one real-world thing — call a service, write a row, charge a card. Activities are where all the messy, non-deterministic, failure-prone work lives. The engine retries them automatically on failure and records each result.
• Worker. Your process that hosts the workflow and activity code. It long-polls the server for work, runs it, and reports back. Workers are stateless and scale horizontally; you can kill them at any time.
• Task Queue. A named queue the server uses to hand tasks to workers. You start a workflow "on" a task queue, and the workers polling that queue pick it up. (Cadence calls this a Task List.)

Here is the whole loop in the Temporal Python SDK. The workflow is plain async code; the side effects are isolated in activities; the worker wires them to a task queue:

from datetime import timedelta
from temporalio import activity, workflow
from temporalio.client import Client
from temporalio.worker import Worker
from temporalio.common import RetryPolicy

# --- Activities: where all the I/O lives (may be non-deterministic) ---
@activity.defn
async def charge_card(order_id: str, amount_cents: int) -> str:
    return await payments.charge(order_id, amount_cents)   # real network call

@activity.defn
async def send_receipt(order_id: str) -> None:
    await email.send(order_id)

# --- Workflow: deterministic orchestration only, no direct I/O ---
@workflow.defn
class CheckoutWorkflow:
    @workflow.run
    async def run(self, order_id: str, amount_cents: int) -> str:
        confirmation = await workflow.execute_activity(
            charge_card, args=[order_id, amount_cents],
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=RetryPolicy(maximum_attempts=5),   # auto-retried
        )
        await workflow.execute_activity(
            send_receipt, order_id,
            start_to_close_timeout=timedelta(seconds=30),
        )
        return confirmation

# --- Worker: long-polls a Task Queue and runs the code above ---
async def main():
    client = await Client.connect("localhost:7233", namespace="default")
    worker = Worker(
        client, task_queue="checkout",
        workflows=[CheckoutWorkflow], activities=[charge_card, send_receipt],
    )
    await worker.run()

Event history & replay

A workflow is not persisted as a snapshot of its variables. It is persisted as an Event History: an append-only log of everything that has happened — WorkflowExecutionStarted, ActivityTaskScheduled, ActivityTaskCompleted, TimerStarted, TimerFired, and so on. The log is the source of truth. The workflow's "current state" is simply whatever you get by replaying that log through the code.

That is how a workflow survives a crash. When a worker dies mid-execution, the server hands the workflow to another worker, which replays the history from the beginning: it re-runs your function, but every step that already has a recorded result returns that result instantly instead of doing the work again. The activity that already completed isn't re-invoked — its recorded output is fed back. A timer that already fired doesn't wait again. When replay reaches the last recorded event, the in-memory state — local variables, position in the code, pending awaits — is exactly what it was before the crash, and execution continues forward.

Why workflow code must be deterministic

Replay only works if re-running the code produces the same sequence of actions it produced the first time. If the original run scheduled "charge card" as its second step, the replay must also reach "charge card" as its second step — otherwise the recorded results no longer line up with what the code is asking for, and the engine raises a non-determinism error rather than corrupt the execution. So workflow code must be deterministic: given the same history, it must always make the same calls in the same order.

That forbids, directly in workflow code, anything whose value can change between runs: the wall clock, random numbers, UUIDs, reading a file or database, iterating a hash map in undefined order, or spawning unmanaged threads. The engine doesn't ban these capabilities — it relocates them. Non-deterministic work goes into activities (whose results get recorded), and for the common cases the SDK provides replay-safe equivalents that read from the history instead of the real world:

@workflow.run
async def run(self) -> None:
    # WRONG — non-deterministic: replay can diverge and fail the task
    now   = datetime.now()          # real wall clock
    token = uuid.uuid4()            # real randomness
    rows  = await db.query(...)     # direct I/O

    # RIGHT — replay-safe SDK APIs, or push the work into an Activity
    now   = workflow.now()             # deterministic time, from history
    token = workflow.uuid4()           # deterministic, seeded from history
    rows  = await workflow.execute_activity(load_rows, ...)

Commands and events: the round trip

The workflow code never writes to the database itself. Instead, when a worker runs a Workflow Task, it replays to the current point, runs forward until the code blocks on something (an activity result, a timer, a signal), and emits Commands describing what it wants to happen next: ScheduleActivityTask, StartTimer, CompleteWorkflowExecution. The server validates those commands, turns them into Events, appends them to the history, and acts on them — dispatching the activity, arming the timer. The result later arrives as another event, which triggers the next Workflow Task, and the cycle repeats. (Cadence calls the worker's instructions Decisions and the task a Decision Task — the same mechanism under an older name.)

This is also what makes a three-day wait free. A workflow.sleep(timedelta(days=3)) doesn't hold a thread for three days; it emits a StartTimer command, the workflow is evicted from memory entirely, and three days later the TimerFired event wakes it on whatever worker is free. Durability and long waits fall out of the same event-log mechanism.

The durable toolkit

On top of workflows and activities sits a small set of primitives that cover most real orchestration. Each is durable and replay-safe, recorded in the same event history.

Signals, queries, and timers compose into the patterns that are painful to hand-roll. Here a subscription bills every month, sleeps durably between charges for as long as the customer stays subscribed, and reacts to an out-of-band cancellation — a process that may live for years across countless deploys and crashes:

@workflow.defn
class SubscriptionWorkflow:
    def __init__(self) -> None:
        self._cancelled = False
        self._status = "active"

    @workflow.run
    async def run(self, user_id: str) -> str:
        while not self._cancelled:
            await workflow.execute_activity(charge_month, user_id, ...)
            # Durable sleep — survives crashes, can span weeks
            await workflow.sleep(timedelta(days=30))
        self._status = "cancelled"
        return self._status

    @workflow.signal                # async write into a running workflow
    def cancel(self) -> None:
        self._cancelled = True

    @workflow.query                 # read-only; must not mutate state
    def status(self) -> str:
        return self._status

The caller drives it through a handle, possibly months apart, from anywhere:

handle = await client.start_workflow(
    SubscriptionWorkflow.run, "user-42",
    id="sub-user-42", task_queue="billing",
)
await handle.signal(SubscriptionWorkflow.cancel)         # … months later
print(await handle.query(SubscriptionWorkflow.status))   # "cancelled"

Sagas: failure that unwinds itself

The checkout that has to undo a charge when shipping ultimately fails is the Saga pattern, and durable execution turns its bookkeeping into ordinary control flow. You record a compensating action after each successful step, and on failure run the compensations in reverse. Because the workflow's local state is durable, the list of "things to undo" survives crashes for free — no separate saga log, no reconciler:

compensations = []
try:
    await workflow.execute_activity(charge_card, ...)
    compensations.append(refund_card)
    await workflow.execute_activity(reserve_inventory, ...)
    compensations.append(release_inventory)
    await workflow.execute_activity(ship_order, ...)
except Exception:
    for undo in reversed(compensations):   # unwind in reverse
        await workflow.execute_activity(undo, ...)
    raise

Inside the server

Workers are stateless; all the durability lives in the server. Both Temporal and Cadence — unsurprisingly, given the shared lineage — decompose that server into the same four internal services, each independently scalable, talking to a pluggable datastore.

• Frontend. A stateless gateway that terminates the gRPC API, authenticates, rate-limits, and routes calls to the other services. It holds no state, so you scale it like any web tier — it is the one service that is not sharded.
• History. The heart. It owns each workflow's mutable state and Event History, drives executions forward, and persists every event. To scale, the workflow id space is sharded across the cluster — each shard owns a slice of the id space.
• Matching. Hosts the user-facing Task Queues and matches pending tasks to the workers long-polling for them. Task queues are created on demand and effectively unlimited.
• Worker (internal/system). Runs the server's own background workflows — replication, archival, periodic system maintenance. Distinct from your worker processes, which live outside the server.

Underneath sits a pluggable persistence layer — Cassandra, MySQL, or PostgreSQL — and a separate visibility store that powers "list and search my workflows." Basic visibility runs on the same database; advanced visibility — filtering millions of executions by custom search attributes — is backed by Elasticsearch.

Changing code on a live workflow

Determinism buys durability, but it bills you at deploy time. A workflow execution started today may still be running in six months, replaying its history against whatever code the worker is running then. If you have edited the workflow in a way that changes the sequence of commands it produces, the replay of an old execution diverges from its recorded history — the non-determinism error from earlier. You cannot freely edit code that has live executions.

Both engines offer two first-party ways out.

Patching

A patch lets old and new logic coexist behind a guarded branch keyed by a change id. The first time an execution hits workflow.patched("id") it records a marker in its history; from then on, that execution always takes the branch matching its marker. Executions that started before the change keep replaying the old path deterministically; new executions take the new one. Once every pre-patch execution has drained, you call deprecate_patch and later delete the dead branch.

@workflow.run
async def run(self, order_id: str) -> None:
    await workflow.execute_activity(reserve_inventory, order_id, ...)
    if workflow.patched("ship-v2"):
        await workflow.execute_activity(ship_v2, order_id, ...)   # new executions
    else:
        await workflow.execute_activity(ship_v1, order_id, ...)   # old, in-flight

Worker Versioning

The newer, and now generally recommended, approach is to version the workers instead of branching the code. You tag a worker deployment with a Build ID and roll it out as a versioned deployment. The server then keeps each workflow on a compatible version: pinned workflows run to completion on the exact version they started on, while auto-upgrade workflows move to the new version as it rolls out. Old and new code run side by side — "rainbow deployments" — with gradual traffic ramping and instant rollback, and no patch branches cluttering the workflow. Temporal's guidance is blunt: if you can run versioned worker deployments, prefer Worker Versioning over patching.

One lineage: Temporal & Cadence

Temporal and Cadence look so alike because they are alike: Temporal is a fork of Cadence, made by the people who wrote Cadence. The story also explains why the two will feel uncannily similar in any sample you read.

Cadence's creators — Maxim Fateev and Samar Abbas — had earlier launched the first public version of AWS Simple Workflow Service (Maxim also led the work that became Amazon SQS). They reunited at Uber starting in 2015 to build Cadence, a "fault-oblivious" workflow engine on the thesis that "any program is a workflow"; it grew to power over a thousand services at Uber. In 2019 the two founded Temporal Technologies around a fork of Cadence, hardening the model and broadening the SDK ecosystem. Uber kept investing in Cadence — it now lives under the open cadence-workflow GitHub organization, was contributed to the CNCF in 2025, and remains in heavy production use there.

The split left a thin layer of renamed vocabulary over an identical model. If you know one engine, this table is most of what you need to read the other:

Where they diverge is positioning, by Cadence's own account: Temporal prioritized ecosystem breadth and a commercial cloud, while Cadence stayed focused on the reliability, multi-tenancy, and cost-efficiency Uber needs at very high scale. The programming model you'd write against is, for most intents, the same.

Running it in production

A durable-execution platform is a stateful distributed system with a datastore behind it, so the operational question is who runs that system — you or a vendor.

• Self-hosted. Both engines are open source: you run the server cluster and operate the backing store (Cassandra/MySQL/PostgreSQL, plus Elasticsearch for advanced visibility). Maximum control; you own capacity planning, upgrades, and the database.
• Managed. Temporal Cloud, from Temporal Technologies, is a hosted Temporal Service with an SLA, namespace management, SAML auth, and audit logging — you bring only workers. Managed Cadence is offered by Instaclustr by NetApp. In both cases your workers still run in your own infrastructure; only the server and datastore are hosted.

SDK reach is one practical difference between the two. Temporal officially ships SDKs for Go, Java, Python, TypeScript, .NET, Ruby, and PHP; Cadence's first-party clients are Go and Java (with community Python/Ruby and the iWF framework on top). The Python in this doc is Temporal's SDK for that reason.

What Hacker News thinks

Durable execution has been chewed over on Hacker News for years. Two threads bracket the arc: the 2019 launch discussion of "Cadence: Uber's Workflow Orchestration Engine" (243 points), and the 2022 "Temporal raises $103M in Series B funding" thread (92 points). The sentiment across both is admiring but far from unanimous.

The praise

• It solves a real, genuinely hard problem. The dominant view is that coordinating long-running, failure-prone, multi-step processes — normally hand-rolled from queues, a status column, and cron — is fragile glue code, and durable execution replaces an entire category of edge cases. A recurring analogy: this is to manual state-and-queue plumbing what garbage collection was to manual memory management.
• Workflows as code beat YAML and DSLs. Strong, repeated preference for writing orchestration as ordinary procedural code over JSON/YAML state machines, on the argument that generic config languages become unmaintainable for non-trivial logic.
• The durability "clicks" once you get it. Commenters describe the guarantee — start a workflow and it's effectively indestructible short of losing the database — as the moment the model wins them over, reaching for hibernation and Kubernetes-reconciler analogies.
• Founder pedigree buys credibility. The team's history (AWS Simple Workflow, Uber Cadence) comes up often; one of the original AWS Step Functions designers even joined the 2019 discussion, lending it unusual authority.

The skepticism

• Self-hosting is operationally heavy — and under-documented. The single most common complaint: running it yourself (a Cassandra/SQL backend, the cluster, monitoring) is a lot, and production self-hosting was felt to be poorly documented — which, fairly or not, pushes people toward the paid cloud.
• It's overkill for simple apps. Many argued a plain SQL backend, a Celery queue, a handful of Airflow DAGs, or cron is enough, and that plenty of teams reach for this imagining a scale they don't have.
• The determinism model has real foot-guns. Replay semantics and the ban on non-deterministic workflow code are called non-obvious and easy to trip over, with debugging harder than an imperative task queue.
• Versioning live workflows is admittedly painful — a point conceded even by commenters from the Temporal side.
• "We tried it and backed out." The most-cited caution came from a team that dropped it over the learning curve and limited visibility, feeling it "shifted the problem from one place to another."

The alternatives: a lighter wave

The complaints in the last section — heavy to operate, overkill for the simple case — are precisely the brief a wave of newer engines set out to fill. Since roughly 2024, and sharply accelerated by AI agents (long-running, failure-prone loops that beg for durability), durable execution has gone from a niche two-horse race to a crowded field. Almost every newcomer defines itself against Temporal's operational weight, and they sort along three axes: a library or a standalone service, your existing database or a purpose-built store, and durability that is explicit in your code or invisible underneath it.

"Just use your database"

DBOS is the sharpest contrast to Temporal: durable execution as an in-process library, not a cluster. You annotate functions with @DBOS.workflow() and @DBOS.step(), and the decorators checkpoint each step's result into Postgres tables; on a crash it replays from the last finished step. There is no separate workflow server — "just your program and Postgres" — and a state transition is a ~1 ms database write rather than a server round-trip. (It grew out of a Stanford/MIT project with Postgres's creator Michael Stonebraker advising.)

from dbos import DBOS

@DBOS.step()                    # a checkpoint: its result is recorded in Postgres
def charge_card(order_id: str) -> str:
    return payments.charge(order_id)

@DBOS.workflow()                # durable: on crash, resumes at the last finished step
def checkout(order_id: str) -> None:
    charge_card(order_id)
    send_receipt(order_id)      # also a @DBOS.step()

Hatchet makes the same Postgres-native bet but keeps the orchestrator as a separate service, treating durable execution as one feature of a broader task queue. Microsoft's pg_durable pushes the idea furthest — durable workflows authored in a SQL DSL and run inside Postgres as an extension — and two widely read 2025 write-ups, Armin Ronacher's "Absurd Workflows" and Gunnar Morling's SQLite engine, show the whole pattern in a few thousand lines over a plain database.

On Hacker News this camp drew one consistent objection: it moves control flow out of code-and-Git and into the database, and makes Postgres the scaling ceiling — "the step journaling is pretty brutal to postgres," said more than one engineer who had built the embedded approach and later ripped it back out to an external system. The equally consistent rebuttal: most services never reach that ceiling, and not running a separate distributed system is worth a great deal.

One binary, or a new runtime

Restate answers "don't run a cluster" a different way: the entire engine is a single self-contained Rust binary with a built-in durable log, so you "download and start and you are done." Its core unit is not a workflow but a durable RPC handler, with virtual objects giving handlers consistent keyed state; the headline claims are low latency and a push model that fits serverless/Lambda better than a pull-based worker. Flawless goes further still: it compiles workflows to WebAssembly and snapshots the WASM stack and heap, so durability is implicit — there are no step wrappers, just code that happens to be resumable. The price is a single-language cage (Rust→WASM) and a curated set of permitted host calls.

Serverless & the managed clouds

Inngest brings durable step.*() calls to an event-driven, push-based serverless runtime, and has proposed StepKit as a backend-agnostic "durable execution standard" that runs on Cloudflare, Netlify, or self-hosted. Around it sit the cloud-native managed offerings, several of which predate the current wave: AWS Step Functions (JSON state machines), Azure Durable Functions (built on the Durable Task Framework — a sibling of the same AWS-SWF lineage that produced Cadence), Google Cloud Workflows, and Cloudflare Workflows. The trade is the familiar one: zero infrastructure in exchange for some vendor lock-in and, often, a DSL instead of code.

Adjacent: the DAG orchestrators

One step away sit the pipeline schedulers — Apache Airflow, Prefect, Dagster — and the script-and-low-code tools Windmill and n8n. These are DAG-first rather than durable-code-first, and aimed at data engineers or business users; the durable-execution founders mostly cede that lane rather than fight for it. Others worth knowing in the broader space include Trigger.dev, Resonate (durable promises), Golem (WASM durable execution), LittleHorse, and Orkes / Netflix Conductor.

Further reading

• Temporal — official site and docs.temporal.io. The concept encyclopedia, SDK guides, and the source of this doc's Temporal terminology and the Python examples.
• temporalio/temporal on GitHub. The server (Go), release notes, and the README's statement of the Cadence fork lineage.
• cadenceworkflow.io and its docs. Cadence's concepts ("fault-oblivious stateful workflow"), the Task List / Domain / Decision vocabulary, and the Cadence-vs-Temporal FAQ.
• cadence-workflow/cadence on GitHub. The Cadence server and releases (current v1.4).
• Uber Engineering — Cadence. The "any program is a workflow" thesis and Uber's production scale, from the team that built it.
• The Hacker News threads: Cadence (2019) and Temporal's Series B (2022) — the source of the sentiment section.
• The alternatives: DBOS, Restate, Hatchet, Inngest, and the managed AWS Step Functions and Azure Durable Functions — the lighter engines surveyed above.