Workflow orchestration architecture — state machine, persistence, retries, versioning

Why architecture matters here

Workflows fail on non-durability (crash loses state), unbounded retries, and version breakage on deploy. Architecture matters because engines handle these for you if used correctly.

The architecture: every piece explained

The top strip is the model. Workflow definition in code or DSL. State machine is the runtime. Persistence makes it durable. Retries + backoff handle transient failures.

The middle row is control. Timers + timeouts for time-based steps. Signals for external input. Compensation undoes on failure. Versioning handles evolution.

The lower rows are ops. Observability per-step trace. Testing replay tests. Ops covers Temporal / Cadence + drills.

End-to-end flow

End-to-end: order workflow starts. Books inventory. Charges card. Ships. If ship fails, compensation reverses charge + release inventory. Server crashes mid-workflow; engine replays from persisted state on restart. Weeks later, code changes; version compatibility ensures running workflows finish under old logic while new ones use new.

What this page owns, and where the neighbours start

This page is about the run: one logical piece of work that spans several steps, survives longer than one model call, and has to end in a defined state even when a step in the middle fails. Everything narrower than that belongs to a neighbour, and re-explaining it here would only give you two half-answers.

Choosing which agent handles an incoming turn is dispatch, not orchestration, and lives in the router deep-dive. How events reach a caller while work is still in progress - ordering, backpressure, cancellation - is streaming. Declaring a tool, shaping its schema, and validating what the model sent is tools in depth. The map of which component does what in the runtime is the core module overview. Teams of model-driven agents handing control to each other is multi-agent architecture, and the tour of the three deterministic composites as language primitives is workflow agents.

What is left, and what you will find below, is the operational half: what a composite guarantees under failure, where the intermediate state physically sits, how a run survives a JVM restart, how it waits days for a human, and how you unwind three committed steps when the fourth one refuses.

A workflow run is not a turn

A turn is bounded by a user message and the final response to it. A workflow run is bounded by an outcome. Those two boundaries coincide often enough that teams build the first version assuming they always do, and then discover the gap the first time a step needs to wait for a warehouse, a payment processor, or a manager's approval.

The distinction shows up in three concrete places. Lifetime: a turn dies with the request thread and the HTTP connection; a run must not. Identity: a turn is identified by a session and an invocation; a run needs its own identifier that you can quote in a support ticket six hours later. Retry semantics: retrying a turn means asking the model again, which is cheap and usually harmless; retrying a run means possibly re-executing steps that already moved money.

Get the identifier right first. If every step writes its result under a key derived from a stable run id, most of the durability work later becomes bookkeeping instead of archaeology. If steps write under keys derived from a timestamp or a retry counter, no amount of engine sophistication will make replay safe.

Three composites, and the guarantee each one actually makes

ADK gives you three agents whose behaviour is fixed in code rather than decided by a model: SequentialAgent, ParallelAgent and LoopAgent. They compose freely, because a composite is itself an agent and can therefore be a child of another composite. The syntax tour belongs to workflow agents; what matters operationally is the guarantee, and the guarantees are narrower than most people assume.

What you are promised

SequentialAgent promises order: child n begins after child n-1 has finished. That is a happens-before relationship, and it is the only reason a later step can read what an earlier one wrote. ParallelAgent promises concurrency: children start together and the composite finishes when they have all finished. LoopAgent promises repetition of its body until an exit condition fires.

What you are not promised

None of the three promises durability. They are in-process control structures, and if the JVM exits mid-run, the run is gone. None of them promises isolation between branches. None of them promises a rollback: a sequential composite that dies on child four leaves the effects of children one through three exactly where they landed. Read them as structured concurrency with a defined shape, not as a transaction manager - and if you need the second thing, you have to build it or buy it.

Where the state between steps lives

Steps communicate through session state, not through return values. An agent that produces something worth keeping writes it under a named key - ADK calls this the output key, spelled output_key in the Python surface - and a later agent reads that key by name, typically by interpolating it into its instruction. The consequence is that the set of key names is the interface between your steps, and it is an interface nothing type-checks for you.

Treat those names with the seriousness you would give a REST contract. Namespace them by step (step.extract.entities rather than result), because a generic key written by two branches of a fan-out is a lost update with no error message. Write the shape down somewhere a reviewer will see it. A one-page table of key, producer, consumer and type catches more integration bugs before merge than any amount of prompt tuning catches after.

Size is the other trap. Session state is carried, serialised and often re-sent; parking a multi-megabyte PDF in it will not fail loudly, it will quietly inflate every subsequent persist. Put bulk payloads in the artifact store and keep a reference in state. State should hold identifiers, decisions and small structured results - the things a later step needs in order to branch.

Deterministic control flow versus letting the model choose

You have two ways to decide what happens next. Encode it - a composite whose shape is fixed at build time. Or delegate it - hand the model a set of options and let it pick. Both are legitimate; the mistake is picking by taste rather than by consequence.

Insist on the deterministic form whenever any of these is true. The order is a policy rather than a preference, such as a screening step that must precede a disbursement step. A step has an irreversible side effect and skipping or repeating it is materially worse than being slow. An auditor will one day ask why the system did what it did, and "the model judged it appropriate" is not an answer you want on the record. The sequence has to be reproducible in a test that must not call a model at all.

Delegate the decision when the branch genuinely depends on the semantics of unstructured input - which specialist should look at this complaint, whether the retrieved passages actually answer the question. Even then, prefer a narrow choice: a model choosing among four labelled branches is a decision you can log, count and alert on; a model free-forming its own plan is one you can only read about afterwards. A useful hybrid is a fixed skeleton with model-chosen leaves - the skeleton keeps the guarantees, the leaves keep the flexibility.

Fan-out, fan-in, and the branch that failed

Fan-out is the easy half. Three enrichment lookups that do not depend on each other run under a ParallelAgent, and the wall-clock cost of the group becomes the cost of its slowest member rather than the sum. With model calls in the branches this is frequently the single largest latency win available: three sequential calls at roughly two seconds each become one span of about two seconds plus scheduling overhead.

Fan-in is where the design decisions hide, and the first one is what the join means when a branch fails. Spell out which of these you want, per fan-out, before you write it:

Whichever you choose, the synthesising step must be able to tell "this branch returned nothing" from "this branch never ran". Write a sentinel into the branch's state key on failure rather than leaving the key absent, otherwise the consumer cannot distinguish a genuine empty result from a hole, and models are alarmingly willing to invent a value for a hole.

The second decision is the concurrency ceiling. A fan-out of eight branches, each of which calls a provider, is eight simultaneous requests against a quota that was sized for one. Bound the width, and expect the provider-side backpressure to be handled where it belongs - see rate limiting and bulkhead isolation.

Loop exits and the runaway guard

A loop body that refines a draft, or retries an extraction until it validates, is one of the most useful shapes in the toolbox and one of the easiest to detonate. It has exactly three ways out, and you should be able to point at all three in the code.

The first is the escalation signal: a child decides the work is done and raises it - in ADK this rides on the event's actions, alongside the state delta - and the loop stops after that iteration completes. The second is the iteration ceiling: the loop's own bound on how many times its body may run, spelled max_iterations in the same Python surface. The third is a failure that propagates out of the body.

The ceiling is not optional and it is not a formality. A loop whose exit condition is judged by a model can fail to fire for reasons that have nothing to do with your logic: the critic is too agreeable, the instruction that describes "good enough" drifted during a prompt edit, the escalation only fires on an exact phrase the model stopped emitting after a version bump. Without a ceiling, each of those becomes an unbounded spend. Set it to the smallest number that has ever actually been needed, plus one - three is a common honest answer for refinement loops - and emit a metric every time the ceiling is what stopped the loop rather than the signal. That counter is the early warning: a rate that climbs from near zero is a prompt regression, visible days before anyone complains about quality.

Guard the exit condition too. If the criterion is "the critic approves", make the criterion a structured field rather than a phrase to be matched, and treat an unparseable critique as a failure to approve rather than as approval.

When the workflow outlives the request

The in-process composites are bounded by the life of the JVM that is running them. The moment a run must survive a deploy, a crash, or a wait measured in hours, the run's position has to exist somewhere outside the heap. There are three ways teams usually get there, in increasing order of cost and capability.

Checkpoint after each step

The cheapest option: after each step commits, persist the run id, the index of the last completed step, and the state keys it produced. On restart, load the record and resume from the next step. This gets you crash tolerance for a linear pipeline and almost nothing else, but it is a genuine improvement over losing the run and it costs one table.

Drive the run from a queue

Each step is a message; completing a step enqueues the next. The broker gives you durability and retry for free, and the agent process becomes stateless, which makes deploys uneventful. The costs are real: message ordering is now your problem, duplicate delivery is now your problem, and a step that fails permanently needs somewhere to go - see dead-letter handling. Committing the state change and the next message atomically is the transactional outbox, covered in the outbox architecture.

Adopt a durable execution engine

Engines in the Temporal and Cadence family persist an event history and rebuild in-memory position by replaying it, which is what lets a workflow "sleep" for a month without holding a thread. You get timers, signals, retry policies and versioning as platform features rather than as code you maintain. The price is a real one: another cluster to run, a programming model with determinism rules your workflow code must obey, and versioning discipline so that runs started under last week's code still finish correctly.

Replaying a step without repeating its effects

Durability and retries together produce a specific hazard: a step whose effect landed but whose completion was never recorded. The process died in the window between charging the card and writing "charged". Any resume mechanism will now run that step again.

The only robust answer is to make step execution idempotent, which means deriving a stable key from the run id and the step identity - never from a clock, never from a retry counter - and having the effect either carry that key to a provider that deduplicates on it, or be recorded in the same transaction that performs it. The mechanics, including the execute-or-replay decision and the lifecycle of the record, are worked through in the idempotency deep-dive; the workflow-level obligation is simply that every step with an external effect must have such a key, and the key must be a pure function of the run and the step.

Two workflow-specific corollaries. A retried step must produce the same state keys as the original attempt would have, or the resumed run diverges from the one that started. And a step inside a loop needs the iteration index in its key, otherwise the second iteration is deduplicated against the first and silently does nothing - a bug that presents as a refinement loop that stops improving.

Human in the loop: parking a workflow that has no thread

Approval steps are where the difference between a request-scoped pipeline and a durable run becomes unavoidable. A run that needs a manager to approve a refund may wait four minutes or four days, and blocking anything for four days is not an implementation, it is an outage waiting for a deploy.

The shape that works is the same one used for any external wait. The step persists the run's position, records what it is waiting for and who can satisfy it, emits a notification, and returns. Nothing is held. A separate inbound path - an API call, a webhook, a signal in engine terms - carries the decision, is matched to the waiting run by its identifier, writes the decision into state, and resumes the run from the recorded position.

Three details decide whether this survives contact with real users. Give the wait a deadline and a defined behaviour when it expires: auto-reject, escalate to a second approver, or cancel the run, but decide, because "wait forever" produces runs that accumulate for months. Make the resume path idempotent, since a manager who clicks approve twice must not advance the run twice. And record the decider's identity and timestamp in state - if the workflow needed human judgement, someone will eventually need to know whose judgement it was.

Compensation when step 4 fails and steps 1-3 already landed

A multi-step run that touches several systems is a distributed transaction without a distributed transaction manager. There is no rollback. What you have instead is compensation: for each step that has an external effect, a second operation that undoes it, and a defined order in which those are executed when a later step fails. This is the saga pattern, and the parts that surprise people are all in the details.

Compensation runs in reverse order of the forward steps: undo four, then three, then two. It has to be at least as reliable as the forward path, because a failed compensation leaves the exact inconsistent state the mechanism existed to prevent - so compensations get retries, and they get idempotency keys of their own. And compensations are rarely true inverses. You do not un-send an email; you send a correction. You do not un-charge a card; you refund it, which is a distinct entry with its own identifier that will appear on the customer's statement. Write the compensating step against what the downstream system can actually do, not against a mental model of undo.

Some steps have no compensation at all. A message published to partners, a report already exported - these are one-way doors. The design response is ordering: push irreversible steps as late in the run as they will go, so that the maximum number of failures happen while unwinding is still possible. When you cannot, the honest alternative is a two-phase shape - reserve first, commit at the end - where the reservation is what expires harmlessly. Tool-level compensating actions for a single call are covered in tools in depth; the run-level ordering question is the one this page cares about.

Timeouts: per step, per branch, per workflow

One timeout is not enough and it is usually placed at the wrong altitude. Three levels each answer a different question.

The step timeout bounds one unit of work - a model call, a tool invocation, an HTTP request to an internal service. It should be set from the observed distribution of that specific step, not from a global default, because a vector search that normally answers in 80 ms and a report generator that normally takes 40 s cannot share a number. The mechanics of cancelling cooperatively in Java, including the interrupt handling that people get wrong, are in tool timeout handling.

The branch timeout bounds a parallel arm so that one slow arm cannot hold the join open indefinitely. It interacts with the join policy above: under best effort, a branch that exceeds it is simply recorded as absent; under all-must-succeed, it fails the group.

The run timeout bounds the whole thing, and it is the one most often missing. Without it, a run can consume every step timeout in turn and still be alive hours later. The invariant that keeps this coherent is that the sum of the timeouts you are willing to spend must not exceed the budget one level up. Where they disagree, the tighter one wins and the looser one is dead configuration - which is worse than no configuration, because someone will tune it and observe nothing. Excluded from the arithmetic, deliberately, are deliberate waits: a human approval step is not a step that is slow, and folding it into the same budget produces runs that cancel themselves overnight.

Reading a workflow run after the fact

The question you will actually be asked is "what happened to run a3f9c1", usually hours later, usually by someone who cannot read logs. Design for that question specifically rather than for dashboards in general.

Make the run id the primary correlation key and stamp it on every log line, every span and every state write the run performs. A trace whose root span is the run and whose children are steps answers most questions on sight: which step was slow, which step retried, where the run stopped. Nested composites should nest as spans, so a parallel group reads as a fan of sibling spans with visibly overlapping time ranges - if they do not overlap, your parallel branch is not actually parallel, and that picture is the fastest way to find out. Span and callback plumbing is covered in observability.

Four metrics carry most of the operational weight, and they are all run-scoped rather than step-scoped: run duration by outcome (completed, failed, compensated, timed out, abandoned); the count of runs that ended by hitting a loop ceiling rather than an exit signal; compensation invocations by step, because a compensation is a step you should not have needed; and the age of the oldest run in a waiting state, which is the single number that reveals runs parked on an approval nobody will ever give.

In-process composites or a durable engine - choosing

Most teams need less than they fear and more than they build. Use the built-in composites when the whole run finishes inside one request, every step is retryable, and total work is seconds. That covers the large majority of agent pipelines: retrieve, then reason, then check, then answer. Adding an orchestration cluster to that is cost you cannot recover.

Move to durable orchestration when any one of four things is true, and note that one is enough. The run waits on a human or an external system for longer than a request should live. Steps have effects that money or law will not let you repeat or lose. Runs must survive deploys, which for a busy service means daily. Or the compensation logic has grown past a couple of steps and is now the most intricate code in the service - at that point you are maintaining a workflow engine that nobody reviewed as one.

The intermediate rung is worth naming because it is under-used: keep the ADK composites for the reasoning shape, and put durability underneath them as a checkpoint table plus a queue. You keep the programming model your agents already use, you get crash tolerance and retry, and you defer the engine until the fourth criterion above genuinely bites. The failure mode to avoid in either direction is a hybrid nobody can describe - half the control flow in a composite, half in ad-hoc retry code, and no single place that says what a run is allowed to do next.