Streaming agent architecture — SSE, partials, interrupt, resume, backpressure

Why architecture matters here

Streaming agents fail on the boring parts. Client disconnects mid-response — is the work still running? Interrupt during a tool call — does the tool complete or abort? Client reconnects — can it resume without repeating the tool call? Slow client — does the server buffer forever?

The architecture matters because these are joint concerns of the runtime, protocol, and state management. Naive implementations lose work; over-engineered ones become impossible to debug.

With the pieces in mind, you can build a streaming agent that survives real network conditions.

The architecture: every piece explained

The top strip is the wire path. Client opens an SSE connection. Agent runtime runs an async loop backed by virtual threads or reactive streams. Model stream emits tokens token-by-token. Tool call events announce start and result — clients can render "agent is running search()".

The middle row is the resilience. Partial output deltas ship as they arrive. Interrupt handler handles user cancels and tears down downstream work (model call, tool invocation) with cancellation propagation. Resume from checkpoint uses a session ID to pick up after disconnects. Backpressure pauses the model stream when the client is slow, propagating pressure upstream.

The lower rows are state and ops. State store persists checkpoints per step so a resume can rehydrate the runtime. Observability tracks stream metrics — first-token latency, tokens/s, interrupt count — and trace across services. Ops handles timeouts, reconnect strategies, and graceful shutdown when the server rolls.

End-to-end flow

End-to-end: a user asks the agent a question. Client opens SSE. Runtime starts a model call; tokens stream to the client. Model calls a tool; runtime emits a tool_start event, runs the tool, emits tool_result. Model continues generating. User's network hiccup; SSE closes. Runtime observes the disconnect; because the user did not cancel, it continues to completion and checkpoints state. User's client reconnects with the same session ID; runtime replays the buffered output. User interrupts; runtime sends cancellation through the model call, aborts, and returns a final message. Metrics show p95 first-token 320 ms.

The stream is the API: Flowable of Event

In ADK for Java the runner does not hand you a response object. Runner.runAsync(...) returns a Flowable<Event> - an RxJava 3 publisher - and everything a turn does becomes an element on that sequence. Text deltas, the function call the model decided to make, the value the tool returned, state mutations, control-flow transfers, the terminal event: one ordered channel. There is no second callback interface to register and no side band where the interesting things happen.

Two consequences fall out of that type immediately, and both surprise people arriving from a blocking HTTP mindset. First, the Flowable is cold. Building it costs nothing and runs nothing; the model is not called until something subscribes. A method that returns the publisher and never subscribes has started no work at all, and a publisher that gets subscribed twice runs the turn twice - two model calls, two tool invocations, two bills. Second, because Flowable implements org.reactivestreams.Publisher, it drops into Reactor and Spring WebFlux without an adapter library: Flux.from(runner.runAsync(...)) is legal, and so is handing it to any Reactive Streams sink.

The anatomy of the Event object itself - its parts, authors, actions, and state deltas - is covered in ADK Events: the agent's event stream. This article is about what the Java stream type does to your code.

What one run emits, and in what order

A single-agent turn with one tool call produces a recognisable shape. A short burst of partial text events as the model starts talking. A non-partial event that carries the aggregated content for that message. An event whose content is a function call. Then, once your runtime has actually executed that tool, its result arrives as its own event. Then partial text again as the model continues with the tool output in context, another aggregate, and finally the event that closes the turn.

The ordering guarantee you can rely on is the Reactive Streams one: onNext calls are serialised and never overlap, so your consumer is single-threaded with respect to event handling even when the producer side is fanning out internally. You never need a lock inside subscribe. What you do not get is a guarantee that the shape above is the only shape - a run with no tool call skips those events entirely, a router turn inserts a transfer, and a multi-agent run interleaves events authored by different agents. Code that switches on position in the sequence breaks; code that switches on the event's own fields does not.

Distinguishing in-progress from settled is the one thing every consumer must get right. Partial events carry the partial flag; the aggregate that follows does not, and finalResponse() tells you an event is the one worth persisting or returning to a non-streaming caller. Append partials, then replace on the aggregate - appending both is the double-render bug that shows the user every sentence twice. The conceptual treatment of partials versus aggregates lives in ADK streaming architecture; the Java-side rule is simply that both flags live on the same object and you must read them before you touch your buffer.

Tool calls arrive on the same stream as text

A function call is not a pause in the stream - it is an event on it. The model emits an event whose content carries a call with a name and arguments; nothing has executed yet. Your runtime dispatches the tool, and when it returns, a function response event lands on the same sequence. From the consumer's seat this is just two more onNext calls between two runs of text.

The operational problem is the gap. A model streaming at forty tokens per second puts an event on the wire every twenty-five milliseconds; a tool that queries a warehouse puts nothing there for four seconds. Nothing in the byte stream distinguishes a tool that is working from a connection that has died, and load balancers enforcing idle timeouts will close the socket out from under a perfectly healthy turn. So forward the call event downstream the instant it arrives - it is what lets the UI render "searching orders..." rather than an empty pane - and schedule a keepalive frame for as long as a tool is outstanding.

Long-running tools deserve a separate decision. If a tool takes minutes, you do not want the turn's stream held open across it; the durable pattern is to return a handle immediately and let the result arrive as a later event against the same session. And if you run several tools concurrently, remember that responses land in completion order, not call order - correlate by the call identifier on the event, never by arrival position.

Backpressure is request(n), not a metaphor

The reason ADK Java's stream type is Flowable and not Observable is that Flowable is backpressure-aware: the subscriber signals demand with request(n) and the producer is not permitted to exceed it. If your subscriber's onNext writes to a servlet output stream and that write blocks because the client's TCP window is full, you simply stop requesting, and the pressure propagates upstream for free. A slow browser throttles the model, which is exactly what you want.

You break that chain the moment you decouple, and the decoupling is usually accidental. observeOn introduces a bounded queue whose default size is small; onBackpressureBuffer() with no argument introduces an unbounded one. Reach for the unbounded form to silence a MissingBackpressureException and you have not fixed the problem, you have converted a fast failure into a heap that grows until the JVM dies - and it will die under exactly the load you were trying to survive.

The correct move is to bound the buffer and decide, explicitly, what overflow means. Not all events are equally droppable: a text delta can be coalesced or dropped because the aggregate that follows carries the full content anyway, but a function call event, a state delta, or the terminal event must never be dropped, because losing one desynchronises the client permanently. That argues for a type-aware overflow policy rather than a blanket strategy.

runner.runAsync(userId, sessionId, message, runConfig)
    // hand off to a sender thread, but with a bounded queue and an
    // overflow policy that only ever discards coalescable deltas
    .onBackpressureBuffer(
        256,
        () -> metrics.counter("stream.overflow").increment(),
        BackpressureOverflowStrategy.DROP_OLDEST)
    .observeOn(Schedulers.io())
    .subscribe(
        event -> sink.send(toFrame(event)),   // may block on a slow client
        error -> sink.completeWithError(error),
        sink::complete);

Note the counter. Overflow is a signal that a consumer is losing a race, and it is the metric that explains user reports of truncated answers long before anyone thinks to look at the model.

Which thread runs your onNext

By default RxJava does no thread switching: your onNext runs on whatever thread emitted the event. For a model stream that is an HTTP client's IO thread. Doing a blocking network write from inside that callback means a slow client is occupying a thread from a pool sized for model calls, and once enough clients are slow, the pool is exhausted and new turns stop starting - a failure that looks like the model provider is down when it is entirely self-inflicted.

The fix is one operator: observeOn moves downstream work onto a chosen scheduler, so the emitting thread returns immediately. Note that subscribeOn is a different decision - it chooses where the subscription and therefore the work starts - and putting observeOn in the wrong position moves the wrong half of the pipeline. Position matters: everything after the operator runs on the new scheduler, everything before it does not.

Two Java-specific notes. Thread-locals do not follow the stream across an observeOn boundary, which is why MDC-based logging and any request-scoped context silently go empty halfway through a streamed turn unless you propagate them deliberately - capture what you need at subscribe time and carry it explicitly. And on a modern JVM, the blocking-write-per-connection model is far cheaper than it used to be, because a virtual thread parked on a socket write costs almost nothing; see virtual threads for Java agents for when that changes the sizing arithmetic.

Cancellation mid-stream, and the tool that is still running

Cancellation in Reactive Streams is a signal you send upstream by disposing the subscription. It is cooperative and asynchronous: dispose() returns immediately and guarantees only that you will receive no further events. It does not reach into a blocking JDBC call inside a tool and stop it. A tool implemented as a blocking HTTP request keeps that request in flight, keeps holding its connection, and keeps costing money until it finishes on its own schedule and discovers nobody is listening.

So make tools observe cancellation on their own terms - a client-level timeout, an interruptible wait, a cancellation token threaded through - and treat the deadline as the real control, not the dispose. Put teardown in doFinally, never in doOnComplete: only doFinally runs on all three terminations - complete, error, and cancel - and a leaked live session or an unreleased connection is precisely the thing that only leaks on the cancel path, which is the path nobody tests.

Then separate the two events people both call cancelling. A user pressing stop is an intentional interruption: seal the message at what they actually saw, persist the truncation, and stop paying for tokens. A closed tab is a transport failure that arrives with no notification at all - the JVM finds out when a write finally fails, which on a half-open TCP connection can be tens of seconds and a whole completion's worth of billed tokens later. Register a completion or timeout callback on the async context so disposal is driven by the container's disconnect detection rather than by your next write attempt.

The hardest case is neither: a tool that already committed a side effect. Cancelling the stream after a refund has posted does not un-post it. That is a compensation problem, not a cancellation problem, and it belongs in the tool's design rather than the stream's.

Errors after the first byte

Streaming destroys the usual error contract. A rate limit or a thrown tool arrives long after the response line went out, and the status code is already committed to 200. Every failure past the first byte therefore has to be carried inside the envelope, which means your frame format needs an error variant designed in from the start - retrofitting one means every deployed client treats the failure as a stream that simply stopped, and the interface sits in a permanent typing state.

In RxJava terms, onError is terminal and exclusive: it replaces completion, and no further onNext can follow it. If you want the client to receive an error frame, you must convert the error into a value before the stream ends - onErrorResumeNext that emits a synthetic error event and then completes normally. Otherwise the exception surfaces in your subscriber's error handler, the SSE connection closes without explanation, and the browser's automatic reconnect quietly starts the whole turn again.

Which is the trap worth stating plainly: retry operators re-subscribe, and re-subscribing a cold Flowable re-runs the turn. A retry(3) wrapped around runAsync after a tool has already executed does not resume from the failure point - it replays the run from the top, including the tool call, which is how a transient 503 turns into three charges on the same card. Retry belongs inside the model client and inside individual tools, where the unit of work is idempotent and small, not around the run.

Resumability: rebuild from the session, not from the socket

When a connection drops mid-turn you face two independent questions, and conflating them is the usual bug. Should the run continue? And what does the client see when it comes back?

A dropped socket is not a cancellation unless you decide it is. Letting the run finish and commit costs tokens for output nobody is watching, but it means a reconnecting client finds a complete answer already in the session. Disposing on disconnect saves the tokens and guarantees a partial conversation. Both are defensible; pick per workload - long analytical turns usually want to finish, chat usually does not - and make it explicit rather than an accident of where your disconnect detection happens to sit.

Recovery should not try to replay the socket. Partial events are transport-only and never reach the session service, so the persisted history is composed entirely of whole messages: reading it back gives you each message either complete or absent, with nothing to reconcile and no fragment to de-duplicate. Reconnecting is therefore an ordinary session read, not a stream-resume protocol - no acknowledgement bookkeeping, no per-event cursor, and it degrades to "the user re-reads the conversation" in the worst case. Durable sessions across process death are covered in ADK session resumption architecture; the transport-level alternative, where a client replays from a last-seen event ID, is covered for MCP in MCP streaming transport architecture.

One thing to design against: mid-turn reconnects arriving while the original run is still alive. Two live streams against one session double-write state deltas. Guard the session with a single-writer lease so the second attach observes rather than runs.

Buffering versus flushing, and where time-to-first-token goes

A streamed turn has exactly one latency number anyone perceives: how long the pane stays empty. On the JVM that number is usually spent after the event has already been produced, in four places that all sit outside the agent code.

Response compression is the most common. A gzip filter buffers until it has enough bytes to compress, which converts a stream into a batch that arrives all at once at the end; either exclude the streaming endpoint or ensure the encoder is flushed per frame. Reverse proxies are the second - a proxy that buffers responses by default holds your events until its own buffer fills, and the fix is a per-route setting or the explicit no-buffering header. The servlet container is the third: without an explicit flush after each write, output sits in the container's buffer. And the container's async timeout is the fourth - a default that is shorter than your longest turn kills streams that were working perfectly.

Once those are correct, there is a genuine tradeoff left. Flushing on every token delta minimises latency and maximises syscalls and frame overhead; coalescing deltas on a short timer - tens of milliseconds - cuts frame count substantially at a cost the eye cannot detect, since a human reads far slower than the model generates. Coalesce the deltas, but flush structural events immediately: a function call event delayed by a timer is a visible stall in the UI, because it is the event that changes what the interface is showing.

Measure time to first byte out of the process, not time to first event inside it. The two diverge by exactly the amount of buffering you have not found yet.

Testing a stream: assert on the sequence, not the final string

Asserting on the concatenated output of a streamed turn tests the model, which is nondeterministic, and misses every bug that actually occurs in streaming: the tool event that never fired, the aggregate that arrived before its partials, the terminal event that was dropped under load. Assert on the shape of the sequence instead.

RxJava gives you this directly. Calling .test() on the Flowable returns a TestSubscriber that records everything, and its assertions - value counts, terminal state, predicate matches on individual positions - are the right vocabulary. Map each event to a small token describing its kind, and the test becomes readable.

List<String> shape = runner.runAsync(userId, sessionId, msg, cfg)
    .map(StreamShape::classify)   // "partial" | "agg" | "call:x" | "resp:x" | "final"
    .distinctUntilChanged()
    .toList()
    .blockingGet();

assertThat(shape).containsSubsequence("call:lookup_order", "resp:lookup_order");
assertThat(shape).endsWith("final");
assertThat(shape.indexOf("agg")).isGreaterThan(shape.indexOf("partial"));

Three invariants are worth encoding as a reusable assertion because they hold for every agent you will write: every function call is followed by exactly one matching response, no event follows the terminal one, and no aggregate precedes the partials it aggregates. Add a cancellation test - dispose after N events and assert the tool's teardown hook ran - because that path has no natural coverage otherwise.

For anything time-dependent, drive it with a test scheduler rather than sleeping, so keepalive and coalescing-timer behaviour is deterministic. The production counterparts are Java-side signals the model dashboards will never show you: the backpressure overflow counter from the snippet above, the occupancy of every observeOn queue you introduced, the count of live subscriptions against the size of the pool that serves them, and dispose-to-teardown latency - the interval between disposing a subscription and the last doFinally completing, which is where leaked sessions and unreleased connections announce themselves. Routing-specific stream fields are a separate concern, handled in Agent Router Architecture in Depth.