Streaming ASR architecture — VAD, encoder, endpointing, partials, rescore

Why architecture matters here

Batch ASR is easy; streaming is hard. Batch has the whole utterance for context; streaming decides moment-to-moment. Poor endpointing cuts users off mid-sentence or holds them waiting. Unstable partials flicker letters in and out. High WER at low latency means agents misunderstand.

The architecture matters because tuning is joint: encoder chunk size affects latency AND accuracy; endpointing threshold affects both cut-off and hold time; rescore adds accuracy at the cost of some latency. Each pair trades off.

With the pipeline mapped, you can tune to your product's latency and accuracy SLOs.

The architecture: every piece explained

The top strip is the ingest and encode. Audio stream arrives at 16 kHz (typical). VAD filters silence and detects speech onset. Feature front-end produces mel-spectrograms or the equivalent for a conformer input. Streaming encoder uses chunked attention so it can emit outputs incrementally with limited lookahead.

The middle row is the decoder path. CTC / transducer decoders produce word / subword outputs monotonically over frames. Partial results are emitted continuously with confidence and word timings. Endpointing decides when the user has finished — silence duration plus intent signals like syntactic completeness. Rescore / LM fusion uses an external language model to correct partials and finalize the transcript with better lexical accuracy.

The lower rows are ecosystem pieces. Diarization + PII attaches speaker labels and scrubs sensitive content. Integration sends transcripts to chat, IVR, or caption UIs with the appropriate contract. Observability tracks word error rate (WER), latency (TTFT and end-of-word), endpoint precision, and partial churn (how much partials change before finalization).

The latency-accuracy tradeoff, stated precisely

Streaming is not a feature you switch on; it is a constraint the model has to be trained under. A streaming recogniser commits to a token for frame t having seen audio only up to t + R, where R is the right-context (lookahead) budget. Every millisecond in R is a millisecond the caller waits, and every millisecond you remove from it costs accuracy, because the evidence that disambiguates a word routinely arrives after the word's acoustic centre. "Recognise speech" and "wreck a nice beach" are separated by what comes next.

Three latencies get conflated and should be measured separately:

• Algorithmic latency - the model's structural lookahead: chunk size plus any per-layer future context. A property of the architecture, identical on every machine.
• Computational latency - wall-clock time to run encoder and decoder over a chunk. Under load this is queueing delay, not FLOPs.
• Emission delay - how long after a word was spoken the model chooses to emit its token, despite already having the evidence. This one is learned behaviour, and it is the term most teams never measure.

Perceived latency is the sum of all three plus transport and endpointing. A system with 320 ms of lookahead, 60 ms of compute, 200 ms of emission delay and a 700 ms silence timeout does not feel like a 320 ms system; it feels like a 1.3 second one. Budget the whole chain or you will optimise the wrong term.

Chunked inference vs fully streaming encoders

A vanilla Transformer or Conformer encoder is non-causal: self-attention at every layer sees the whole utterance and depthwise convolutions look symmetrically forwards and backwards. Both have to be constrained before the model can run online, and there are two families of constraint.

Block processing (chunked attention)

Split the feature sequence into fixed blocks - 160 to 640 ms after subsampling is the usual range - and mask attention so each frame sees its own block, a bounded number of previous blocks, and nothing after. Algorithmic latency is then about one chunk. Chunking stays popular because full bidirectional attention survives inside the chunk, which recovers most of the offline model's accuracy. The cost is a sawtooth latency profile: the first frame of a block waits a whole chunk, the last waits almost nothing. Quote p95, not the mean.

Frame-synchronous (limited lookahead)

Give every frame a sliding window of left context and a small fixed right context. Latency becomes uniform instead of sawtooth, but per-layer lookahead accumulates: seventeen layers each allowed one future 40 ms frame give you 680 ms of effective lookahead, not 40 ms. This is the most common streaming-latency bug in the field. Chunked masking avoids it because all layers share the same boundary.

State caching is what makes it real time

Neither scheme streams if you re-run the encoder over the whole utterance on every chunk. That is quadratic work; it keeps up for ten seconds and then falls behind permanently. A real streaming encoder carries per-connection state: the attention key/value cache for the left-context window, a ring buffer for each depthwise convolution, and the subsampling front-end's overlap frames. Getting the convolution cache wrong is the classic silent failure - the model runs, the transcript is readable, and word error rate is two points worse than the offline evaluation because every chunk boundary is fed zeros instead of the previous chunk's tail.

Streaming decoder families - and why transducers win

The encoder decides what you may look at; the decoder decides when you may speak. Three families can be made to run online, and they are not equally good at it.

CTC

CTC scores each frame independently given the encoder output, with a blank symbol absorbing frames that emit nothing. Greedy CTC decoding is an argmax and a dedup, so it is trivially streamable and the cheapest option per frame. Its weakness is the conditional independence assumption: no internal language model, so it produces phonetically plausible nonsense on rare words and needs external LM fusion to compete. Its spiky frame-level emission does make word timings easy to extract, which is a genuine advantage for captioning and for generating the alignments other models train against.

Neural transducer (RNN-T and successors)

The transducer adds a prediction network over tokens emitted so far and a small joint network combining it with the current encoder frame, defining a monotonic alignment lattice over (frame, token) pairs. That structure is the streaming problem's native shape: alignment never goes backwards, frames are consumed one at a time, and the model may emit zero, one, or several tokens per frame. It gets a language model for free from the predictor, needs no attention over the past, and its per-session state is small. That is why transducers are the default in production streaming stacks.

Attention encoder-decoder with monotonic alignment

Plain attention-based encoder-decoders are structurally offline: cross attention is defined over the complete encoder output, so the model is free to consult the end of the utterance while emitting the first word. Monotonic variants - monotonic chunkwise attention, monotonic multihead attention - constrain each head to a forward-only pointer with local attention around it, and those do stream. They are harder to train stably and rarer in production. The pragmatic alternative is to leave the offline model offline and use it as a second pass that rescores the streaming first pass once the utterance closes.

Emission delay and the right-context budget

A transducer trained with plain sequence loss learns to cheat. The loss cares that the token sequence is right, not when each token appeared, so the model discovers that delaying emission by a few frames buys more acoustic evidence and lowers training loss. The result is a model with 200 ms of algorithmic latency and 500 ms of observed word emission latency, and no configuration knob that explains the gap.

Two mitigations are standard. A delay penalty added to the transducer loss (the FastEmit family) reweights the lattice toward earlier-emitting paths, trading a small WER regression for a large latency win. Alignment restriction computes a reference alignment with a cheap CTC or HMM model and then masks the transducer lattice to a band around it, forbidding the model from emitting more than a few frames late. Alignment restriction also cuts training memory sharply, since the joint-network tensor dominates and the band prunes most of it.

Measure emission delay rather than inferring it: take force-aligned reference word end times, subtract them from the timestamp at which each token first appeared in a partial, and report the distribution. The median tells you how the system feels; p90 tells you why a caller occasionally thinks it has hung. Report the first word of an utterance separately - it is usually the worst, because the predictor has no context yet.

Partials, finals, and hypothesis instability

A streaming recogniser emits two kinds of result. Partials are the current best hypothesis and are revisable at any time. Finals are committed and never change. Most integration bugs come from treating a partial as a final.

Partials are unstable for a structural reason: beam search keeps several hypotheses alive and the one currently on top can lose. The tail of the transcript flickers while the head settles. Quantify it as churn - tokens retracted or rewritten per second, or edit distance between successive partials normalised by length. Single-digit percentages are normal; 30 percent means the beam is too wide, the LM fusion weight is too high, or the model is running with less lookahead than it was trained for.

• Hold back the unstable tail. Emit only the prefix common to every hypothesis in the beam, or suppress the last N tokens. Costs latency, buys calm.
• Stability by agreement. Freeze a prefix once it has survived K consecutive partials unchanged, and flag it as stable in the wire format so clients can render it differently.
• Render, do not retype. In captions, show the stable prefix solid and the volatile tail greyed. Users tolerate a visibly provisional tail far better than text that rewrites itself.
• Never trigger side effects on a partial. A tool call or database lookup fired from a partial will eventually fire on a hypothesis that gets retracted. See the streaming agent contract for how partial and final events reach application code.

Endpointing is a decoder decision, not only a VAD decision

Endpointing answers "has the user finished?". Acoustic voice activity detection answers "is there speech energy right now?". Conflating the two is why so many voice agents interrupt people. The VAD side - framing, neural frame scoring, hysteresis, hangover, pre-roll, semantic turn detection - is covered in depth in Voice Activity Detection architecture. What follows is only the recogniser's contribution.

The recogniser knows things an energy detector cannot. A transducer emitting a long uninterrupted run of blanks is stronger evidence of end-of-speech than low frame energy, because it accounts for what was said, not merely that sound stopped. Many production systems train an explicit end-of-query token into the transducer vocabulary so the acoustic model itself proposes the endpoint, then fuse that proposal with VAD state and a timeout ladder.

Treat endpointing as a controller with tiered timeouts rather than one threshold: a short window (roughly 300 to 500 ms) when the hypothesis looks syntactically complete and the model has proposed end-of-query; a longer one (800 ms to 1.5 s) mid-phrase, after a filler, or after a determiner with no noun; and a hard maximum-utterance timer so a stuck stream cannot hold a session open forever. Let the error asymmetry drive tuning. A late endpoint costs the user a pause; an early endpoint truncates the utterance and destroys words that nothing downstream can recover. Tune against truncation, not against mean latency.

Beam search and biasing under streaming constraints

The beam

Offline beam search can expand the lattice and choose at the end. A streaming beam has to produce a usable answer at every chunk boundary, in bounded time and memory, for hundreds of concurrent sessions. Four constraints follow. The beam advances chunk by chunk and cannot backtrack past a committed prefix. A transducer may emit several tokens on one frame, so a max_symbols_per_frame guard is mandatory - without it a badly conditioned model loops on a single frame and the stream stalls. Prefixes that converge to the same token sequence must be merged and their probabilities summed, or duplicates crowd the beam and the effective width collapses. And beam width is a latency knob as much as an accuracy knob, since each extra hypothesis is another prediction-network step per frame; 4 to 8 is typical for streaming against 16 or more offline, and the accuracy return above 8 is small.

If you add an external language model by shallow fusion, remember the transducer already carries an internal LM in its prediction network. Naive addition double-counts the language prior and biases toward frequent words; internal language model estimation subtracts the predictor's own prior before adding the external one. It matters most on domain vocabulary - exactly where you added the LM to help.

Hotwords and contextual biasing

Every deployment has words the model has barely seen: a contact list, a product catalogue, drug names, the twelve options this IVR accepts. Shallow-fusion biasing compiles the phrase list into a subword prefix trie or weighted transducer and adds a bonus to hypotheses advancing along an arc, with a matching penalty when a partially matched phrase is abandoned so half-matches do not keep a free boost. It needs no retraining and can be swapped per request. The failure mode is over-biasing: push the weight up and the recogniser starts hearing the phrase list everywhere. Keep weights modest, apply the bonus only once enough of the phrase has matched to be discriminative, and cap the list - quality degrades somewhere in the hundreds to low thousands of entries. Neural biasing, where phrase embeddings are attended over by the model itself, scales further and misfires less, at the cost of training the capability in rather than configuring it on.

Running the stream: transport, buffering, GPU batching

Chunk assembly and backpressure

Audio arrives in 20 ms packets over WebRTC, or in whatever framing the client chose over a WebSocket or gRPC bidirectional stream; the model wants 160 to 640 ms chunks. The server therefore keeps an inference buffer on top of the network jitter buffer, and the two should not be confused. Smoothing arrival variance is the jitter buffer's job, covered in jitter buffer architecture and the wider real-time audio pipeline; choosing the wire itself is covered in audio streaming protocols compared. The inference buffer's only job is to fill exactly one model chunk and fire.

Two behaviours must be decided explicitly. On a gap, do you pad with silence to keep the model's clock honest, or wait and let real time drift away from audio time? Padding is usually right, because emission timestamps and endpoint timers are derived from audio time. And when a client sends faster than real time - a batch uploader replaying a file down a streaming socket - it will happily saturate a GPU meant for live callers, so streaming endpoints need a per-connection rate check and real backpressure, not a bigger queue.

stream:
  sample_rate_hz: 16000
  packet_ms: 20            # what the client sends
  chunk_ms: 320            # what the encoder consumes
  left_context_chunks: 4   # cached KV, bounded per session
  on_gap: pad_silence      # keep audio-clock == wall-clock
  max_realtime_factor: 1.5 # reject file-dump clients

decode:
  beam: 5
  max_symbols_per_frame: 3
  partial_interval_ms: 320
  stable_after_n_partials: 2

endpoint:
  complete_hypothesis_ms: 400
  midphrase_ms: 1200
  max_utterance_s: 30

Batching many streams on one GPU

One session's chunk is a tiny GPU workload and running it alone wastes the device. Throughput comes from batching the current chunk of many independent sessions into a single forward pass, which is harder than offline batching because each session carries its own encoder cache, predictor state, and beam. The batch is a gather over per-session state rather than a contiguous tensor, and its membership changes every chunk as sessions join and leave. The practical scheduler collects arriving chunks for a few milliseconds - far below the chunk period - forms a batch, gathers state, runs one step, then scatters state back.

Capacity planning therefore follows from per-session state size, not model size: weights are shared across the batch, but every concurrent session pays for its own attention cache and convolution buffers, and that footprint times the concurrency target is what decides how many streams fit on a card. Put admission control at session start, where you can still refuse cleanly - a stream that is accepted and then starved produces a broken transcript instead of an error.

When not to stream

Streaming buys interactivity and costs accuracy, complexity, and money. If nobody is waiting on the transcript, do not pay for it. Post-call analytics, media archives, podcast indexing, and compliance transcription are offline jobs where a full-context encoder-decoder sees the whole utterance and wins outright on word error rate; see ASR architecture for the offline pipeline shape.

Beware the middle option that looks free: calling an offline model repeatedly on a growing window, re-decoding the last few seconds each time. Partials do appear, so it looks like streaming, but cost per second of audio grows with window length, word timings jump, and the partials are unstable in a way no stability heuristic fixes because each pass is an independent decode. Fine as a prototype, poor as a product.

What large deployments converge on is two-pass: a streaming transducer produces low-latency partials and a provisional final, then once the endpoint fires a stronger non-causal model - or a neural LM rescoring the first pass's lattice - produces a corrected final within a couple of hundred milliseconds the user never perceives, because the turn is already over. Streaming latency with near-offline accuracy, at the price of running two models. Whether that trade is worth it depends entirely on whether anything downstream reads the corrected transcript.

End-to-end flow

End-to-end: a user says "what time is my flight tomorrow?" Audio streams in; VAD marks speech onset at 120 ms. Encoder emits chunked outputs; decoder produces partial "what", then "what time", then "what time is my flight tomorrow". Endpointing detects 400 ms of silence + syntactic completeness and closes the utterance. Rescore applies a small LM and the final transcript is confirmed. Total latency to final: 950 ms. Metrics: WER 4.2%, partial churn 6%, endpoint precision 0.97. Integration sends transcript to the agent runtime. If a caller trails off, endpointing waits longer than the base threshold because syntactic completeness is not reached.