Why architecture matters here
Batch ASR is easy; streaming is hard. Batch has the whole utterance for context; streaming decides moment-to-moment. Poor endpointing cuts users off mid-sentence or holds them waiting. Unstable partials flicker letters in and out. High WER at low latency means agents misunderstand.
The architecture matters because tuning is joint: encoder chunk size affects latency AND accuracy; endpointing threshold affects both cut-off and hold time; rescore adds accuracy at the cost of some latency. Each pair trades off.
With the pipeline mapped, you can tune to your product's latency and accuracy SLOs.
The architecture: every piece explained
The top strip is the ingest and encode. Audio stream arrives at 16 kHz (typical). VAD filters silence and detects speech onset. Feature front-end produces mel-spectrograms or the equivalent for a conformer input. Streaming encoder uses chunked attention so it can emit outputs incrementally with limited lookahead.
The middle row is the decoder path. CTC / transducer decoders produce word / subword outputs monotonically over frames. Partial results are emitted continuously with confidence and word timings. Endpointing decides when the user has finished — silence duration plus intent signals like syntactic completeness. Rescore / LM fusion uses an external language model to correct partials and finalize the transcript with better lexical accuracy.
The lower rows are ecosystem pieces. Diarization + PII attaches speaker labels and scrubs sensitive content. Integration sends transcripts to chat, IVR, or caption UIs with the appropriate contract. Observability tracks word error rate (WER), latency (TTFT and end-of-word), endpoint precision, and partial churn (how much partials change before finalization).
The latency-accuracy tradeoff, stated precisely
Streaming is not a feature you switch on; it is a constraint the model has to be trained under. A streaming recogniser commits to a token for frame t having seen audio only up to t + R, where R is the right-context (lookahead) budget. Every millisecond in R is a millisecond the caller waits, and every millisecond you remove from it costs accuracy, because the evidence that disambiguates a word routinely arrives after the word's acoustic centre. "Recognise speech" and "wreck a nice beach" are separated by what comes next.
Three latencies get conflated and should be measured separately:
- Algorithmic latency - the model's structural lookahead: chunk size plus any per-layer future context. A property of the architecture, identical on every machine.
- Computational latency - wall-clock time to run encoder and decoder over a chunk. Under load this is queueing delay, not FLOPs.
- Emission delay - how long after a word was spoken the model chooses to emit its token, despite already having the evidence. This one is learned behaviour, and it is the term most teams never measure.
Perceived latency is the sum of all three plus transport and endpointing. A system with 320 ms of lookahead, 60 ms of compute, 200 ms of emission delay and a 700 ms silence timeout does not feel like a 320 ms system; it feels like a 1.3 second one. Budget the whole chain or you will optimise the wrong term.
Chunked inference vs fully streaming encoders
A vanilla Transformer or Conformer encoder is non-causal: self-attention at every layer sees the whole utterance and depthwise convolutions look symmetrically forwards and backwards. Both have to be constrained before the model can run online, and there are two families of constraint.
Block processing (chunked attention)
Split the feature sequence into fixed blocks - 160 to 640 ms after subsampling is the usual range - and mask attention so each frame sees its own block, a bounded number of previous blocks, and nothing after. Algorithmic latency is then about one chunk. Chunking stays popular because full bidirectional attention survives inside the chunk, which recovers most of the offline model's accuracy. The cost is a sawtooth latency profile: the first frame of a block waits a whole chunk, the last waits almost nothing. Quote p95, not the mean.
Frame-synchronous (limited lookahead)
Give every frame a sliding window of left context and a small fixed right context. Latency becomes uniform instead of sawtooth, but per-layer lookahead accumulates: seventeen layers each allowed one future 40 ms frame give you 680 ms of effective lookahead, not 40 ms. This is the most common streaming-latency bug in the field. Chunked masking avoids it because all layers share the same boundary.
State caching is what makes it real time
Neither scheme streams if you re-run the encoder over the whole utterance on every chunk. That is quadratic work; it keeps up for ten seconds and then falls behind permanently. A real streaming encoder carries per-connection state: the attention key/value cache for the left-context window, a ring buffer for each depthwise convolution, and the subsampling front-end's overlap frames. Getting the convolution cache wrong is the classic silent failure - the model runs, the transcript is readable, and word error rate is two points worse than the offline evaluation because every chunk boundary is fed zeros instead of the previous chunk's tail.
Streaming decoder families - and why transducers win
The encoder decides what you may look at; the decoder decides when you may speak. Three families can be made to run online, and they are not equally good at it.
CTC
CTC scores each frame independently given the encoder output, with a blank symbol absorbing frames that emit nothing. Greedy CTC decoding is an argmax and a dedup, so it is trivially streamable and the cheapest option per frame. Its weakness is the conditional independence assumption: no internal language model, so it produces phonetically plausible nonsense on rare words and needs external LM fusion to compete. Its spiky frame-level emission does make word timings easy to extract, which is a genuine advantage for captioning and for generating the alignments other models train against.
Neural transducer (RNN-T and successors)
The transducer adds a prediction network over tokens emitted so far and a small joint network combining it with the current encoder frame, defining a monotonic alignment lattice over (frame, token) pairs. That structure is the streaming problem's native shape: alignment never goes backwards, frames are consumed one at a time, and the model may emit zero, one, or several tokens per frame. It gets a language model for free from the predictor, needs no attention over the past, and its per-session state is small. That is why transducers are the default in production streaming stacks.
Attention encoder-decoder with monotonic alignment
Plain attention-based encoder-decoders are structurally offline: cross attention is defined over the complete encoder output, so the model is free to consult the end of the utterance while emitting the first word. Monotonic variants - monotonic chunkwise attention, monotonic multihead attention - constrain each head to a forward-only pointer with local attention around it, and those do stream. They are harder to train stably and rarer in production. The pragmatic alternative is to leave the offline model offline and use it as a second pass that rescores the streaming first pass once the utterance closes.
Emission delay and the right-context budget
A transducer trained with plain sequence loss learns to cheat. The loss cares that the token sequence is right, not when each token appeared, so the model discovers that delaying emission by a few frames buys more acoustic evidence and lowers training loss. The result is a model with 200 ms of algorithmic latency and 500 ms of observed word emission latency, and no configuration knob that explains the gap.
Two mitigations are standard. A delay penalty added to the transducer loss (the FastEmit family) reweights the lattice toward earlier-emitting paths, trading a small WER regression for a large latency win. Alignment restriction computes a reference alignment with a cheap CTC or HMM model and then masks the transducer lattice to a band around it, forbidding the model from emitting more than a few frames late. Alignment restriction also cuts training memory sharply, since the joint-network tensor dominates and the band prunes most of it.
Measure emission delay rather than inferring it: take force-aligned reference word end times, subtract them from the timestamp at which each token first appeared in a partial, and report the distribution. The median tells you how the system feels; p90 tells you why a caller occasionally thinks it has hung. Report the first word of an utterance separately - it is usually the worst, because the predictor has no context yet.
Partials, finals, and hypothesis instability
A streaming recogniser emits two kinds of result. Partials are the current best hypothesis and are revisable at any time. Finals are committed and never change. Most integration bugs come from treating a partial as a final.
Partials are unstable for a structural reason: beam search keeps several hypotheses alive and the one currently on top can lose. The tail of the transcript flickers while the head settles. Quantify it as churn - tokens retracted or rewritten per second, or edit distance between successive partials normalised by length. Single-digit percentages are normal; 30 percent means the beam is too wide, the LM fusion weight is too high, or the model is running with less lookahead than it was trained for.
- Hold back the unstable tail. Emit only the prefix common to every hypothesis in the beam, or suppress the last N tokens. Costs latency, buys calm.
- Stability by agreement. Freeze a prefix once it has survived K consecutive partials unchanged, and flag it as stable in the wire format so clients can render it differently.
- Render, do not retype. In captions, show the stable prefix solid and the volatile tail greyed. Users tolerate a visibly provisional tail far better than text that rewrites itself.
- Never trigger side effects on a partial. A tool call or database lookup fired from a partial will eventually fire on a hypothesis that gets retracted. See the streaming agent contract for how partial and final events reach application code.