Structured Output Architecture for LLMs in Depth

Why architecture matters here

Structured output architecture matters because LLM prose is unreliable to parse. A model that returns "Sure, the answer is yes" needs regex; another day it says "Certainly, yes it is." Structured output makes this deterministic: {"answer": true}. The class of bugs disappears.

Cost is modest and often net-negative. Structured output may cost a fraction more per call (function calling adds tokens for schema) but eliminates retries from parse failures, which is a bigger cost.

Reliability jumps. Retry-on-error patterns can push parse success rates from 95% (best-effort prose) to 99.9%+ (structured with retries). At scale this is the difference between "sometimes fails" and "reliable production."

The reliability spectrum - from asking nicely to guaranteeing syntax

Five different mechanisms get bundled under "structured output", and they do not make the same promise. Prose instruction ("reply with JSON and nothing else") is a request the model usually honours: often enough to matter, you get a markdown fence, a "Here is the JSON:" preamble, or a trailing paragraph of commentary. Few-shot exemplars raise adherence by showing the shape instead of describing it, and fix field naming and value formatting far better than they fix syntax. JSON mode moves the guarantee into the serving stack: the bytes will parse. Schema-attached tool calling also tells the model which keys exist, in a request shape it was trained on.

Only the fifth, constrained decoding, changes the kind of claim you can make. The first four are probabilities you improve by prompting harder. Constrained decoding makes an invalid document unrepresentable, because the tokens that would produce one are removed before the sampler draws. That is an invariant rather than a success rate - and a purely syntactic one, which is the source of most disappointment with it.

The architecture: every piece explained

Walk the diagram top to bottom.

Model + prompt. Standard LLM call with system + user messages. System prompt says the response format is required.

Schema. JSON Schema or a Pydantic model. Defines fields, types, required, and constraints (enum, min/max).

Provider Feature. OpenAI's function calling / structured outputs, Anthropic tool use with schema, Gemini function calling. Provider constrains the model to produce valid JSON per the schema.

Constrained Decoding. At token level, the sampler restricts to tokens that continue a valid JSON matching the schema. Guarantees valid output but subtly affects distribution.

Post-parse Validation. After receipt, parse and validate against the schema. Even with provider constraints, semantic constraints (business rules, cross-field) may fail. Retry with the error as feedback.

Instructor / Guardrails / Outlines. Libraries that abstract provider differences. Instructor uses Pydantic + retries. Guardrails supports validation and re-ask. Outlines does grammar-guided decoding.

Retry with error. If parse or validation fails, resend the prompt with the specific error and instructions to fix. Usually succeeds on retry.

Fallback. If retries exhaust, degrade gracefully — simpler schema, freeform response with wrapper, or human escalation.

Streaming Structured Output. Some providers stream partial JSON. Client can render as fields fill.

Observability. Parse success rate, retry rate, per-schema cost. Regression detection.

End-to-end structured output flow

Trace a call. You have a support-classification task. Schema: {ticket_id: str, category: enum, urgency: 1-5, needs_human: bool, summary: str}.

Using Instructor with OpenAI: define Pydantic model TicketClassification with those fields. Call client.chat.completions.create(response_model=TicketClassification, messages=[...]).

Instructor sends the schema as a function-call. Provider constrains decoding. Model returns valid JSON.

Instructor parses into TicketClassification. Pydantic validates types and constraints. All pass; return the object.

Retry scenario: model returns urgency=7 (outside range). Pydantic raises ValidationError. Instructor sends a re-ask with the specific error: "urgency must be between 1 and 5." Model corrects; second attempt validates.

Streaming scenario: use response_model with streaming. As tokens arrive, Instructor incrementally parses; user sees the ticket_id appear first, then category fills in, etc.

Observability: log parse_success_rate = 99.4% over the last hour; retry_rate = 3%. Alert if retry rate crosses 10% — probably a schema or prompt bug.

JSON mode is not a schema

JSON mode and strict schema enforcement get conflated, and the gap is where production bugs live. JSON mode guarantees one thing: the response parses. It says nothing about which keys appear, their types, whether a required field is present, or whether an enum value is one you defined. A model in JSON mode can legitimately return {"result": {"answer": "yes"}} where your parser expects {"answer": true}, or a bare array at the top level.

Strict schema mode compiles the schema into the constraint itself, so key set, types, enum membership and required-field presence are enforced at decode time. That changes what you must build downstream: JSON mode still needs full structural validation plus a repair path, a strict schema needs only semantic validation. Strict implementations support a restricted JSON Schema subset - unbounded recursion and open-ended pattern properties are commonly rejected. Prefer a stack that errors on an unsupported schema over one that silently falls back to unconstrained generation.

How constrained decoding works - a mask over the logits

At every decoding step the model emits a logit vector over the entire vocabulary, typically 32k to 250k entries. Constrained decoding sits between that vector and the sampler: it computes a boolean mask of tokens whose byte expansion could still extend into a valid document, sets every other logit to negative infinity, and lets sampling proceed. Temperature and top-p then apply to the surviving set, so the distribution is renormalised over legal continuations only.

The mask comes from an automaton. The schema or grammar is compiled once into a state machine: a regular constraint becomes a finite-state machine, while JSON with arbitrary nesting needs a stack, so implementations use a pushdown automaton or an FSM plus a bounded depth counter. Each state carries the set of tokens that may follow.

state = automaton.start()
for _ in range(max_tokens):
    logits = model.forward(prefix)           # [vocab_size]
    logits[~automaton.mask(state)] = -inf    # illegal tokens cannot be drawn
    tok = sample(softmax(logits / T))
    state = automaton.advance(state, tok)
    prefix.append(tok)

Note what this is not: no regex over finished text, no JSON repair pass, no retry - an invalid character is never generated in the first place. One useful side effect is that when a state admits exactly one continuation, such as the remaining bytes of a required key name or a closing brace, the decoder can emit those tokens without sampling and prefill them in a single batched pass, so a schema-heavy response can finish in fewer forward passes than the same text generated freely.

Tokenizer boundaries, and why the mask must be cached

Grammars are defined over characters; models emit tokens, and the two do not line up. One BPE token can expand to ", " plus the opening quote of the next key, straddling three grammar terminals. So "which token is allowed" is not "which character is allowed": you must ask, for every vocabulary entry, whether its byte expansion legally continues the current partial parse. Naively that is 100k+ parse attempts per generated token, which would dwarf the forward pass.

Layered caching avoids it. The schema compiles to an automaton once, keyed by a hash, ideally at deploy time rather than per request. Each state's vocabulary bitmask is then computed on first visit and interned, so hot states are reused across steps and across every later request with the same schema. Warm, mask overhead is a low single-digit percentage of decode time; cold, or with many regex-constrained string fields exploding the state count, it can rival the model itself at small batch sizes. Two sharp edges remain: each sequence in a batch sits in its own state, so masking is per-sequence work that must overlap the forward pass to stay cheap, and forcing a token boundary the tokenizer would not have chosen pushes the model off-distribution - which is why good implementations back up and re-tokenize the forced prefix rather than splicing tokens blindly.

What constrained decoding does not fix

The guarantee is shape, not truth. {"invoice_total": 4210.00} is schema-valid whether or not that number appears anywhere in the document you passed in. Field-level hallucination is untouched, and a schema-valid wrong answer is more dangerous than a malformed one precisely because it sails through your parser.

A schema can also manufacture hallucination. If a field is required and the model has no evidence for it, the grammar forbids skipping it, so it invents a value. A constraint with no legal way to say "not present" converts abstention into confident fabrication: use nullable types and an "unknown" member in every enum that can legitimately be unknown.

There is a quality cost when the grammar fights the model. Masking renormalises over a subset of the distribution, and if that subset excludes what the model wanted to say next - a key order it did not prefer, an answer before it has reasoned - you are sampling from a low-probability region. Symptoms are blander, shorter field values and measurably worse reasoning on tasks that benefit from thinking out loud. Give reasoning somewhere legal to live: decoding is left to right, so a reasoning string declared before the verdict fields lets the model think inside the schema, and a two-call split works when even that is too tight. Constrained decoding also does not prevent truncation - hitting the token limit mid-object yields an object that never closes, a distinct failure class from a validation error that needs its own budget headroom and its own alert.

Schema design that models handle well

Schemas are prompts. The serialized schema sits in the model's context, so every name and description is instruction, and the shape decides how much bookkeeping the model does while it writes.

Flat beats deeply nested - every level is another bracket to track and another chance for a field to land in the wrong object; two levels is a comfortable ceiling, and a genuinely tree-shaped result is usually better as several calls. Enums beat free strings for anything categorical: they collapse the allowed continuations to a handful of tokens and eliminate spelling and casing drift. Names should carry units (total_amount_usd_cents, not amount), arrays should carry maxItems so an unbounded list cannot become a 300-item confabulation, and evidence fields belong before verdict fields.

{ "category": {"type": "string", "enum": ["billing","outage","howto","other"]},
  "urgency":  {"type": "integer", "minimum": 1, "maximum": 5},
  "evidence": {"type": "string", "description": "quote from the ticket"},
  "account_id": {"type": ["string","null"],
                 "description": "null if not stated in the ticket"},
  "needs_human": {"type": "boolean"} }

For where the schema sits relative to the instructions and the input, see prompt delimiters; for versioning schemas and prompts as artifacts, see prompt template libraries.

Validation and repair loops as the fallback

You validate even under strict constraints, because the grammar cannot see your business rules: end_date >= start_date, an SKU that must exist in the catalogue, a total that must equal the sum of line items. None of that is expressible in a schema, so it is a post-parse check by construction.

Treat repair as a budget, not a loop. Each attempt is a full forward pass over the whole prompt plus the whole output, so one repair roughly doubles that request's cost and latency; cap at one or two and then degrade, because a third attempt seldom converges when the first two failed the same way. Keep the error message narrow - failing field path, value received, rule broken. Dumping the schema or a stack trace invites the model to rewrite fields that were already correct, while feeding the invalid object back as the assistant turn makes it edit rather than regenerate. Most importantly, a sustained repair rate above a few percent is a schema smell rather than a model failure: an ambiguous field, a rule the schema cannot satisfy, no way to say "unknown". Fix the schema; do not tune the retry count.

Streaming partial structured output

Partial JSON is not JSON. The prefix {"title": "Ada", "sco fails every strict parser, so streaming needs a tolerant incremental parser that speculatively closes open strings and objects and yields a best-effort snapshot per chunk. That snapshot is not safe to use blindly: a value whose closing quote has not arrived can still change meaning with the next token, so an enum rendered early can flash the wrong label.

Two rules keep it sane. Render a field only once its value is closed, and never let a partial object trigger a side effect - writes, tool calls and commits wait for the object to close and validate. Field order becomes a UI decision, since decoding is sequential: short user-visible fields first, long free text last. Streaming also composes badly with repair, because you may have already shown a field the repaired object changes.

Evaluation - schema validity is not field accuracy

Two orthogonal metrics, and teams routinely ship the wrong one. Schema-validity rate - the fraction of responses that parse and validate - is roughly 100 percent by construction under constrained decoding, which makes it useless as a quality signal and excellent as a regression alarm: it should only move on truncation or on a config change that quietly turned constraints off.

Field-level accuracy against a labelled set is the real measure, reported per field rather than averaged over the object, because object-level exact match tells you only that something broke. Alongside it, track abstention quality on optional fields (did it emit null when the value was genuinely absent?), a hallucination rate for extraction fields, and a confusion matrix per enum, where a badly named category shows up immediately. Add truncation rate, repair rate, tokens per object and the p99 latency delta against unconstrained decoding, then gate schema and prompt changes on a nightly run over a fixed input set. For the grounding checks that sit alongside this, see hallucination guardrails.