LLM Structured Output: Get Reliable JSON from Any Language Model

Why Structured Output Matters

Every production application that uses a language model eventually hits the same problem: the model returns text, but the application needs data. You need a JSON object with specific fields. You need an array of items with typed properties. You need a classification label from a fixed set of options. You need a boolean decision with a confidence score. The model can generate all of these as text, but text is not data. Text that looks like JSON might have a missing comma, an extra quote, a field spelled differently than your code expects, or a value of the wrong type. In a demo, you fix these manually. In production, they crash your pipeline at 3 AM.

Before structured output existed, developers used three workarounds, all of them fragile. The first was prompt engineering: you told the model "respond in JSON with these exact fields" and hoped for the best. This works about 85-95% of the time depending on the model, which means it fails hundreds or thousands of times per day in any system handling real traffic. The second was post-processing: you wrote regex parsers, JSON repair libraries, or retry logic that attempted to extract valid data from whatever the model returned. This adds latency, complexity, and a whole class of bugs that have nothing to do with your actual business logic. The third was function calling, where you defined tools and the model returned structured parameters for those tools. This works well for tool invocation but is semantically awkward when you just want the model to return structured data rather than call a function.

Structured output eliminates all three workarounds. You define a JSON schema that describes exactly what you want, and the model's token generation is constrained so that every token it produces is guaranteed to result in a response that conforms to your schema. Not "usually conforms" or "conforms if prompted correctly," but mathematically guaranteed to conform. OpenAI's structured outputs achieve 100% schema compliance in their evaluations. Anthropic's implementation uses grammar-based constrained decoding to achieve the same guarantee. This is not a prompt trick, it is a change in how the model generates tokens.

The impact on production reliability is enormous. Teams that previously spent 20-30% of their LLM integration code on parsing, validation, and retry logic can remove nearly all of it. Error rates from schema violations drop to zero. The model's output becomes a typed data structure that your application can trust, the same way it trusts the response from a well-designed API. This is what makes LLMs viable as components in data pipelines, not just chat interfaces, and it connects directly to how AI memory systems store and retrieve structured knowledge.

How Structured Output Works

The mechanism behind structured output is called constrained decoding, sometimes called grammar-guided generation or schema-constrained generation. To understand it, you need to understand how language models generate text. At each step, the model produces a probability distribution over its entire vocabulary, every possible next token. Normally, the model samples from this distribution (with temperature, top-p, or other sampling parameters), and any token can be selected. With constrained decoding, the system masks out any token that would make the output violate the schema, before sampling occurs. The model can only choose from tokens that keep the output on a valid path through the schema.

Consider a simple schema: an object with a field "sentiment" that must be one of "positive", "negative", or "neutral". When the model reaches the point where it needs to generate the value for the "sentiment" field, the system masks out every token except those that begin "positive", "negative", or "neutral". The model cannot choose "somewhat positive" or "mixed" or any other value, because those tokens are masked. This is enforced at the infrastructure level, not through prompting, which is why the guarantee is absolute rather than probabilistic.

The underlying technology varies by provider. OpenAI uses what they describe as a Context-Free Grammar (CFG) engine that compiles the JSON schema into a finite state machine and uses that machine to mask invalid tokens at each generation step. Anthropic compiles the JSON schema into a grammar and restricts token generation during inference. Google's Gemini uses a similar constrained decoding approach. The details differ, but the principle is the same: the schema is compiled into a set of constraints that are applied during token generation, not after.

There are limits to what schemas can express. Most providers support JSON Schema draft 2020-12 with restrictions: a maximum nesting depth (typically 5 levels), all properties must be listed in the "required" array when using strict mode, recursive schemas may not be supported or may be limited in depth, and some schema features like patternProperties or conditional schemas may not be available. These limitations exist because the constrained decoding engine needs to compile the schema into a finite representation that can be evaluated at each token generation step. Despite these limits, the supported schema features cover the vast majority of production use cases. The page on designing schemas for structured output covers how to work within these constraints effectively.

JSON Mode vs Schema Enforcement

There is a critical distinction between JSON mode and structured output that trips up many developers. JSON mode, introduced by OpenAI in late 2023, guarantees that the model's response is valid JSON syntax, meaning it can be parsed by a JSON parser without errors. It does not guarantee that the JSON has any particular structure. The model might return {"answer": "yes"} when you expected {"sentiment": "positive", "confidence": 0.95}. Valid JSON, wrong structure.

Structured output with a JSON schema goes further: it guarantees not just valid JSON syntax but conformance to a specific schema. The response will have exactly the fields you specified, with exactly the types you specified, with values constrained to the enums you specified. This is the difference between "the response is parseable" and "the response matches the contract." JSON mode is the legacy approach, and structured output with a schema is the production standard in 2026.

The practical advice is straightforward: if you are building anything that will run in production, use schema-enforced structured output, not JSON mode. JSON mode is only appropriate for exploratory work where you want flexible JSON output and do not need a specific structure. For everything else, define a schema and let the constrained decoding engine enforce it. The detailed comparison in the page on JSON mode vs structured output covers the edge cases and migration path from one to the other.

Provider Support in 2026

All three major LLM providers now support schema-enforced structured output, though the implementations differ in syntax, capabilities, and maturity.

OpenAI was the first to ship structured output in August 2024 and has the most mature implementation. You set response_format to {"type": "json_schema", "json_schema": {...}} with "strict": true. Their CFG engine achieves 100% schema compliance. Schema support includes objects, arrays, enums, nested types up to 5 levels deep, and anyOf for union types. All properties must be in the required array when strict mode is enabled, and optional fields are expressed using anyOf with a null type. The implementation works across GPT-4o, GPT-4o-mini, GPT-5, and GPT-5.2.

Anthropic launched structured output in beta in November 2025 and moved to general availability in 2026. The parameter is output_config.format (previously output_format in the beta), and it supports both JSON schema enforcement for direct responses and strict tool use with "strict": true on tool definitions. Supported models include Claude Opus 4.6, Sonnet 4.6, Sonnet 5, and the full Opus/Sonnet/Haiku 4.5 family. The implementation uses grammar-based constrained decoding, achieving the same mathematical guarantee as OpenAI's approach. Anthropic also supports structured output through the Agent SDK.

Google Gemini supports structured output through the response_schema parameter in the generation config. Their implementation has been available since 2024 and supports enum constraints, nested objects, and arrays. Gemini's structured output works across their model family and is also available through Vertex AI for enterprise deployments.

Beyond the major providers, structured output is available through OpenRouter, Azure OpenAI, Amazon Bedrock (for both Anthropic and other models), and several open-source inference engines like vLLM and TensorRT-LLM that implement their own constrained decoding. The page on structured output across providers provides a detailed feature comparison with code examples for each.

Validation and Error Handling

Even with schema enforcement guaranteeing structural validity, you still need application-level validation. The schema guarantees that the field "price" is a number, but it does not guarantee that the number is positive or within a reasonable range. The schema guarantees that "email" is a string, but it does not validate that the string is a properly formatted email address. Schema enforcement eliminates structural errors but not semantic errors, and production systems need both layers.

The standard approach in 2026 is to pair schema-enforced structured output with a validation library. In Python, Pydantic is the dominant choice: you define a Pydantic model with field validators, use that model's JSON schema as the structured output schema, and then validate the model's response through Pydantic after receiving it. This gives you both the structural guarantee from the LLM provider and the semantic validation from Pydantic's field validators, custom validators, and model validators. In TypeScript, Zod serves the same role, defining schemas that both constrain the LLM output and validate the result. The page on Pydantic and Zod for LLM output validation covers both ecosystems in detail.

The Instructor library sits on top of this stack and adds automatic retry logic. When a Pydantic validation fails, Instructor automatically feeds the validation error back to the model and asks it to try again, typically fixing the issue in one retry. This is invaluable for semantic validation failures that the schema cannot catch. A model that gets it right 95% of the time will fail thousands of times per day in a high-volume system, and automatic retry turns those failures into invisible latency bumps instead of broken records. The page on the Instructor library covers the full feature set and when to use it versus raw structured output.

When validation does fail, even after retries, your system needs a fallback. Common strategies include logging the failure for human review, using a default value, falling back to a more capable model, or returning the raw text for manual processing. The page on handling validation errors covers these strategies with production-tested patterns.

Production Patterns

Structured output unlocks several production patterns that are impractical without it. Data extraction from unstructured text becomes reliable: you can point a model at a contract, a resume, a medical record, or an invoice and get back a typed data structure with specific fields, every time. Classification becomes deterministic: the model must choose from your predefined categories, not invent new ones. Multi-step pipelines become composable: the output of one model call, guaranteed to match a schema, becomes the input to the next processing step without any parsing uncertainty.

The most common production patterns are extraction pipelines, where the model extracts structured data from documents; classification systems, where the model categorizes inputs into predefined buckets; decision engines, where the model returns a structured decision with reasoning; and quality evaluation, where the model scores or grades content according to a rubric. Each of these patterns benefits from structured output because the downstream processing code can trust the shape of the data it receives.

A pattern that connects structured output to retrieval is structured extraction in RAG pipelines. When a RAG system retrieves documents, the model needs to extract specific facts from those documents and present them in a consistent format. Without structured output, the model might present the same fact differently across different retrieval contexts, making aggregation and comparison difficult. With structured output, every extracted fact follows the same schema, making it possible to merge, deduplicate, and compare facts across documents programmatically. The page on structured output in RAG pipelines covers this integration pattern.

Cost and Performance Impact

A common question is whether structured output costs more than free-form generation. The answer has two parts. First, the schema itself adds tokens to the request, typically 100-500 tokens depending on schema complexity, because the schema definition is included in the system prompt or request parameters. This is a small overhead that is usually negligible compared to the input tokens from the actual content. Second, constrained decoding can slightly reduce generation speed because the masking step adds computation at each token. In practice, this slowdown is minimal, typically under 10%, and is offset by the elimination of retry logic that you would otherwise need.

The net cost impact is usually positive: you save money by eliminating retries, reducing parsing failures that require human intervention, and removing the defensive code that otherwise inflates development and maintenance costs. For high-volume systems processing thousands of requests per hour, the reliability improvement alone justifies any marginal increase in per-request cost. The page on token cost of structured output breaks down the math for different schema sizes and volume levels.

Structured Output vs Function Calling

Structured output and function calling are closely related features that are often confused. Both produce structured data from a language model, but they serve different purposes and have different semantics. Function calling is designed for tool invocation: you define tools the model can call, and the model decides when to call them and generates the parameters for the call. The structured data is a side effect of the tool call, not the primary output. Structured output is designed for direct data responses: you tell the model to respond in a specific format, and the model's entire response is the structured data.

The practical difference matters. With function calling, the model decides whether to call a tool at all, it might respond with text instead. With structured output, the entire response is guaranteed to be the structured data, there is no ambiguity about whether the model "decided" to use the format. For data extraction, classification, and evaluation tasks, structured output is the cleaner abstraction because you always want structured data, not sometimes.

Some providers blur the line. Anthropic's strict tool use ("strict": true on tool definitions) functions similarly to structured output by guaranteeing the tool parameters match the schema. The Instructor library can use either function calling or structured output as the underlying mechanism, abstracting the distinction. For the developer, the important point is the guarantee: if you need every response to be structured data matching a specific schema, use structured output mode, not function calling, because function calling adds a layer of indirection (the model choosing to call a tool) that you do not need.

The AI tool use pillar covers function calling in depth, including when tool invocation is the right abstraction (agent loops, multi-step tasks, external API calls). The rule of thumb: use function calling when the model needs to decide which action to take, use structured output when the model needs to return data in a specific format.

Structured Output and Memory Systems

Structured output and AI memory are deeply connected. A memory system stores facts, preferences, and learned knowledge in structured formats, and it retrieves them in structured formats. When an AI assistant learns something about a user, it needs to store that knowledge as a structured record with fields for the fact, confidence, source, and timestamp, not as a blob of text. Structured output is what makes this reliable: the model extracts knowledge from a conversation, returns it in the exact schema the memory system expects, and the memory system stores it without parsing errors.

Adaptive Recall uses structured extraction at every stage of its memory pipeline. When new information enters the system, a model extracts entities, relationships, and facts using structured output schemas. When the system retrieves memories, the results are structured records with confidence scores, temporal information, and provenance metadata. When the system consolidates or updates memories, the operations are performed on structured data, not raw text. This is why structured output is not just a convenience feature for memory systems, it is a foundational requirement. Without reliable structured extraction, every memory operation becomes probabilistic, and probabilities do not compound well across thousands of operations.

The connection extends to knowledge graphs, where structured output is used to extract entities and relationships from text in a consistent schema that maps directly to graph nodes and edges. The knowledge graphs pillar and the entity extraction pillar both rely on structured output patterns for their core operations.

Core Concepts

Foundations

Validation and Libraries

Implementation