What Is Structured Output from LLMs?

The Problem Structured Output Solves

Language models generate text one token at a time, sampling from a probability distribution over their vocabulary at each step. Without any constraints, the model can produce any sequence of tokens, which means it can produce any string. When you prompt a model to "return JSON with a field called sentiment that is positive, negative, or neutral," the model usually complies. But "usually" is a specific number: depending on the model and the complexity of the schema, prompt-based JSON generation succeeds 85-96% of the time. That failure rate is invisible in development and devastating in production.

A system processing 10,000 requests per day with a 95% success rate generates 500 failures daily. Each failure requires either a retry (adding latency and cost), a fallback path (adding code complexity), or manual intervention (adding operational overhead). These failures are not bugs in your code. They are the inherent unreliability of asking a probabilistic text generator to produce deterministic data structures. The failures are varied and unpredictable: missing commas, extra closing braces, fields spelled differently than expected, values of the wrong type, extra fields that your parser does not expect, or truncated output that is valid JSON up to a point but incomplete.

Structured output solves this by changing how the model generates tokens. Instead of generating freely and hoping the result matches your schema, the system compiles your schema into a set of constraints and enforces those constraints during generation. At each token, any token that would violate the schema is masked out before sampling occurs. The model can only produce tokens that keep the output on a valid path through the schema. The result is not "the model usually returns valid JSON," it is "the model cannot return anything other than schema-valid JSON."

How Constrained Decoding Works

The core mechanism is called constrained decoding, and it operates at the token generation level, the lowest level of the model's output pipeline. Here is the process step by step.

First, you provide a JSON schema to the API along with your prompt. This schema defines the exact structure of the response you want: which fields, what types, which values are allowed. Second, the provider's system compiles this schema into an internal representation, typically a finite state machine or context-free grammar, that can evaluate at each token whether the output so far is still on a valid path toward a complete, schema-compliant response. Third, during generation, at each token step, the system checks which tokens in the vocabulary would keep the output valid. All other tokens are masked, meaning their probability is set to zero. The model samples only from the remaining valid tokens. Fourth, this process repeats until the model generates a complete response that satisfies the schema.

The result is a response that is guaranteed to parse as valid JSON and guaranteed to conform to your schema. Not "almost guaranteed" or "guaranteed if you prompt correctly," but guaranteed by the mechanics of the generation process. The model physically cannot produce an invalid response because invalid tokens are removed from its options at every step.

This is different from post-processing or validation. Post-processing catches errors after the model has already generated bad output, which means you have already spent the tokens and latency on that failed generation. Constrained decoding prevents errors from occurring in the first place, which is fundamentally more efficient and reliable.

What Schemas Can Express

Structured output schemas are JSON Schema definitions, typically supporting a subset of the JSON Schema specification. The supported features cover the majority of production use cases:

Object types define a set of named fields, each with its own type. You can specify that a response must have a "name" field of type string, an "age" field of type integer, and a "tags" field of type array. Every response will have exactly these fields.

Array types define ordered lists of items, optionally with constraints on the item type, minimum length, and maximum length. You can specify an array of strings, an array of objects with specific fields, or a nested array of arrays.

Enum types constrain a string or number field to a specific set of allowed values. If your schema says the "status" field must be one of "active", "inactive", or "pending", the model cannot return any other value. This is the mechanism behind reliable classification, covered in detail on the classification and enum outputs page.

Nested objects allow schemas within schemas, typically up to 5 levels deep. You can define an object that contains an array of objects, each of which contains another object, creating rich hierarchical data structures.

Union types (using anyOf) allow a field to be one of several types. The most common use is making fields optional by allowing either the field's type or null. In strict mode with OpenAI, all fields must be listed in the required array, so optional fields are expressed as {"anyOf": [{"type": "string"}, {"type": "null"}]}.

What schemas cannot currently express includes regex pattern constraints on strings, arbitrary numeric ranges (minimum/maximum on numbers is not universally supported), conditional schemas (if/then/else), and deeply recursive structures. These limitations exist because the constrained decoding engine needs to represent the schema as a finite structure that can be evaluated at each token. The page on designing schemas for structured output shows how to work within these constraints and when to move validation to a post-processing layer.

Structured Output vs Prompting for JSON

Before structured output existed, the standard approach was to include instructions in the prompt: "Respond in JSON format with the following fields: name (string), age (integer), city (string)." This approach relies entirely on the model's instruction-following ability, which is strong but not perfect. The failure modes are numerous.

The model might return the JSON wrapped in markdown code fences. It might include explanatory text before or after the JSON. It might use single quotes instead of double quotes. It might return a field called "Name" instead of "name". It might return age as a string "25" instead of an integer 25. It might omit a field it considers irrelevant. It might add extra fields it considers helpful. Each of these is a valid model behavior from the model's perspective, because the model is generating text, not data.

Structured output eliminates every one of these failure modes. The response is pure JSON, no code fences, no explanatory text, no variations in field names, no type mismatches, no missing fields, no extra fields. The schema is the contract, and the response satisfies the contract.

There is still a role for prompt engineering with structured output. The schema constrains the structure, but the quality of the content within that structure still depends on the prompt. A schema that requires a "summary" field of type string will always get a string, but whether that string is a good summary depends on how you prompt the model. The schema handles structure; the prompt handles quality. Both matter for production applications.

When to Use Structured Output

Use structured output whenever your application consumes the model's response as data rather than displaying it as text. This includes data extraction from documents, classification into predefined categories, decision-making with structured reasoning, form generation, entity extraction, sentiment analysis with scores, content evaluation with rubrics, and any pipeline where the model's output feeds into downstream processing.

Do not use structured output when the model's response is meant to be read by a human as free-form text. A conversational chatbot response, a creative writing output, or a code generation result are all cases where free-form text is the appropriate output format. If you need a mix, such as a structured response that includes a free-text explanation field, you can use a schema with a string field for the explanation and structured fields for the data.

The rule of thumb: if you would write a JSON parser, a regex extractor, or a post-processing function to extract data from the model's response, you should use structured output instead. It does the same job with zero code, zero maintenance, and zero failure rate.