Structured Outputs: Reliable LLM JSON

To make a large language model return something you can ship, stop asking for a good answer and start requiring a valid object: define a JSON Schema with required fields and enums, pass it as a real output contract (not a suggestion), validate every response before it touches your UI, and design an explicit fallback for partial or malformed output.

Prose answers are for humans; JSON Schema and constrained decoding are for machines. Require a schema, validate before render, and always define what happens when a field is missing.

From "Good Answer" to "Valid Object": Why Free Text Breaks in Production

A model can write a fluent, correct-sounding answer and still be useless to your product, because "useless" isn't about quality — it's about whether the response is a shape your code was built to consume. Production systems don't read prose; they read fields, types, and keys that match what the parser expects.

In a demo, a slightly-off answer is charming — you read it, get the gist, and move on. In production, a slightly-off answer is a stack trace. If your rendering code expects data.priority to be one of "low" | "medium" | "high" and the model instead writes "I'd say this is probably high priority, though it could go either way," your UI has nothing to render. Not a bad answer — no answer your code can use at all.

This is the gap most teams miss moving from a chatbot demo to a real pipeline. A good answer, in the human sense, is judged on tone, accuracy, and helpfulness. A shippable object is judged on a different axis: does it deserialize, does it satisfy every required key, and does every value fall inside the type or enum your downstream code expects? Those are binary. There's no partial credit for "directionally correct JSON."

Independent LLM researcher and developer-tools writer Simon Willison has made a similar point repeatedly in his public notes on model APIs: a syntactically valid response and a semantically usable one are different guarantees, and conflating them is where most "the AI is unreliable" bug reports actually originate. The model didn't fail; the contract between the model and the code was never written down.

If you're still tuning wording and tone rather than output shape, our complete guide to prompt design covers the fundamentals this piece builds on. This article picks up specifically where "good prompt" meets "system that has to parse the result."

Common gaps between a demo and a shippable pipeline:

Free text wrapped around the JSON ("Sure, here's the object: {...}")
Field names the model invents or subtly renames mid-conversation
Enums it treats as suggestions ("high-ish", "medium to high")
Fields silently dropped when a response runs long or gets truncated

Three Ways to Constrain What a Model Returns

Three approaches force a model toward a shape you can use: parsing prose with regex or a second model call, prompting with a JSON Schema and asking nicely, and constrained decoding, which makes invalid tokens mathematically impossible to generate. Each trades reliability against flexibility and cost differently.

Approach	How it forces shape	Typical failure mode	Best for
Prose parsing (regex / second LLM call)	Ask for JSON in the prompt, extract it after the fact	Breaks on stray text, markdown fences, trailing commas	Quick prototypes, low-stakes internal tools
JSON Schema prompting (`response_format`, tool/function calling)	Model is shown a schema and instructed — sometimes enforced server-side — to match it	Model can still omit optional-seeming fields or hedge inside string values	Most production features
Constrained decoding (grammar-based sampling, e.g. `Outlines`, `Guidance`, local grammars)	Invalid tokens are removed from the sampling distribution during generation	Can slow generation or reduce reasoning quality on complex tasks	High-volume pipelines where syntactic validity can't fail

OpenAI's Structured Outputs feature and Anthropic's tool-use API both sit in the middle row: they don't touch the sampler directly, but they raise the odds of a valid response from "usually" to "almost always" by combining schema-aware prompting with server-side checking against your JSON Schema.

Tool or function calling is worth calling out as its own sub-case, because it reframes the problem usefully: instead of asking the model to "write JSON," you define a function signature and let the model choose to "call" it with arguments. Product teams tend to find this easier to reason about, because a function signature is a vocabulary that product and engineering already share, while a loose instruction like "respond in JSON" is one more piece of prose that can drift as the surrounding prompt gets edited over time.

Constrained decoding is the most reliable option syntactically, but it isn't free. Research such as the 2024 paper Let Me Speak Freely? found that rigidly enforcing a JSON format at generation time can measurably reduce accuracy on reasoning-heavy tasks compared to letting a model reason in free text before it structures the final answer — the model spends part of its reasoning budget fighting the grammar instead of solving the problem. The practical takeaway: constrain the final answer, not the model's thinking.

Whichever method you pick, the schema you hand the model is functionally part of your spec — which is the whole argument behind why the system prompt is the new PRD. The shape of the output is a requirement, not a suggestion, and it belongs in the same document engineers read to build the parser.

The Failing Case: One Missing Field, One Broken Screen

Here's a concrete failure: a feature-triage UI renders a colored priority badge by reading ticket.priority, and the model — asked in prose to "assess urgency" — returns a paragraph with no such field, so ticket.priority is undefined and the badge component throws before the rest of the page renders.

A naive first attempt at JSON doesn't fully fix it:

{
  "ticket_id": "TCK-4821",
  "summary": "Checkout button unresponsive on Safari",
  "priority": "pretty urgent I think"
}

That third field is the crash. Your component expects one of a small set of enum values to pick a color and label. It gets a sentence instead. In a typical component tree, that means Badge either throws on an unrecognized variant prop or, worse, silently renders nothing — and nobody notices until a support ticket says "the dashboard is blank for some tickets."

A real fix defines the shape up front:

{
  "ticket_id": { "type": "string" },
  "summary": { "type": "string" },
  "priority": { "type": "string", "enum": ["low", "medium", "high", "urgent"] },
  "owner": { "type": ["string", "null"] }
}

with "required": ["ticket_id", "summary", "priority"] and "additionalProperties": false.

Passing this schema to the model, via response_format or tool calling, makes the malformed case far less likely. But "far less likely" still isn't "never" — models occasionally drop a required field when a response is truncated by a token limit, or hallucinate an enum value that isn't in your list. That's why the schema alone isn't the fix; the fix is a validation layer — Pydantic in Python, Zod in TypeScript, or a schema validator like Ajv — that checks every response before it reaches the render path, the same way you'd never trust an unvalidated payload from a third-party API.

This same failure shows up anywhere free-form input meets a fixed UI, including voice. A spoken product journal entry has to be transcribed and then mapped onto fields like situation, emotion, and follow_up; skip the schema step and you get the same kind of blank card, just triggered by a microphone instead of a text box. We cover the transcription side of that pipeline in our guide to designing multimodal and voice AI products.

A Schema-Design Checklist You Can Reuse

A schema that actually holds up in production makes four decisions explicitly: which fields are required versus optional, which values are constrained enums instead of open strings, what a partial or failed extraction returns instead of a crash, and how the schema itself is versioned as your product changes.

Mark fields required, not implied. If your UI cannot render without priority, it belongs in the schema's required array — not just mentioned in a prompt instruction the model can ignore under pressure.
Prefer enums over free text for anything your code branches on. Severity, status, category, sentiment: if a switch statement or a colored badge depends on the value, constrain it to a fixed list rather than trusting the model to stay consistent across a thousand calls.
Give every optional field an explicit "empty" state. Use null rather than omitting the key, so your parser has exactly one shape to check for absence instead of two.
Set additionalProperties: false. This catches the model inventing a plausible-looking field that nothing downstream reads — a common source of silent bugs.
Design a fallback object, not just a fallback message. Decide in advance what a "degraded" response looks like — for example { "status": "needs_review", "raw_text": "...", "confidence": "low" } — a shape your UI already knows how to render as a review-queue item, rather than an unhandled exception.
Version the schema. Once real data depends on a shape, changing a field name is a breaking change; treat schema edits with the same discipline as an API contract, changelog included.

Field behavior in your code	Use this type	Not this
Drives a `switch`, color, icon, or routing decision	`enum` of exact string literals	Open `string`
Feeds a chart axis or a calculation	`number` with `minimum`/`maximum`	A `string` that "usually" looks numeric
May legitimately be absent	`["string", "null"]`, key still present	Omitting the key entirely
Free-form user-facing text (summary, notes)	`string`, no enum	Forcing prose into a rigid enum it doesn't fit

Jon Postel's robustness principle — "be conservative in what you send, liberal in what you accept" — is exactly the right frame here: demand a strict shape from the model, but build your parser to accept and gracefully route anything that falls short, rather than assuming strictness alone will save you.

Treat the Model Like an Untrusted API: Validate, Retry, Fall Back

Once you have a schema, treat every model response the way you'd treat a payload from a third-party API you don't control: validate it before use, retry with the specific error on failure, and log every rejection so you see drift before customers do.

The retry loop is simple and effective: if validation fails, send the exact error back to the model ("missing required field: priority") and ask it to correct only that field. This recovers most failures on a second call, but a second call is not free — it roughly doubles latency and token cost for that request. Whether that trade is worth it is a genuine product-economics question, not just an engineering one; our piece on the economics of running AI features in production goes deeper on how retry rates quietly show up in your per-request cost.

Validation is also your last line of defense, not just a formatting nicety. An unvalidated field can carry more than a type error — a model that's been prompt-injected or has drifted off-task can smuggle instructions, unexpected values, or sensitive data through a field your code trusts implicitly. Treating schema validation as a guardrail, not just a parser, is part of the broader discipline covered in our guide to AI safety and guardrails for production systems.

A minimal version of that loop, independent of language or SDK, looks roughly like this:

response = call_model(prompt, schema)
result = validate(response, schema)

if result.errors:
    response = call_model(prompt, schema, prior_errors=result.errors)
    result = validate(response, schema)

if result.errors:
    return fallback_object(raw=response, status="needs_review")

return result.data

That is the entire pattern: one retry with the specific error attached, then a defined degraded object rather than an exception. Nothing about it is exotic, which is exactly the point — it's boring, repeatable infrastructure, not a prompting trick you have to re-derive for every feature.

A validation routine worth standardizing across your team:

Validate immediately after the model call, before any business logic touches the object.
Reject and retry (once, maybe twice) rather than silently coercing bad values into defaults.
Log every validation failure with the raw model output — your best early signal that a prompt or model upgrade quietly changed behavior.
Cap retries and always have a terminal fallback, so a stubborn failure degrades gracefully instead of hanging the request.

Where a Defined Schema Pays Off Beyond the Prompt

The discipline of defining required fields, enums, and fallbacks doesn't stop at the LLM call — it's the same discipline that makes any data shippable downstream, from the API contract your frontend consumes to the tables your analytics team eventually queries.

This is one reason schema-first thinking is worth building as a habit rather than a one-off prompting trick. In Prodinja's prototype, the Data Modelling tool works from the other direction: you define entities and their fields once, and it turns them into real SQL DDL — a concrete reminder that a defined schema, not a well-worded description, is what actually makes an output shippable to a database, an API, or a UI component. The same instinct that makes a CREATE TABLE statement useful is the instinct that makes a model's JSON response useful: name the fields, type them, and decide what's required before anything downstream has to guess.

Zoom out further and this becomes a data-strategy question, not just a prompting one — the schema you enforce on a model's output is one more contract your organization now has to govern and keep consistent. Our guide to treating schema and data quality as part of your AI data strategy covers how these contracts multiply once several AI features are writing into the same systems.

Key Takeaways

A "good answer" is judged by humans; a "shippable object" is judged by a parser — optimize for the second once you leave the demo.
JSON Schema prompting via response_format or tool calling is the practical default for most production features; reserve full constrained decoding for high-volume pipelines where syntactic validity can't fail.
Every field your UI depends on to render should be required in the schema, and constrained to an enum if your code branches on it.
Design the fallback object before you ship, not after the first crash — a needs_review shape beats an unhandled exception.
Validate every response like an untrusted API payload: reject, retry with the specific error, log failures, and cap retries with a graceful terminal fallback.
Constraining format at generation time isn't free — it can trade off reasoning quality and latency, so constrain the final answer, not the model's thinking.

Frequently Asked Questions

What's the difference between JSON mode and structured outputs?

"JSON mode" only guarantees the response is syntactically valid JSON — it says nothing about which fields exist or what values they hold. "Structured outputs," the term used for schema-constrained generation, goes further: you supply a JSON Schema, and the API works to make the response match that schema's required fields, types, and enums, not just valid syntax. Treat JSON mode as a floor, and structured outputs as the actual contract.

Can a model still return invalid output even with a JSON Schema?

Yes. Schema-aware prompting and server-enforced structured outputs reduce failures dramatically but don't reach zero. Truncation from token limits, edge cases in deeply nested schemas, and provider-side quirks can all still produce a response that fails validation, which is why a client-side validator and a retry-with-fallback strategy remain necessary even on the most reliable structured-output API you can find.

Should I use constrained decoding or just prompt with a schema?

For most product teams, prompting with a schema through a provider's built-in structured-output or tool-calling feature is the right default — it's simpler to maintain and sits close enough to full reliability for most use cases. Reach for grammar-based constrained decoding, using libraries like Outlines or Guidance, only when you're running high volume through a self-hosted model and syntactic validity truly cannot fail.

How do I handle a model that returns a partial object?

Design for it explicitly rather than treating it as an edge case: define a degraded schema variant — for example a status: "needs_review" object carrying whatever fields did come back — and route anything that fails full validation into that shape instead of the primary render path. That turns an outage-shaped bug into a queue your team can clear on their own schedule.

Do enums make the model less accurate?

Not meaningfully, for well-chosen categories — enums usually make classification more consistent, because you've removed the model's need to invent its own phrasing. The real risk runs the other way: an enum list that's too narrow for reality forces the model to force-fit an answer, so review your enum values periodically against real outputs and add an "other" value paired with a free-text companion field rather than expanding the enum indefinitely.