Getting Reliable Structured Outputs From LLMs

Structured outputs from LLMs are the point where "AI feature" turns into "actual code that has to work." A summary that's off by a word is fine. A JSON payload with a missing field breaks the next function in the pipeline.

This post is the working version of how to get reliable structured outputs from LLMs in production. What providers actually enforce, where they fake it, and the validation you need on your side regardless of which provider you use.

The three approaches, briefly

Prompt-only JSON

Ask the model in the prompt to return JSON matching a schema. It'll try. It'll often succeed. It'll sometimes wrap the JSON in a code fence, add explanations before it, or drop a required field. Reliability sits somewhere in the low 90s on well-behaved schemas.

JSON mode

Provider-level guarantee that the output is valid JSON. OpenAI, Anthropic, and most modern providers offer this. It guarantees parse-ability. It does not guarantee that your specific schema is followed.

Structured outputs / schema-enforced

Provider-level guarantee that the output matches a JSON schema you supply. OpenAI calls this "structured outputs." Anthropic's tool use approximates it. When available, this is the highest-reliability option — the provider constrains the generation to match your schema.

Even with structured outputs, validate on your side. Providers change behavior. Schemas evolve. Trust but verify.

Tool calls vs pure structured output

Tool calls (function calling) are a specific case of structured output. The model produces a call to a named function with typed arguments. Under the hood, it's a JSON payload matching a schema — the same enforcement machinery.

Use tool calls when the LLM is deciding what to do next in an agent flow. Use pure structured outputs when the LLM is producing data that downstream code will consume directly. The distinction is more UX than technical.

Design schemas that don't fight the model

Prefer flat over nested

Deep nesting is where structured output goes wrong. A flat object with 15 fields is easier to generate correctly than a three-level nested tree. When you can, flatten.

Enums where possible

If a field can be one of four values, make it an enum. This eliminates free-form generation on that field and dramatically reduces error rate. Especially useful for classifications, priorities, and categories.

Constraints, minimally

String-length caps, integer ranges, and required fields all work. Regex patterns often don't work well — they get followed inconsistently. If a field needs a specific format (a phone number, an ID), validate after generation and prompt for a retry if it fails.

Don't over-constrain

The tighter the schema, the more often the model boxes itself into a corner where it can't produce valid output. A schema requiring five specific keys, each with three sub-fields, may fail on inputs where fewer are relevant. Make optional what's actually optional.

Validate on your side

Regardless of which approach you use, validate the parsed output before handing it to downstream code. Zod, Pydantic, io-ts — whatever fits your stack. The validator is your last line of defense.

When validation fails, log the raw output. That log is priceless when debugging. The prompt might need a tweak. The schema might be too tight. The model might be having a bad day. You can't tell without the raw output.

Handling failures

Retry with the error

If validation fails, send the model the validation error and ask it to fix the output. This works surprisingly often. One retry, not five. If it fails twice, the schema or prompt is the issue.

Best-effort parsing

Sometimes 90% of the fields are right and one field is malformed. Depending on the feature, best-effort parsing (keep the good fields, drop or default the bad one) is better than failing the whole request. Design this deliberately — silent partial failure is worse than a loud one.

Graceful degradation

If the structured output is unusable, fall back to plain text and show that instead. Not a perfect UX but better than an error.

Where teams get structured outputs wrong

Trusting the model without validation

Even with schema-enforced outputs, you'll get a schema-valid response that's semantically wrong. Empty strings, "N/A" where a number was expected, obvious hallucinations in required fields. Validation catches malformed. Domain logic catches wrong.

Schemas evolved without prompt updates

Add a field to the schema, forget to update the prompt, watch reliability plummet. Keep them in the same file, review together, and don't ship one without the other.

Ambitious schemas at v1

A schema with 30 fields, deep nesting, and dozens of enums is unlikely to be right on the first try. Start smaller. Add fields as you validate they work.

Not measuring reliability

Track schema-validation failure rate as a metric. If it drifts above 1–2%, something changed — the model, the input distribution, or the prompt. Alert on it.

The prompt still matters

Structured outputs don't relieve you of writing a good prompt. The schema constrains shape. The prompt controls content. Both need care.

Include an example in the prompt. Not "here's a JSON template" but "here's an input, and here's the output we'd expect." Few-shot examples move quality noticeably even under strict schemas.

Cost and latency

Structured outputs cost slightly more than plain text because the schema counts against tokens. Latency is usually similar or slightly higher. Both effects are small.

Tool-call chains, where the model calls tools sequentially, compound the latency. Design chains carefully. Every extra tool call is another round-trip.

Mistakes teams make

Skipping schema-enforced options. If your provider supports structured outputs and you're using prompt-only JSON, you're leaving reliability on the table.

Retrying without changing anything. Retrying the exact same call rarely fixes it. Include the validation error in the retry prompt.

Silent partial failures. Best-effort parsing can hide real issues. Log every case, review weekly.

Not versioning the schema. When the schema changes, downstream code needs to know. Version and coordinate.

Trade-offs

Tighter schemas mean higher reliability and less flexibility. Retries add latency and cost. Best-effort parsing improves UX but can mask bugs. Schema-enforced outputs are more reliable but tie you closer to a provider's specifics.

Pick the trade-offs per feature. Not globally.

Common misconceptions

"JSON mode guarantees my schema." It doesn't. It guarantees valid JSON. Schema conformance is a different feature.

"Structured outputs eliminate hallucinations." They constrain shape, not content. The model can still return schema-valid nonsense.

"Any downstream code can trust the LLM output." Never. Validate every time.

Frequently asked questions

Should I use JSON mode or structured outputs?

Structured outputs when your provider supports them. JSON mode as a fallback. Prompt-only as last resort.

How do I handle validation failures?

Retry once with the validation error included in the prompt. If it fails again, log and fall back gracefully.

Do I still need Zod or Pydantic?

Yes. Provider guarantees don't replace application-level validation.

Can I use structured outputs with streaming?

Yes, but partial JSON parsing gets tricky. Design the schema so partial parses render something usable.

What's the best schema shape?

Flat, with enums where possible, and only the fields you actually need. Add nesting only where the domain requires it.

Conclusion

Reliable structured outputs from LLMs are less about clever prompting and more about picking the right provider feature, designing schemas that don't fight the model, and validating on your side no matter what. When code depends on the shape, treat the LLM output like any other untrusted input.

Ship structured outputs behind validation. Monitor the failure rate. Version the schema. Retry once, then degrade gracefully. That's the whole pattern.

If you're building AI features where the output has to feed code and you're not sure your current setup will hold, the AI Audit covers reliability and integration as first-class questions.