R_REDDYX.XYZ
Agents2026-05-25

Structured Output in 2026: Making LLMs Return Clean JSON

Structured Output in 2026: Making LLMs Return Clean JSON
TL;DR: Parsing free-form text from LLM with regex — path to hell. In 2026 we have three working layers: native structured output with schema guarantee (OpenAI, Anthropic, local engines), constrained decoding at token level, and validation via Pydantic on top of everything. Below — what to use, when, and how not to catch 3% broken responses in prod.

WHY "ASK FOR JSON IN PROMPT" IS NOT A SOLUTION

2023 classic: "Respond strictly in JSON format, no markdown, no explanations". Model nods and complies in 95% of cases. Problem is in the remaining 5%. At a stream of a million requests that's tens of thousands of failed pipelines: sometimes it wraps response in ```json ... ```, sometimes adds "Sure, here's your JSON:", sometimes puts trailing comma, sometimes cuts object in the middle due to token limit.

Free text is a probabilistic process. As long as decoder is free to choose any next token, it will sooner or later choose one that breaks parser. The only reliable way — take away from model the very ability to generate invalid output. That's exactly what modern structured output does.

THREE LEVELS OF FORMAT CONTROL

Don't confuse these things — they're often lumped together under the word "JSON mode", but guarantees are different.

  • JSON mode — engine guarantees output will be syntactically valid JSON. But what fields and types are in it — doesn't guarantee. Model may return {"result": 42} instead of expected {"score": 42, "reason": "..."}.
  • Structured output / schema-guided — you pass JSON Schema, and engine guarantees compliance with structure: those fields, those types, those enum values. This is what's needed in 90% of prod tasks.
  • Constrained decoding — low-level mechanism under the hood. At each generation step grammar (usually compiled from schema into finite automaton) masks logits of forbidden tokens. Model physically cannot step outside grammar.

Upper layer is convenient, lower is powerful. Ideally you use high-level SDK, and constrained decoding works invisibly.

HOW IT WORKS UNDER THE HOOD

Mechanics are simpler than they seem. JSON Schema compiles into grammar (regular or context-free), from which automaton is built. At each generation step current automaton state is known — hence the set of tokens allowed next is known. Logits of all other tokens are zeroed (mask), softmax computed only over allowed ones.

Result: if schema requires after key "age" a colon and number, model simply doesn't have token " or letters in distribution. Broken JSON becomes impossible in principle, not just unlikely.

Libraries like XGrammar, Outlines, and llguidance brought overhead nearly to zero — mask can be precomputed in parallel with forward pass. Per community measurements overhead on structured generation today is within single-digit percentages, not 2-3x as it was a couple years ago. Fresh engine releases and grammar compilers conveniently tracked in REDDYX catalog.

COMPARISON OF APPROACHES

ApproachStructure GuaranteeWhere it worksOverheadWhen to use
Prompt "return JSON"NoneEverywhere0Prototype, one-off script
JSON modeOnly validityMost APIs~0When fields not critical
Structured output (schema)Full (fields + types)OpenAI, Anthropic, vLLM, OllamaLowProd, default choice
Function / tool callingFull (arguments = schema)All major providersLowAgents, tool calling
Constrained decoding locallyFull, down to grammarvLLM, llama.cpp, TGISingle-digit %Own models, complex grammars

PYDANTIC AS THE SINGLE SOURCE OF TRUTH

The main shift in thinking in 2026: you don't write JSON Schema manually. You describe a Pydantic-model, and the schema is generated from it automatically. The same model serves as a contract for LLM, a validator on output, and a type for the rest of the code. DRY in its purest form.

Working example with OpenAI SDK and Pydantic — the model will guaranteedly return an object of the required shape, and parse will return an already typed Python object:

from pydantic import BaseModel, Field
from openai import OpenAI

client = OpenAI()

class Ticket(BaseModel):
    priority: str = Field(description="one of: low, medium, high, urgent")
    category: str
    summary: str = Field(max_length=120)
    needs_human: bool

completion = client.chat.completions.parse(
    model="gpt-4.1",
    messages=[
        {"role": "system", "content": "Классифицируй обращение в саппорт."},
        {"role": "user", "content": "Всё упало, платёж списался дважды!!!"},
    ],
    response_format=Ticket,
)

ticket = completion.choices[0].message.parsed
print(ticket.priority, ticket.needs_human)  # -> urgent True
print(type(ticket))  # -> Ticket, не dict, не строка

Want to stay ahead with such tools — subscribe to the Telegram channel REDDYX AI: new repositories and breakdowns every 30-60 minutes.

New repositories every 30 minutes

REDDYX AI scans GitHub 24/7 and ships the best AI/ML/Web3 projects to Telegram.

Join on Telegram

← All articles

Structured Output in 2026: Making LLMs Return Clean JSON | REDDYX AI