WHY "ASK FOR JSON IN PROMPT" IS NOT A SOLUTION
2023 classic: "Respond strictly in JSON format, no markdown, no explanations". Model nods and complies in 95% of cases. Problem is in the remaining 5%. At a stream of a million requests that's tens of thousands of failed pipelines: sometimes it wraps response in ```json ... ```, sometimes adds "Sure, here's your JSON:", sometimes puts trailing comma, sometimes cuts object in the middle due to token limit.
Free text is a probabilistic process. As long as decoder is free to choose any next token, it will sooner or later choose one that breaks parser. The only reliable way — take away from model the very ability to generate invalid output. That's exactly what modern structured output does.
THREE LEVELS OF FORMAT CONTROL
Don't confuse these things — they're often lumped together under the word "JSON mode", but guarantees are different.
- JSON mode — engine guarantees output will be syntactically valid JSON. But what fields and types are in it — doesn't guarantee. Model may return
{"result": 42}instead of expected{"score": 42, "reason": "..."}. - Structured output / schema-guided — you pass JSON Schema, and engine guarantees compliance with structure: those fields, those types, those enum values. This is what's needed in 90% of prod tasks.
- Constrained decoding — low-level mechanism under the hood. At each generation step grammar (usually compiled from schema into finite automaton) masks logits of forbidden tokens. Model physically cannot step outside grammar.
Upper layer is convenient, lower is powerful. Ideally you use high-level SDK, and constrained decoding works invisibly.
HOW IT WORKS UNDER THE HOOD
Mechanics are simpler than they seem. JSON Schema compiles into grammar (regular or context-free), from which automaton is built. At each generation step current automaton state is known — hence the set of tokens allowed next is known. Logits of all other tokens are zeroed (mask), softmax computed only over allowed ones.
Result: if schema requires after key "age" a colon and number, model simply doesn't have token " or letters in distribution. Broken JSON becomes impossible in principle, not just unlikely.
Libraries like XGrammar, Outlines, and llguidance brought overhead nearly to zero — mask can be precomputed in parallel with forward pass. Per community measurements overhead on structured generation today is within single-digit percentages, not 2-3x as it was a couple years ago. Fresh engine releases and grammar compilers conveniently tracked in REDDYX catalog.
COMPARISON OF APPROACHES
| Approach | Structure Guarantee | Where it works | Overhead | When to use |
|---|---|---|---|---|
| Prompt "return JSON" | None | Everywhere | 0 | Prototype, one-off script |
| JSON mode | Only validity | Most APIs | ~0 | When fields not critical |
| Structured output (schema) | Full (fields + types) | OpenAI, Anthropic, vLLM, Ollama | Low | Prod, default choice |
| Function / tool calling | Full (arguments = schema) | All major providers | Low | Agents, tool calling |
| Constrained decoding locally | Full, down to grammar | vLLM, llama.cpp, TGI | Single-digit % | Own models, complex grammars |
PYDANTIC AS THE SINGLE SOURCE OF TRUTH
The main shift in thinking in 2026: you don't write JSON Schema manually. You describe a Pydantic-model, and the schema is generated from it automatically. The same model serves as a contract for LLM, a validator on output, and a type for the rest of the code. DRY in its purest form.
Working example with OpenAI SDK and Pydantic — the model will guaranteedly return an object of the required shape, and parse will return an already typed Python object:
from pydantic import BaseModel, Field
from openai import OpenAI
client = OpenAI()
class Ticket(BaseModel):
priority: str = Field(description="one of: low, medium, high, urgent")
category: str
summary: str = Field(max_length=120)
needs_human: bool
completion = client.chat.completions.parse(
model="gpt-4.1",
messages=[
{"role": "system", "content": "Классифицируй обращение в саппорт."},
{"role": "user", "content": "Всё упало, платёж списался дважды!!!"},
],
response_format=Ticket,
)
ticket = completion.choices[0].message.parsed
print(ticket.priority, ticket.needs_human) # -> urgent True
print(type(ticket)) # -> Ticket, не dict, не строка
Want to stay ahead with such tools — subscribe to the Telegram channel REDDYX AI: new repositories and breakdowns every 30-60 minutes.