Structured Outputs Guarantee a Grammar, Not Your Schema

Structured outputs turn a JSON Schema into a hard promise: the model cannot emit text that breaks your shape. That part is true, and it beats asking politely and parsing the wreckage. But the promise covers less of your schema than your schema does.

Several keywords you probably use never reach the model at all. minimum is one. maxLength is another. They stay in your file, and they still look like rules.

Key takeaways#

• Structured outputs constrain generation to a grammar, and a grammar can only forbid the next token.
• Your JSON Schema mixes shape rules with value assertions. Only the shape rules compile.
• Anthropic's raw API rejects an unsupported keyword with a 400. Its SDKs instead drop the keyword and check it after the fact.
• llama.cpp takes a third route and skips unsupported features silently.
• A truncated response is still grammatical. It is a valid prefix, not a valid document, so keep checking stop_reason.

What do structured outputs actually constrain?#

They constrain the shape of the output, not its values. At each step the decoder masks out any token that would break the schema, so the reply is always parseable and always has the right types and keys. Value rules such as minimum are a separate matter.

The mechanism is narrow, and every limit falls out of it.

At each step a language model produces one score per token in its vocabulary. Guided generation adds a mask over those scores. Willard and Louf write it as a boolean mask m applied elementwise to the logits, which zeroes the probability of every token that would break the constraint.

The model then samples from what survives. So the naive version costs a full pass over the vocabulary per token, which their paper puts at a fixed cost of order N. Here N is the vocabulary size, and it is "often on the order of 10^4 or larger".

Their contribution is an index built from a finite state machine, and it brings the average cost down to order 1. That is why structured outputs are cheap enough to leave switched on.

Now look at what the mask can express. It answers exactly one question: given the tokens so far, which tokens are legal next? That is a decision about a position in a string, not a decision about meaning.

Why the subset exists#

The JSON Schema specification is blunt about the split. Its validation vocabulary — minimum, maxLength, uniqueItems and the rest — is described as keywords that "are all assertions without any annotation behavior". An assertion is a test you run against a finished value.

Here is the mismatch. A grammar decides what may come next. An assertion inspects what already arrived.

Some assertions survive the translation anyway. type and required and enum are really shape rules wearing assertion clothes, so a compiler can turn each one into a pattern. An enum of three strings becomes three alternatives, and a required key becomes a branch the grammar has to take.

Others cannot survive, and uniqueItems shows why best. To enforce it, the mask would have to remember every item it already emitted and compare the next one against all of them.

A finite state machine has no such memory, and that is the definition of finite. Arbitrary minimum is nearly as bad, since you would need arithmetic over a partially written number.

This is not a vendor cutting corners. It is a property of the technique, so you should expect the same gap everywhere the technique appears.

Three implementations, three different answers#

The gap is universal. What each implementation does about it is not, and the difference matters more than the gap itself.

I ranked these by one criterion: how quickly a wrong assumption becomes visible. On that measure the 400 wins outright, because a build that fails on Tuesday costs an hour. A field that quietly drifts out of range costs however long it takes someone to notice.

Anthropic documents the raw behaviour plainly: "If you use an unsupported feature, you'll receive a 400 error with details." Meanwhile the llama.cpp grammar guide states the opposite convention just as plainly: "Unsupported features are skipped silently."

Neither is wrong for its context. A local grammar sampler that refused every schema with a maximum in it would be tiresome. Still, if you move code between the two, that one line separates a caught bug from a shipped one.

A worked example#

Take a schema for a product record. Nothing exotic:

{
  "type": "object",
  "properties": {
    "sku": { "type": "string", "maxLength": 12 },
    "price_cents": { "type": "integer", "minimum": 1 },
    "tags": { "type": "array", "items": { "type": "string" }, "uniqueItems": true }
  },
  "required": ["sku", "price_cents", "tags"],
  "additionalProperties": false
}

Four of those constraints compile. The object shape, the three required keys, the types, and additionalProperties: false all become grammar. The model cannot return a fifth key, and it cannot put a string in price_cents.

Three do not compile: maxLength, minimum and uniqueItems.

Send that schema to the raw API and you get a 400 naming them. Send it through the Python or TypeScript SDK and something more interesting happens, because the SDK rewrites your schema before sending it.

First it removes the unsupported constraints. Then it updates the description with the constraint text, so minimum: 100 becomes the sentence "Must be at least 100". Finally it validates the response against your original schema once that response arrives.

Read what that does to minimum. It goes in as a hard bound and comes out as a hint in a description plus a check afterwards. Anthropic is candid about the trade: "Claude receives a simplified schema, but your code still enforces all constraints through validation."

Such a design is reasonable. It is also a completely different guarantee, and the word "guarantee" does not distinguish the two.

One constraint cannot be violated. The other can be, and then your code raises. Given a retry loop, that second kind costs a round trip; without one, it throws in production.

The same machinery guards tool inputs#

Most writing on this topic treats it as a response-formatting feature. That undersells it, because the same compiler runs on your tool definitions, and that is where the tier boundary does real damage.

Setting strict: true on a tool constrains the arguments the model may pass you. Requirements match the response case: every object needs additionalProperties: false and an explicit required list. In return the tool input validates exactly, so your handler can skip its defensive parsing.

Now consider the blast radius. A malformed response fails in one place, in front of you, on the request you just made.

A malformed tool argument travels instead. It reaches a function that writes to a database, calls a payment API, or edits a file, and it arrives looking like a normal call.

Say you put minimum: 1 on a quantity parameter. Through an SDK that rewrites schemas, that bound may become a description, so a zero or a negative number can reach your handler. Where the response-level story ends with your validator raising, the tool-level story ends with your code running.

Therefore validate tool arguments inside the handler regardless of what the schema says. Treat strict: true as a strong filter on shape rather than a contract about values. The grammar has already spared you the malformed-JSON case, which was most of the work. Your remaining check is small, and it is the one that protects the side effects.

What structured outputs cost at runtime#

Constrained decoding is not free on the first call, and the pricing model is unusual enough to plan around.

Anthropic compiles your schema into a grammar artifact, and that compilation adds latency to the first request using a given schema. Afterwards the compiled grammar stays cached for 24 hours from last use, so steady traffic pays the cost once.

Invalidation rules are the part worth designing for. Changing the schema structure invalidates the cache, and so does changing the set of tools in the request. Changing only a name or description does not.

That last exemption is more useful than it looks. It means you can reword a field description, including the constraint text an SDK injects, without paying to recompile.

Watch a second cache too, because the two interact badly. Changing output_config.format invalidates the prompt cache for that conversation thread.

So a service that swaps schemas per request discards its prompt cache on every call. That usually costs far more than the grammar compile it was worrying about, so group requests by schema where you can.

The failure mode, and how to detect it#

Here is the failure that catches people, and it survives every layer above.

Grammars guarantee that each token is a legal continuation. They do not guarantee the model reaches the end.

Hit your token ceiling midway through an object and you get exactly what the grammar promised: a valid prefix of a conforming document. A prefix of valid JSON is not valid JSON.

So "no more parse errors" holds only when generation finishes. Check stop_reason before you parse, every time, and treat truncation as its own outcome rather than a parse failure. A refusal behaves the same way, because a model that declines has not produced your object either.

Detecting the quieter failure takes more work. When a value-level assertion goes missing, nothing throws, and you simply start seeing numbers you thought were impossible.

Two habits catch it early. First, validate a sample of real responses against the original schema using a plain validator, not the one in the generation path. Our JSON Schema validator runs in the browser, so you can paste a suspicious response beside the schema you meant to enforce and see which keyword fails.

Second, log the schema you actually sent whenever an SDK rewrites it. That diff between intended and sent is the entire bug.

What post-hoc validation gets right#

The older approach was to prompt for JSON, parse it, validate it, and retry on failure. Constrained decoding is better at the first two steps and no better at the third, which is worth being honest about.

Validation-and-retry has one real advantage: it treats every keyword the same. Your validator does not care whether a rule is expressible as a grammar, so uniqueItems gets the same enforcement as type. There is no supported-features table to memorise and no silent tier boundary.

Its cost is equally real. Retries burn latency and tokens, they occasionally never converge, and syntax errors used to be the common case rather than the rare one.

Combining both is the sensible answer, and it is what the SDKs already do. Let the grammar handle shape, because it does that perfectly and for free, and let a validator handle values, because nothing else can.

What you should not do is run only the first and describe it as the second. For the general rules behind the validator half, we covered them in the JSON Schema validation guide.

Where this advice is wrong#

Three places, and I would rather name them than let you find them.

Tiers are not fixed. This unsupported list is an implementation limit, not a theorem, and vendors keep chipping at it.

Anthropic already supports minItems for the values 0 and 1, which is a small crack in the array-constraint wall. Meanwhile llama.cpp supports minimum for integers, where digit patterns stay bounded. So check the current table rather than trusting this article's snapshot.

I have overstated how often this bites. Most schemas are mostly shape. If yours is types, enums and required keys, every rule you wrote compiles and none of this applies to you. The gap only matters once you start expressing business rules in the schema.

Moving everything to a validator has its own cost. A rule enforced in two places drifts. A rewritten schema is also harder to reason about than one that is simply wrong in an obvious way. On a small internal tool, a single validator on your side stays honest more easily than a split enforcement story.

The short version#

Structured outputs compile your schema to a grammar, and a grammar can only decide which token comes next. Shape rules make that translation, while value assertions do not, because a mask cannot compare against something it already wrote.

Three implementations handle the leftovers three ways, and only the raw API tells you at once. So check stop_reason before parsing, validate real responses against the schema you meant, and remember that structured outputs guarantee the shape rather than the contents.

Sources: Anthropic: structured outputs, JSON Schema validation vocabulary, draft 2020-12, Willard and Louf, Efficient Guided Generation for LLMs, llama.cpp GBNF grammar guide