A new benchmark of 12 large-language-model (LLM) APIs finds that almost every service returns JSON that matches the requested schema, but a sizable minority spews factually wrong values. The token cost of the same schema swings from a few dozen tokens to nearly five thousand. Developers building extraction pipelines or data-driven agents can no longer trust “schema-valid” as a proxy for “correct”.

Why the test matters

API providers have been touting “structured output” as a way to eliminate parsing errors. The promise is simple: give the model a JSON schema, and it will fill in the fields without you writing fragile post-processing code. In practice, many production systems already rely on this guarantee to avoid crashes and keep downstream analytics pipelines clean. When the guarantee holds only half-true, bugs creep in silently, and cost calculations based on token usage go wildly off.

The good news: schemas are now mostly enforced

  • Most models in the test suite produced JSON that passed a strict validator.
  • Constrained decoding – models that lock the decoder to the schema cannot emit stray characters, so malformed payloads are practically extinct.
  • Parse-error rates – developers no longer need to wrap every call in try-catch blocks for JSON syntax failures.

The bad news: validity ≠ accuracy

A valid shape does not guarantee a valid value. Four of the twelve models—DeepSeek V4, Qwen, and GLM-5.2 (the latter two appearing under two different names in the report)—produced perfectly formed JSON that contained the wrong numbers when the “thinking” (or chain-of-thought) mode was turned on.

  • On the Qwen model, a simple arithmetic extraction went from 1 correct answer out of 16 with reasoning enabled to 8 correct out of 8 when reasoning was disabled.
  • DeepSeek V4 Pro showed a similar swing: extraction accuracy rose from 1/8 to 7/8 once the model stopped trying to explain its steps.

The extra reasoning step interferes with the constrained decoder, letting the model drift into hallucination while still respecting the outer brackets.

The ugly side: token-cost surprises and ignored parameters

  • Claude’s response format – when accessed through OpenAI-compatible endpoints, Claude completely ignored the response_format flag, returning 0 % schema-compliant output. The model does support structured calls, but only via Anthropic’s native tool-call interface.
  • Schema token inflation – a modest 12 KB schema costs 30 tokens on DeepSeek, yet the same payload ate 4,959 tokens on Claude.
  • Billing inconsistencies – some providers count the schema as part of the prompt, charging for every token it consumes; others treat it as a free overlay. At scale, the schema bill can outstrip the cost of the model’s generated content.

What developers should do now

  1. Validate values, not just shapes – a schema validator will not catch an incorrect numeric answer even though it fits the expected type. Add domain-specific checks (range, unit, cross-field consistency).
  2. Turn off chain-of-thought for extraction on DeepSeek, Qwen, and GLM when you need reliable field filling. The extra reasoning step is optional, not required for correctness.
  3. Audit token usage – log how many tokens each request consumes, including the schema portion, and compare bills across vendors before committing to large-scale deployments.
  4. Test portability – a schema that works on OpenAI may be silently dropped on Gemini or Claude. Run a quick sanity check on each target platform before shipping code.

Counter-point from the vendors

Some providers argue that “thinking” mode is a developer choice meant for tasks where explanation outweighs raw extraction accuracy. Claude ignores the response_format flag and suggests using Anthropic native tool calls instead. Those explanations are technically correct, but they shift the burden onto developers to know which mode to pick and how to budget for hidden token fees.

Bottom line

A JSON schema is no longer a safety net; it’s just a shape. Ensure the data inside matches reality, keep an eye on hidden token costs, and remember that a model’s “thinking” can corrupt even the cleanest-looking output.