OpenAI announced that its GPT-5.6 Sol model hit a 38.3 percent score on the ARC-AGI-3 logic benchmark, outpacing Anthropic’s Claude Opus 5 at 30.2 percent. The claim sparked an immediate technical dispute because the same model scored only 7.8 percent when run through the benchmark’s official test harness.

Why the ARC-AGI-3 score matters

ARC-AGI-3 probes a model’s ability to solve brand-new, reasoning-heavy problems rather than rely on memorised patterns. Researchers cite the scores as a proxy for “general-intelligence” progress, so a ten-point lead looks like a material stride toward human-like problem solving. That alone explains why the headline rippled through the AI community.

The hidden lever: Retained Reasoning and Compaction

OpenAI did not evaluate GPT-5.6 Sol with the public test harness. Instead it used its own Responses API with two proprietary options:

  • Retained Reasoning – keeps the model’s chain of thought alive across multiple steps, preventing the loss of intermediate logic that occurs when each step is treated in isolation.
  • Compaction – compresses earlier context instead of cutting it off, letting the model reference a longer, summarised history.

When those settings are active, the model stitches together longer arguments and reuses earlier deductions, inflating performance on multi-step tasks. In the official harness, which discards reasoning after each action, the same model’s score collapses to 7.8 percent, placing it well below Claude Opus 5.

OpenAI argues that a benchmark should measure the whole inference pipeline, not just the raw neural weights. From that perspective, the “wrapper” – the API logic that preserves and compacts context – belongs to the system being evaluated.

The fairness debate

François Chollet, co-founder of the ARC Prize that sponsors the benchmark, weighed in. He distinguished between custom harnesses built to “solve” a benchmark and the general-purpose API settings any user can enable. Chollet noted that the original ARC Prize testing environment used an older OpenAI-style completions API, which lacked the advanced features now available in Anthropic’s Claude API. That mismatch could have disadvantaged OpenAI in earlier rounds.

He stopped short of declaring the comparison invalid. Chollet said a head-to-head claim remains acceptable if the exact settings and any associated costs are disclosed. Transparency, he argued, lets the community assess whether the gap stems from model architecture, engineering tricks, or both.

Who wins, who loses

If the higher score is accepted at face value, OpenAI gains a public perception boost and a stronger bargaining position in enterprise licensing talks. Claude’s team can point to the raw-model performance on the standardized harness as evidence that their architecture is more efficient when stripped of extra engineering layers.

Researchers and developers who rely on benchmark rankings to guide investments may find the episode unsettling. The case shows that as models grow larger, the software that orchestrates their inference can become decisive. Companies that ship sophisticated inference stacks at low marginal cost could dominate headline scores without a superior underlying model.

What’s next for ARC-AGI-3 and other benchmarks

The dispute will likely push ARC Prize organizers toward tighter rules about permissible inference configurations. Possible responses include:

  • Publishing a “baseline” score that reflects performance with the official harness alone.
  • Requiring participants to submit a cost analysis of any additional settings, so that a high score can be weighed against extra compute or engineering overhead.
  • Adding a separate “system-level” track that rewards clever use of context-management features.

Such moves would force a clearer separation between raw model capability and engineering optimisation, making future claims easier to compare.

Counter-argument: Benchmarks should reflect real-world usage

One side argues that the strict “raw-model-only” view is unrealistic. In production, developers routinely layer caching, context summarisation, and other tricks onto language models. If a benchmark aims to predict practical utility, it should allow – or even encourage – those optimisations. From that angle, OpenAI’s Retained Reasoning and Compaction are legitimate tools that any competitor could adopt, provided they are publicly documented.

批评人士反驳称，如果没有统一的基准，评分就会变成一个移动的目标，从而削弱了基准作为客观衡量标准的地位。生态效度（模拟真实部署场景）与方法论纯粹性（孤立模型本身）之间的张力，加剧了当前的争议。

核心观点

GPT-5.6 Sol 的说法凸显了 AI 评估领域日益增长的分歧：模型本身的原始能力，与提取这些能力的工程生态系统之间的博弈。只要业界坚持要求完全披露推理设置及其成本，这场辩论就会变得更加激烈，而不是模糊我们对迈向通用人工智能进程的观察。