OpenAI says its GPT-5.6 Sol model hit a 38.3 % success rate on the ARC-AGI-3 logical-reasoning benchmark, outpacing Anthropic’s Claude Opus 5 at 30.2 %. The lead appears only when the model runs behind OpenAI’s custom Responses API, which adds two inference-time tricks the official test harness does not allow.

The lead-up: a benchmark arms race

ARC-AGI-3 probes a model’s ability to solve novel, multi-step problems without relying on memorized facts. Earlier this year Anthropic announced a four-fold jump on the benchmark, putting Opus 5 at the top of the public leaderboard. That result set a new reference point for “raw” model capability because the official harness discards any intermediate reasoning after each step, forcing the model to start fresh every time.

OpenAI’s claim lands in the same competitive climate. By publishing a higher score, the company signals that its latest generation can not only match but exceed its chief rival’s logical performance. The headline, however, masks a technical nuance that is reshaping how the community reads benchmark tables.

What the numbers really mean

When GPT-5.6 Sol is evaluated under the same official ARC-AGI-3 harness that gave Opus 5 its 30.2 % result, the model collapses to a 7.8 % success rate. The swing isn’t a glitch; it’s the result of two engineered features OpenAI bundles with the model:

  • Retained Reasoning – instead of wiping the chain-of-thought after each sub-task, the system keeps the intermediate text alive, letting the model refer back to earlier deductions and build a cumulative argument.
  • Compaction – rather than truncating older context to stay within token limits, the wrapper compresses prior steps into a short summary, preserving the logical thread while keeping the prompt size manageable.

Both mechanisms live in the proprietary Responses API. By feeding the model a richer, continuous context, OpenAI effectively augments the model’s “memory” and lets it reuse its own reasoning. The official harness, by contrast, enforces a strict “stateless” mode that strips away those advantages.

Why the methodology matters

The episode highlights a growing split in AI evaluation: raw model scores versus system-level performance. OpenAI argues that a benchmark should reflect the whole stack developers will actually deploy – model, inference pipeline, and memory management. From that view, a 38.3 % score tells a product team that the combined offering can solve more real-world tasks than a model evaluated in isolation.

Critics say this muddies scientific progress. If every lab adds bespoke wrappers, the leaderboard becomes a patchwork of engineering tricks rather than a clean comparison of model intelligence. The official ARC-AGI-3 harness exists to level the playing field, ensuring a higher number signals a genuinely stronger underlying model, not a smarter wrapper.

Stakes for developers and investors

For startups building AI-driven products, the distinction is practical. A model that can retain reasoning and compact context may need less prompt engineering, lower inference latency, and fewer API calls to reach a solution. That translates into cheaper compute bills and smoother user experiences. Companies that ship a turnkey system – model plus optimized inference layer – gain a competitive edge where speed and cost matter.

Investors watch benchmark climbs as proxies for future revenue. A headline-grabbing lead can boost a firm’s valuation, even if the underlying model’s raw capability matches rivals. The debate over what the numbers represent therefore influences how capital flows across the AI sector.

What to watch next

  • Standard-setter responses – benchmark organizers may tighten rules around “external” inference tricks or add a separate “system” track that explicitly allows them. Their response will shape the next wave of public leaderboards.
  • Open-source wrappers – if the community releases its own implementations of retained reasoning and compaction, the advantage could become a baseline feature rather than a proprietary edge.
  • Real-world testbeds – companies increasingly post performance on proprietary problem sets (e.g., internal code-generation suites). Those results, while less comparable, will start to matter more than a single public benchmark.

Counter-argument: is this “gaming” the benchmark?

يصف بعض الباحثين استخدام واجهات برمجة التطبيقات (APIs) المخصصة بأنه "تلاعب بالمعايير" (benchmark gaming)، بحجة أنه يضخم النتائج دون تحقيق تقدم في فهم النماذج الأساسية. ويشيرون إلى أن انهيار أداء النموذج إلى خانة الآحاد تحت القيود الصارمة يعني أنه لا يزال بعيداً عن هدف الذكاء الاصطناعي العام (AGI). ويكمن القلق في أن المجال قد يبدأ في مكافأة الاختصارات الهندسية بدلاً من القفزات الحقيقية في قدرات الاستنتاج.

من جانبها، تؤكد OpenAI أن الخط الفاصل بين النموذج والنظام أصبح اصطناعياً بشكل متزايد. فنادراً ما تعمل تطبيقات الذكاء الاصطناعي الحديثة في فراغ؛ بل تعمل دائماً خلف طبقة خدمة (serving layer) تتولى عمليات التخزين المؤقت (caching)، وربط السياق (context stitching)، ومعالجة المخرجات اللاحقة (output post-processing). ومن هذا المنظور، فإن قياس خط المعالجة (pipeline) بأكمله يعد أكثر صدقاً في التعبير عما يختبره المستخدمون فعلياً.

الخلاصة

تُظهر قصة GPT-5.6 Sol أن انتصارات المعايير اليوم تعتمد على هندسة الاستدلال (inference engineering) بقدر اعتمادها على حجم النموذج. وسوف يحدد ما إذا كان المجتمع سيتبنى درجات مستوى النظام أو سيحد من استخدام الأغلفة المخصصة (custom wrappers) كيفية قراءتنا للعنوان القادم في "لوحة الصدارة" (leaderboard). وبالنسبة لأي شخص يقوم ببناء منتجات الذكاء الاصطناعي أو تمويلها، فإن الدرس واضح: الأرقام المجردة مهمة، ولكن البرمجيات المحيطة بها يمكن أن تكون العامل الحاسم في الأداء على أرض الواقع.