While Cerebras builds custom AI chips, French startup Kog squeezes performance out of the GPUs enterprises already own. By rewriting low-level software for existing datacenter hardware, Kog eliminates the latency bottlenecks that slow AI workflows.

Challenging the Necessity of Custom AI Silicon

The AI industry has pivoted to purpose-built chips for inference. Kog's CEO, Gaël Delalleau, says the belief that standard GPUs can’t handle decoding is wrong. Modern GPUs—Nvidia H200, AMD MI300X—offer memory bandwidth that current software leaves idle.

Kog’s Kog Inference Engine (KIE) targets "extremely fast single-request decoding," a must-have for real-time apps. In a recent demo the company hit 3,000 tokens per second (TPS) with Laneformer 2B, an open-source 2-billion-parameter model tuned for their stack. The demo uses a small model, but Kog plans to extend those gains to much larger LLMs.

A "Hacker" Approach to GPU Engineering

Unlike hardware-agnostic providers such as ZML, Kog digs deep. The team reverse-engineers GPUs at the assembly and binary levels, treating the silicon as a set of physical laws to master.

That low-level focus extracts every ounce of efficiency, but it costs time. With a lean crew of 11 engineers, Kog spends weeks or months dissecting each new GPU architecture before adding support. The method delivers top-tier performance but limits how quickly Kog can cover new hardware.

Targeting High-Value AI Use Cases

Kog focuses on sectors where latency equals lost revenue:

  • Software engineering: Tools like Claude Code stall for minutes. Kog aims to turn "waiting hours" into near-instant results.
  • Generative app/game design: Prompt-to-app pipelines need rapid iteration to keep users engaged and revenue flowing.

The company’s next milestone is a 10× speedup on its first major large-scale model. Backed by Scaleway, Bpifrance and the French Tech 2030 program, Kog positions itself as a key player in Europe’s AI sovereignty drive.

Key Takeaways

  • Optimization over hardware: Kog pushes the memory bandwidth of Nvidia H200 and AMD MI300X GPUs to the limit with extreme low-level software engineering.
  • Breaking the latency barrier: The startup targets a 30× boost in LLM inference speed for real-time professional workflows such as automated coding and generative design.
  • Deep-level engineering: By adopting a "hacker" mindset, Kog reverse-engineers GPU assembly and binary code to achieve performance that standard stacks cannot reach.

Kog, a French startup, says its Kog Inference Engine aims to run large-language-model (LLM) decoding up to 30 times faster on the same Nvidia H200 or AMD MI300X GPUs that data-center operators already own.

Why the push for speed matters now

LLM inference now throttles products like code-completion assistants, on-the-fly content generators, and interactive game-design tools. In those settings, the gap between a user’s prompt and the model’s reply directly impacts productivity and revenue. High-level APIs such as CUDA leave a large slice of GPU memory bandwidth idle, especially during token-by-token decoding, which dominates real-time use cases.

Kog’s low-level answer

Kog skips conventional software layers and talks straight to the silicon. Its engineers reverse-engineer the GPU at the assembly level, treating the hardware as a set of constraints to obey rather than a black box to abstract. The result is a custom execution path that keeps data flowing through the GPU’s memory pipes at near-full capacity.

In a recent demo the company ran Laneformer 2B—a 2-billion-parameter open-source model tuned for its stack—and reported a high token throughput. The model is modest compared with larger commercial systems, but the speed gain shows what a purpose-built software stack can extract from the same hardware.

The engineering trade-off

The upside comes with a steep cost curve. Kog’s 11-engineer team spends weeks or months dissecting each new GPU architecture before it can be supported. That depth yields high performance on the targeted cards, but it also means Kog cannot instantly add support for every new GPU that hits the market. Customers benefit only if their fleets include the Nvidia H200 or AMD MI300X GPUs Kog has already optimized.

Who stands to gain

  • 软件工程工具 – 诸如 Claude Code 之类的代码生成产品，在模型处理请求时经常会停顿数分钟。如果能将等待时间缩短至几秒钟，将使交互式开发变得更加流畅。
  • 生成式设计流水线 – 将文本提示词转化为应用原型或游戏资产的工作室需要快速迭代，以保持创作者的参与度。更快的解码速度可以缩短设计周期并提高转化率。

这两个领域都会将延迟直接转化为流失的可计费工时或用户流失，因此 30 倍的速度提升可能成为决定性的竞争优势。

更宏观的视角

Kog 的策略与构建定制化 AI 芯片的行业趋势背道而驰。像 Cerebras 这样的公司正投入数十亿美元研发旨在提高每瓦吞吐量的芯片。Kog 则认为，当今的 GPU 已经拥有足够的解码内存带宽；缺失的部分是能够真正利用这些带宽的软件。如果这一主张在大规模应用中成立，开发者就可以推迟昂贵的硬件升级，转而依靠软件升级来实现近乎实时的推理。

潜在的阻力

  • 维护负担 – 每一代新的 GPU 都需要进行全新的逆向工程。随着市场的多样化，这个小团队可能会面临人手不足的问题。
  • 向更大模型的扩展性 – 演示使用的是一个 2B 参数的模型。将同样的技术扩展到规模大得多的系统时，可能会遇到内存容量限制，或者需要尚未披露的其他工程技巧。
  • 替代路径 – 云服务提供商已经提供了捆绑了专用芯片和软件的推理优化实例。对于某些工作负载，Kog 技术栈带来的边际收益可能并不足以抵消托管服务的便利性。

值得关注的动向

  • 首次大模型部署 – Kog 表示，其下一个里程碑是在“主流大规模模型”上实现 10 倍的加速。2B 参数的演示模型与企业级模型之间的差距，将是对该方法最清晰的考验。
  • 硬件支持的扩展 – 是否增加对更新的 GPU 或其他加速器的支持，将预示其底层模型能否跟上硬件快速迭代的节奏。

总结

Kog 表明，通过从底层重写软件栈，从现有 GPU 中压榨出更多性能是可能的。LLM 解码速度提升 30 倍的前景可能会重塑开发者对 AI 服务定价和架构的设计方式——前提是该公司能够维持为适配每一款新芯片所需的密集工程投入。