PIVOT, a new inference-time trick for sparse-attention models, cuts the indexer cost by up to four-fold and trims overall latency by roughly 1.6×, all without touching the model’s weights. It works by grouping queries and letting a single “proxy” query do the heavy lifting.

Why sparse attention stalls at 100 K tokens

Sparse attention was meant to let transformers look at very long sequences—think 100 K tokens or more—while keeping compute affordable. In practice the promised speed-up evaporates once the sequence grows past a few tens of thousands of tokens. The culprit is the indexer, which scores every token against every query to decide which tokens belong in the sparse pattern. Its work scales as O(L²) (L = sequence length), so at 100 K tokens the indexer alone dominates runtime and erodes any benefit from sparsity.

The two observations behind PIVOT

Researchers found that adjacent queries almost always pick the same top-k tokens—about 90 % overlap. That means a single representative query can stand in for a whole batch of neighbours and still surface a useful candidate set. PIVOT capitalises on this by:

  1. Grouping a fixed number of consecutive queries (size g).
  2. Averaging the group to create a proxy query.
  3. Running the indexer once on the proxy instead of once per original query.
  4. Refining the proxy’s candidate list for each member of the group.

The math drops from O(L²) to O(L² / g). With a group size of eight, the indexer runs eight times fewer full scans.

Two operating modes

  • PIVOT-Refine keeps the dense indexer’s accuracy while delivering about a three-fold speed boost on the indexer stage.
  • PIVOT-Reuse pushes speed even further, sacrificing a small amount of accuracy for the maximum throughput gain.

Benchmarks on the DeepSeek-V3.2 and GLM-5.1 models show a consistent four-times indexer speedup and a 1.6× reduction in end-to-end latency when PIVOT is applied during inference.

Plug-and-play implementation

The technique arrives as a reference implementation that can be dropped into any existing Dynamic Sparse Attention (DSA) pipeline. It requires no weight changes, so models trained with standard sparse-attention recipes work unchanged. The only caveat is that the reference code runs on generic GPU kernels; production deployments will need hand-tuned Triton or CUDA kernels to hit the full speed potential.

What the trade-offs look like

PIVOT-Reuse’s speed advantage comes with a modest dip in attention quality, which may matter for tasks that are highly sensitive to exact token selection. Teams must weigh that loss against their latency budget. Additionally, the need for custom kernels adds engineering overhead for organisations without GPU-kernel expertise.

What to watch next

PIVOT shows that a clever re-ordering of work—grouping queries and sharing a proxy scan—can resurrect the promise of sparse attention for truly long contexts, delivering tangible speed gains without the cost of retraining.