Linear and Sparse Attention

Attention is at the heart of LLMs, and (as with human nature) we are always finding ways to become more efficient with our attention.

Linear and sparse attention are two approaches to improving the efficiency of LLMs that have recently been popularized by two open-source models. Kimi focuses on linear attention with Kimi Delta Attention (KDA), whereas DeepSeek has introduced an improved sparse-attention mechanism with DeepSeek Sparse Attention (DSA).

Kimi Delta Attention

Kimi K3 has recently taken the spotlight as a frontier-approaching open source model scaled to 2.8T parameters, with Kimi Delta Attention at its heart.

Kimi Delta Attention builds on a lineage of linear-attention variants that the authors map out in their previous paper, Kimi Linear. The progression runs from DeltaNet to GatedDeltaNet, and then to Kimi Delta Attention.

DeepSeek Sparse Attention

Sparse attention is another approach to solving the quadratic attention problem. In its simplest form, sparse attention can be thought of as a sliding window or filter on the global attention mechanism.

DeepSeek Sparse Attention (DSA), introduced with DeepSeek-V3.2, uses a two-stage pipeline:

• Lightning indexer: score all tokens for relevance.
• Top-k selection: compute attention using only the highest-scoring tokens.

The increased complexity of sparse selection allows the model to learn more context-aware filtering procedures while retaining quality.

DSA in Other Contexts: GLM 5.2

GLM 5.2 is another open-source model that leverages DSA alongside an optimization called IndexShare.

Whereas vanilla DSA has an indexer and top-k selection at each layer, IndexShare (as the name suggests) shares the indexer across layers.