DeepSeek Sparse Attention uses an indexer and token selector to keep a learned subset of the visible prefix. A selected token can be far outside a fixed local window.

Architecture gallery

Figure 1. GLM-5 combines DeepSeek Sparse Attention with Multi-head Latent Attention (Original source The Big LLM Architecture Comparison).

Summary

Learn a sparse attention pattern

Practical benefit

Reduce long-context attention work

Example architectures

DeepSeek V3.2, GLM-5, and GLM-5.2

Learned sparsity versus a fixed window

Figure 2. Full causal attention reads the visible prefix, sliding-window attention keeps a local block, and DSA learns a subset (Original source From DeepSeek V3 to V3.2: Architecture, Sparse Attention, and RL Updates).

The indexer and selector

The lightning indexer scores an earlier position s for the current position t as follows:

\[I_{t,s} = \sum_{j=1}^{H^I} w_{t,j}\,\operatorname{ReLU}\!\left(q_{t,j}\cdot k_s\right).\]

# t = current position, s = earlier positions, j = indexer head
scores[s] = sum(w[t,j] * relu(dot(q[t,j], k[s])) for j in heads)
selected  = top_k(scores, k=2048)
output    = sparse_attention(query_t, selected)  # O(Lk) over the sequence

Figure 3. The indexer scores the prefix before the selector retains the top entries (Original source From DeepSeek V3 to V3.2: Architecture, Sparse Attention, and RL Updates).

How DSA and MLA fit together

MLA compresses what is stored in the KV cache. DSA changes which stored positions the attention operation reads.

Figure 4. DeepSeek V3.2 pairs MLA with DSA (Original source From DeepSeek V3 to V3.2: Architecture, Sparse Attention, and RL Updates).

The k in top-k, not to be confused with the k used for keys in the equation, is set to 2048 in the released model code. The indexer and token selector result in each token attending to a few past tokens that the model has learned to consider most relevant, rather than all tokens or a fixed local window.

The goal was not to improve performance over the dense base model but to reduce performance degradation from the sparse attention mechanism while benefiting from improved efficiency. GLM-5 later adopted DSA and MLA. GLM-5.2 adds IndexShare so that nearby layers can reuse selected positions.

Sources

The Big LLM Architecture Comparison A Dream of Spring for Open-Weight LLMs DeepSeek V3.2 technical article DeepSeek V3.2 paper GLM-5 paper

Back to architecture gallery