So, what is sliding window attention? If we think of regular self-attention as a global attention mechanism, since each sequence element can access every other sequence element, then we can think of sliding window attention as local attention, because here we restrict the context size around the current query position.

Some architectures combine these local layers with occasional global attention layers so that information can still propagate across the entire sequence.

Architecture gallery From-scratch chapter

A comparison between regular attention (left) and sliding window attention (right) (Original source The Big LLM Architecture Comparison).

What changes

Selected layers only attend to a recent window instead of the entire context

Why use it

Local layers use less computation and keep less cached context; occasional global layers retain full-context access

Example architectures

Gemma 3 27B, OLMo 3 32B, Xiaomi MiMo-V2-Flash, Arcee Trinity, Step 3.5 Flash, and Tiny Aya

Gemma 3 as a reference point

For instance, Gemma 2 uses a hybrid attention mechanism that combines sliding window (local) and global attention in a 1:1 ratio. Each token can attend to a 4k-token window of nearby context.

Where Gemma 2 used sliding window attention in every other layer, Gemma 3 has a 5:1 ratio, meaning there is only 1 full attention layer for every 5 local layers. Gemma 3 also reduced the sliding window size from 4096 to 1024.

According to the Gemma 3 ablation study, this more aggressive use of sliding window attention has minimal impact on modeling performance.

The Gemma 3 ablation study suggests that the smaller window and more aggressive local-to-global ratio have little effect on perplexity. Underlying paper Gemma 3 article (Original source The Big LLM Architecture Comparison).

The ratio and window size

The local-to-global layer pattern and the attention window size determine how aggressively a model uses SWA. The gallery includes several examples:

• Gemma 3 and Xiaomi use a 5:1 local-to-global pattern.
• OLMo 3 and Arcee Trinity use a 3:1 pattern.
• Xiaomi also uses a window size of 128, which is much smaller, and therefore more aggressive, than Gemma’s 1024.

SWA is essentially a knob that can be tuned more or less aggressively.

The long-context savings come from turning many full-attention layers into local ones, which reduces how much cached context those layers need to consider (Original source LLMs-from-scratch SWA materials).

Why it often appears with GQA

Please note that sliding window attention can be used with both multi-head attention and grouped-query attention (GQA); Gemma 3 uses GQA.

The two mechanisms change different parts of the cache. SWA limits how much context each local layer keeps. GQA reduces the number of cached key and value heads per token.

Sources

The Big LLM Architecture Comparison A Dream of Spring for Open-Weight LLMs LLMs-from-scratch SWA chapter

Back to architecture gallery