Grouped-query attention is an attention variant derived from standard multi-head attention. It was introduced in the 2023 paper GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints by Joshua Ainslie and colleagues.

Unlike MHA, where each head has its own set of keys and values, GQA groups multiple query heads to share the same key and value projections. This reduces how many keys and values have to be stored and retrieved from the KV cache during inference.

Architecture gallery From-scratch code

A comparison between MHA and GQA. Here, the group size is 2, where a key and value pair is shared among 2 query heads (Original source The Big LLM Architecture Comparison).

What changes

Several query heads share the same key and value projections

Why use it

Fewer K/V heads make the KV cache smaller and reduce memory bandwidth use

Example architectures

Dense: Llama 3 8B, Qwen3 4B, Gemma 3 27B, Mistral Small 3.1 24B, SmolLM3 3B, and Tiny Aya 3.35B.
Sparse: Llama 4 Maverick, Qwen3 235B-A22B, Step 3.5 Flash 196B, and Sarvam 30B.

Why GQA became popular

The core idea behind GQA is to reduce the number of key and value heads by sharing them across multiple query heads. This (1) lowers the model’s parameter count and (2) reduces the memory bandwidth usage for key and value tensors during inference since fewer keys and values need to be stored and retrieved from the KV cache.

(If you are curious how GQA looks in code, see my GPT-2 to Llama 3 conversion guide for a version without KV cache and my KV-cache variant here.)

While GQA is mainly a computational-efficiency workaround for MHA, ablation studies (such as those in the original GQA paper and the Llama 2 paper) show it performs comparably to standard MHA in terms of LLM modeling performance.

How the memory savings work

For bf16 keys and values, the per-layer cache size is:

2 × sequence length × number of K/V heads × head dimension × 2 bytes

MHA uses as many K/V heads as query heads, whereas multi-query attention uses one. GQA is in between. This is also why the savings grow with sequence length.

Lower is better. Once the context window grows, the KV-cache savings become more pronounced (Original source LLMs-from-scratch GQA materials).

Why GQA still matters in 2026

More advanced variants such as multi-head latent attention (MLA) can offer better modeling performance at the same KV efficiency levels, based on the ablation studies in the DeepSeek-V2 paper. However, MLA is more complicated to implement.

GQA remains appealing because it is robust, easier to implement, and also easier to train (since there are fewer hyperparameter tunings necessary, based on my experience).

Among the releases in my Spring Architectures article, only MiniMax M2.5 and Nanbeige 4.1 stayed very classic here, using GQA without any other efficiency tweak. The smaller Sarvam 30B model also uses classic GQA, whereas the larger 105B variant switched to DeepSeek-style MLA.

GQA shares key and value heads, whereas MLA compresses the state stored in the cache. Both reduce the same memory bottleneck in different ways (Original source A Dream of Spring for Open-Weight LLMs).

Sources

Ainslie et al. (2023), GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints The Big LLM Architecture Comparison A Dream of Spring for Open-Weight LLMs LLMs-from-scratch GQA materials

Back to architecture gallery