Back to Gemma 4 26B-A4B in the gallery

Google

Sparse Gemma 4 variant that keeps the local:global attention backbone while swapping dense FFNs for MoE layers.

Gemma 4 26B-A4B architecture diagram
Gemma 4 26B-A4B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
25.2B total, 3.8B active (15.1% active)
Context (tokens)
262,144
Vocabulary
262,144 (~262k)
License
Apache License 2.0
Date
2026-04-02
Decoder type
Sparse MoE
Attention
GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
Layer mix
25 sliding-window + 5 global
KV cache / token (bf16) info
212.5 KiB (compact global K=V layout) · High
Key detail
Uses 128 total experts with only 8 routed plus 1 shared expert active per token.

Training and benchmarks

Pretraining data
Not disclosed No per-model pretraining-token count is published. Gemma 4 model card
Post-training methods
Instruction tuning IT variant; the report refers to Gemma 3-like post-training without itemizing alignment stages. Gemma 4 report
AA Intelligence Index v4.3.2 info
Total score 13.1 General N/A Scientific 5.7 Coding N/A Agents N/A

Further reading and tools