Gemma 4 26B-A4B Architecture
Back to Gemma 4 26B-A4B in the gallery
Sparse Gemma 4 variant that keeps the local:global attention backbone while swapping dense FFNs for MoE layers.
Architecture and specifications
- Scale
- 25.2B total, 3.8B active (15.1% active)
- Context (tokens)
- 262,144
- Vocabulary
- 262,144 (~262k)
- License
- Apache License 2.0
- Date
- 2026-04-02
- Decoder type
- Sparse MoE
- Attention
- GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
- Layer mix
- 25 sliding-window + 5 global
- KV cache / token (bf16) info
- 212.5 KiB (compact global K=V layout) · High
- Key detail
- Uses 128 total experts with only 8 routed plus 1 shared expert active per token.
Training and benchmarks
- Pretraining data
- Not disclosed No per-model pretraining-token count is published. Gemma 4 model card
- Post-training methods
- Instruction tuning IT variant; the report refers to Gemma 3-like post-training without itemizing alignment stages. Gemma 4 report