Gemma 4 E4B Architecture
Back to Gemma 4 E4B in the gallery
Larger Gemma 4 edge variant keeps the same multimodal hybrid recipe but doubles width and KV heads for a stronger 128K mobile checkpoint. Uses per-layer embeddings, which add small layer-specific token vectors without scaling the full compute path, so its compute footprint is closer to 4.5B than a full 8B dense model.
Architecture and specifications
- Scale
- 8B parameters (4.5B effective)
- Context (tokens)
- 131,072
- Vocabulary
- 262,144 (~262k)
- License
- Apache License 2.0
- Date
- 2026-04-02
- Decoder type
- Dense
- Attention
- GQA with QK-Norm, cross-layer KV sharing, p-RoPE on global layers, and 5:1 sliding-window/global attention
- Layer mix
- 35 sliding-window + 7 global
- KV cache / token (bf16) info
- 56 KiB · Low
- Key detail
- The first 24 layers produce the KV cache for all 42 layers. Each layer uses 2 KV heads and a 10,240-wide GELU MLP.
Training and benchmarks
- Pretraining data
- Not disclosed No per-model pretraining-token count is published. Gemma 4 model card
- Post-training methods
- Instruction tuning IT variant; the report refers to Gemma 3-like post-training without itemizing alignment stages. Gemma 4 report