Gemma 3 27B Architecture
Back to Gemma 3 27B in the gallery
Gemma's flagship text stack leans on local attention more aggressively than Gemma 2.
Architecture and specifications
- Scale
- 27B parameters
- Context (tokens)
- 131,072
- Vocabulary
- 262,144 (~262k)
- License
- Gemma Terms of Use + Gemma Prohibited Use Policy
- Date
- 2025-03-11
- Decoder type
- Dense
- Attention
- GQA with QK-Norm and 5:1 sliding-window/global attention
- Layer mix
- 52 sliding-window + 10 global
- KV cache / token (bf16) info
- 496 KiB · Very high
- Key detail
- Built around a 27B sweet spot with heavier local attention and a large 262k multilingual vocabulary.
Training and benchmarks
- Pretraining data
- 14T tokens Includes text and image data; pretraining uses knowledge distillation. Gemma 3 report
- Post-training methods
- Distillation, RLHF, RLVR IT checkpoint uses teacher distillation and RL with human-feedback reward models, code execution, and math ground truth. Gemma 3 report