Back to Gemma 3 27B in the gallery

Google

Gemma's flagship text stack leans on local attention more aggressively than Gemma 2.

Gemma 3 27B architecture diagram
Gemma 3 27B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
27B parameters
Context (tokens)
131,072
Vocabulary
262,144 (~262k)
License
Gemma Terms of Use + Gemma Prohibited Use Policy
Date
2025-03-11
Decoder type
Dense
Attention
GQA with QK-Norm and 5:1 sliding-window/global attention
Layer mix
52 sliding-window + 10 global
KV cache / token (bf16) info
496 KiB · Very high
Key detail
Built around a 27B sweet spot with heavier local attention and a large 262k multilingual vocabulary.

Training and benchmarks

Pretraining data
14T tokens Includes text and image data; pretraining uses knowledge distillation. Gemma 3 report
Post-training methods
Distillation, RLHF, RLVR IT checkpoint uses teacher distillation and RL with human-feedback reward models, code execution, and math ground truth. Gemma 3 report
AA Intelligence Index v4.3.2 info
Total score 4.9 General 6.9 Scientific 2.2 Coding 11.6 Agents 0.0

Further reading and tools