Gemma 3 270M Architecture
Back to Gemma 3 270M in the gallery
Tiny Gemma 3 variant that preserves the family's local-global attention recipe at a toy scale.
Architecture and specifications
- Scale
- 270M parameters
- Context (tokens)
- 32,768
- Vocabulary
- 262,144 (~262k)
- License
- Gemma Terms of Use + Gemma Prohibited Use Policy
- Date
- 2025-08-14
- Decoder type
- Dense
- Attention
- Multi-query attention with QK-Norm and 5:1 sliding-window/global attention
- Layer mix
- 15 sliding-window + 3 global
- KV cache / token (bf16) info
- 18 KiB · Very low
- Key detail
- Keeps the Gemma 3 stack shape while shrinking down to 4 attention heads, a single KV head, and the same 262k vocabulary.
Training and benchmarks
- Pretraining data
- 6T tokens Pretraining budget for the 270M model. Gemma 3 270M model card
- Post-training methods
- Instruction tuning An IT checkpoint accompanies the pretrained model; its specific alignment stages are not disclosed. Gemma 3 270M release