Back to Gemma 3 270M in the gallery

Google

Tiny Gemma 3 variant that preserves the family's local-global attention recipe at a toy scale.

Gemma 3 270M architecture diagram
Gemma 3 270M architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
270M parameters
Context (tokens)
32,768
Vocabulary
262,144 (~262k)
License
Gemma Terms of Use + Gemma Prohibited Use Policy
Date
2025-08-14
Decoder type
Dense
Attention
Multi-query attention with QK-Norm and 5:1 sliding-window/global attention
Layer mix
15 sliding-window + 3 global
KV cache / token (bf16) info
18 KiB · Very low
Key detail
Keeps the Gemma 3 stack shape while shrinking down to 4 attention heads, a single KV head, and the same 262k vocabulary.

Training and benchmarks

Pretraining data
6T tokens Pretraining budget for the 270M model. Gemma 3 270M model card
Post-training methods
Instruction tuning An IT checkpoint accompanies the pretrained model; its specific alignment stages are not disclosed. Gemma 3 270M release
AA Intelligence Index v4.3.2 info
Total score 5.1 General N/A Scientific 1.8 Coding N/A Agents N/A

Further reading and tools