Back to Llama 3.2 1B in the gallery

Meta

Small dense Llama baseline in the Qwen comparison, with fewer layers but more width.

Llama 3.2 1B architecture diagram
Llama 3.2 1B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
1B parameters
Context (tokens)
131,072
License
Llama Community License Agreement (variant-specific)
Date
2024-09-25
Decoder type
Dense
Attention
GQA
Layer mix
16 GQA
KV cache / token (bf16) info
32 KiB · Low
Key detail
Wider architecture with more heads than Qwen3 0.6B.

Training and benchmarks

Pretraining data
Up to 9T tokens Pretraining-only figure reported for the 1B and 3B models. Pruning and teacher-logit distillation are part of their pretraining. Model card
Post-training methods
SFT, DPO The Instruct variant uses repeated SFT, rejection sampling, and DPO. These alignment stages apply to the instruct counterpart of the gallery's base configuration. Model card
AA Intelligence Index v4.3.2 info
Total score 4.8 General N/A Scientific 2.7 Coding N/A Agents N/A

Further reading and tools