Llama 3.2 1B Architecture
Back to Llama 3.2 1B in the gallery
Meta
Small dense Llama baseline in the Qwen comparison, with fewer layers but more width.
Architecture and specifications
- Scale
- 1B parameters
- Context (tokens)
- 131,072
- License
- Llama Community License Agreement (variant-specific)
- Date
- 2024-09-25
- Decoder type
- Dense
- Attention
- GQA
- Layer mix
- 16 GQA
- KV cache / token (bf16) info
- 32 KiB · Low
- Key detail
- Wider architecture with more heads than Qwen3 0.6B.
Training and benchmarks
- Pretraining data
- Up to 9T tokens Pretraining-only figure reported for the 1B and 3B models. Pruning and teacher-logit distillation are part of their pretraining. Model card
- Post-training methods
- SFT, DPO The Instruct variant uses repeated SFT, rejection sampling, and DPO. These alignment stages apply to the instruct counterpart of the gallery's base configuration. Model card