Llama 3.2 3B Architecture
Back to Llama 3.2 3B in the gallery
Meta
Small Llama baseline used to show how closely Nanbeige follows the mainstream compact decoder recipe.
Architecture and specifications
- Scale
- 3B parameters
- Context (tokens)
- 131,072
- License
- Llama 3.2 Community License Agreement
- Date
- 2024-09-25
- Decoder type
- Dense
- Attention
- GQA
- Layer mix
- 28 GQA
- KV cache / token (bf16) info
- 112 KiB · Moderate
- Key detail
- Reference small-model Llama architecture with tied embeddings.
Training and benchmarks
- Pretraining data
- Up to 9T tokens Pretraining-only figure reported for the 1B and 3B models. Pruning and teacher-logit distillation are part of their pretraining. Model card
- Post-training methods
- SFT, DPO The Instruct variant uses repeated SFT, rejection sampling, and DPO. These alignment stages apply to the instruct counterpart of the gallery's base configuration. Model card