Back to Llama 3.2 3B in the gallery

Meta

Small Llama baseline used to show how closely Nanbeige follows the mainstream compact decoder recipe.

Llama 3.2 3B architecture diagram
Llama 3.2 3B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
3B parameters
Context (tokens)
131,072
License
Llama 3.2 Community License Agreement
Date
2024-09-25
Decoder type
Dense
Attention
GQA
Layer mix
28 GQA
KV cache / token (bf16) info
112 KiB · Moderate
Key detail
Reference small-model Llama architecture with tied embeddings.

Training and benchmarks

Pretraining data
Up to 9T tokens Pretraining-only figure reported for the 1B and 3B models. Pruning and teacher-logit distillation are part of their pretraining. Model card
Post-training methods
SFT, DPO The Instruct variant uses repeated SFT, rejection sampling, and DPO. These alignment stages apply to the instruct counterpart of the gallery's base configuration. Model card
AA Intelligence Index v4.3.2 info
Total score 5.7

Further reading and tools