Llama 3 8B Architecture
Back to Llama 3 8B in the gallery
Meta
Reference dense Llama stack used to contrast OLMo 2's normalization and attention choices.
Architecture and specifications
- Scale
- 8B parameters
- Context (tokens)
- 8,192
- License
- Llama 3 Community License Agreement
- Date
- 2024-04-18
- Decoder type
- Dense
- Attention
- GQA with RoPE
- Layer mix
- 32 GQA
- KV cache / token (bf16) info
- 128 KiB · Moderate
- Key detail
- Pre-norm baseline; wider than OLMo 2 at a similar scale.
Training and benchmarks
- Pretraining data
- >15T tokens Pretraining budget for the original April 2024 Llama 3 release. Model card
- Post-training methods
- SFT, DPO, RLHF The Instruct variant uses SFT, rejection sampling, PPO, and DPO with preference rankings. These stages apply to the instruct counterpart of the gallery's base configuration. Release post