Back to Llama 3 8B in the gallery

Meta

Reference dense Llama stack used to contrast OLMo 2's normalization and attention choices.

Llama 3 8B architecture diagram
Llama 3 8B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
8B parameters
Context (tokens)
8,192
License
Llama 3 Community License Agreement
Date
2024-04-18
Decoder type
Dense
Attention
GQA with RoPE
Layer mix
32 GQA
KV cache / token (bf16) info
128 KiB · Moderate
Key detail
Pre-norm baseline; wider than OLMo 2 at a similar scale.

Training and benchmarks

Pretraining data
>15T tokens Pretraining budget for the original April 2024 Llama 3 release. Model card
Post-training methods
SFT, DPO, RLHF The Instruct variant uses SFT, rejection sampling, PPO, and DPO with preference rankings. These stages apply to the instruct counterpart of the gallery's base configuration. Release post
AA Intelligence Index v4.3.2 info
Total score 4.8 General N/A Scientific 2.5 Coding N/A Agents N/A

Further reading and tools