Back to OLMo 2 7B in the gallery

Ai2

Transparent dense model that keeps classic MHA and pushes normalization changes for training stability.

OLMo 2 7B architecture diagram
OLMo 2 7B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
7B parameters
Context (tokens)
4,096
License
Apache License 2.0
Date
2024-11-25
Decoder type
Dense
Attention
MHA with QK-Norm
Layer mix
32 MHA
KV cache / token (bf16) info
512 KiB · Very high
Key detail
Uses inside-residual post-norm instead of the usual pre-norm layout.

Training and benchmarks

Pretraining data
4.05T tokens per training branch 4T initial pretraining tokens, then three separate 50B-token mid-training branches whose weights are averaged. OLMo 2 technical report
Post-training methods
SFT, DPO, RLVR The 7B Instruct release follows the Tulu 3 pipeline of supervised fine-tuning, direct preference optimization, and verifiable-reward RL. OLMo 2 technical report
AA Intelligence Index v4.3.2 info
Total score 5.6

Further reading and tools