OLMo 2 7B Architecture
Back to OLMo 2 7B in the gallery
Ai2
Transparent dense model that keeps classic MHA and pushes normalization changes for training stability.
Architecture and specifications
- Scale
- 7B parameters
- Context (tokens)
- 4,096
- License
- Apache License 2.0
- Date
- 2024-11-25
- Decoder type
- Dense
- Attention
- MHA with QK-Norm
- Layer mix
- 32 MHA
- KV cache / token (bf16) info
- 512 KiB · Very high
- Key detail
- Uses inside-residual post-norm instead of the usual pre-norm layout.
Training and benchmarks
- Pretraining data
- 4.05T tokens per training branch 4T initial pretraining tokens, then three separate 50B-token mid-training branches whose weights are averaged. OLMo 2 technical report
- Post-training methods
- SFT, DPO, RLVR The 7B Instruct release follows the Tulu 3 pipeline of supervised fine-tuning, direct preference optimization, and verifiable-reward RL. OLMo 2 technical report