OLMo 3 7B Architecture
Back to OLMo 3 7B in the gallery
Ai2
New transparent Allen AI model that keeps OLMo's post-norm flavor while modernizing context handling.
Architecture and specifications
- Scale
- 7B parameters
- Context (tokens)
- 65,536
- License
- Apache License 2.0
- Date
- 2025-11-20
- Decoder type
- Dense
- Attention
- MHA with QK-Norm and 3:1 sliding-window/global attention
- Layer mix
- 24 sliding-window + 8 global
- KV cache / token (bf16) info
- 512 KiB · Very high
- Key detail
- Retains post-norm, keeps MHA, and applies YaRN only on global layers.
Training and benchmarks
- Pretraining data
- 6.08T tokens 5.93T initial pretraining, 100B mid-training, and 50B context-extension tokens. The 7B model uses no branch merging. OLMo 3 7B base model card
- Post-training methods
- SFT, DPO, RLVR The released Think and Instruct counterparts use SFT, DPO, and RLVR. The Base checkpoint itself has no post-training. OLMo 3 7B base model card