Back to OLMo 3 7B in the gallery

Ai2

New transparent Allen AI model that keeps OLMo's post-norm flavor while modernizing context handling.

OLMo 3 7B architecture diagram
OLMo 3 7B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
7B parameters
Context (tokens)
65,536
License
Apache License 2.0
Date
2025-11-20
Decoder type
Dense
Attention
MHA with QK-Norm and 3:1 sliding-window/global attention
Layer mix
24 sliding-window + 8 global
KV cache / token (bf16) info
512 KiB · Very high
Key detail
Retains post-norm, keeps MHA, and applies YaRN only on global layers.

Training and benchmarks

Pretraining data
6.08T tokens 5.93T initial pretraining, 100B mid-training, and 50B context-extension tokens. The 7B model uses no branch merging. OLMo 3 7B base model card
Post-training methods
SFT, DPO, RLVR The released Think and Instruct counterparts use SFT, DPO, and RLVR. The Base checkpoint itself has no post-training. OLMo 3 7B base model card
AA Intelligence Index v4.3.2 info
Total score 5.2 General N/A Scientific 2.9 Coding N/A Agents N/A

Further reading and tools