OLMo 3 32B Architecture
Back to OLMo 3 32B in the gallery
Ai2
Scaled-up OLMo 3 keeps the same block design but moves to grouped-query attention.
Architecture and specifications
- Scale
- 32B parameters
- Context (tokens)
- 65,536
- License
- Apache License 2.0
- Date
- 2025-11-20
- Decoder type
- Dense
- Attention
- GQA with QK-Norm and 3:1 sliding-window/global attention
- Layer mix
- 48 sliding-window + 16 global
- KV cache / token (bf16) info
- 256 KiB · High
- Key detail
- Keeps post-norm while scaling width and applying YaRN only on global layers.
Training and benchmarks
- Pretraining data
- 5.7T tokens per training branch 5.5T initial pretraining tokens, two 100B-token mid-training branches merged together, then 100B context-extension tokens. Aggregate training across both branches is 5.8T. OLMo 3 32B base model card
- Post-training methods
- SFT, DPO, RLVR The 32B Think release uses supervised fine-tuning, direct preference optimization, and verifiable-reward RL. OLMo 3 32B Think model card