Back to OLMo 3 32B in the gallery

Ai2

Scaled-up OLMo 3 keeps the same block design but moves to grouped-query attention.

OLMo 3 32B architecture diagram
OLMo 3 32B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
32B parameters
Context (tokens)
65,536
License
Apache License 2.0
Date
2025-11-20
Decoder type
Dense
Attention
GQA with QK-Norm and 3:1 sliding-window/global attention
Layer mix
48 sliding-window + 16 global
KV cache / token (bf16) info
256 KiB · High
Key detail
Keeps post-norm while scaling width and applying YaRN only on global layers.

Training and benchmarks

Pretraining data
5.7T tokens per training branch 5.5T initial pretraining tokens, two 100B-token mid-training branches merged together, then 100B context-extension tokens. Aggregate training across both branches is 5.8T. OLMo 3 32B base model card
Post-training methods
SFT, DPO, RLVR The 32B Think release uses supervised fine-tuning, direct preference optimization, and verifiable-reward RL. OLMo 3 32B Think model card
AA Intelligence Index v4.3.2 info
Total score 6.5 General N/A Scientific 3.2 Coding N/A Agents N/A

Further reading and tools