Back to Qwen3 8B in the gallery

Alibaba Cloud

Dense Qwen3 baseline used here to show how little OLMo 3 changed the overall decoder recipe.

Qwen3 8B architecture diagram
Qwen3 8B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
8B parameters
Context (tokens)
32,768 native (131,072 with YaRN)
License
Apache License 2.0
Date
2025-04-28
Decoder type
Dense
Attention
GQA with QK-Norm
Layer mix
36 GQA
KV cache / token (bf16) info
144 KiB · Moderate
Key detail
Reference Qwen3 dense stack with QK-Norm and 8 KV heads.

Training and benchmarks

Pretraining data
~36T tokens Reported Qwen3 pretraining budget. Technical report
Post-training methods
SFT, Distillation The 8B model uses off-policy response distillation followed by on-policy teacher-logit alignment. Technical report
AA Intelligence Index v4.3.2 info
Total score 6.0 General N/A Scientific 1.0 Coding N/A Agents N/A

Further reading and tools