Qwen3 8B Architecture
Back to Qwen3 8B in the gallery
Alibaba Cloud
Dense Qwen3 baseline used here to show how little OLMo 3 changed the overall decoder recipe.
Architecture and specifications
- Scale
- 8B parameters
- Context (tokens)
- 32,768 native (131,072 with YaRN)
- License
- Apache License 2.0
- Date
- 2025-04-28
- Decoder type
- Dense
- Attention
- GQA with QK-Norm
- Layer mix
- 36 GQA
- KV cache / token (bf16) info
- 144 KiB · Moderate
- Key detail
- Reference Qwen3 dense stack with QK-Norm and 8 KV heads.
Training and benchmarks
- Pretraining data
- ~36T tokens Reported Qwen3 pretraining budget. Technical report
- Post-training methods
- SFT, Distillation The 8B model uses off-policy response distillation followed by on-policy teacher-logit alignment. Technical report