Qwen3 4B Architecture
Back to Qwen3 4B in the gallery
Alibaba Cloud
Mid-size dense Qwen3 model used here as a clean baseline against SmolLM3 and Tiny Aya.
Architecture and specifications
- Scale
- 4B parameters
- Context (tokens)
- 32,768 native (131,072 with YaRN)
- License
- Apache License 2.0
- Date
- 2025-04-28
- Decoder type
- Dense
- Attention
- GQA with QK-Norm
- Layer mix
- 36 GQA
- KV cache / token (bf16) info
- 144 KiB · Moderate
- Key detail
- Compact Qwen3 dense stack with QK-Norm and a 151k vocabulary.
Training and benchmarks
- Pretraining data
- ≈36T tokens Family pretraining budget; excludes post-training. Qwen3 technical report
- Post-training methods
- SFT, Distillation Instruction/reasoning counterpart uses response-supervised and on-policy teacher distillation. Base checkpoints have no post-training. Qwen3 technical report