Back to Qwen3 4B in the gallery

Alibaba Cloud

Mid-size dense Qwen3 model used here as a clean baseline against SmolLM3 and Tiny Aya.

Qwen3 4B architecture diagram
Qwen3 4B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
4B parameters
Context (tokens)
32,768 native (131,072 with YaRN)
License
Apache License 2.0
Date
2025-04-28
Decoder type
Dense
Attention
GQA with QK-Norm
Layer mix
36 GQA
KV cache / token (bf16) info
144 KiB · Moderate
Key detail
Compact Qwen3 dense stack with QK-Norm and a 151k vocabulary.

Training and benchmarks

Pretraining data
≈36T tokens Family pretraining budget; excludes post-training. Qwen3 technical report
Post-training methods
SFT, Distillation Instruction/reasoning counterpart uses response-supervised and on-policy teacher distillation. Base checkpoints have no post-training. Qwen3 technical report
AA Intelligence Index v4.3.2 info
Total score 6.6

Further reading and tools