Qwen3 30B-A3B Architecture
Back to Qwen3 30B-A3B in the gallery
Alibaba Cloud
Small sparse Qwen3 model that sits close to GPT-OSS in active size but uses a deeper stack.
Architecture and specifications
- Scale
- 30B total, 3B active (10% active)
- Context (tokens)
- 32,768 native (131,072 with YaRN)
- License
- Apache License 2.0
- Date
- 2025-04-28
- Decoder type
- Sparse MoE
- Attention
- GQA
- Layer mix
- 48 GQA
- KV cache / token (bf16) info
- 96 KiB · Moderate
- Key detail
- Deeper, narrower MoE alternative to GPT-OSS without a shared expert.
Training and benchmarks
- Pretraining data
- ≈36T tokens Family pretraining budget; excludes post-training. Qwen3 technical report
- Post-training methods
- SFT, Distillation Instruction/reasoning counterpart uses response-supervised and on-policy teacher distillation. Base checkpoints have no post-training. Qwen3 technical report