Kimi K3 Architecture
Back to Kimi K3 in the gallery
Moonshot AI
2.8T sparse hybrid that combines a 3:1 KDA and Gated MLA stack with Block Attention Residuals and Stable LatentMoE. It is also native multimodal, but the figure focuses on the text portion.
Architecture and specifications
- Scale
- 2.8T total, 104B active (3.7% active)
- Context (tokens)
- 1,048,576
- License
- Kimi K3 License
- Date
- 2026-07-27
- Decoder type
- Sparse hybrid
- Attention
- 3:1 Kimi Delta Attention and Gated MLA with NoPE
- Layer mix
- 69 KDA + 24 Gated MLA
- KV cache / token (bf16) info
- 27 KiB · Low
- Key detail
- Uses eight Block Attention Residual groups and Stable LatentMoE with 16 of 896 routed experts active per token.
Training and benchmarks
- Pretraining data
- Not disclosed The report describes native multimodal pretraining and context extension without publishing the total training tokens. Technical report
- Post-training methods
- SFT, RL, Distillation SFT is followed by domain- and effort-specialized RL, then multi-teacher on-policy distillation into one model. Technical report
- AA Intelligence Index v5.0 info
- Total score N/A Knowledge Work N/A Coding N/A Science 20.7 General N/A