Back to Kimi K3 in the gallery

Moonshot AI

2.8T sparse hybrid that combines a 3:1 KDA and Gated MLA stack with Block Attention Residuals and Stable LatentMoE. It is also native multimodal, but the figure focuses on the text portion.

Kimi K3 architecture diagram
Kimi K3 architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
2.8T total, 104B active (3.7% active)
Context (tokens)
1,048,576
License
Kimi K3 License
Date
2026-07-27
Decoder type
Sparse hybrid
Attention
3:1 Kimi Delta Attention and Gated MLA with NoPE
Layer mix
69 KDA + 24 Gated MLA
KV cache / token (bf16) info
27 KiB · Low
Key detail
Uses eight Block Attention Residual groups and Stable LatentMoE with 16 of 896 routed experts active per token.

Training and benchmarks

Pretraining data
Not disclosed The report describes native multimodal pretraining and context extension without publishing the total training tokens. Technical report
Post-training methods
SFT, RL, Distillation SFT is followed by domain- and effort-specialized RL, then multi-teacher on-policy distillation into one model. Technical report
AA Intelligence Index v5.0 info
Total score N/A Knowledge Work N/A Coding N/A Science 20.7 General N/A
AA Intelligence Index v4.3.2 info
Total score 43.6 General 45.8 Scientific 35.2 Coding 36.1 Agents 51.9

Further reading and tools