Mistral Small 4 Architecture
Back to Mistral Small 4 in the gallery
Mistral AI
Multimodal Mistral Small refresh that jumps from the older dense 24B stack to an MLA-based sparse MoE design.
Architecture and specifications
- Scale
- 119B total, 6.63B active (5.6% active)
- Context (tokens)
- 262,144
- License
- Apache License 2.0
- Date
- 2026-03-16
- Decoder type
- Sparse MoE
- Attention
- MLA
- Layer mix
- 36 MLA
- KV cache / token (bf16) info
- 22.5 KiB · Very low
- Key detail
- Uses 128 experts with 4 routed plus 1 shared expert active per token while unifying instruct, reasoning, and vision.
Training and benchmarks
- Pretraining data
- Not disclosed The model card does not report the pretraining-token total. Mistral Small 4 model card
- Post-training methods
- Not disclosed The model card describes hybrid reasoning and instruction following but does not identify the post-training methods. Mistral Small 4 model card