Back to Sarvam 105B in the gallery

Sarvam AI

Larger Sarvam variant keeps the sparse MoE layout but switches from GQA to MLA.

Sarvam 105B architecture diagram
Sarvam 105B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
105B total, 10.3B active (9.8% active)
Context (tokens)
131,072
License
Apache License 2.0
Date
2026-03-03
Decoder type
Sparse MoE
Attention
MLA with KV LayerNorm and NoPE + RoPE
Layer mix
32 MLA
KV cache / token (bf16) info
36 KiB · Low
Key detail
Large vocabulary and strong Indic language support carried into the larger MLA-based sparse MoE variant.

Training and benchmarks

Pretraining data
12T tokens Reported pretraining budget. Training overview
Post-training methods
SFT, RLVR RL combines verifiable outcomes with rubric-based rewards, using asynchronous group-relative optimization. Training overview
AA Intelligence Index v4.3.2 info
Total score 8.8 General N/A Scientific 5.5 Coding N/A Agents N/A

Further reading and tools