Sarvam 105B Architecture
Back to Sarvam 105B in the gallery
Sarvam AI
Larger Sarvam variant keeps the sparse MoE layout but switches from GQA to MLA.
Architecture and specifications
- Scale
- 105B total, 10.3B active (9.8% active)
- Context (tokens)
- 131,072
- License
- Apache License 2.0
- Date
- 2026-03-03
- Decoder type
- Sparse MoE
- Attention
- MLA with KV LayerNorm and NoPE + RoPE
- Layer mix
- 32 MLA
- KV cache / token (bf16) info
- 36 KiB · Low
- Key detail
- Large vocabulary and strong Indic language support carried into the larger MLA-based sparse MoE variant.
Training and benchmarks
- Pretraining data
- 12T tokens Reported pretraining budget. Training overview
- Post-training methods
- SFT, RLVR RL combines verifiable outcomes with rubric-based rewards, using asynchronous group-relative optimization. Training overview