Soofi S 30B-A3B Architecture
Back to Soofi S 30B-A3B in the gallery
SOOFI Consortium
German-English base model that retrains the Nemotron 3 Nano architecture on 26.68T tokens with a fully documented data mixture.
Architecture and specifications
- Scale
- 31.6B total, 3.2B active (10.1% active)
- Context (tokens)
- 1,048,576
- License
- Custom license (preview)
- Date
- 2026-07-10
- Decoder type
- Hybrid MoE
- Attention
- Mostly Mamba-2 with 6 GQA layers and no positional embeddings (NoPE)
- Layer mix
- 6 GQA + 23 Mamba-2 + 23 MoE
- KV cache / token (bf16) info
- 6 KiB · Very low
- Key detail
- Reuses the Nemotron 3 Nano architecture without modification, isolating the effect of a German-English data and training recipe.
Training and benchmarks
- Pretraining data
- ~26.68T tokens Consumed tokens across pretraining, annealing, and context extension; the report rounds this to ~27T. Technical report
- Post-training methods
- None (base model) This entry is Soofi-S-Base. Instruction and reasoning post-training are described as future work. Technical report