Nemotron 3 Super 120B-A12B Architecture
Back to Nemotron 3 Super 120B-A12B in the gallery
NVIDIA
The Super variant scales up Nano and adds both latent experts and native speculative decoding support.
Architecture and specifications
- Scale
- 120B total, 12B active (10% active)
- Context (tokens)
- 1,000,000
- License
- NVIDIA Nemotron Open Model License
- Date
- 2026-03-11
- Decoder type
- Hybrid MoE
- Attention
- Mostly Mamba-2 with a few GQA layers using NoPE
- Layer mix
- 8 GQA + 40 Mamba-2 + 40 MoE
- KV cache / token (bf16) info
- 8 KiB · Very low
- Key detail
- Adds latent-space MoE and shared-weight MTP for fast inference.
Training and benchmarks
- Pretraining data
- 25T + 51B long-context tokens The main pretraining run uses 25T tokens. Two context-extension phases add 34B and 17B tokens. Nemotron 3 Super technical report
- Post-training methods
- SFT, RLVR, RLHF Supervised fine-tuning, multi-environment and software-engineering RL with verifiable rewards, then human-feedback alignment. Nemotron 3 Super technical report