Back to Nemotron 3.5 Super VL 120B-A12B in the gallery

NVIDIA

NVIDIA's vision-language model extends the Nemotron 3 Super backbone with C-RADIO vision encoding and shared-weight multi-token prediction. The figure shows the language backbone, with the vision encoder and MTP omitted.

Nemotron 3.5 Super VL 120B-A12B architecture diagram
Nemotron 3.5 Super VL 120B-A12B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
120B total, 12B active (10% active)
Context (tokens)
1,048,576
Vocabulary
131,072
License
OpenMDW 1.1
Date
2026-10-15
Decoder type
Hybrid MoE
Attention
Mostly Mamba-2 with eight GQA layers using NoPE
Layer mix
8 GQA + 40 Mamba-2 + 40 LatentMoE; GQA + MoE MTP predictor
KV cache / token (bf16) info
8 KiB · Very low
Key detail
Routes each token to 22 of 512 experts in a 1,024-dimensional latent space, alongside one shared expert at the full 4,096-dimensional width.

Training and benchmarks

Pretraining data
1.2094T additional tokens Continued from Nemotron 3 Super Base with 1T distillation, 100B Indic, 35B context-extension, 65.4B agentic mid-training, and 9B context-recovery tokens. Separate 100B MTP-head warmup freezes the backbone. Nemotron 3.5 technical report
Post-training methods
SFT, RLVR, RLHF, Distillation Vision-language SFT, verifiable-reward RL, human-feedback RL, and multi-teacher on-policy distillation. Nemotron 3.5 technical report
AA Intelligence Index v5.0 info
Total score 13.6 Knowledge Work 19.5 Coding 3.7 Science 5.4 General 25.9
AA Intelligence Index v4.3.2 info
Total score 26.5 General 32.6 Scientific 20.6 Coding 23.9 Agents 26.0

Further reading and tools