Nemotron 3.5 Lightning 30B-A3B Architecture
Back to Nemotron 3.5 Lightning 30B-A3B in the gallery
NVIDIA
NVIDIA's 30B hybrid model keeps 3B parameters active and extends the Nemotron 3 Nano backbone with a 1M-token context window and multi-token prediction.
Architecture and specifications
- Scale
- 30B total, 3B active (10% active)
- Context (tokens)
- 1,048,576
- Vocabulary
- 131,072
- License
- OpenMDW 1.1
- Date
- 2026-08-11
- Decoder type
- Hybrid MoE
- Attention
- Mostly Mamba-2 with six GQA layers using NoPE
- Layer mix
- 6 GQA + 23 Mamba-2 + 23 MoE; GQA + MoE MTP predictor
- KV cache / token (bf16) info
- 6 KiB · Very low
- Key detail
- Adds a shared GQA-plus-MoE MTP predictor to the 52-layer Nemotron 3 Nano backbone.
Training and benchmarks
- Pretraining data
- 1.021T additional tokens Continued from Nemotron 3 Nano Base with 1T distillation tokens and 21B long-context tokens. A separate 100B-token MTP-head warmup freezes the backbone. Nemotron 3.5 technical report
- Post-training methods
- SFT, RLVR Single-stage text-only supervised fine-tuning followed by reinforcement learning with verifiable rewards. Nemotron 3.5 technical report