Nemotron 3 Nano 30B-A3B Architecture
Back to Nemotron 3 Nano 30B-A3B in the gallery
NVIDIA
NVIDIA's Nano model is the most extreme transformer-state-space hybrid in the gallery.
Architecture and specifications
- Scale
- 30B total, 3B active (10% active)
- Context (tokens)
- 1,000,000
- License
- NVIDIA Nemotron Open Model License
- Date
- 2025-12-04
- Decoder type
- Hybrid MoE
- Attention
- Mostly Mamba-2 with a few GQA layers using NoPE
- Layer mix
- 6 GQA + 23 Mamba-2 + 23 MoE
- KV cache / token (bf16) info
- 6 KiB · Very low
- Key detail
- Interleaves Mamba-2 and MoE blocks, using attention only sparingly.
Training and benchmarks
- Pretraining data
- 25T + 121B long-context tokens The main pretraining run uses 25T tokens, followed by 121B tokens of long-context continued pretraining. Nemotron 3 Nano technical report
- Post-training methods
- SFT, RLVR, RLHF Supervised fine-tuning, multi-environment reinforcement learning with verifiable rewards, and human-feedback alignment. Nemotron 3 Nano technical report