Nemotron 3 Nano 4B Architecture
Back to Nemotron 3 Nano 4B in the gallery
NVIDIA
Compact on-device hybrid that compresses Nemotron Nano 9B v2 into a mostly Mamba-2 stack with only four attention layers.
Architecture and specifications
- Scale
- 4B parameters
- Context (tokens)
- 262,144
- License
- NVIDIA Nemotron Open Model License
- Date
- 2026-03-16
- Decoder type
- Dense hybrid
- Attention
- GQA with NoPE in 4 attention layers
- Layer mix
- 4 GQA + 21 Mamba-2 + 17 FFN
- KV cache / token (bf16) info
- 16 KiB · Very low
- Key detail
- Its 17 feed-forward blocks apply an up-projection, squared ReLU, and a down-projection. The separate Mamba-2 blocks have their own gate.
Training and benchmarks
- Pretraining data
- 213B recovery-distillation tokens Compressed from Nano 9B v2, descended from a 20T-token pretrained 12B model. The 4B recovery stages use 63B and 150B tokens; this is additional recovery training, not a from-scratch pretraining total. NVIDIA Nano 4B training details · NVIDIA Nemotron Nano 2 technical report
- Post-training methods
- Distillation, SFT, RLVR Compression recovery distillation, two SFT stages, and three RLVR stages for instruction following and tool use. NVIDIA Nano 4B training details