Back to Nemotron 3 Nano 4B in the gallery

NVIDIA

Compact on-device hybrid that compresses Nemotron Nano 9B v2 into a mostly Mamba-2 stack with only four attention layers.

Nemotron 3 Nano 4B architecture diagram
Nemotron 3 Nano 4B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
4B parameters
Context (tokens)
262,144
License
NVIDIA Nemotron Open Model License
Date
2026-03-16
Decoder type
Dense hybrid
Attention
GQA with NoPE in 4 attention layers
Layer mix
4 GQA + 21 Mamba-2 + 17 FFN
KV cache / token (bf16) info
16 KiB · Very low
Key detail
Its 17 feed-forward blocks apply an up-projection, squared ReLU, and a down-projection. The separate Mamba-2 blocks have their own gate.

Training and benchmarks

Pretraining data
213B recovery-distillation tokens Compressed from Nano 9B v2, descended from a 20T-token pretrained 12B model. The 4B recovery stages use 63B and 150B tokens; this is additional recovery training, not a from-scratch pretraining total. NVIDIA Nano 4B training details · NVIDIA Nemotron Nano 2 technical report
Post-training methods
Distillation, SFT, RLVR Compression recovery distillation, two SFT stages, and three RLVR stages for instruction following and tool use. NVIDIA Nano 4B training details
AA Intelligence Index v5.0 info
Total score 1.8 Knowledge Work 0.2 Coding 0.0 Science 0.0 General 7.1
AA Intelligence Index v4.3.2 info
Total score 4.8 General 8.8 Scientific 2.4 Coding 8.0 Agents 0.1

Further reading and tools