Back to Nemotron 3 Nano 30B-A3B in the gallery

NVIDIA

NVIDIA's Nano model is the most extreme transformer-state-space hybrid in the gallery.

Nemotron 3 Nano 30B-A3B architecture diagram
Nemotron 3 Nano 30B-A3B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
30B total, 3B active (10% active)
Context (tokens)
1,000,000
License
NVIDIA Nemotron Open Model License
Date
2025-12-04
Decoder type
Hybrid MoE
Attention
Mostly Mamba-2 with a few GQA layers using NoPE
Layer mix
6 GQA + 23 Mamba-2 + 23 MoE
KV cache / token (bf16) info
6 KiB · Very low
Key detail
Interleaves Mamba-2 and MoE blocks, using attention only sparingly.

Training and benchmarks

Pretraining data
25T + 121B long-context tokens The main pretraining run uses 25T tokens, followed by 121B tokens of long-context continued pretraining. Nemotron 3 Nano technical report
Post-training methods
SFT, RLVR, RLHF Supervised fine-tuning, multi-environment reinforcement learning with verifiable rewards, and human-feedback alignment. Nemotron 3 Nano technical report
AA Intelligence Index v4.3.2 info
Total score 6.8 General N/A Scientific 2.3 Coding N/A Agents N/A

Further reading and tools