Back to Dory1.0 4B in the gallery

NVIDIA

NVIDIA's dense looped transformer applies a shared 20-block core twice between 8-block input and output stacks, using normalized GPT (nGPT) residual updates.

Dory1.0 4B architecture diagram
Dory1.0 4B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
4B parameters
Context (tokens)
262,144
Vocabulary
131,072 (~131k)
License
OpenMDW 1.1
Date
2026-10-16
Decoder type
Dense
Attention
Gated GQA; 3:1 sliding-window/full, RoPE
Layer mix
8 input + 20 shared × 2 passes + 8 output (56 applications)
KV cache / token (bf16) info
224 KiB · High
Key detail
Reinjects the input-stack output before each recurrent pass. The 36 distinct blocks give 56 block applications by default, with L2-normalized residual updates and a 1,024-token attention window.

Training and benchmarks

Pretraining data
6T tokens + 300B context-extension tokens The model card reports pretraining and context extension separately. Dory 1.0 model card
Post-training methods
SFT, RL, Distillation SFT uses teacher-generated data, followed by multi-environment GRPO for reasoning, coding, tool use, and instruction following. Dory 1.0 model card

Further reading and tools