DeepSeek R1 Architecture
Back to DeepSeek R1 in the gallery
DeepSeek
Reasoning-tuned DeepSeek model built on the V3 architecture rather than a new base design.
Architecture and specifications
- Scale
- 671B total, 37B active (5.5% active)
- Context (tokens)
- 128,000
- License
- MIT License
- Date
- 2025-01-20
- Decoder type
- Sparse MoE
- Attention
- MLA
- Layer mix
- 61 MLA
- KV cache / token (bf16) info
- 68.6 KiB · Low
- Key detail
- Architecture matches DeepSeek V3, including the MTP-1 path; the main change is the reasoning-oriented training recipe.
Training and benchmarks
- Pretraining data
- 14.8T tokens (inherited base) R1 starts from DeepSeek V3 Base; this is the base pretraining count, excluding R1 post-training. DeepSeek V3 report · DeepSeek R1 report
- Post-training methods
- SFT, RLVR, RLHF Cold-start SFT, reasoning GRPO, rejection-sampling SFT, then RL with verifiable rewards and human-preference reward models. DeepSeek R1 report