VibeThinker-3B Architecture
Back to VibeThinker-3B in the gallery
Qwen2.5-Coder-based dense reasoner whose gains mainly come from verifier-driven post-training rather than a new block design.
Architecture and specifications
- Scale
- 3B parameters
- Context (tokens)
- 131,072
- Vocabulary
- 151,936 (~152k)
- License
- MIT License
- Date
- 2026-06-15
- Decoder type
- Dense
- Attention
- 8:1 GQA with RoPE
- Layer mix
- 36 GQA
- KV cache / token (bf16) info
- 36 KiB · Low
- Key detail
- Standard Qwen2.5-Coder-style dense stack strengthened by curriculum SFT, MGPO RL, Long2Short RL, offline self-distillation, and Instruct RL.
Training and benchmarks
- Pretraining data
- 5.5T tokens (base-model coding training) Inherited Qwen2.5-Coder-3B budget; VibeThinker adds post-training rather than a new reported pretraining run. Technical report · Model card
- Post-training methods
- SFT, RLVR, Distillation, RL Two-stage SFT, verifiable reasoning RL, offline self-distillation, then instruction RL with rubric rewards. Technical report