SmolLM3 3B Architecture
Back to SmolLM3 3B in the gallery
Hugging Face
Compact dense model that experiments with leaving out positional encodings in selected layers.
Architecture and specifications
- Scale
- 3B parameters
- Context (tokens)
- 131,072
- License
- Apache License 2.0
- Date
- 2025-06-19
- Decoder type
- Dense
- Attention
- GQA with periodic NoPE layers
- Layer mix
- 36 GQA
- KV cache / token (bf16) info
- 72 KiB · Low
- Key detail
- Every fourth layer omits RoPE to test a NoPE-style cadence.
Training and benchmarks
- Pretraining data
- 11.2T tokens Reported pretraining total; reasoning mid-training adds 140B tokens separately. Model card
- Post-training methods
- SFT, APO For SmolLM3-3B Instruct, SFT is followed by Anchored Preference Optimization and checkpoint merging. The linked Base checkpoint precedes these stages. Training recipe