Back to SmolLM3 3B in the gallery

Hugging Face

Compact dense model that experiments with leaving out positional encodings in selected layers.

SmolLM3 3B architecture diagram
SmolLM3 3B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
3B parameters
Context (tokens)
131,072
License
Apache License 2.0
Date
2025-06-19
Decoder type
Dense
Attention
GQA with periodic NoPE layers
Layer mix
36 GQA
KV cache / token (bf16) info
72 KiB · Low
Key detail
Every fourth layer omits RoPE to test a NoPE-style cadence.

Training and benchmarks

Pretraining data
11.2T tokens Reported pretraining total; reasoning mid-training adds 140B tokens separately. Model card
Post-training methods
SFT, APO For SmolLM3-3B Instruct, SFT is followed by Anchored Preference Optimization and checkpoint merging. The linked Base checkpoint precedes these stages. Training recipe

Further reading and tools