Nanbeige 4.1 3B Architecture
Back to Nanbeige 4.1 3B in the gallery
BOSS Zhipin
Small on-device oriented model that stays close to Llama 3.2 while nudging the scaling choices.
Architecture and specifications
- Scale
- 3B parameters
- Context (tokens)
- 262,144
- License
- Apache License 2.0
- Date
- 2026-02-10
- Decoder type
- Dense
- Attention
- GQA
- Layer mix
- 32 GQA
- KV cache / token (bf16) info
- 64 KiB · Low
- Key detail
- Llama-like stack without tying input embeddings to the output layer.
Training and benchmarks
- Pretraining data
- 23T tokens (inherited base) Uses Nanbeige4-3B-Base, whose four pretraining phases total 23T tokens. The 4.1 release adds post-training. Nanbeige 4 base model card · Nanbeige 4.1 technical report
- Post-training methods
- SFT, RLHF, RLVR SFT, human-preference reward-model RL, pairwise RL, and reasoning/code RL with executable correctness checks. Nanbeige 4.1 technical report