Step 3.5 Flash 196B Architecture
Back to Step 3.5 Flash 196B in the gallery
StepFun
Throughput-oriented MoE model that stays competitive with much larger DeepSeek-style systems.
Architecture and specifications
- Scale
- 196B total, 11B active (5.6% active)
- Context (tokens)
- 262,144
- License
- Apache License 2.0
- Date
- 2026-02-01
- Decoder type
- Sparse MoE
- Attention
- GQA with 96 sliding-window and 64 global query heads, and 8 KV heads in each layer
- Layer mix
- 33 sliding-window + 12 global (+ 3 optional MTP layers)
- KV cache / token (bf16) info
- 180 KiB · High
- Key detail
- Uses MTP-3 during both training and inference for unusually high throughput.
Training and benchmarks
- Pretraining data
- ~17.6T tokens + 750B mid-training tokens Section 4.2 gives 14.6T + 3T pretraining tokens. The introduction's 17.2T figure conflicts with that detailed schedule. Technical report
- Post-training methods
- SFT, RLVR, RLHF, Distillation Two-stage SFT, domain RL with verifiable and human-preference rewards, and self-distillation. Technical report