Back to Step 3.5 Flash 196B in the gallery

StepFun

Throughput-oriented MoE model that stays competitive with much larger DeepSeek-style systems.

Step 3.5 Flash 196B architecture diagram
Step 3.5 Flash 196B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
196B total, 11B active (5.6% active)
Context (tokens)
262,144
License
Apache License 2.0
Date
2026-02-01
Decoder type
Sparse MoE
Attention
GQA with 96 sliding-window and 64 global query heads, and 8 KV heads in each layer
Layer mix
33 sliding-window + 12 global (+ 3 optional MTP layers)
KV cache / token (bf16) info
180 KiB · High
Key detail
Uses MTP-3 during both training and inference for unusually high throughput.

Training and benchmarks

Pretraining data
~17.6T tokens + 750B mid-training tokens Section 4.2 gives 14.6T + 3T pretraining tokens. The introduction's 17.2T figure conflicts with that detailed schedule. Technical report
Post-training methods
SFT, RLVR, RLHF, Distillation Two-stage SFT, domain RL with verifiable and human-preference rewards, and self-distillation. Technical report
AA Intelligence Index v4.3.2 info
Total score 17.0 General N/A Scientific 13.4 Coding N/A Agents N/A

Further reading and tools