Solar Open 2 Architecture
Back to Solar Open 2 in the gallery
Upstage
Upstage's sparse hybrid alternates one gated GQA layer with three Kimi Delta Attention layers and reaches a 1M-token context without positional embeddings.
Architecture and specifications
- Scale
- 250B total, 15B active (6% active)
- Context (tokens)
- 1,048,576
- Vocabulary
- 196,608 (~197k)
- License
- Upstage Solar License
- Date
- 2026-07-22
- Decoder type
- Sparse hybrid
- Attention
- 3:1 Kimi Delta Attention and gated GQA with NoPE
- Layer mix
- 12 gated GQA + 36 Kimi Delta Attention
- KV cache / token (bf16) info
- 48 KiB · Low
- Key detail
- Uses a repeating gated-GQA/KDA hybrid stack, kernel-4 ShortConv in KDA, and top-8 routing over 320 experts plus one shared expert.
Training and benchmarks
- Pretraining data
- ~12T additional tokens After selective weight transfer from Solar Open 1, the stages consume 10T + 1T + 0.9T tokens. Technical report
- Post-training methods
- SFT, RLVR, Distillation SFT and multi-domain verifiable RL are followed by specialist training and multi-teacher on-policy distillation. Technical report