Back to Solar Open 2 in the gallery

Upstage

Upstage's sparse hybrid alternates one gated GQA layer with three Kimi Delta Attention layers and reaches a 1M-token context without positional embeddings.

Solar Open 2 architecture diagram
Solar Open 2 architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
250B total, 15B active (6% active)
Context (tokens)
1,048,576
Vocabulary
196,608 (~197k)
License
Upstage Solar License
Date
2026-07-22
Decoder type
Sparse hybrid
Attention
3:1 Kimi Delta Attention and gated GQA with NoPE
Layer mix
12 gated GQA + 36 Kimi Delta Attention
KV cache / token (bf16) info
48 KiB · Low
Key detail
Uses a repeating gated-GQA/KDA hybrid stack, kernel-4 ShortConv in KDA, and top-8 routing over 320 experts plus one shared expert.

Training and benchmarks

Pretraining data
~12T additional tokens After selective weight transfer from Solar Open 1, the stages consume 10T + 1T + 0.9T tokens. Technical report
Post-training methods
SFT, RLVR, Distillation SFT and multi-domain verifiable RL are followed by specialist training and multi-teacher on-policy distillation. Technical report
AA Intelligence Index v4.3.2 info
Total score 24.7 General N/A Scientific 17.1 Coding N/A Agents N/A

Further reading and tools