Back to Motif 3 Beta in the gallery

Motif Technologies

Large sparse MoE that combines Grouped Differential Latent Attention, per-expert PolyNorm, and modified mHC in a 53-layer decoder.

Motif 3 Beta architecture diagram
Motif 3 Beta architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
314B total, 13B active (4.1% active)
Context (tokens)
262,144
Vocabulary
220,160 (~220k)
License
Motif Technologies Non-Commercial License
Date
2026-07-21
Decoder type
Sparse MoE
Attention
GDLA with MLA-style latent projections, RoPE, and 3:1 sliding-window/full attention
Layer mix
39 sliding-window + 14 full GDLA
KV cache / token (bf16) info
530 KiB · Very high
Key detail
Routes each token through 8 of 384 experts plus one shared expert, with per-expert PolyNorm and a 3:1 sliding-window/full GDLA pattern.

Training and benchmarks

Pretraining data
Not disclosed This is an intermediate beta checkpoint. The later Motif 3 report gives about 12.5T tokens for the final release, without a separate beta budget. Motif 3 Beta model card · Motif 3 final-release report
Post-training methods
Not disclosed The beta card gives no training recipe. The later final-release SFT and distillation pipeline is not explicitly attributed to this checkpoint. Motif 3 Beta model card · Motif 3 final-release report
AA Intelligence Index v4.3.2 info
Total score 32.3 General N/A Scientific 23.2 Coding N/A Agents N/A

Further reading and tools