Back to Muse Spark Open 816B-A42B in the gallery

Meta

Meta's multimodal MoE model combines 62 latent-expert blocks with gated GQA, local-only RoPE, global NoPE, and learned residual gates; the figure focuses on the text decoder.

Muse Spark Open 816B-A42B architecture diagram
Muse Spark Open 816B-A42B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
816B total, 42B active (5.1% active)
Context (tokens)
1,048,576
Vocabulary
202,048 (~202k)
License
Muse Model License
Date
2026-10-15
Decoder type
Sparse MoE
Attention
Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention
Layer mix
46 sliding-window + 16 global gated GQA
KV cache / token (bf16) info
248 KiB · High
Key detail
Selects 8 of 256 experts, compresses the expert path from 8,192 to 4,096 dimensions, normalizes each expert output before combining, and uses no shared expert.

Training and benchmarks

Pretraining data
Not disclosed The card lists multimodal data sources but does not provide a training-set size or token total. Muse Spark Open model card
Post-training methods
Not disclosed Additional safety post-training is confirmed, but the model card does not specify SFT, preference-optimization, or RL methods. Muse Spark Open model card
AA Intelligence Index v5.0 info
Total score 24.7 Knowledge Work 37.0 Coding 7.8 Science 12.1 General 41.7
AA Intelligence Index v4.3.2 info
Total score 38.9 General 45.0 Scientific 29.4 Coding 33.3 Agents 42.8

Further reading and tools