Llama 4 Maverick Architecture
Back to Llama 4 Maverick in the gallery
Meta
Meta's large MoE follows the DeepSeek V3 playbook but with a more conventional attention stack.
Architecture and specifications
- Scale
- 400B total, 17B active (4.3% active)
- Context (tokens)
- 1,000,000
- License
- Llama 4 Community License Agreement
- Date
- 2025-04-05
- Decoder type
- Sparse MoE
- Attention
- GQA with RoPE in chunked layers and NoPE in full-attention layers
- Layer mix
- 36 chunked + 12 full GQA
- KV cache / token (bf16) info
- 192 KiB · High
- Key detail
- Alternates dense and MoE blocks and uses fewer, larger experts than DeepSeek V3.
Training and benchmarks
- Pretraining data
- ~22T tokens Pretraining-token count reported for Maverick, including its multimodal training data. Model card
- Post-training methods
- SFT, RL, DPO The Maverick Instruct recipe uses SFT, online RL, and a final lightweight DPO stage. Release post