Grok 2.5 270B Architecture
Back to Grok 2.5 270B in the gallery
xAI
Rare production-model release that shows an older MoE style with fewer, larger experts.
Architecture and specifications
- Scale
- 270B parameters
- Context (tokens)
- 131,072
- License
- Grok 2 Community License Agreement
- Date
- 2025-08-22
- Decoder type
- Sparse MoE
- Attention
- GQA
- Layer mix
- 64 GQA
- KV cache / token (bf16) info
- 256 KiB · High
- Key detail
- Adds an always-on SwiGLU path that effectively behaves like a shared expert.
Training and benchmarks
- Pretraining data
- Not disclosed The released Grok 2 checkpoint does not publish its training-data size. Grok 2 model card
- Post-training methods
- Not disclosed The release is post-trained, but its model card does not specify SFT, DPO, or reward methods. Grok 2 model card