GLM-4.5 355B Architecture
Back to GLM-4.5 355B in the gallery
Z.ai
Agent-oriented instruction/reasoning hybrid that borrows DeepSeek's dense-prefix MoE layout.
Architecture and specifications
- Scale
- 355B total, 32B active (9% active)
- Context (tokens)
- 131,072
- License
- MIT License
- Date
- 2025-07-28
- Decoder type
- Sparse MoE
- Attention
- GQA with QK-Norm
- Layer mix
- 92 GQA
- KV cache / token (bf16) info
- 368 KiB · Very high
- Key detail
- Starts with three dense layers before MoE routing, keeps a shared expert, and uses MTP during training.
Training and benchmarks
- Pretraining data
- 23T tokens Combined pretraining and mid-training budget. GLM-4.5 report
- Post-training methods
- SFT, RLVR, RLHF, RLAIF, Distillation SFT and expert RL cover reasoning, agents, and chat; expert capabilities are distilled into the final model. GLM-4.5 report