Tencent Hy3-preview 295B-A21B Architecture
Back to Tencent Hy3-preview 295B-A21B in the gallery
Tencent
Tencent Hunyuan's 295B sparse MoE preview combines a dense first FFN layer, GQA with QK-Norm, and one extra MTP layer for faster generation.
Architecture and specifications
- Scale
- 295B total, 21B active (7.1% active)
- Context (tokens)
- 262,144
- License
- Tencent Hy Community License Agreement
- Date
- 2026-04-23
- Decoder type
- Sparse MoE
- Attention
- GQA with QK-Norm
- Layer mix
- 80 GQA + 1 MTP layer
- KV cache / token (bf16) info
- 320 KiB · Very high
- Key detail
- Uses 192 routed experts, 8 routed plus 1 shared expert active per token, sigmoid routing with expert bias, and a 3.8B MTP layer.
Training and benchmarks
- Pretraining data
- Not disclosed The model card and release post describe pretraining without a token count. Model card · Release post
- Post-training methods
- RL Tencent reports reinforcement learning for coding and agentic tasks; the full alignment recipe and reward types are not specified. Model card