Tencent Hy4-preview 770B-A49B Architecture
Back to Tencent Hy4-preview 770B-A49B in the gallery
Tencent
Tencent's 770B sparse MoE combines gated DeepSeek Sparse Attention, cross-layer index reuse, and four-stream identity Hyper-Connections.
Architecture and specifications
- Scale
- 770B backbone, 49B active (6.4% active); plus 10B MTP
- Context (tokens)
- 1,048,576
- Vocabulary
- 120,832 (~121k)
- License
- Apache License 2.0
- Date
- 2026-08-28
- Decoder type
- Sparse MoE
- Attention
- Gated MLA with DeepSeek Sparse Attention and IndexCache
- Layer mix
- 78 gated MLA/DSA + 1 MTP layer
- KV cache / token (bf16) info
- 93 KiB · Moderate
- Key detail
- Uses 256 routed experts with 8 routed plus 1 shared expert active per token, a top-2,048 DSA indexer, cross-layer index reuse, and four iHC residual streams.
Training and benchmarks
- Pretraining data
- Not disclosed The release card says training data was scaled up but does not give a token count. Model card
- Post-training methods
- RL Core contributor Chenchen Zhang reports reinforcement-learning work on Hy4 preview. The full post-training recipe and reward types are not disclosed. Core contributor's project description · Model card