Back to Tencent Hy4-preview 770B-A49B in the gallery

Tencent

Tencent's 770B sparse MoE combines gated DeepSeek Sparse Attention, cross-layer index reuse, and four-stream identity Hyper-Connections.

Tencent Hy4-preview 770B-A49B architecture diagram
Tencent Hy4-preview 770B-A49B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
770B backbone, 49B active (6.4% active); plus 10B MTP
Context (tokens)
1,048,576
Vocabulary
120,832 (~121k)
License
Apache License 2.0
Date
2026-08-28
Decoder type
Sparse MoE
Attention
Gated MLA with DeepSeek Sparse Attention and IndexCache
Layer mix
78 gated MLA/DSA + 1 MTP layer
KV cache / token (bf16) info
93 KiB · Moderate
Key detail
Uses 256 routed experts with 8 routed plus 1 shared expert active per token, a top-2,048 DSA indexer, cross-layer index reuse, and four iHC residual streams.

Training and benchmarks

Pretraining data
Not disclosed The release card says training data was scaled up but does not give a token count. Model card
Post-training methods
RL Core contributor Chenchen Zhang reports reinforcement-learning work on Hy4 preview. The full post-training recipe and reward types are not disclosed. Core contributor's project description · Model card

Further reading and tools