Manifold-constrained hyper-connections (mHC) change the residual connections inside a transformer block. They replace the single residual stream with several parallel residual streams and learned mappings between them, then constrain those mappings to keep signal mixing stable.

This goes back to a research paper that the DeepSeek team shared on 31 Dec 2025. The experiments included models up to 27B parameters. DeepSeek V4-Pro and V4-Flash subsequently adopted mHC with four parallel residual streams.

The main idea behind mHC here is to modernize the design of the residual connections inside the transformer block, which is refreshing, because architecture tweaks are usually focused on the attention mechanism, normalization layer placement, and MoE parts.

Architecture gallery Gated Residuals Article section mHC paper Hyper-connections paper

Figure 1. DeepSeek V4-Pro uses four parallel residual streams around each attention and MoE sublayer. Pre-mixing forms the sublayer input before RMSNorm, while post-mixing distributes its output for addition to the mixed residual streams. A separate mHC head combines the four streams before final RMSNorm (Architecture gallery; V4 implementation).

From residual connections to hyper-connections

mHC is based on previous work on hyper-connections by Zhu et al. from 2024. Hyper-connections essentially modify the single residual stream inside the transformer block by replacing it with several parallel residual streams and learned mappings between them.

(For those new to residual connections, I made a video on residual neural networks many years ago, where I explained the general mechanism.)

The idea behind hyper-connections is to widen the residual stream. We can think of this as keeping several parallel residual streams, with an additional Res Mapping linear transformation that mixes them across layers. Since the attention or MoE layer itself still operates on the normal hidden size, hyper-connections also add a Pre Mapping that combines the parallel residual streams into one normal hidden vector for the layer, and a Post Mapping that distributes the layer output back across the parallel residual streams.

Figure 2. Regular transformer block compared with a transformer block using hyper-connections. Pre Mapping reads from the widened state, the sublayer runs at the normal width, and Post Mapping writes the output back into the widened state (Original source mHC: Manifold-Constrained Hyper-Connections).

The figure focuses on the attention-layer portion of the transformer block, but the same concept applies to the second residual branch around the MoE layer.

The purpose of hyper-connections is to make the residual pathway more expressive without making the actual attention or MoE layer wider. The extra mixing operates across the small residual-stream axis, for example, n = 4 in DeepSeek V4, for each hidden feature. Predicting the mixing weights also adds computation, but the attention and MoE sublayers still run at their normal hidden width.

In the original hyper-connections paper, the dense 7B OLMo experiment goes from 13.36G to 13.38G FLOPs per token, which is basically unchanged. In terms of reported gains, there were modest (but consistent) improvements.

(However, only looking at FLOPs is a bit simplistic. The widened residual state still has to be stored, moved through memory, and mixed. So the practical overhead also depends on memory traffic and the implementation.)

How Attention Residuals and mHC differ

Both methods change the residual path, but they do it in different directions.

• Attention Residuals: select and combine outputs from earlier depths.
• mHC: maintain and mix several residual streams at the current depth.

Attention Residuals use learned attention weights over earlier sublayer outputs for the same token. mHC keeps several residual streams at the current depth and constrains how those streams exchange information.

What the manifold constraint adds

The main change from regular hyper-connections (HC) to manifold-constrained hyper-connections (mHC) is that the mappings are no longer left unconstrained. In regular HC, the Res Mapping is a learned matrix that mixes the parallel residual streams, but stacking many such matrices can amplify or shrink signals unpredictably.

In mHC, this residual mapping is projected onto the manifold of doubly stochastic matrices, meaning all entries are non-negative and each row and column sums to 1. This makes the residual mixing behave more like a stable redistribution of information across streams. The Pre Mapping and Post Mapping are also constrained to be non-negative and bounded, which avoids cancellation when reading from and writing back into the widened residual state.

The released V4 configurations use 20 Sinkhorn iterations to approximate the doubly stochastic constraint. This constraint applies to the residual mixing matrix; the pre- and post-mixing weights use separate bounds.

Figure 3. mHC keeps the parallel residual streams from hyper-connections but constrains the stream-mixing weights. The Res Mapping becomes doubly stochastic, while the Pre Mapping and Post Mapping are bounded and non-negative (Original source mHC: Manifold-Constrained Hyper-Connections).

mHC in DeepSeek V4

Both V4-Pro and V4-Flash set hc_mult to 4. For each token, this gives four vectors with 7,168 entries each in Pro or 4,096 entries each in Flash. The input embedding is copied into the four streams before the first transformer block.

The mHC mappings perform the following operations.

Mapping	Streams	Operation
mHC pre	4 to 1	Forms a weighted sum of the input streams, which then passes through RMSNorm and the attention or MoE sublayer.
mHC res	4 to 4	Mixes the same input streams through a learned, constrained 4-by-4 matrix along the residual path.
mHC post	1 to 4	Scales and distributes the sublayer output across four streams for addition to the mixed residual input.

The V4 implementation computes the pre-, post-, and residual-mixing weights from the input to each sublayer. After the attention update, the resulting four streams become the input to both the MoE pre-mix and its residual branch.

After the last block, a separate learned mHC head combines the four streams into one vector. Final RMSNorm and the vocabulary projection follow this reduction. The head mapping sits outside the repeated transformer block.

Single-Pass mHC in DeepSeek V4.1

DeepSeek V4.1-Flash keeps four residual streams around every attention and MoE sublayer. Each stream has 5,120 entries per token. Its Single-Pass mHC changes when the pre-mixing weights are computed. A sublayer combines its current input streams using pre-mixing weights produced by the preceding sublayer. It computes its own residual- and post-mixing weights from the current input, along with the pre-mixing weights for the next sublayer.

This shift allows the deployment kernel to combine the residual update, input mixing, and coefficient prediction in one pass through the residual state. In the released implementation, the final four-to-one reduction uses the last sublayer’s pre-mixing weights, followed by final RMSNorm and the vocabulary projection.

Training overhead

In the mHC paper, the DeepSeek team’s optimized implementation (with fusion, recomputation, and pipeline scheduling) adds 6.7% training time overhead for 4 residual streams (n = 4) compared to the single-stream baseline. This is a result for their training setup; the widened residual state still affects memory use and data movement.

mHC changes the residual path around each sublayer. V4’s CSA/HCA attention determines how the attention sublayer reads the context. These mechanisms address separate parts of the transformer block.

Sources

Recent Developments in LLM Architectures mHC paper Hyper-connections paper DeepSeek V4 technical report DeepSeek V4-Pro config.json DeepSeek V4-Flash config.json DeepSeek V4 mHC implementation DeepSeek V4.1 technical report

Back to architecture gallery