Manifold-constrained hyper-connections (mHC) change the residual connections inside a transformer block. They replace the single residual stream with several parallel residual streams and learned mappings between them, then constrain those mappings to keep signal mixing stable.

This goes back to a research paper that the DeepSeek team shared on 31 Dec 2025. However, in this paper, the technique was only tested on an experimental 27B scale model. Now, we see it in their flagship release, which is a good sign that this idea actually works well in production.

The main idea behind mHC here is to modernize the design of the residual connections inside the transformer block, which is refreshing, because architecture tweaks are usually focused on the attention mechanism, normalization layer placement, and MoE parts.

Architecture gallery Article section mHC paper Hyper-connections paper

Figure 17. DeepSeek V4-Pro places mHC mixers around the attention and MoE sublayers. The model uses 4 parallel residual streams while keeping the attention and MoE sublayers at their normal hidden width (Original source Recent Developments in LLM Architectures).

From residual connections to hyper-connections

mHC is based on previous work on hyper-connections by Zhu et al. from 2024. Hyper-connections essentially modify the single residual stream inside the transformer block by replacing it with several parallel residual streams and learned mappings between them.

(For those new to residual connections, I made a video on residual neural networks many years ago, where I explained the general mechanism.)

The idea behind hyper-connections is to widen the residual stream. We can think of this as keeping several parallel residual streams, with an additional Res Mapping linear transformation that mixes them across layers. Since the attention or MoE layer itself still operates on the normal hidden size, hyper-connections also add a Pre Mapping that combines the parallel residual streams into one normal hidden vector for the layer, and a Post Mapping that distributes the layer output back across the parallel residual streams.

Figure 18. Regular transformer block compared with a transformer block using hyper-connections. Pre Mapping reads from the widened state, the sublayer runs at the normal width, and Post Mapping writes the output back into the widened state (Original source mHC: Manifold-Constrained Hyper-Connections).

The figure focuses on the attention-layer portion of the transformer block, but the same concept applies to the second residual branch around the MoE layer.

The purpose of hyper-connections is to make the residual pathway more expressive without making the actual attention or MoE layer wider. This is only mildly more expensive in FLOPs because the extra mappings operate over the small residual-stream axis, for example, n = 4 in DeepSeek V4, not over a huge hidden dimension.

In the original hyper-connections paper, the 7B OLMo MoE experiment goes from 13.36G to 13.38G FLOPs per token, which is basically unchanged. In terms of reported gains, there were modest (but consistent) improvements.

(However, only looking at FLOPs is a bit simplistic. The widened residual state still has to be stored, moved through memory, mixed, etc. So the practical overhead can come more from memory traffic and implementation complexity than from arithmetic, which is not explicitly measured.)

What the manifold constraint adds

The main change from regular hyper-connections (HC) to manifold-constrained hyper-connections (mHC) is that the mappings are no longer left unconstrained. In regular HC, the Res Mapping is a learned matrix that mixes the parallel residual streams, but stacking many such matrices can amplify or shrink signals unpredictably.

In mHC, this residual mapping is projected onto the manifold of doubly stochastic matrices, meaning all entries are non-negative and each row and column sums to 1. This makes the residual mixing behave more like a stable redistribution of information across streams. The Pre Mapping and Post Mapping are also constrained to be non-negative and bounded, which avoids cancellation when reading from and writing back into the widened residual state.

Figure 20. mHC keeps the parallel residual streams from hyper-connections but constrains the stream-mixing weights. The Res Mapping becomes doubly stochastic, while the Pre Mapping and Post Mapping are bounded and non-negative (Original source mHC: Manifold-Constrained Hyper-Connections).

In the mHC paper, using a 27B parameter model for the experiments, the DeepSeek team’s optimized implementation (with fusion, recomputation, and pipeline scheduling) adds only 6.7% additional training time overhead for 4 residual streams (n = 4) throughout all transformer blocks compared to the single-stream baseline.

To sum up, HC/mHC changes how information is carried around these layers by replacing the single residual stream with several interacting residual streams, with additional stability constraints in mHC. It pairs with the CSA/HCA attention changes, which modify other parts of the transformer block.

Sources

Recent Developments in LLM Architectures mHC paper Hyper-connections paper DeepSeek V4 technical report DeepSeek V4-Pro config.json

Back to architecture gallery