Looped Depth Sharing
Looped depth sharing increases a transformer’s effective depth by applying the same layer stack more than once. The attention, feed-forward, and normalization weights are reused on each pass. This adds computation without adding another set of transformer-layer parameters.
Nanbeige 4.2 uses 22 distinct transformer layers and runs them twice. It therefore performs 44 transformer-block applications while storing weights for only 22 layers.
What is shared
The transformer-layer weights are reused across passes
Effective depth
22 distinct layers become 44 block applications in Nanbeige 4.2
Example architectures
How The Loop Works
Let the 22-layer transformer stack be a function (F). A regular model applies it once:
hidden states -> F -> output
The looped model feeds the result through the same function again:
hidden states -> F -> F -> output
Both occurrences of (F) use the same parameters. However, they receive different hidden states. The first pass builds an intermediate representation, and the second pass processes that representation further.
This is weight sharing across depth. It differs from cross-layer KV sharing, which reuses key and value tensors to reduce cache growth.
Where The Idea Came From
The earliest clear Transformer architecture based on repeatedly applying shared layers was the Universal Transformer, introduced in 2018. It recurrently applies the same self-attentive transformation across computational depth. The authors also included an optional adaptive halting mechanism that lets different token positions use different amounts of computation.
The term looped Transformer became prominent later in the 2023 paper Looped Transformers as Programmable Computers. The work placed a fixed 13-layer Transformer in a loop and showed that it could execute iterative algorithms and emulate basic computer programs.
Nanbeige 4.2 uses the same general shared-depth principle for a language model, with two fixed passes through its 22-layer stack.
The Tradeoff
Looped depth sharing keeps the parameter count smaller than a model with 44 distinct layers, but it does not make the second pass free. Training and inference require additional computation. Nanbeige 4.2 also keeps separate KV-cache state for the two passes, so its cache scales with the 44 block applications rather than only the 22 distinct layers.
The Nanbeige 4.2 report (Section 2.1) says that two passes gave the best tradeoff in its experiments. The two-pass setup retained approximately 75% of the token efficiency of a standard transformer while providing a capacity gain. More passes produced only marginal improvements and made training slower and less stable.
The report does not provide the underlying benchmark table for this loop-depth study. It therefore supports the two-pass choice qualitatively, but does not isolate the exact gain from changing 22 applications into 44.
Sources