Nanbeige 4.2 3B uses looped depth sharing. This basically means it runs the same 22-layer (=transformer block) stack twice. So, it extends the 22-layer architecture to 44 layers, but without duplicating the weights. (2x the transformer block compute but the same memory footprint for the transformer weights.)

Why? The info is a bit sparse, but section 2.1 of the Nanbeige 4.2 technical report says two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. More passes gave barely any gains but made the training much slower and much more expensive.

Anyway, this is the architecture detail I found most interesting in Nanbeige 4.2.

Architecture gallery Nanbeige 4.2 config Technical report

Nanbeige 4.2 sends the hidden states through the same 22-layer stack twice. The green return path shows the second pass through the shared weights.

What is shared

The transformer-layer weights are reused across passes

Effective depth

22 distinct layers become 44 block applications in Nanbeige 4.2

Example architectures

Nanbeige 4.2 3B

Sources

Nanbeige 4.2 technical report Nanbeige 4.2 configuration

Back to architecture gallery