PolyNorm
PolyNorm is a trainable activation that mixes separately normalized linear, quadratic, and cubic transformations of the same preactivation. In Motif 3 Beta, it replaces the SiLU part of the usual SwiGLU-style feed-forward module.
The separate normalization step is important. Squaring and cubing can change activation magnitudes rapidly. PolyNorm RMS-normalizes each power before learning how strongly to mix the three terms.
Learned quantities
Three scalar mixture coefficients and one bias per activation instance
Where it sits
On the gate projection before elementwise multiplication with the up projection
Example architectures
From SwiGLU To PolyNorm
A gated feed-forward module starts with two projections of the same hidden state. Let g = W_gate h and u = W_up h. A SwiGLU-style module computes
output = W_down(SiLU(g) ⊙ u)
Motif keeps the same gate, up, and down projections but changes the activation on the gate branch:
output = W_down(s · PolyNorm(g) ⊙ u)
Here, s is an output scale. Motif 3 Beta sets it to 0.5. The elementwise product with u still provides the gating behavior, so PolyNorm changes the activation shape without changing the feed-forward matrix dimensions.
The Normalized Polynomial
The Motif implementation first defines an RMS normalization over the last tensor dimension:
norm(z) = z / sqrt(mean(z²) + ε)
It then mixes three normalized powers:
PolyNorm(x) = a₃ · norm(x³) + a₂ · norm(x²) + a₁ · norm(x) + b
The three a values and the bias b are learned. Normalizing each power separately prevents the cubic term from dominating only because its raw magnitude grows faster. The coefficients can instead learn the relative contribution of the linear, quadratic, and cubic shapes.
Motif applies a sigmoid to the three learned coefficient parameters, which constrains each effective a value to the interval from 0 to 1. It also clamps the learned bias to ±0.5 in the routed experts. These are Motif-specific stabilization choices rather than requirements of the general polynomial-composition idea.
Grouped PolyNorm In The MoE Layer
Motif 3 Beta has 384 routed experts and selects eight for each token. Each routed expert has an independent row of three PolyNorm coefficients plus one bias. This adds 1,536 learned scalars per MoE layer for the routed experts, which is tiny compared with their projection matrices.
The expert-specific coefficients let two experts use different activation shapes even when their feed-forward modules have the same structure. Only the selected experts run for a token, and a separate shared expert is evaluated alongside them.
Per-expert activations make the implementation more specialized. A grouped matrix-multiplication path must preserve the expert identity so it can select the matching PolyNorm parameters. The Motif model code includes an explicit per-expert path, while the official Motif activation package provides a fused grouped PolyNorm operation for CUDA.
What The PolyCom Paper Tested
The Polynomial Composition Activations paper introduced PolyNorm together with a related PolyReLU activation. The main experiments covered a dense 1B-parameter model and a sparse model with 7B total parameters and about 1B active parameters.
In those experiments, the polynomial activations reached lower validation loss and perplexity than the tested fixed activations and improved several downstream evaluation results. The paper’s order ablation found that third- and fourth-order variants converged similarly, making the cheaper third-order form a practical choice.
Motif adapts this third-order PolyNorm form for a much larger MoE model. Its per-expert parameter grouping, sigmoid-constrained coefficients, bias clamp, and output scaling are implementation choices visible in the released Motif model code.
Costs And Tradeoffs
PolyNorm adds only four learned scalars per activation instance, but parameter count is not the main cost. Computing three powers, three RMS normalizations, and their weighted sum requires more elementwise work and reductions than applying SiLU once.
Efficient inference therefore benefits from fusing the normalization, polynomial mixture, gating multiplication, and expert selection. Without a suitable fused kernel, the activation can introduce extra memory traffic or require a slower expert-dispatch path.
PolyNorm also does not reduce the large matrix multiplications in an MoE layer. It changes the activation function used inside each expert. Whether that added flexibility is worth the extra kernel complexity remains an empirical architecture choice.
Sources