The main difference between dense Qwen variants and MoE-style Qwen variants is how the feed-forward computation is used at each token.

In a dense model, every token goes through the same feed-forward block in every layer. That makes the execution pattern simple and predictable.

In an MoE model, the feed-forward block is replaced by multiple experts, and a router activates only some of them per token.

The Qwen overview in the repo shows that the family includes both dense and MoE-style variants rather than one single architecture form

So the tradeoff is:

  • dense Qwen: simpler execution, easier reasoning about latency, all parameters active in the dense path
  • MoE Qwen: much larger total capacity, but only a subset of experts active per token

This is why MoE variants can have very large total parameter counts without paying the full dense-compute cost on every step.

The MoE figure in the repo explains the underlying mechanism: many experts are stored, but routing keeps token-level compute sparse

Dense variants are often easier to deploy and reason about. MoE variants can offer a better quality-efficiency tradeoff at larger scale, but they come with more routing complexity and implementation detail.

So the difference is not that MoE Qwen stops being a transformer. It is that the feed-forward part of the transformer becomes sparse and routed instead of fully dense.

In short, dense Qwen variants use the same full feed-forward path for every token, while MoE-style variants route tokens through a subset of experts, trading simpler execution for larger total capacity and sparse compute.