I seriously think US open-weight models wouldn’t be this good without Chinese open-weight models and vice versa. And proprietary models wouldn’t be this good (and cheap) without open-weight models.

Competition is a good thing.

Timeline of US and Chinese LLMs from 2021 to 2026, with arrows tracing rotary embeddings, grouped-query attention, and other ideas between models

LLaMA uses rotary embeddings from RoFormer. In the other direction, DeepSeek LLM 67B uses grouped-query attention from Google, and so on. Click the figure to enlarge.

References

Here are the references for the connections shown in the figure above. (Note that the figure is not a comprehensive list, but just a selection.)

Model Which idea did it use?
LLaMA (2023) LLaMA uses RoFormer’s rotary position embeddings to implement the popular RoPE method, which encodes token positions by rotating queries and keys in each attention layer. (Also see my full article, Understanding Positional Embeddings in Modern LLMs)
DeepSeek LLM 67B (2023) Grouped-query attention, is now a standard technique where query heads share sets of keys and values. (For more info, see my GQA page)
OLMoE (2024) DeepSeekMoE’s popularized the fine-grained expert design, which OLMoE (and many others) adapted.
DeepSeek-V3 (2024) DeepSeek adapted Meta’s multi-token prediction objective. During training, an extra module predicts one token beyond the usual next-token target.
Kimi K2 (2025) DeepSeek-V2’s multi-head latent attention (MLA), popularized the MLA mechanism as alternative to GQA+sliding window approaches for KV-cache reduction. Here, keys and values share a compressed representation in the cache. See my MLA page for more details.
Qwen3-Next (2025) Qwen3-Next was one of the first architectures to use an attention hybrid design with the gated delta rule from Gated DeltaNet. Also see my page on Hybrid Attention.
Trinity Large (2026) Arcee’s Trinity uses a learned sigmoid gate from Qwen’s gated attention work to scale the attention output before the output projection.
Instella-MoE (2026) AMD’s Instella combines DeepSeek-style MLA with attention output gating. The input determines how much of the MLA output passes through.
Marin Mochi (2026) Open Athena uses Jianlin Su’s quantile balancing method. It bins the raw router scores to estimate how to adjust each expert’s routing bias.

Originally posted on Substack Notes.