The US-China Open-Weight Feedback Loop
I seriously think US open-weight models wouldn’t be this good without Chinese open-weight models and vice versa. And proprietary models wouldn’t be this good (and cheap) without open-weight models.
Competition is a good thing.
References
Here are the references for the connections shown in the figure above. (Note that the figure is not a comprehensive list, but just a selection.)
| Model | Which idea did it use? |
|---|---|
| LLaMA (2023) | LLaMA uses RoFormer’s rotary position embeddings to implement the popular RoPE method, which encodes token positions by rotating queries and keys in each attention layer. (Also see my full article, Understanding Positional Embeddings in Modern LLMs) |
| DeepSeek LLM 67B (2023) | Grouped-query attention, is now a standard technique where query heads share sets of keys and values. (For more info, see my GQA page) |
| OLMoE (2024) | DeepSeekMoE’s popularized the fine-grained expert design, which OLMoE (and many others) adapted. |
| DeepSeek-V3 (2024) | DeepSeek adapted Meta’s multi-token prediction objective. During training, an extra module predicts one token beyond the usual next-token target. |
| Kimi K2 (2025) | DeepSeek-V2’s multi-head latent attention (MLA), popularized the MLA mechanism as alternative to GQA+sliding window approaches for KV-cache reduction. Here, keys and values share a compressed representation in the cache. See my MLA page for more details. |
| Qwen3-Next (2025) | Qwen3-Next was one of the first architectures to use an attention hybrid design with the gated delta rule from Gated DeltaNet. Also see my page on Hybrid Attention. |
| Trinity Large (2026) | Arcee’s Trinity uses a learned sigmoid gate from Qwen’s gated attention work to scale the attention output before the output projection. |
| Instella-MoE (2026) | AMD’s Instella combines DeepSeek-style MLA with attention output gating. The input determines how much of the MLA output passes through. |
| Marin Mochi (2026) | Open Athena uses Jianlin Su’s quantile balancing method. It bins the raw router scores to estimate how to adjust each expert’s routing bias. |
Originally posted on Substack Notes.
Read Next
Open October LLM Architecture Roundup
A roundup of ten open-weight LLM architecture releases from October 2026, covering attention mechanisms, hybrid layers, and MoE designs.
Implementing LLM Architectures From Scratch
Short note linking a talk on implementing LLM architectures from scratch and comparing new open-weight model implementations against references.
