A curated reference for modern language-model architectures

Compare architecture diagrams, implementation links, release dates, attention mechanisms, decoder types, and compact fact sheets without digging through individual posts.

126 models

Nov 18 last updated

The gallery focuses on text-only LLMs and language-model backbones. For multimodal releases, cards usually describe the text decoder/backbone rather than every vision, audio, or image-generation component.

Corrections are welcome in the issue tracker. If the latest changes are missing, try a hard reload (macOS: Command + Shift + R; Windows: Ctrl + F5).

Your support helps me maintain and expand the gallery. The downloads below are a thank-you for paid Ahead of AI subscribers.

Architecture images

Download all diagrams and use the poster builder to make your own poster.

Download Images

Model metadata

Download model specs and info as YAML for your own analyses.

Download Metadata
Subscribe

Architecture diff tool

Select two models to compare their architectures

If you want to compare two architectures side by side instead of browsing the gallery, use this diff tool. You can use the selectors here or the Model A / Model B actions on each card.

GPT-2 XL (1.5B)

by OpenAI 🇺🇸

Late-2019 dense baseline included here as a reference point for how much decoder stacks have changed since GPT-2.

Scale
1.5B parameters
Context (tokens)
1,024
License
OpenAI "Modified MIT" license
Date
2019-11-05
Decoder type
Dense
Attention
MHA with learned absolute positional embeddings
Layer mix
48 MHA
KV cache / token (bf16) info
300 KiB · High
Key detail
Classic GPT-2 recipe with dropout, GELU, LayerNorm, and full multi-head attention.
Pretraining data
40 GB of text; over 8M documents WebText corpus size after filtering and deduplication. The report does not provide a total training-token count. GPT-2 report
Post-training methods
None (base model) Original pretrained language model; no instruction or preference post-training in this release. GPT-2 report

Related concepts

Compare

Reference dense Llama stack used to contrast OLMo 2's normalization and attention choices.

Scale
8B parameters
Context (tokens)
8,192
License
Llama 3 Community License Agreement
Date
2024-04-18
Decoder type
Dense
Attention
GQA with RoPE
Layer mix
32 GQA
KV cache / token (bf16) info
128 KiB · Moderate
Key detail
Pre-norm baseline; wider than OLMo 2 at a similar scale.
Pretraining data
>15T tokens Pretraining budget for the original April 2024 Llama 3 release. Model card
Post-training methods
SFT, DPO, RLHF The Instruct variant uses SFT, rejection sampling, PPO, and DPO with preference rankings. These stages apply to the instruct counterpart of the gallery's base configuration. Release post
AA Intelligence Index v4.3.2 info
Total score 4.8 General N/A Scientific 2.5 Coding N/A Agents N/A

Related concepts

Compare

Llama 3.2 (1B)

by Meta 🇺🇸

Small dense Llama baseline in the Qwen comparison, with fewer layers but more width.

Scale
1B parameters
Context (tokens)
128,000
License
Llama Community License Agreement (variant-specific)
Date
2024-09-25
Decoder type
Dense
Attention
GQA
Layer mix
16 GQA
KV cache / token (bf16) info
32 KiB · Low
Key detail
Wider architecture with more heads than Qwen3 0.6B.
Pretraining data
Up to 9T tokens Pretraining-only figure reported for the 1B and 3B models. Pruning and teacher-logit distillation are part of their pretraining. Model card
Post-training methods
SFT, DPO The Instruct variant uses repeated SFT, rejection sampling, and DPO. These alignment stages apply to the instruct counterpart of the gallery's base configuration. Model card
AA Intelligence Index v4.3.2 info
Total score 4.8 General N/A Scientific 2.7 Coding N/A Agents N/A

Related concepts

Compare

OLMo 2 (7B)

by Ai2 🇺🇸

Transparent dense model that keeps classic MHA and pushes normalization changes for training stability.

Scale
7B parameters
Context (tokens)
4,096
License
Apache License 2.0
Date
2024-11-25
Decoder type
Dense
Attention
MHA with QK-Norm
Layer mix
32 MHA
KV cache / token (bf16) info
512 KiB · Very high
Key detail
Uses inside-residual post-norm instead of the usual pre-norm layout.
Pretraining data
4.05T tokens per training branch 4T initial pretraining tokens, then three separate 50B-token mid-training branches whose weights are averaged. OLMo 2 technical report
Post-training methods
SFT, DPO, RLVR The 7B Instruct release follows the Tulu 3 pipeline of supervised fine-tuning, direct preference optimization, and verifiable-reward RL. OLMo 2 technical report
AA Intelligence Index v4.3.2 info
Total score 5.6

Related concepts

Compare

DeepSeek V3 (671B)

by DeepSeek 🇨🇳

DeepSeek's flagship template kicked off the recent wave of large open MoE models.

Scale
671B total, 37B active (5.5% active)
Context (tokens)
128,000
License
DeepSeek License Agreement v1.0
Date
2024-12-26
Decoder type
Sparse MoE
Attention
MLA
Layer mix
61 MLA
KV cache / token (bf16) info
68.6 KiB · Low
Key detail
Uses a dense prefix, shared expert, and MTP-1 objective to keep a very large model practical.
Pretraining data
14.8T tokens Base pretraining budget, before post-training. DeepSeek V3 report
Post-training methods
SFT, RLVR, RL, Distillation SFT includes R1-generated reasoning data. GRPO combines rule-based math/code rewards with learned reward models. DeepSeek V3 report
AA Intelligence Index v4.3.2 info
Total score 8.5 General 15.3 Scientific 1.5 Coding 17.9 Agents 0.1

Related concepts

Compare

DeepSeek R1 (671B)

by DeepSeek 🇨🇳

Reasoning-tuned DeepSeek model built on the V3 architecture rather than a new base design.

Scale
671B total, 37B active (5.5% active)
Context (tokens)
128,000
License
MIT License
Date
2025-01-20
Decoder type
Sparse MoE
Attention
MLA
Layer mix
61 MLA
KV cache / token (bf16) info
68.6 KiB · Low
Key detail
Architecture matches DeepSeek V3, including the MTP-1 path; the main change is the reasoning-oriented training recipe.
Pretraining data
14.8T tokens (inherited base) R1 starts from DeepSeek V3 Base; this is the base pretraining count, excluding R1 post-training. DeepSeek V3 report · DeepSeek R1 report
Post-training methods
SFT, RLVR, RLHF Cold-start SFT, reasoning GRPO, rejection-sampling SFT, then RL with verifiable rewards and human-preference reward models. DeepSeek R1 report
AA Intelligence Index v4.3.2 info
Total score 11.4 General 22.2 Scientific 4.5 Coding 19.2 Agents 0.0

Related concepts

Compare

Gemma's flagship text stack leans on local attention more aggressively than Gemma 2.

Scale
27B parameters
Context (tokens)
128,000
Vocabulary
262,144 (~262k)
License
Gemma Terms of Use + Gemma Prohibited Use Policy
Date
2025-03-11
Decoder type
Dense
Attention
GQA with QK-Norm and 5:1 sliding-window/global attention
Layer mix
52 sliding-window + 10 global
KV cache / token (bf16) info
496 KiB · Very high
Key detail
Built around a 27B sweet spot with heavier local attention and a large 262k multilingual vocabulary.
Pretraining data
14T tokens Includes text and image data; pretraining uses knowledge distillation. Gemma 3 report
Post-training methods
Distillation, RLHF, RLVR IT checkpoint uses teacher distillation and RL with human-feedback reward models, code execution, and math ground truth. Gemma 3 report
AA Intelligence Index v4.3.2 info
Total score 4.9 General 6.9 Scientific 2.2 Coding 11.6 Agents 0.0

Related concepts

Compare

Mistral Small 3.1 (24B)

by Mistral AI 🇫🇷

Fast dense 24B model that drops the sliding-window setup used in older Mistral releases.

Scale
24B parameters
Context (tokens)
128,000
License
Apache License 2.0
Date
2025-03-18
Decoder type
Dense
Attention
Standard GQA
Layer mix
40 GQA
KV cache / token (bf16) info
160 KiB · Moderate
Key detail
Latency-focused design with a smaller KV cache and fewer layers than Gemma 3 27B.
Pretraining data
Not disclosed No pretraining-token total is given for this version. Mistral Small 3.1 Instruct model card
Post-training methods
Instruction tuning The released Instruct counterpart is instruction-tuned; the detailed supervised, preference, or RL recipe is not specified. Mistral Small 3.1 Instruct model card
AA Intelligence Index v4.3.2 info
Total score 7.1 General 12.8 Scientific 2.2 Coding 13.9 Agents 0.2

Related concepts

Compare

Llama 4 Maverick (400B)

by Meta 🇺🇸

Meta's large MoE follows the DeepSeek V3 playbook but with a more conventional attention stack.

Scale
400B total, 17B active (4.3% active)
Context (tokens)
1,000,000
License
Llama 4 Community License Agreement
Date
2025-04-05
Decoder type
Sparse MoE
Attention
GQA with RoPE in chunked layers and NoPE in full-attention layers
Layer mix
36 chunked + 12 full GQA
KV cache / token (bf16) info
192 KiB · High
Key detail
Alternates dense and MoE blocks and uses fewer, larger experts than DeepSeek V3.
Pretraining data
~22T tokens Pretraining-token count reported for Maverick, including its multimodal training data. Model card
Post-training methods
SFT, RL, DPO The Maverick Instruct recipe uses SFT, online RL, and a final lightweight DPO stage. Release post
AA Intelligence Index v4.3.2 info
Total score 9.3 General 18.8 Scientific 2.5 Coding 15.9 Agents 0.0

Related concepts

Compare

Qwen3 (0.6B)

by Alibaba Cloud 🇨🇳

Tiny current-generation Qwen model that trades width for more depth and a low memory footprint.

Scale
0.6B parameters
Context (tokens)
32,768
License
Apache License 2.0
Date
2025-04-28
Decoder type
Dense
Attention
GQA
Layer mix
28 GQA
KV cache / token (bf16) info
112 KiB · Moderate
Key detail
Deeper than Llama 3.2 1B and unusually practical for local experiments and teaching.
Pretraining data
≈36T tokens Family pretraining budget; excludes post-training. Qwen3 technical report
Post-training methods
SFT, Distillation Instruction/reasoning counterpart uses response-supervised and on-policy teacher distillation. Base checkpoints have no post-training. Qwen3 technical report
AA Intelligence Index v4.3.2 info
Total score 4.8 General N/A Scientific 2.5 Coding N/A Agents N/A

Related concepts

Compare

Qwen3 (235B-A22B)

by Alibaba Cloud 🇨🇳

Large sparse Qwen variant that stays very close to DeepSeek V3 while removing the shared expert.

Scale
235B total, 22B active (9.4% active)
Context (tokens)
128,000
License
Apache License 2.0
Date
2025-04-28
Decoder type
Sparse MoE
Attention
GQA with QK-Norm
Layer mix
94 GQA
KV cache / token (bf16) info
188 KiB · High
Key detail
High-capacity MoE design optimized for serving efficiency without a shared expert.
Pretraining data
≈36T tokens Family pretraining budget; excludes post-training. Qwen3 technical report
Post-training methods
SFT, RLVR, RLHF Cold-start/fusion SFT, reasoning RL, and general RL with human-preference rewards. Qwen3 technical report
AA Intelligence Index v4.3.2 info
Total score 8.3 General N/A Scientific 2.1 Coding N/A Agents N/A

Related concepts

Compare

Qwen3 (30B-A3B)

by Alibaba Cloud 🇨🇳

Small sparse Qwen3 model that sits close to GPT-OSS in active size but uses a deeper stack.

Scale
30B total, 3B active (10% active)
Context (tokens)
128,000
License
Apache License 2.0
Date
2025-04-28
Decoder type
Sparse MoE
Attention
GQA
Layer mix
48 GQA
KV cache / token (bf16) info
96 KiB · Moderate
Key detail
Deeper, narrower MoE alternative to GPT-OSS without a shared expert.
Pretraining data
≈36T tokens Family pretraining budget; excludes post-training. Qwen3 technical report
Post-training methods
SFT, Distillation Instruction/reasoning counterpart uses response-supervised and on-policy teacher distillation. Base checkpoints have no post-training. Qwen3 technical report
AA Intelligence Index v4.3.2 info
Total score 6.6 General N/A Scientific 2.3 Coding N/A Agents N/A

Related concepts

Compare

Large dense Qwen3 model that serves as the clearest like-for-like comparison for OLMo 3 32B.

Scale
32B parameters
Context (tokens)
128,000
License
Apache License 2.0
Date
2025-04-28
Decoder type
Dense
Attention
GQA with QK-Norm
Layer mix
64 GQA
KV cache / token (bf16) info
256 KiB · High
Key detail
Reference dense Qwen stack with QK-Norm and 8 KV heads.
Pretraining data
≈36T tokens Family pretraining budget; excludes post-training. Qwen3 technical report
Post-training methods
SFT, RLVR, RLHF Cold-start/fusion SFT, reasoning RL, and general RL with human-preference rewards. Qwen3 technical report
AA Intelligence Index v4.3.2 info
Total score 7.3

Related concepts

Compare

Mid-size dense Qwen3 model used here as a clean baseline against SmolLM3 and Tiny Aya.

Scale
4B parameters
Context (tokens)
32,768
License
Apache License 2.0
Date
2025-04-28
Decoder type
Dense
Attention
GQA with QK-Norm
Layer mix
36 GQA
KV cache / token (bf16) info
144 KiB · Moderate
Key detail
Compact Qwen3 dense stack with QK-Norm and a 151k vocabulary.
Pretraining data
≈36T tokens Family pretraining budget; excludes post-training. Qwen3 technical report
Post-training methods
SFT, Distillation Instruction/reasoning counterpart uses response-supervised and on-policy teacher distillation. Base checkpoints have no post-training. Qwen3 technical report
AA Intelligence Index v4.3.2 info
Total score 6.6

Related concepts

Compare

Dense Qwen3 baseline used here to show how little OLMo 3 changed the overall decoder recipe.

Scale
8B parameters
Context (tokens)
128,000
License
Apache License 2.0
Date
2025-04-28
Decoder type
Dense
Attention
GQA with QK-Norm
Layer mix
36 GQA
KV cache / token (bf16) info
144 KiB · Moderate
Key detail
Reference Qwen3 dense stack with QK-Norm and 8 KV heads.
Pretraining data
~36T tokens Reported Qwen3 pretraining budget. Technical report
Post-training methods
SFT, Distillation The 8B model uses off-policy response distillation followed by on-policy teacher-logit alignment. Technical report
AA Intelligence Index v4.3.2 info
Total score 6.0 General N/A Scientific 1.0 Coding N/A Agents N/A

Related concepts

Compare

SmolLM3 (3B)

by Hugging Face 🇺🇸

Compact dense model that experiments with leaving out positional encodings in selected layers.

Scale
3B parameters
Context (tokens)
131,072
License
Apache License 2.0
Date
2025-06-19
Decoder type
Dense
Attention
GQA with periodic NoPE layers
Layer mix
36 GQA
KV cache / token (bf16) info
72 KiB · Low
Key detail
Every fourth layer omits RoPE to test a NoPE-style cadence.
Pretraining data
11.2T tokens Reported pretraining total; reasoning mid-training adds 140B tokens separately. Model card
Post-training methods
SFT, APO For SmolLM3-3B Instruct, SFT is followed by Anchored Preference Optimization and checkpoint merging. The linked Base checkpoint precedes these stages. Training recipe

Related concepts

Compare

Kimi K2 (1T)

by Moonshot AI 🇨🇳

Trillion-parameter Moonshot model that essentially scales the DeepSeek V3 recipe upward.

Scale
1T total, 32B active (3.2% active)
Context (tokens)
128,000
License
Modified MIT License
Date
2025-07-10
Decoder type
Sparse MoE
Attention
MLA
Layer mix
61 MLA
KV cache / token (bf16) info
68.6 KiB · Low
Key detail
More experts and fewer MLA heads than DeepSeek V3.
Pretraining data
15.5T tokens Total tokens used to pretrain the released Kimi K2 base. Technical report
Post-training methods
SFT, RLVR Kimi-K2-Instruct uses SFT and RL with verifiable rewards, plus self-critique rubric rewards for open-ended tasks. The gallery architecture is shared with the base model. Technical report
AA Intelligence Index v4.3.2 info
Total score 12.7 General N/A Scientific 3.7 Coding N/A Agents N/A

Related concepts

Compare

GLM-4.5 (355B)

by Z.ai 🇨🇳

Agent-oriented instruction/reasoning hybrid that borrows DeepSeek's dense-prefix MoE layout.

Scale
355B total, 32B active (9% active)
Context (tokens)
128,000
License
MIT License
Date
2025-07-28
Decoder type
Sparse MoE
Attention
GQA with QK-Norm
Layer mix
92 GQA
KV cache / token (bf16) info
368 KiB · Very high
Key detail
Starts with three dense layers before MoE routing, keeps a shared expert, and uses MTP during training.
Pretraining data
23T tokens Combined pretraining and mid-training budget. GLM-4.5 report
Post-training methods
SFT, RLVR, RLHF, RLAIF, Distillation SFT and expert RL cover reasoning, agents, and chat; expert capabilities are distilled into the final model. GLM-4.5 report
AA Intelligence Index v4.3.2 info
Total score 12.8 General N/A Scientific 6.5 Coding N/A Agents N/A

Related concepts

Compare

GPT-OSS (120B)

by OpenAI 🇺🇸

Larger gpt-oss variant keeps the same alternating-attention recipe as the 20B model.

Scale
117B total, 5.1B active (4.4% active)
Context (tokens)
128,000
License
Apache License 2.0
Date
2025-08-04
Decoder type
Sparse MoE
Attention
GQA with alternating sliding-window and global layers
Layer mix
18 sliding-window + 18 global
KV cache / token (bf16) info
72 KiB · Low
Key detail
Shared architectural template scaled up for OpenAI's flagship open-weight release.
Pretraining data
Trillions of tokens; exact count undisclosed Text-only pretraining; no per-model token total is published. gpt-oss model card
Post-training methods
SFT, RL SFT followed by high-compute reasoning and tool-use RL, with deliberative alignment and instruction-hierarchy training. gpt-oss release
AA Intelligence Index v4.3.2 info
Total score 11.6 General 18.8 Scientific 10.4 Coding 17.0 Agents 1.8

Related concepts

Compare

GPT-OSS (20B)

by OpenAI 🇺🇸

OpenAI's smaller open-weight MoE model favors width and alternating local/global attention.

Scale
21B total, 3.6B active (17.1% active)
Context (tokens)
128,000
License
Apache License 2.0
Date
2025-08-04
Decoder type
Sparse MoE
Attention
GQA with alternating sliding-window and global layers
Layer mix
12 sliding-window + 12 global
KV cache / token (bf16) info
48 KiB · Low
Key detail
Wider and shallower than Qwen3, with attention bias and sink mechanisms.
Pretraining data
Trillions of tokens; exact count undisclosed Text-only pretraining; no per-model token total is published. gpt-oss model card
Post-training methods
SFT, RL SFT followed by high-compute reasoning and tool-use RL, with deliberative alignment and instruction-hierarchy training. gpt-oss release
AA Intelligence Index v4.3.2 info
Total score 9.0 General 12.8 Scientific 6.2 Coding 19.4 Agents 0.0

Related concepts

Compare

Tiny Gemma 3 variant that preserves the family's local-global attention recipe at a toy scale.

Scale
270M parameters
Context (tokens)
32,768
Vocabulary
262,144 (~262k)
License
Gemma Terms of Use + Gemma Prohibited Use Policy
Date
2025-08-14
Decoder type
Dense
Attention
Multi-query attention with QK-Norm and 5:1 sliding-window/global attention
Layer mix
15 sliding-window + 3 global
KV cache / token (bf16) info
18 KiB · Very low
Key detail
Keeps the Gemma 3 stack shape while shrinking down to 4 attention heads, a single KV head, and the same 262k vocabulary.
Pretraining data
6T tokens Pretraining budget for the 270M model. Gemma 3 270M model card
Post-training methods
Instruction tuning An IT checkpoint accompanies the pretrained model; its specific alignment stages are not disclosed. Gemma 3 270M release
AA Intelligence Index v4.3.2 info
Total score 5.1 General N/A Scientific 1.8 Coding N/A Agents N/A

Related concepts

Compare

Grok 2.5 (270B)

by xAI 🇺🇸

Rare production-model release that shows an older MoE style with fewer, larger experts.

Scale
270B parameters
Context (tokens)
131,072
License
Grok 2 Community License Agreement
Date
2025-08-22
Decoder type
Sparse MoE
Attention
GQA
Layer mix
64 GQA
KV cache / token (bf16) info
256 KiB · High
Key detail
Adds an always-on SwiGLU path that effectively behaves like a shared expert.
Pretraining data
Not disclosed The released Grok 2 checkpoint does not publish its training-data size. Grok 2 model card
Post-training methods
Not disclosed The release is post-trained, but its model card does not specify SFT, DPO, or reward methods. Grok 2 model card

Related concepts

Compare

Qwen3 Next (80B-A3B)

by Alibaba Cloud 🇨🇳

Efficiency-focused Qwen refresh that swaps standard attention for a DeltaNet-attention hybrid.

Scale
80B total, 3B active (3.8% active)
Context (tokens)
262,144
License
Apache License 2.0
Date
2025-09-09
Decoder type
Sparse hybrid
Attention
3:1 Gated DeltaNet and Gated Attention
Layer mix
12 gated attention + 36 DeltaNet
KV cache / token (bf16) info
24 KiB · Very low
Key detail
Adds many more experts, a shared expert, native 262k context, and MTP for speculative decoding.
Pretraining data
15T tokens Base pretraining count reported for Qwen3-Next-80B-A3B. Model card
Post-training methods
RL The release describes RL for the Instruct and Thinking variants without detailing the full recipe. Release blog
AA Intelligence Index v4.3.2 info
Total score 9.6 General N/A Scientific 3.8 Coding N/A Agents N/A

Compare

MiniMax M2 (230B)

by MiniMax 🇨🇳

MiniMax's flagship returns to full attention and looks like a leaner, sparser cousin of Qwen3.

Scale
230B total, 10B active (4.3% active)
Context (tokens)
196,608
License
Modified MIT License
Date
2025-10-23
Decoder type
Sparse MoE
Attention
GQA with QK-Norm and partial RoPE
Layer mix
62 GQA
KV cache / token (bf16) info
248 KiB · High
Key detail
Uses per-layer QK-Norm, much sparser MoE routing than Qwen3, and MTP during training.
Pretraining data
29.2T tokens (M2-series base) The series report gives 19.9T tokens in the constant phase and 9.3T in decay and context extension. It does not separate pretraining budgets by M2-series checkpoint. Technical report
Post-training methods
SFT, RLVR The M2-series recipe uses SFT and mixed-domain RL with executable task rewards, plus process and completion-time rewards. The recipe evolves across M2, M2.5, and M2.7. Technical report
AA Intelligence Index v4.3.2 info
Total score 18.6 General N/A Scientific 7.3 Coding N/A Agents N/A

Related concepts

Compare

Kimi Linear (48B-A3B)

by Moonshot AI 🇨🇳

Linear-attention hybrid that keeps a transformer backbone but replaces most full-attention layers.

Scale
48B total, 3B active (6.3% active)
Context (tokens)
1,000,000
License
MIT License
Date
2025-10-30
Decoder type
Sparse hybrid
Attention
3:1 Kimi Delta Attention and MLA
Layer mix
7 MLA + 20 Kimi Delta Attention
KV cache / token (bf16) info
7.9 KiB · Very low
Key detail
Uses NoPE in MLA layers, channel-wise KDA gating, and kernel-4 ShortConv preprocessing for Q, K, and V in KDA layers.
Pretraining data
5.7T tokens Budget for the final released checkpoint; the report's 1.4T-token runs are architecture comparisons. Technical report
Post-training methods
SFT, RL The Instruct variant uses multi-stage SFT followed by RL on math, code, and STEM tasks, with concurrent supervised training to preserve general capabilities. Technical report
AA Intelligence Index v4.3.2 info
Total score 7.3

Compare

Scaled-up OLMo 3 keeps the same block design but moves to grouped-query attention.

Scale
32B parameters
Context (tokens)
65,536
License
Apache License 2.0
Date
2025-11-20
Decoder type
Dense
Attention
GQA with QK-Norm and 3:1 sliding-window/global attention
Layer mix
48 sliding-window + 16 global
KV cache / token (bf16) info
256 KiB · High
Key detail
Keeps post-norm while scaling width and applying YaRN only on global layers.
Pretraining data
5.7T tokens per training branch 5.5T initial pretraining tokens, two 100B-token mid-training branches merged together, then 100B context-extension tokens. Aggregate training across both branches is 5.8T. OLMo 3 32B base model card
Post-training methods
SFT, DPO, RLVR The 32B Think release uses supervised fine-tuning, direct preference optimization, and verifiable-reward RL. OLMo 3 32B Think model card
AA Intelligence Index v4.3.2 info
Total score 6.5 General N/A Scientific 3.2 Coding N/A Agents N/A

Related concepts

Compare

New transparent Allen AI model that keeps OLMo's post-norm flavor while modernizing context handling.

Scale
7B parameters
Context (tokens)
65,536
License
Apache License 2.0
Date
2025-11-20
Decoder type
Dense
Attention
MHA with QK-Norm and 3:1 sliding-window/global attention
Layer mix
24 sliding-window + 8 global
KV cache / token (bf16) info
512 KiB · Very high
Key detail
Retains post-norm, keeps MHA, and applies YaRN only on global layers.
Pretraining data
6.08T tokens 5.93T initial pretraining, 100B mid-training, and 50B context-extension tokens. The 7B model uses no branch merging. OLMo 3 7B base model card
Post-training methods
SFT, DPO, RLVR The released Think and Instruct counterparts use SFT, DPO, and RLVR. The Base checkpoint itself has no post-training. OLMo 3 7B base model card
AA Intelligence Index v4.3.2 info
Total score 5.2 General N/A Scientific 2.9 Coding N/A Agents N/A

Related concepts

Compare

DeepSeek V3.2 (671B)

by DeepSeek 🇨🇳

DeepSeek's successor keeps the V3 template but adds sparse attention to cut long-context costs.

Scale
671B total, 37B active (5.5% active)
Context (tokens)
128,000
License
MIT License
Date
2025-12-01
Decoder type
Sparse MoE
Attention
MLA with DeepSeek Sparse Attention
Layer mix
61 MLA
KV cache / token (bf16) info
68.6 KiB · Low
Key detail
Adds sparse attention while retaining the DeepSeek V3-style MTP path.
Pretraining data
943.7B continued-training tokens + 2.1B indexer warm-up Additional DSA training from V3.1-Terminus. The report does not give a cumulative pretraining total. DeepSeek V3.2 report
Post-training methods
SFT, RLVR, RL, Distillation Specialist distillation and mixed GRPO; rule-based reasoning/agent rewards and rubric-based generative rewards. Tool-use cold start uses SFT. DeepSeek V3.2 report
AA Intelligence Index v4.3.2 info
Total score 16.0 General N/A Scientific 6.0 Coding N/A Agents N/A

Related concepts

Compare

Mistral Large 3 (673B)

by Mistral AI 🇫🇷

Mistral's new flagship effectively adopts the DeepSeek architecture and retunes the expert sizes.

Scale
673B total, 41B active (6.1% active)
Context (tokens)
262,144
License
Apache License 2.0
Date
2025-12-02
Decoder type
Sparse MoE
Attention
MLA
Layer mix
61 MLA
KV cache / token (bf16) info
68.6 KiB · Low
Key detail
Near-clone of DeepSeek V3 with larger experts, fewer routed experts, and multimodal support.
Pretraining data
Not disclosed The model card describes training from scratch but does not report a training-token total. Mistral Large 3 model card
Post-training methods
Instruction tuning The released model is instruction-tuned; the card does not specify its supervised, preference, or RL training stages. Mistral Large 3 model card
AA Intelligence Index v4.3.2 info
Total score 9.3 General 17.1 Scientific 2.1 Coding 18.3 Agents 0.2

Related concepts

Compare

Nemotron 3 Nano (30B-A3B)

by NVIDIA 🇺🇸

NVIDIA's Nano model is the most extreme transformer-state-space hybrid in the gallery.

Scale
30B total, 3B active (10% active)
Context (tokens)
1,000,000
License
NVIDIA Nemotron Open Model License
Date
2025-12-04
Decoder type
Hybrid MoE
Attention
Mostly Mamba-2 with a few GQA layers using NoPE
Layer mix
6 GQA + 23 Mamba-2 + 23 MoE
KV cache / token (bf16) info
6 KiB · Very low
Key detail
Interleaves Mamba-2 and MoE blocks, using attention only sparingly.
Pretraining data
25T + 121B long-context tokens The main pretraining run uses 25T tokens, followed by 121B tokens of long-context continued pretraining. Nemotron 3 Nano technical report
Post-training methods
SFT, RLVR, RLHF Supervised fine-tuning, multi-environment reinforcement learning with verifiable rewards, and human-feedback alignment. Nemotron 3 Nano technical report
AA Intelligence Index v4.3.2 info
Total score 6.8 General N/A Scientific 2.3 Coding N/A Agents N/A

Related concepts

Compare

Xiaomi MiMo-V2-Flash (309B)

by Xiaomi 🇨🇳

Large MoE model that pushes sliding-window attention harder than most contemporaries.

Scale
309B total, 15B active (4.9% active)
Context (tokens)
262,144
License
MIT License
Date
2025-12-16
Decoder type
Sparse MoE
Attention
5:1 sliding-window/global attention
Layer mix
40 sliding-window + 8 global
KV cache / token (bf16) info
144 KiB · Moderate
Key detail
Uses an unusually small 128-token local window plus multi-token prediction.
Pretraining data
27T tokens Reported pretraining budget. Model card
Post-training methods
SFT, RL, Distillation, RLVR SFT, domain-specialized RL, and multi-teacher on-policy distillation. Verifiable tasks use programmatic and LLM verification; subjective helpfulness and safety use rubric-based LLM rewards. Technical report
AA Intelligence Index v4.3.2 info
Total score 16.0 General N/A Scientific 4.3 Coding N/A Agents N/A

Related concepts

Compare

GLM-4.7 (355B)

by Z.ai 🇨🇳

Immediate GLM predecessor that stays closer to the older GLM-4.5 style before the MLA shift.

Scale
355B total, 32B active (9% active)
Context (tokens)
202,752
License
MIT License
Date
2025-12-22
Decoder type
Sparse MoE
Attention
GQA with QK-Norm
Layer mix
92 GQA
KV cache / token (bf16) info
368 KiB · Very high
Key detail
Serves as the pre-MLA, pre-sparse-attention baseline and uses an MTP-3-style setup.
Pretraining data
23T tokens (inherited GLM-4.5 base) The team confirms GLM-4.5, 4.6, and 4.7 share a base model; 23T is its pretraining and mid-training budget. GLM-4.5 report · GLM team AMA
Post-training methods
SFT, RL Reprocessed SFT data followed by RL using the slime framework; the full reward recipe is not specified. GLM team AMA
AA Intelligence Index v4.3.2 info
Total score 17.4 General N/A Scientific 3.2 Coding N/A Agents N/A

Related concepts

Compare

Arcee AI Trinity Large (400B)

by Arcee AI 🇺🇸

Arcee's flagship blends several efficiency tricks into a DeepSeek-like coarse MoE design; the AA score shown here refers to the Trinity Large Thinking checkpoint.

Scale
400B total, 13B active (3.3% active)
Context (tokens)
512,000
License
Apache License 2.0
Date
2026-01-27
Decoder type
Sparse MoE
Attention
GQA with gated attention and 3:1 sliding-window/global attention
Layer mix
45 sliding-window + 15 global
KV cache / token (bf16) info
240 KiB · High
Key detail
Combines QK-Norm, RoPE+NoPE, sandwich norm, and a coarse-grained MoE.
Pretraining data
17T tokens Full Base checkpoint, including its mid-training anneals; the earlier TrueBase checkpoint used 10T. Trinity Large report · Trinity Large Thinking model card
Post-training methods
SFT, RL Preview uses SFT; the later Thinking checkpoint adds instruction tuning and agentic RL. Trinity Large report · Trinity Large Thinking model card
AA Intelligence Index v4.3.2 info
Total score 10.8 General 16.6 Scientific 8.4 Coding 20.6 Agents 0.2

Compare

GLM-5 (744B)

by Z.ai 🇨🇳

Huge GLM refresh that adopts both MLA and DeepSeek Sparse Attention for flagship-scale inference.

Scale
744B total, 40B active (5.4% active)
Context (tokens)
202,752
License
MIT License
Date
2026-02-11
Decoder type
Sparse MoE
Attention
MLA with DeepSeek Sparse Attention
Layer mix
78 MLA
KV cache / token (bf16) info
87.8 KiB · Moderate
Key detail
Bigger than GLM-4.7, with more experts, fewer layers, and DeepSeek-style MTP.
Pretraining data
28.5T tokens Combined pretraining and mid-training budget for the base model. GLM-5 report
Post-training methods
SFT, RLVR, RL, Distillation SFT, reasoning/agentic/general RL, and on-policy cross-stage distillation. General RL combines rule-based and model-based rewards. GLM-5 report
AA Intelligence Index v4.3.2 info
Total score 21.8 General N/A Scientific 3.8 Coding N/A Agents N/A

Related concepts

Compare

Nemotron 3 Super (120B-A12B)

by NVIDIA 🇺🇸

The Super variant scales up Nano and adds both latent experts and native speculative decoding support.

Scale
120B total, 12B active (10% active)
Context (tokens)
1,000,000
License
NVIDIA Nemotron Open Model License
Date
2026-03-11
Decoder type
Hybrid MoE
Attention
Mostly Mamba-2 with a few GQA layers using NoPE
Layer mix
8 GQA + 40 Mamba-2 + 40 MoE
KV cache / token (bf16) info
8 KiB · Very low
Key detail
Adds latent-space MoE and shared-weight MTP for fast inference.
Pretraining data
25T + 51B long-context tokens The main pretraining run uses 25T tokens. Two context-extension phases add 34B and 17B tokens. Nemotron 3 Super technical report
Post-training methods
SFT, RLVR, RLHF Supervised fine-tuning, multi-environment and software-engineering RL with verifiable rewards, then human-feedback alignment. Nemotron 3 Super technical report
AA Intelligence Index v4.3.2 info
Total score 12.8 General 22.1 Scientific 12.0 Coding 18.1 Agents 0.6

Compare

Gemma 4 (31B)

by Google 🇺🇸

Dense Gemma 4 scales the family to a 256K-context multimodal checkpoint without changing the core local-global recipe much.

Scale
30.7B parameters
Context (tokens)
256,000
Vocabulary
262,144 (~262k)
License
Apache License 2.0
Date
2026-04-02
Decoder type
Dense
Attention
GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
Layer mix
50 sliding-window + 10 global
KV cache / token (bf16) info
840 KiB · Very high
Key detail
Carries Gemma's unusual pre/post-norm stack into a larger 31B dense model with 256K context.
Pretraining data
Not disclosed No per-model pretraining-token count is published. Gemma 4 model card
Post-training methods
Instruction tuning IT variant; the report refers to Gemma 3-like post-training without itemizing alignment stages. Gemma 4 report
AA Intelligence Index v4.3.2 info
Total score 13.9 General N/A Scientific 5.9 Coding N/A Agents N/A

Related concepts

Compare

Gemma 4 (26B-A4B)

by Google 🇺🇸

Sparse Gemma 4 variant that keeps the local:global attention backbone while swapping dense FFNs for MoE layers.

Scale
25.2B total, 3.8B active (15.1% active)
Context (tokens)
256,000
Vocabulary
262,144 (~262k)
License
Apache License 2.0
Date
2026-04-02
Decoder type
Sparse MoE
Attention
GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
Layer mix
25 sliding-window + 5 global
KV cache / token (bf16) info
210 KiB · High
Key detail
Uses 128 total experts with only 8 routed plus 1 shared expert active per token.
Pretraining data
Not disclosed No per-model pretraining-token count is published. Gemma 4 model card
Post-training methods
Instruction tuning IT variant; the report refers to Gemma 3-like post-training without itemizing alignment stages. Gemma 4 report
AA Intelligence Index v4.3.2 info
Total score 13.1 General N/A Scientific 5.7 Coding N/A Agents N/A

Related concepts

Compare

Gemma 4 (12B)

by Google 🇺🇸

Dense Gemma 4 12B Unified uses an encoder-free multimodal input path, while its text decoder keeps the 31B-style local-global attention recipe.

Scale
12B parameters
Context (tokens)
262,144
Vocabulary
262,144 (~262k)
License
Apache License 2.0
Date
2026-06-03
Decoder type
Dense
Attention
GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
Layer mix
40 sliding-window + 8 global
KV cache / token (bf16) info
328 KiB · Very high
Key detail
Figure shows the 48-layer text decoder; the full 12B Unified model also projects audio, image, and video inputs directly into the decoder input stream.
Pretraining data
Not disclosed No per-model pretraining-token count is published. Gemma 4 model card
Post-training methods
Instruction tuning IT variant; the report refers to Gemma 3-like post-training without itemizing alignment stages. Gemma 4 report
AA Intelligence Index v4.3.2 info
Total score 9.4 General N/A Scientific 3.2 Coding N/A Agents N/A

Related concepts

Compare

Llama 3.2 (3B)

by Meta 🇺🇸

Small Llama baseline used to show how closely Nanbeige follows the mainstream compact decoder recipe.

Scale
3B parameters
Context (tokens)
128,000
License
Llama 3.2 Community License Agreement
Date
2024-09-25
Decoder type
Dense
Attention
GQA
Layer mix
28 GQA
KV cache / token (bf16) info
112 KiB · Moderate
Key detail
Reference small-model Llama architecture with tied embeddings.
Pretraining data
Up to 9T tokens Pretraining-only figure reported for the 1B and 3B models. Pruning and teacher-logit distillation are part of their pretraining. Model card
Post-training methods
SFT, DPO The Instruct variant uses repeated SFT, rejection sampling, and DPO. These alignment stages apply to the instruct counterpart of the gallery's base configuration. Model card
AA Intelligence Index v4.3.2 info
Total score 5.7

Related concepts

Compare

Qwen3 Coder Flash (30B-A3B)

by Alibaba Cloud 🇨🇳

Coding-tuned Qwen model that keeps a straightforward grouped-query MoE stack instead of the newer hybrid-attention variants.

Scale
30B total, 3.3B active (11% active)
Context (tokens)
256,000
License
Apache License 2.0
Date
2025-07-31
Decoder type
Sparse MoE
Attention
GQA
Layer mix
48 GQA
KV cache / token (bf16) info
96 KiB · Moderate
Key detail
Uses 128 experts with 8 active per token and a native 256k context window for coding workloads.
Pretraining data
Not disclosed The 30B card omits a token count. The 7.5T figure in the initial Coder release describes the 480B model. Model card · Coder release
Post-training methods
Not disclosed The 30B card confirms post-training but does not give a separate method breakdown. Model card
AA Intelligence Index v4.3.2 info
Total score 9.6 General N/A Scientific 1.9 Coding N/A Agents N/A

Related concepts

Compare

Kimi K2.5 (1T)

by Moonshot AI 🇨🇳

Native-multimodal Moonshot flagship that keeps the K2/DeepSeek-style MoE layout and pushes native context to 256k.

Scale
1T total, 32B active (3.2% active)
Context (tokens)
256,000
License
Modified MIT License
Date
2026-01-27
Decoder type
Sparse MoE
Attention
MLA
Layer mix
61 MLA
KV cache / token (bf16) info
68.6 KiB · Low
Key detail
Keeps the 384-expert K2 backbone, but adds multimodal capabilities (not shown) and doubles the native context length.
Pretraining data
~15T additional tokens Continued pretraining from Kimi K2 on a mixed visual and text corpus; this excludes K2's earlier pretraining. Technical report
Post-training methods
SFT, RLVR SFT and RL combine verifiable outcome rewards with generative reward models. Parallel-agent RL trains the orchestrator to delegate work. Technical report
AA Intelligence Index v4.3.2 info
Total score 19.4 General N/A Scientific 6.9 Coding N/A Agents N/A

Related concepts

Compare

Step 3.5 Flash (196B)

by StepFun 🇨🇳

Throughput-oriented MoE model that stays competitive with much larger DeepSeek-style systems.

Scale
196B total, 11B active (5.6% active)
Context (tokens)
262,144
License
Apache License 2.0
Date
2026-02-01
Decoder type
Sparse MoE
Attention
GQA with 3:1 sliding-window attention
Layer mix
36 sliding-window + 12 global
KV cache / token (bf16) info
192 KiB · High
Key detail
Uses MTP-3 during both training and inference for unusually high throughput.
Pretraining data
~17.6T tokens + 750B mid-training tokens Section 4.2 gives 14.6T + 3T pretraining tokens. The introduction's 17.2T figure conflicts with that detailed schedule. Technical report
Post-training methods
SFT, RLVR, RLHF, Distillation Two-stage SFT, domain RL with verifiable and human-preference rewards, and self-distillation. Technical report
AA Intelligence Index v4.3.2 info
Total score 17.0 General N/A Scientific 13.4 Coding N/A Agents N/A

Related concepts

Compare

Nanbeige 4.1 (3B)

by BOSS Zhipin 🇨🇳

Small on-device oriented model that stays close to Llama 3.2 while nudging the scaling choices.

Scale
3B parameters
Context (tokens)
262,144
License
Apache License 2.0
Date
2026-02-10
Decoder type
Dense
Attention
GQA
Layer mix
32 GQA
KV cache / token (bf16) info
64 KiB · Low
Key detail
Llama-like stack without tying input embeddings to the output layer.
Pretraining data
23T tokens (inherited base) Uses Nanbeige4-3B-Base, whose four pretraining phases total 23T tokens. The 4.1 release adds post-training. Nanbeige 4 base model card · Nanbeige 4.1 technical report
Post-training methods
SFT, RLHF, RLVR SFT, human-preference reward-model RL, pairwise RL, and reasoning/code RL with executable correctness checks. Nanbeige 4.1 technical report
AA Intelligence Index v4.3.2 info
Total score 8.4 General N/A Scientific 5.4 Coding N/A Agents N/A

Related concepts

Compare

MiniMax-M2.5 (230B)

by MiniMax 🇨🇳

Popular 230B coder that opts for a classic architecture instead of the newer hybrid-attention ideas.

Scale
230B total, 10B active (4.3% active)
Context (tokens)
196,608
License
Modified MIT License
Date
2026-02-12
Decoder type
Sparse MoE
Attention
GQA with QK-Norm
Layer mix
62 GQA
KV cache / token (bf16) info
248 KiB · High
Key detail
Deliberately avoids sliding-window or linear-attention hybrids while keeping a 10B active path and three MTP modules.
Pretraining data
29.2T tokens (M2-series base) The series report gives 19.9T tokens in the constant phase and 9.3T in decay and context extension. It does not separate pretraining budgets by M2-series checkpoint. Technical report
Post-training methods
SFT, RLVR The M2-series recipe uses SFT and mixed-domain RL with executable task rewards, plus process and completion-time rewards. The recipe evolves across M2, M2.5, and M2.7. Technical report
AA Intelligence Index v4.3.2 info
Total score 22.8 General N/A Scientific 10.8 Coding N/A Agents N/A

Related concepts

Compare

Tiny Aya (3.35B)

by Cohere 🇨🇦

Compact multilingual model from Cohere with a rare parallel transformer block.

Scale
3.35B parameters
Context (tokens)
8,192
License
Creative Commons Attribution-NonCommercial 4.0
Date
2026-02-13
Decoder type
Dense
Attention
GQA with 3:1 sliding-window attention
Layer mix
27 sliding-window + 9 global
KV cache / token (bf16) info
72 KiB · Low
Key detail
Runs attention and the MLP in parallel while mixing RoPE with NoPE.
Pretraining data
6T tokens Pretraining budget for Tiny Aya Base. Technical report
Post-training methods
SFT, Preference tuning, Model merging Global and regional variants add SFT; Global adds minimal preference tuning, and regional releases merge checkpoints. Tiny Aya Base precedes these stages. Technical report
AA Intelligence Index v4.3.2 info
Total score 4.8 General N/A Scientific 2.6 Coding N/A Agents N/A

Related concepts

Compare

Ling 2.5 (1T)

by InclusionAI 🇨🇳

Trillion-parameter long-context model that swaps DeltaNet for Lightning Attention.

Scale
1T total, 63B active (6.3% active)
Context (tokens)
256,000
License
MIT License
Date
2026-02-15
Decoder type
Sparse hybrid
Attention
Lightning Attention plus MLA
Layer mix
10 MLA + 70 Lightning Attention
KV cache / token (bf16) info
11.2 KiB · Very low
Key detail
Uses a 7:1 linear-attention/MLA ratio and a much larger 63B active path.
Pretraining data
9T additional tokens Continued pretraining after retrofitting the Ling 2.0 base. The card separately describes a pretraining corpus expanded from 20T to 29T tokens. Model card
Post-training methods
RL Reported methods include correctness and redundancy rewards, bidirectional RL feedback, and agentic RL in interactive environments. Model card

Related concepts

Compare

Qwen3.5 (397B)

by Alibaba Cloud 🇨🇳

Mainline Qwen refresh that brings the Next-style hybrid attention into the flagship series.

Scale
397B total, 17B active (4.3% active)
Context (tokens)
262,144
License
Apache License 2.0
Date
2026-02-16
Decoder type
Sparse hybrid
Attention
3:1 Gated DeltaNet and Gated Attention
Layer mix
15 gated attention + 45 DeltaNet
KV cache / token (bf16) info
30 KiB · Low
Key detail
Turns the former Qwen3-Next side branch into the new core design with 512 experts and 17B active parameters.
Pretraining data
Not disclosed The model card describes multimodal pretraining without a token total. Model card
Post-training methods
RL Large-scale RL across agent environments is confirmed; the card does not specify all post-training stages. Model card
AA Intelligence Index v4.3.2 info
Total score 21.4 General N/A Scientific 10.3 Coding N/A Agents N/A

Compare

Sarvam (30B)

by Sarvam AI 🇮🇳

Reasoning-oriented Indian-language sparse MoE that keeps GQA at the smaller size.

Scale
30B total, 2.4B active (8% active)
Context (tokens)
131,072
License
Apache License 2.0
Date
2026-03-03
Decoder type
Sparse MoE
Attention
GQA with QK-Norm
Layer mix
19 GQA
KV cache / token (bf16) info
19 KiB · Very low
Key detail
Large vocabulary and strong Indic language support paired with a reasoning-focused sparse MoE design.
Pretraining data
16T tokens Reported pretraining budget. Training overview
Post-training methods
SFT, RLVR RL combines verifiable outcomes with rubric-based rewards, using asynchronous group-relative optimization. Training overview
AA Intelligence Index v4.3.2 info
Total score 6.6 General N/A Scientific 3.9 Coding N/A Agents N/A

Related concepts

Compare

Sarvam (105B)

by Sarvam AI 🇮🇳

Larger Sarvam variant keeps the sparse MoE layout but switches from GQA to MLA.

Scale
105B total, 10.3B active (9.8% active)
Context (tokens)
131,072
License
Apache License 2.0
Date
2026-03-03
Decoder type
Sparse MoE
Attention
MLA with KV LayerNorm and NoPE + RoPE
Layer mix
32 MLA
KV cache / token (bf16) info
36 KiB · Low
Key detail
Large vocabulary and strong Indic language support carried into the larger MLA-based sparse MoE variant.
Pretraining data
12T tokens Reported pretraining budget. Training overview
Post-training methods
SFT, RLVR RL combines verifiable outcomes with rubric-based rewards, using asynchronous group-relative optimization. Training overview
AA Intelligence Index v4.3.2 info
Total score 8.8 General N/A Scientific 5.5 Coding N/A Agents N/A

Related concepts

Compare

Phi-4 (14B)

by Microsoft 🇺🇸

Microsoft's 14B dense Phi refresh stays close to Phi-3-medium but swaps its sliding-window attention for full-context GQA and a larger tokenizer.

Scale
14B parameters
Context (tokens)
16,384
License
MIT License
Date
2024-12-12
Decoder type
Dense
Attention
GQA with RoPE
Layer mix
40 GQA
KV cache / token (bf16) info
200 KiB · High
Key detail
Classic pre-norm RMSNorm stack with GQA, 40 heads, 10 KV heads, and a 100,352-token vocabulary.
Pretraining data
9.8T tokens The model card reports 9.8T training tokens. Phi-4 model card
Post-training methods
SFT, DPO Supervised fine-tuning followed by pivotal-token and judge-guided DPO stages. Phi-4 technical report
AA Intelligence Index v4.3.2 info
Total score 5.9 General N/A Scientific 1.9 Coding N/A Agents N/A

Related concepts

Compare

xLSTM (7B)

by NXAI 🇦🇹

Recurrent 7B language model that replaces self-attention with xLSTM blocks built around matrix memory.

Scale
7B parameters
Context (tokens)
No explicit limit
License
NXAI Community License Agreement
Date
2025-03-17
Decoder type
Recurrent
Attention
No self-attention; mLSTM recurrent layers with matrix memory
Layer mix
32 mLSTM
KV cache / token (bf16) info
0 B · No cache
Key detail
Stateful recurrent architecture aimed at fast long-context inference without an explicit context window.
Pretraining data
~2.3T tokens DCLM and selected high-quality data. Model card
Post-training methods
None (base model) The linked xLSTM-7b release is a pretrained completion model. Model card

Related concepts

Compare

GLM-4.5-Air (106B)

by Z.ai 🇨🇳

Compact GLM-4.5 companion that keeps the same agent-oriented sparse MoE recipe at a smaller serving footprint.

Scale
106B total, 12B active (11.3% active)
Context (tokens)
128,000
License
MIT License
Date
2025-07-28
Decoder type
Sparse MoE
Attention
GQA
Layer mix
46 GQA
KV cache / token (bf16) info
184 KiB · High
Key detail
Shrinks the GLM-4.5 layout to 46 layers, keeps the dense warmup, and inherits the MTP training recipe.
Pretraining data
Not disclosed The report gives 23T tokens for GLM-4.5 but does not separately establish Air's budget. GLM-4.5 report
Post-training methods
SFT, RLVR, RLHF, RLAIF, Distillation The series recipe combines SFT, expert RL with rule-based, human, and AI feedback, and expert distillation. GLM-4.5 report
AA Intelligence Index v4.3.2 info
Total score 11.1 General N/A Scientific 3.5 Coding N/A Agents N/A

Related concepts

Compare

Ouro-Thinking (2.6B)

by ByteDance 🇨🇳

ByteDance's dense LoopLM reuses one 48-layer MHA stack across four recurrent passes and trains an exit gate over the intermediate representations.

Scale
2.6B parameters
Context (tokens)
65,536
Vocabulary
49,152 (~49k)
License
Apache License 2.0
Date
2025-10-29
Decoder type
Dense
Attention
MHA with RoPE
Layer mix
48 shared MHA layers × 4 passes
KV cache / token (bf16) info
1.5 MiB · Very high
Key detail
Reuses the same 48-layer MHA stack across four passes, yielding 192 block applications, with sandwich RMSNorm and a learned exit gate.
Pretraining data
≈7.7T tokens The report's rounded total covers initial pretraining, continued training, context extension, and mid-training before reasoning SFT. Ouro technical report
Post-training methods
SFT Reasoning SFT on approximately 8.3M examples produces the Thinking release. Separate exploratory RLVR runs did not outperform the final SFT checkpoint. Ouro technical report

Compare

INTELLECT-3 (106B)

by Prime Intellect 🇺🇸

Large-scale RL post-training of GLM-4.5-Air that keeps the compact 106B sparse MoE backbone intact.

Scale
106B total, 12B active (11.3% active)
Context (tokens)
128,000
License
MIT License
Date
2025-11-26
Decoder type
Sparse MoE
Attention
GQA
Layer mix
46 GQA
KV cache / token (bf16) info
184 KiB · High
Key detail
Keeps the GLM-4.5-Air architecture unchanged and shifts the capability profile through SFT plus large-scale RL.
Pretraining data
Not disclosed Post-trained from GLM-4.5-Air-Base. A pretraining-token total for that Air base is not disclosed in the checked sources. Technical report · Model card
Post-training methods
SFT, RLVR Two SFT phases are followed by large-scale RL with verifiable rewards across reasoning and agentic tasks. Technical report
AA Intelligence Index v4.3.2 info
Total score 10.6 General N/A Scientific 6.7 Coding N/A Agents N/A

Related concepts

Compare

LongCat-Flash-Lite (68.5B-A3B)

by Meituan 🇨🇳

Meituan's LongCat-Next text backbone is a lightweight MoE that shifts a large share of parameters into N-gram embeddings while keeping sparse inference around 3B active parameters.

Scale
68.5B total, ~3B active avg (4.4% active)
Context (tokens)
256,000
Vocabulary
131,072
License
MIT License
Date
2026-01-28
Decoder type
Sparse MoE
Attention
MLA with RoPE + NoPE
Layer mix
14 logical layers / 28 MLA sublayers
KV cache / token (bf16) info
31.5 KiB · Low
Key detail
Uses N-gram embedding expansion, Shortcut MoE routing over 256 routed plus 128 zero/identity experts, and top-k 12 expert choices per token.
Pretraining data
11T + 1.5T midtraining tokens 11T tokens of initial pretraining are followed by 1.5T tokens of midtraining and context extension, for 12.5T before SFT. Technical report
Post-training methods
SFT The report states that the released chat model is supervised fine-tuned after pretraining and midtraining. Technical report
AA Intelligence Index v4.3.2 info
Total score 11.5 General N/A Scientific 2.9 Coding N/A Agents N/A

Related concepts

Compare

Mistral Small 4 (119B)

by Mistral AI 🇫🇷

Multimodal Mistral Small refresh that jumps from the older dense 24B stack to an MLA-based sparse MoE design.

Scale
119B total, 6.63B active (5.6% active)
Context (tokens)
256,000
License
Apache License 2.0
Date
2026-03-16
Decoder type
Sparse MoE
Attention
MLA
Layer mix
36 MLA
KV cache / token (bf16) info
22.5 KiB · Very low
Key detail
Uses 128 experts with 4 routed plus 1 shared expert active per token while unifying instruct, reasoning, and vision.
Pretraining data
Not disclosed The model card does not report the pretraining-token total. Mistral Small 4 model card
Post-training methods
Not disclosed The model card describes hybrid reasoning and instruction following but does not identify the post-training methods. Mistral Small 4 model card
AA Intelligence Index v5.0 info
Total score 4.3 Knowledge Work 0.2 Coding 0.0 Science 0.0 General 17.0
AA Intelligence Index v4.3.2 info
Total score 11.3 General 21.1 Scientific 5.1 Coding 19.4 Agents 0.2

Related concepts

Compare

Nemotron 3 Nano (4B)

by NVIDIA 🇺🇸

Compact on-device hybrid that compresses Nemotron Nano 9B v2 into a mostly Mamba-2 stack with only four attention layers.

Scale
4B parameters
Context (tokens)
262,144
License
NVIDIA Nemotron Open Model License
Date
2026-03-16
Decoder type
Dense hybrid
Attention
GQA with NoPE in 4 attention layers
Layer mix
4 GQA + 21 Mamba-2 + 17 FFN
KV cache / token (bf16) info
16 KiB · Very low
Key detail
Uses a 42-layer stack with 21 Mamba-2 blocks, 17 ReLU² FFNs, and just 4 GQA layers.
Pretraining data
213B recovery-distillation tokens Compressed from Nano 9B v2, descended from a 20T-token pretrained 12B model. The 4B recovery stages use 63B and 150B tokens; this is additional recovery training, not a from-scratch pretraining total. NVIDIA Nano 4B training details · NVIDIA Nemotron Nano 2 technical report
Post-training methods
Distillation, SFT, RLVR Compression recovery distillation, two SFT stages, and three RLVR stages for instruction following and tool use. NVIDIA Nano 4B training details
AA Intelligence Index v5.0 info
Total score 1.8 Knowledge Work 0.2 Coding 0.0 Science 0.0 General 7.1
AA Intelligence Index v4.3.2 info
Total score 4.8 General 8.8 Scientific 2.4 Coding 8.0 Agents 0.1

Related concepts

Compare

MiniMax M2.7 (230B)

by MiniMax 🇨🇳

Agent-focused 230B follow-up built for self-evolution, agent harnesses, and productivity workflows.

Scale
230B total, 10B active (4.3% active)
Context (tokens)
204,800
License
MiniMax M2.7 Non-Commercial License
Date
2026-03-18
Decoder type
Sparse MoE
Attention
GQA with QK-Norm
Layer mix
62 GQA
KV cache / token (bf16) info
248 KiB · High
Key detail
Keeps the M2.5-style 62-layer sparse MoE stack and adds three MTP modules for speculative decoding support.
Pretraining data
29.2T tokens (M2-series base) The shared base recipe uses 19.9T constant-phase and 9.3T decay/long-context tokens. Separate budgets for M2, M2.5, and M2.7 are not given. MiniMax M2 series technical report
Post-training methods
SFT, RLVR Supervised fine-tuning and multi-domain RL, including executable verification for coding and agent tasks. Other domains also use learned rewards. MiniMax M2 series technical report
AA Intelligence Index v4.3.2 info
Total score 22.8 General 34.7 Scientific 15.1 Coding 25.1 Agents 14.7

Related concepts

Compare

Gemma 4 (E2B)

by Google 🇺🇸

Smallest Gemma 4 edge model keeps the family's hybrid attention stack and adds native audio on a phone-scale multimodal footprint. Uses per-layer embeddings, which add small layer-specific token vectors without scaling the full compute path, so its compute footprint is closer to 2.3B than a full 5.1B dense model.

Scale
5.1B parameters (2.3B effective)
Context (tokens)
128,000
Vocabulary
262,144 (~262k)
License
Apache License 2.0
Date
2026-04-02
Decoder type
Dense
Attention
Multi-query attention with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 4:1 sliding-window/global attention
Layer mix
28 sliding-window + 7 global
KV cache / token (bf16) info
35 KiB · Low
Key detail
Uses a double-wide GELU MLP plus a single KV head to stay light enough for offline edge deployments.
Pretraining data
Not disclosed No per-model pretraining-token count is published. Gemma 4 model card
Post-training methods
Instruction tuning IT variant; the report refers to Gemma 3-like post-training without itemizing alignment stages. Gemma 4 report
AA Intelligence Index v4.3.2 info
Total score 6.5 General N/A Scientific 2.4 Coding N/A Agents N/A

Related concepts

Compare

Gemma 4 (E4B)

by Google 🇺🇸

Larger Gemma 4 edge variant keeps the same multimodal hybrid recipe but doubles width and KV heads for a stronger 128K mobile checkpoint. Uses per-layer embeddings, which add small layer-specific token vectors without scaling the full compute path, so its compute footprint is closer to 4.5B than a full 8B dense model.

Scale
8B parameters (4.5B effective)
Context (tokens)
128,000
Vocabulary
262,144 (~262k)
License
Apache License 2.0
Date
2026-04-02
Decoder type
Dense
Attention
GQA with QK-Norm, unified K/V on global layers, p-RoPE on global layers, and 5:1 sliding-window/global attention
Layer mix
35 sliding-window + 7 global
KV cache / token (bf16) info
84 KiB · Moderate
Key detail
Steps up to a 42-layer stack with 2 KV heads while keeping the same edge-oriented local/global template.
Pretraining data
Not disclosed No per-model pretraining-token count is published. Gemma 4 model card
Post-training methods
Instruction tuning IT variant; the report refers to Gemma 3-like post-training without itemizing alignment stages. Gemma 4 report
AA Intelligence Index v4.3.2 info
Total score 7.5 General N/A Scientific 2.6 Coding N/A Agents N/A

Related concepts

Compare

DeepSeek V4-Flash (284B)

by DeepSeek 🇨🇳

DeepSeek's efficient V4 preview keeps the million-token architecture while reducing the MoE scale to 284B parameters and 13B active parameters.

Scale
284B total, 13B active (4.6% active)
Context (tokens)
1,048,576
License
MIT License
Date
2026-04-24
Decoder type
Sparse MoE
Attention
MLA-style CSA/HCA with mHC
Layer mix
43 CSA/HCA
KV cache / token (bf16) info
5.4 KiB · Very low
Key detail
Uses 256 experts, 6 routed plus 1 shared expert per token, hash-based routing, compressed attention, and the V4 MTP path.
Pretraining data
32T tokens Includes staged context extension to 1M tokens. DeepSeek V4 report
Post-training methods
SFT, RLVR, RL, Distillation Domain experts receive SFT and GRPO, using verifiers and generative rewards, then are combined through on-policy distillation. DeepSeek V4 report
AA Intelligence Index v4.3.2 info
Total score 34.3 General 31.8 Scientific 27.6 Coding 31.2 Agents 43.5

Related concepts

Compare

DeepSeek V4-Pro (1.6T)

by DeepSeek 🇨🇳

DeepSeek's flagship V4 preview scales to 1.6T parameters and introduces compressed sparse attention plus manifold-constrained hyper-connections for million-token contexts.

Scale
1.6T total, 49B active (3.1% active)
Context (tokens)
1,048,576
License
MIT License
Date
2026-04-24
Decoder type
Sparse MoE
Attention
MLA-style CSA/HCA with mHC
Layer mix
61 CSA/HCA
KV cache / token (bf16) info
7.7 KiB · Very low
Key detail
Uses 384 experts, 6 routed plus 1 shared expert per token, hash-based routing, compressed attention caches, and the V4 MTP path.
Pretraining data
33T tokens Includes staged context extension to 1M tokens. DeepSeek V4 report
Post-training methods
SFT, RLVR, RL, Distillation Domain experts receive SFT and GRPO, using verifiers and generative rewards, then are combined through on-policy distillation. DeepSeek V4 report
AA Intelligence Index v5.0 info
Total score 23.1 Knowledge Work 36.3 Coding 15.9 Science 12.8 General 27.1
AA Intelligence Index v4.3.2 info
Total score 36.0 General 34.4 Scientific 29.5 Coding 32.6 Agents 44.2

Related concepts

Compare

Laguna XS.2 (33B)

by Poolside 🇺🇸

Poolside's open-weight agentic coding MoE uses mixed sliding-window/global attention and activates only 3B of 33B parameters per token.

Scale
33B total, 3B active (9.1% active)
Context (tokens)
262,144
License
Apache License 2.0
Date
2026-04-28
Decoder type
Sparse MoE
Attention
Gated GQA with QK-Norm and 3:1 sliding-window/global attention
Layer mix
30 sliding-window + 10 global
KV cache / token (bf16) info
160 KiB · Moderate
Key detail
Uses per-layer query-head counts, a 512-token SWA window, sigmoid MoE routing, and one shared expert alongside the top-8 routed experts.
Pretraining data
>30T tokens Pretraining tokens seen, sampled from a pool of about 27T unique tokens. The corpus size and training budget are different quantities. Technical report
Post-training methods
SFT, RLVR Supervised instruction tuning is followed by reinforcement learning with verifiable rewards using CISPO. Technical report

Compare

ZAYA1-8B (8.4B)

by Zyphra 🇺🇸

Zyphra's compact reasoning MoE pairs CCA/GQA attention updates with top-1 routed expert FFN updates to keep fewer than 1B parameters active per token.

Scale
8.4B total, 760M active/token (9% active)
Context (tokens)
131,072
Vocabulary
262,272
License
Apache License 2.0
Date
2026-05-06
Decoder type
Sparse MoE
Attention
CCA with 4:1 GQA, RoPE, and Q/K L2 norm
Layer mix
40 CCA/GQA attention + 40 top-1 MoE FFN entries
KV cache / token (bf16) info
40 KiB · Low
Key detail
Key detail is CCA/GQA attention, which performs attention in a compressed latent space while keeping a 4:1 GQA layout. Sidenote: the config lists 80 alternating layer entries, equivalent to 40 CCA/GQA attention updates paired with 40 top-1 MoE FFN updates.
Pretraining data
12T tokens + 1.2T mid-training tokens Base phases consume 8T + 4T; 32K mid-training adds 1.2T. The separate SFT stage consumes 660B tokens. Technical report
Post-training methods
SFT, RLVR, RLHF SFT, three verifiable reasoning RL stages, then behavioral RLHF for chat and instruction following. Technical report

Related concepts

Compare

GLM-5.1 (744B)

by Z.ai 🇨🇳

Post-trained GLM refresh that keeps the GLM-5 backbone intact but targets stronger long-horizon agentic coding.

Scale
744B total, 40B active (5.4% active)
Context (tokens)
202,752
License
MIT License
Date
2026-04-07
Decoder type
Sparse MoE
Attention
MLA with DeepSeek Sparse Attention
Layer mix
78 MLA
KV cache / token (bf16) info
87.8 KiB · Moderate
Key detail
Architecture stays aligned with GLM-5, including its MTP-capable backbone; the main shift is post-training.
Pretraining data
Not disclosed The official model card does not give a version-specific pretraining-token total. GLM-5.1 model card
Post-training methods
Not disclosed The official model card and documentation do not itemize version-specific training stages. GLM-5.1 documentation
AA Intelligence Index v4.3.2 info
Total score 26.1 General 34.7 Scientific 17.3 Coding 23.4 Agents 25.3

Related concepts

Compare

Qwen3.6 (35B-A3B)

by Alibaba Cloud 🇨🇳

Compact Qwen3.6 open-weight MoE that keeps the Qwen3.5 hybrid Gated DeltaNet/Gated Attention recipe while activating only about 3B parameters.

Scale
35B total, 3B active (8.6% active)
Context (tokens)
262,144
License
Apache License 2.0
Date
2026-04-15
Decoder type
Sparse hybrid
Attention
3:1 Gated DeltaNet and Gated Attention
Layer mix
10 gated attention + 30 DeltaNet
KV cache / token (bf16) info
20 KiB · Very low
Key detail
Uses 256 experts with 8 routed plus 1 shared expert active inside a 40-layer hybrid stack.
Pretraining data
Not disclosed The model card does not report the pretraining token count. Model card
Post-training methods
Not disclosed The card identifies a post-trained checkpoint without naming the methods. Model card
AA Intelligence Index v4.3.2 info
Total score 18.2 General 28.7 Scientific 11.3 Coding 18.3 Agents 12.7

Compare

Kimi K2.6 (1T)

by Moonshot AI 🇨🇳

Native-multimodal K2.5 successor that keeps the same 1T sparse MoE backbone while targeting stronger long-horizon coding, design, and agent orchestration.

Scale
1T total, 32B active (3.2% active)
Context (tokens)
256,000
License
Modified MIT License
Date
2026-04-20
Decoder type
Sparse MoE
Attention
MLA
Layer mix
61 MLA
KV cache / token (bf16) info
68.6 KiB · Low
Key detail
Uses the same text architecture as Kimi K2.5, with the main change coming from the multimodal and agentic training recipe.
Pretraining data
Not disclosed The version-specific model card and release post do not give a pretraining-token total. Model card · Release post
Post-training methods
Not disclosed The checked release sources describe capabilities without disclosing a version-specific post-training recipe. Model card · Release post
AA Intelligence Index v4.3.2 info
Total score 27.0 General 38.6 Scientific 22.7 Coding 26.0 Agents 19.1

Related concepts

Compare

Qwen3.6 (27B)

by Alibaba Cloud 🇨🇳

Dense Qwen3.6 model that keeps the Qwen3.5-style Gated DeltaNet/Gated Attention hybrid stack while replacing MoE blocks with dense FFNs.

Scale
27B parameters
Context (tokens)
262,144
License
Apache License 2.0
Date
2026-04-22
Decoder type
Dense hybrid
Attention
3:1 Gated DeltaNet and Gated Attention
Layer mix
16 gated attention + 48 DeltaNet
KV cache / token (bf16) info
64 KiB · Low
Key detail
Uses a 64-layer dense hybrid layout with 48 DeltaNet layers and 16 full-attention layers.
Pretraining data
Not disclosed The model card does not report the pretraining token count. Model card
Post-training methods
Not disclosed The card identifies a post-trained checkpoint without naming the methods. Model card
AA Intelligence Index v4.3.2 info
Total score 21.4 General 31.5 Scientific 12.1 Coding 21.4 Agents 17.8

Compare

Xiaomi MiMo-V2.5 (310B)

by Xiaomi 🇨🇳

Omnimodal sparse MoE model that extends the MiMo-V2-Flash backbone with vision and audio encoders.

Scale
310B total, 15B active (4.8% active)
Context (tokens)
1,048,576
License
MIT License
Date
2026-04-22
Decoder type
Sparse omnimodal MoE
Attention
5:1 sliding-window/global attention
Layer mix
39 sliding-window + 9 global
KV cache / token (bf16) info
144 KiB · Moderate
Key detail
Adds native image, video, and audio encoders to the MiMo-V2-Flash-style sparse MoE backbone with MTP support.
Pretraining data
~48T tokens Reported total for text and multimodal pretraining. Model card
Post-training methods
SFT, RL, Distillation SFT, agentic RL, and multi-teacher on-policy distillation (MOPD). Model card
AA Intelligence Index v4.3.2 info
Total score 21.8 General 30.4 Scientific 15.5 Coding 21.9 Agents 17.7

Related concepts

Compare

Xiaomi MiMo-V2.5-Pro (1.02T)

by Xiaomi 🇨🇳

Trillion-parameter sparse MoE model for long-horizon agentic and software-engineering tasks.

Scale
1.02T total, 42B active (4.1% active)
Context (tokens)
1,048,576
Vocabulary
152,576 (~152k)
License
MIT License
Date
2026-04-22
Decoder type
Sparse MoE
Attention
GQA with 6:1 sliding-window/global attention
Layer mix
60 sliding-window + 10 global
KV cache / token (bf16) info
350 KiB · Very high
Key detail
Scales the MiMo-V2.5 Pro stack to 70 layers, 384 routed experts, top-8 routing, and a 1M-token context window.
Pretraining data
27T tokens Reported pretraining budget for the Pro model. Model card
Post-training methods
SFT, RL, Distillation SFT, domain-specialized RL teachers, and multi-teacher on-policy distillation (MOPD). Model card
AA Intelligence Index v4.3.2 info
Total score 26.0 General 34.6 Scientific 19.8 Coding 25.3 Agents 22.2

Related concepts

Compare

Ling 2.6 (1T)

by InclusionAI 🇨🇳

Ling 2.5 successor that keeps the Lightning Attention and MLA hybrid stack while adding an MTP layer for multi-token prediction.

Scale
1T total, 63B active (6.3% active)
Context (tokens)
262,144
License
MIT License
Date
2026-04-23
Decoder type
Sparse hybrid
Attention
Lightning Attention plus MLA
Layer mix
10 MLA + 70 Lightning Attention + 1 MTP layer
KV cache / token (bf16) info
11.2 KiB · Very low
Key detail
Keeps the 7:1 linear-attention/MLA ratio and adds one multi-token-prediction layer for speculative decoding.
Pretraining data
~9.6T additional tokens Architectural migration and continued pretraining from the Ling 2.0 base; the inherited base training is separate from this budget. Technical report
Post-training methods
SFT, RLVR, Distillation SFT and specialist RL include verifiable reasoning and tool-use rewards. Specialist capabilities are then distilled into a unified Ling 2.6 model. Technical report
AA Intelligence Index v4.3.2 info
Total score 17.0 General N/A Scientific 4.5 Coding N/A Agents N/A

Related concepts

Compare

Tencent Hy3-preview (295B-A21B)

by Tencent 🇨🇳

Tencent Hunyuan's 295B sparse MoE preview combines a dense first FFN layer, GQA with QK-Norm, and one extra MTP layer for faster generation.

Scale
295B total, 21B active (7.1% active)
Context (tokens)
262,144
License
Tencent Hy Community License Agreement
Date
2026-04-23
Decoder type
Sparse MoE
Attention
GQA with QK-Norm
Layer mix
80 GQA + 1 MTP layer
KV cache / token (bf16) info
320 KiB · Very high
Key detail
Uses 192 routed experts, 8 routed plus 1 shared expert active per token, sigmoid routing with expert bias, and a 3.8B MTP layer.
Pretraining data
Not disclosed The model card and release post describe pretraining without a token count. Model card · Release post
Post-training methods
RL Tencent reports reinforcement learning for coding and agentic tasks; the full alignment recipe and reward types are not specified. Model card
AA Intelligence Index v4.3.2 info
Total score 17.0 General N/A Scientific 3.6 Coding N/A Agents N/A

Related concepts

Compare

Granite 4.1 (30B)

by IBM 🇺🇸

IBM's dense 30B Granite 4.1 model keeps a straightforward long-context GQA transformer stack and improves tool calling, instruction following, and chat behavior through post-training.

Scale
30B parameters
Context (tokens)
131,072
Vocabulary
100,352
License
Apache License 2.0
Date
2026-04-29
Decoder type
Dense
Attention
GQA with RoPE
Layer mix
64 GQA
KV cache / token (bf16) info
256 KiB · High
Key detail
Uses a 4096-wide dense stack, 32 query heads, 8 KV heads, SwiGLU MLPs, shared input/output embeddings, and Granite-specific MuP scaling factors.
Pretraining data
Approximately 15T tokens Multi-stage pretraining, including long-context extension. Granite 4.1 training details
Post-training methods
SFT, RLHF, RL SFT on about 4.1M examples, then multi-domain, chat RLHF, identity, and math RL using GRPO with DAPO loss. Granite 4.1 training details
AA Intelligence Index v4.3.2 info
Total score 7.4 General N/A Scientific 2.1 Coding N/A Agents N/A

Related concepts

Compare

Command A+ (218B-A25B)

by Cohere 🇨🇦

Cohere's open sparse MoE model scales the Command A parallel transformer block to 218B total parameters while keeping 25B active per token.

Scale
218B total, 25B active/token (11.5% active)
Context (tokens)
128K input, 64K output
Vocabulary
262,144
License
Apache License 2.0
Date
2026-05-20
Decoder type
Sparse MoE
Attention
16:1 GQA with 3:1 sliding-window/global attention
Layer mix
24 sliding-window + 8 global GQA/MoE
KV cache / token (bf16) info
128 KiB · Moderate
Key detail
Runs attention and MoE in parallel, which Cohere reports gives similar quality to a sequential block with better throughput.
Pretraining data
Not disclosed The model card does not report a pretraining token count. Model card
Post-training methods
Distillation Quantization-aware distillation (QAD) is confirmed for the W4A4 release. The broader alignment recipe is not disclosed. Model card
AA Intelligence Index v5.0 info
Total score 6.2 Knowledge Work 0.6 Coding 0.2 Science 1.0 General 23.2
AA Intelligence Index v4.3.2 info
Total score 13.1 General 26.4 Scientific 6.1 Coding 19.5 Agents 0.5

Related concepts

Compare

LFM2.5 (1.2B)

by Liquid AI 🇺🇸

Liquid AI's 1.2B LFM2.5 model keeps the 16-layer hybrid LIV convolution and GQA stack at a wider hidden size.

Scale
1.2B parameters
Context (tokens)
32,768
Vocabulary
65,536 (~65k)
License
LFM Open License v1.0
Date
2026-05-28
Decoder type
Dense hybrid
Attention
LIV convolution blocks plus GQA with QK-Norm
Layer mix
10 LIV conv + 6 GQA
KV cache / token (bf16) info
12 KiB · Very low
Key detail
Keeps the 16-layer LFM2.5 hybrid stack but doubles the hidden size to 2048 and uses 32 query heads.
Pretraining data
28T tokens Pretraining budget for the 1.2B base used by the Thinking checkpoint. Model card
Post-training methods
SFT, DPO, RLVR The Thinking variant uses SFT, DPO, and domain-specific RL with verifiable rewards; the final recipe also merges RL checkpoints. Release post
AA Intelligence Index v4.3.2 info
Total score 5.2 General N/A Scientific 3.1 Coding N/A Agents N/A

Related concepts

Compare

LFM2.5 (350M)

by Liquid AI 🇺🇸

Liquid AI's compact LFM2.5 model combines double-gated LIV convolution blocks with periodic GQA layers.

Scale
350M parameters
Context (tokens)
32,768
Vocabulary
65,536 (~65k)
License
LFM Open License v1.0
Date
2026-05-28
Decoder type
Dense hybrid
Attention
LIV convolution blocks plus GQA with QK-Norm
Layer mix
10 LIV conv + 6 GQA
KV cache / token (bf16) info
12 KiB · Very low
Key detail
Uses a 16-layer hybrid stack with 10 LIV convolution layers and 6 GQA layers.
Pretraining data
28T tokens The pretraining budget was extended from 10T to 28T tokens. Model card · Release post
Post-training methods
RL The release confirms large-scale multi-stage reinforcement learning without a complete breakdown of the alignment methods. Release post

Related concepts

Compare

LFM2.5 (8B-A1B)

by Liquid AI 🇺🇸

Liquid AI's sparse LFM2.5 model combines LIV convolution, GQA, and MoE routing with 4 of 32 experts active per token.

Scale
8.3B total, 1.5B active (18.1% active)
Context (tokens)
128,000
Vocabulary
128,000
License
LFM Open License v1.0
Date
2026-05-28
Decoder type
Sparse hybrid
Attention
LIV convolution blocks plus GQA with QK-Norm and MoE
Layer mix
18 LIV conv + 6 GQA + MoE FFNs
KV cache / token (bf16) info
12 KiB · Very low
Key detail
Uses 24 layers, 32 MoE experts with 4 active per token, and leaves the first two layers dense before sparse routing begins.
Pretraining data
38T tokens The release reports pretraining scaled from 12T to 38T tokens. It also describes 2T and 400B context-extension stages without clearly separating them from that total. Release post
Post-training methods
RL, Preference optimization Large-scale RL includes a targeted hallucination-reduction stage. A separate preference-optimization stage reduces reasoning loops; its objective is not named DPO. Release post
AA Intelligence Index v4.3.2 info
Total score 7.2 General N/A Scientific 3.4 Coding N/A Agents N/A

Related concepts

Compare

JetBrains Mellum2 Thinking (12B-A2.5B)

by JetBrains 🇨🇿

JetBrains' Mellum2 Thinking is a compact reasoning-tuned sparse MoE for code and agentic workflows.

Scale
12B total, 2.5B active (20.8% active)
Context (tokens)
131,072
Vocabulary
98,304 (~98k)
License
Apache License 2.0
Date
2026-06-01
Decoder type
Sparse MoE
Attention
GQA with 3:1 sliding-window/full attention
Layer mix
21 sliding-window + 7 full GQA
KV cache / token (bf16) info
56 KiB · Low
Key detail
Uses 64 experts with 8 active per token, a 1024-token sliding window, and YaRN scaling on full-attention layers.
Pretraining data
~10.6T + 117B context-extension tokens Approximately 10.6T tokens in the main three-phase pretraining curriculum, followed by a separate approximately 117B-token long-context stage. Technical report
Post-training methods
SFT, RLVR The Thinking checkpoint uses SFT followed by RL with verifiable rewards, using GRPO across code, math, tool use, and other tasks. Technical report

Related concepts

Compare

Nemotron 3 Ultra (550B-A55B)

by NVIDIA 🇺🇸

NVIDIA's Ultra model scales the Nemotron 3 hybrid stack to 550B total parameters while keeping 55B active per token.

Scale
550B total, 55B active (10% active)
Context (tokens)
1,000,000
License
NVIDIA Nemotron Open Model License
Date
2026-06-04
Decoder type
Hybrid MoE
Attention
Mostly Mamba-2 with a few GQA layers using NoPE
Layer mix
12 GQA + 48 Mamba-2 + 48 MoE
KV cache / token (bf16) info
12 KiB · Very low
Key detail
Scales the latent-MoE hybrid to 108 layers with 48 Mamba-2, 48 latent MoE, and 12 GQA layers.
Pretraining data
20T + 33B long-context tokens The main pretraining run uses 20T tokens, followed by 33B tokens of long-context continued pretraining. Nemotron 3 Ultra technical report
Post-training methods
SFT, RLVR, RLHF, Distillation SFT and multi-environment RLVR train the student. Specialized teachers, including RLHF-trained chat and alignment teachers, are consolidated through multi-teacher on-policy distillation. Nemotron 3 Ultra technical report
AA Intelligence Index v5.0 info
Total score 11.8 Knowledge Work 12.9 Coding 0.9 Science 6.0 General 27.4
AA Intelligence Index v4.3.2 info
Total score 22.9 General 34.1 Scientific 15.8 Coding 20.4 Agents 18.5

Related concepts

Compare

North Mini Code (30B-A3B)

by Cohere 🇨🇦

Cohere's compact agentic coding MoE adapts the parallel Cohere2-MoE block to a 30B total, 3B active model.

Scale
30B total, 3B active (10% active)
Context (tokens)
256K input, 64K output
Vocabulary
262,144
License
Apache License 2.0
Date
2026-06-05
Decoder type
Sparse MoE
Attention
8:1 GQA with 3:1 sliding-window/global attention
Layer mix
36 sliding-window + 13 global GQA/MoE
KV cache / token (bf16) info
98 KiB · Moderate
Key detail
Runs attention and MoE in parallel, with RoPE in sliding-window layers, NoPE in global layers, and top-8 routing over 128 experts.
Pretraining data
Not disclosed The model card and release article do not report a pretraining token count. Training overview
Post-training methods
SFT, RLVR Two-stage cascaded SFT followed by asynchronous RL with verifiable rewards for coding agents. Training overview
AA Intelligence Index v4.3.2 info
Total score 9.9 General 15.9 Scientific 5.7 Coding 19.6 Agents 0.2

Related concepts

Compare

Kimi K2.7 Code (1T)

by Moonshot AI 🇨🇳

Coding-focused K2.6 successor that keeps the same 1T sparse MoE text backbone and emphasizes long-horizon software engineering agents.

Scale
1T total, 32B active (3.2% active)
Context (tokens)
256,000
License
Modified MIT License
Date
2026-06-12
Decoder type
Sparse MoE
Attention
MLA
Layer mix
61 MLA
KV cache / token (bf16) info
68.6 KiB · Low
Key detail
Keeps the Kimi K2.5/K2.6 text architecture and shifts the release focus toward coding-agent workflows.
Pretraining data
Not disclosed The release card does not disclose a version-specific pretraining-token total. Model card
Post-training methods
Not disclosed The release card does not specify the post-training methods used for this checkpoint. Model card
AA Intelligence Index v4.3.2 info
Total score 25.8 General 33.1 Scientific 22.5 Coding 24.4 Agents 22.0

Related concepts

Compare

MiniMax M3 (428B)

by MiniMax 🇨🇳

Native multimodal MiniMax model that adds sparse attention to an M2-style MoE backbone for 1M-token context.

Scale
428B total, 23B active (5.4% active)
Context (tokens)
1,048,576
License
MiniMax Community License
Date
2026-06-13
Decoder type
Sparse MoE
Attention
GQA with QK-Norm and MiniMax Sparse Attention
Layer mix
3 full GQA + 57 MiniMax Sparse Attention
KV cache / token (bf16) info
120 KiB · Moderate
Key detail
Uses three dense prefix layers, then 57 sparse-attention MoE layers with 128 routed experts and 4 routed plus 1 shared expert active per token.
Pretraining data
Not disclosed The release describes native multimodal pretraining but gives no token total for the production 428B model. MiniMax M3 release
Post-training methods
RL, Rejection sampling Proof and verifier experts use RL with generative judges. Rejection-sampled fine-tuning trains a repair expert. These experts enter M3's merged training; the complete recipe is unspecified. MiniMax MaxProof training details
AA Intelligence Index v5.0 info
Total score 18.7 Knowledge Work 23.9 Coding 13.9 Science 10.9 General 26.1
AA Intelligence Index v4.3.2 info
Total score 29.2 General 36.3 Scientific 21.3 Coding 24.6 Agents 30.7

Related concepts

Compare

VibeThinker-3B (3B)

by Weibo 🇨🇳

Qwen2.5-Coder-based dense reasoner whose gains mainly come from verifier-driven post-training rather than a new block design.

Scale
3B parameters
Context (tokens)
131,072
Vocabulary
151,936 (~152k)
License
MIT License
Date
2026-06-15
Decoder type
Dense
Attention
8:1 GQA with RoPE
Layer mix
36 GQA
KV cache / token (bf16) info
36 KiB · Low
Key detail
Standard Qwen2.5-Coder-style dense stack strengthened by curriculum SFT, MGPO RL, Long2Short RL, offline self-distillation, and Instruct RL.
Pretraining data
5.5T tokens (base-model coding training) Inherited Qwen2.5-Coder-3B budget; VibeThinker adds post-training rather than a new reported pretraining run. Technical report · Model card
Post-training methods
SFT, RLVR, Distillation, RL Two-stage SFT, verifiable reasoning RL, offline self-distillation, then instruction RL with rubric rewards. Technical report

Related concepts

Compare

GLM-5.2 (744B)

by Z.ai 🇨🇳

Long-context GLM refresh that keeps the 744B sparse MoE backbone and adds IndexShare for 1M-token DSA inference.

Scale
744B total, 40B active (5.4% active)
Context (tokens)
1,048,576
License
MIT License
Date
2026-06-17
Decoder type
Sparse MoE
Attention
MLA with DeepSeek Sparse Attention and IndexShare
Layer mix
78 MLA/DSA
KV cache / token (bf16) info
87.8 KiB · Moderate
Key detail
Reuses each full DSA indexer result across the next three layers, making 1M-token sparse attention cheaper.
Pretraining data
Not disclosed The model card does not publish a pretraining-token total for this version. GLM-5.2 model card
Post-training methods
RL, Distillation Agentic RL includes PPO-based CompactionRL. Parallel on-policy distillation merges more than ten expert models. CompactionRL report · GLM-5.2 release
AA Intelligence Index v4.3.2 info
Total score 33.7 General 36.9 Scientific 31.0 Coding 26.1 Agents 37.5

Related concepts

Compare

Laguna XS 2.1 (33B)

by Poolside 🇺🇸

Poolside's 33B Laguna refresh extends the sparse MoE architecture to 262k context while keeping about 3B parameters active per token.

Scale
33B total, 3B active (9.1% active)
Context (tokens)
262,144
Vocabulary
100,352 (~100k)
License
OpenMDW License 1.1
Date
2026-07-02
Decoder type
Sparse MoE
Attention
Gated GQA with QK-Norm and 3:1 sliding-window/global attention
Layer mix
30 sliding-window + 10 global GQA
KV cache / token (bf16) info
160 KiB · Moderate
Key detail
Uses 256 routed experts with top-8 routing plus one shared expert, a dense first layer, and per-layer query-head counts.
Pretraining data
Not disclosed The version-specific release sources do not give a token total; the linked earlier report covers XS.2. Model card · Release post
Post-training methods
RL The model card confirms reinforcement learning but does not publish the complete version-specific post-training recipe. Model card

Compare

Soofi S (30B-A3B)

by SOOFI Consortium 🇩🇪

German-English base model that retrains the Nemotron 3 Nano architecture on 26.68T tokens with a fully documented data mixture.

Scale
31.6B total, 3.2B active (10.1% active)
Context (tokens)
1,048,576
License
Custom license (preview)
Date
2026-07-10
Decoder type
Hybrid MoE
Attention
Mostly Mamba-2 with 6 GQA layers and no positional embeddings (NoPE)
Layer mix
6 GQA + 23 Mamba-2 + 23 MoE
KV cache / token (bf16) info
6 KiB · Very low
Key detail
Reuses the Nemotron 3 Nano architecture without modification, isolating the effect of a German-English data and training recipe.
Pretraining data
~26.68T tokens Consumed tokens across pretraining, annealing, and context extension; the report rounds this to ~27T. Technical report
Post-training methods
None (base model) This entry is Soofi-S-Base. Instruction and reasoning post-training are described as future work. Technical report

Related concepts

Compare

Inkling (975B)

by Thinking Machines Lab 🇺🇸

Thinking Machines Lab's native multimodal model combines a 66-layer sparse MoE decoder with a 5:1 mix of local and global GQA.

Scale
975B total, 41B active (4.2% active)
Context (tokens)
1,048,576
Vocabulary
200,058 (~200k)
License
Apache License 2.0
Date
2026-07-15
Decoder type
Sparse MoE
Attention
GQA with QK-Norm and learned relative-position bias in a 5:1 sliding-window/global pattern
Layer mix
55 sliding-window + 11 global GQA
KV cache / token (bf16) info
484 KiB · Very high
Key detail
Uses learned relative-position bias instead of RoPE and adds four short convolutions to each decoder layer.
Pretraining data
45T tokens Multimodal pretraining on text, images, audio, and video. Training overview
Post-training methods
SFT, RL Synthetic-data SFT bootstrap followed by large-scale asynchronous RL across multiple domains. Training overview
AA Intelligence Index v5.0 info
Total score 12.5 Knowledge Work 13.5 Coding 2.8 Science 7.2 General 26.5
AA Intelligence Index v4.3.2 info
Total score 25.0 General 36.4 Scientific 18.7 Coding 24.0 Agents 18.7

Related concepts

Compare

BTL-3 (27B)

by Bad Theory Labs 🇳🇬

Agent-oriented Qwen3.6-27B derivative that adds a rank-32 LoRA adapter for coding, structured tool use, and long multi-turn execution.

Scale
27B base + 934 MB LoRA adapter
Context (tokens)
262,144
Vocabulary
248,320 (~248k)
License
Apache License 2.0
Date
2026-07-20
Decoder type
Dense hybrid
Attention
3:1 Gated DeltaNet and Gated Attention
Layer mix
16 gated attention + 48 DeltaNet
KV cache / token (bf16) info
64 KiB · Low
Key detail
Retains Qwen3.6-27B's 64-layer hybrid stack while adapting its attention, DeltaNet, and FFN projections through rank-32 LoRA.
Pretraining data
Not disclosed Qwen3.6-27B derivative; its model card does not disclose a pretraining token count. BTL-3 model card
Post-training methods
RL, LoRA Released as the RL-0013 rank-32 LoRA adapter. The card does not specify the reward source or a separate SFT stage. BTL-3 model card

Compare

Antares (1B)

by Cisco 🇺🇸

Cisco Foundation AI's compact terminal agent uses a dense Granite 4.0 backbone and is post-trained with SFT and GRPO for vulnerability localization.

Scale
1B parameters
Context (tokens)
131,072
Vocabulary
100,352
License
Apache License 2.0
Date
2026-07-21
Decoder type
Dense
Attention
GQA with RoPE
Layer mix
40 GQA
KV cache / token (bf16) info
80 KiB · Moderate
Key detail
Retains Granite 4.0's MuP scaling multipliers and tied input/output embeddings, then adds SFT and GRPO for terminal-based vulnerability localization.
Pretraining data
Not disclosed Initialized from Granite 4.0 1B; the Antares report does not give an inherited pretraining-token total. Antares report
Post-training methods
SFT, RLVR, Distillation Teacher-generated SFT trajectories, followed by GRPO with vulnerability-localization rewards against ground truth. Antares report

Related concepts

Compare

Laguna S 2.1 (118B)

by Poolside 🇺🇸

Poolside's 118B sparse MoE scales the Laguna XS recipe to 48 layers and 1M context while activating about 8B parameters per token.

Scale
118B total, 8B active (6.8% active)
Context (tokens)
1,048,576
Vocabulary
100,352 (~100k)
License
OpenMDW License 1.1
Date
2026-07-21
Decoder type
Sparse MoE
Attention
Gated GQA with QK-Norm and 3:1 sliding-window/global attention
Layer mix
36 sliding-window + 12 global GQA
KV cache / token (bf16) info
192 KiB · High
Key detail
Uses 256 routed experts with top-10 routing plus one shared expert, a dense first layer, and per-layer query-head counts.
Pretraining data
Not disclosed Uses the same pretraining data as XS 2.1, but the release post does not give a token count for either run. Release post
Post-training methods
SFT, RLVR SFT with partly synthetic data is followed by RL on harder tasks, including software changes checked against executable test suites. Release post

Compare

Motif 3 Beta (314B)

by Motif Technologies 🇰🇷

Large sparse MoE that combines Grouped Differential Latent Attention, per-expert PolyNorm, and modified mHC in a 53-layer decoder.

Scale
314B total, 13B active (4.1% active)
Context (tokens)
262,144
Vocabulary
220,160 (~220k)
License
Motif Technologies Non-Commercial License
Date
2026-07-21
Decoder type
Sparse MoE
Attention
GDLA with MLA-style latent projections, RoPE, and 3:1 sliding-window/full attention
Layer mix
39 sliding-window + 14 full GDLA
KV cache / token (bf16) info
530 KiB · Very high
Key detail
Routes each token through 8 of 384 experts plus one shared expert, with per-expert PolyNorm and a 3:1 sliding-window/full GDLA pattern.
Pretraining data
Not disclosed This is an intermediate beta checkpoint. The later Motif 3 report gives about 12.5T tokens for the final release, without a separate beta budget. Motif 3 Beta model card · Motif 3 final-release report
Post-training methods
Not disclosed The beta card gives no training recipe. The later final-release SFT and distillation pipeline is not explicitly attributed to this checkpoint. Motif 3 Beta model card · Motif 3 final-release report
AA Intelligence Index v4.3.2 info
Total score 32.3 General N/A Scientific 23.2 Coding N/A Agents N/A

Related concepts

Compare

Nanbeige 4.2 (3B)

by BOSS Zhipin 🇨🇳

Compact agentic model that runs the same 22-layer dense transformer stack twice, resulting in 44 transformer blocks (layers) without duplicating the layer weights.

Scale
4B total, 3B non-embedding
Context (tokens)
262,144
Vocabulary
166,144 (~166k)
License
Apache License 2.0
Date
2026-07-21
Decoder type
Dense
Attention
GQA with RoPE
Layer mix
22 shared GQA layers × 2 passes
KV cache / token (bf16) info
176 KiB · High
Key detail
Reuses the same 22-layer stack for a second pass, giving 44 block applications with 3B non-embedding parameters.
Pretraining data
28T tokens The new looped architecture was pretrained from scratch on 28T tokens. Nanbeige 4.2 technical report
Post-training methods
SFT, RLHF, RLVR Three-stage SFT, two-stage RLHF, reasoning RL with correctness and length rewards, and agentic RL using executable task checks and action rubrics. Nanbeige 4.2 technical report

Compare

Solar Open 2 (250B)

by Upstage 🇰🇷

Upstage's sparse hybrid alternates one gated GQA layer with three Kimi Delta Attention layers and reaches a 1M-token context without positional embeddings.

Scale
250B total, 15B active (6% active)
Context (tokens)
1,048,576
Vocabulary
196,608 (~197k)
License
Upstage Solar License
Date
2026-07-22
Decoder type
Sparse hybrid
Attention
3:1 Kimi Delta Attention and gated GQA with NoPE
Layer mix
12 gated GQA + 36 Kimi Delta Attention
KV cache / token (bf16) info
48 KiB · Low
Key detail
Uses a repeating gated-GQA/KDA hybrid stack, kernel-4 ShortConv in KDA, and top-8 routing over 320 experts plus one shared expert.
Pretraining data
~12T additional tokens After selective weight transfer from Solar Open 1, the stages consume 10T + 1T + 0.9T tokens. Technical report
Post-training methods
SFT, RLVR, Distillation SFT and multi-domain verifiable RL are followed by specialist training and multi-teacher on-policy distillation. Technical report
AA Intelligence Index v4.3.2 info
Total score 24.7 General N/A Scientific 17.1 Coding N/A Agents N/A

Compare

Kimi K3 (2.8T)

by Moonshot AI 🇨🇳

2.8T sparse hybrid that combines a 3:1 KDA and Gated MLA stack with Block Attention Residuals and Stable LatentMoE. It is also native multimodal, but the figure focuses on the text portion.

Scale
2.8T total, 104B active (3.7% active)
Context (tokens)
1,048,576
License
Kimi K3 License
Date
2026-07-27
Decoder type
Sparse hybrid
Attention
3:1 Kimi Delta Attention and Gated MLA with NoPE
Layer mix
69 KDA + 24 Gated MLA
KV cache / token (bf16) info
27 KiB · Low
Key detail
Uses eight Block Attention Residual groups and Stable LatentMoE with 16 of 896 routed experts active per token.
Pretraining data
Not disclosed The report describes native multimodal pretraining and context extension without publishing the total training tokens. Technical report
Post-training methods
SFT, RL, Distillation SFT is followed by domain- and effort-specialized RL, then multi-teacher on-policy distillation into one model. Technical report
AA Intelligence Index v5.0 info
Total score N/A Knowledge Work N/A Coding N/A Science 20.7 General N/A
AA Intelligence Index v4.3.2 info
Total score 43.6 General 45.8 Scientific 35.2 Coding 36.1 Agents 51.9

Compare

Ling 3.0 Flash (124B)

by InclusionAI 🇨🇳

Hybrid MoE that alternates five Kimi Delta Attention layers with one gated MLA layer.

Scale
124B total, 5.1B active (4.1% active)
Context (tokens)
262,144
Vocabulary
157,184 (~157k)
License
MIT License
Date
2026-08-04
Decoder type
Sparse hybrid
Attention
5:1 Kimi Delta Attention and gated MLA
Layer mix
35 Kimi Delta Attention + 7 gated MLA + 1 MTP layer
KV cache / token (bf16) info
7.9 KiB · Very low
Key detail
Uses fine-grained diagonal KDA gating, top-8 routing over 512 experts plus one shared expert, and one MTP layer.
Pretraining data
>10T tokens (training corpus) The official disclosure gives a lower bound on the preprocessed text corpus for the Ling 3.0 family, including Flash. It does not give tokens seen during pretraining. Training data summary
Post-training methods
RL Ant's developer documentation attributes improved long-horizon tool calling to reinforcement learning with stronger checks. The full alignment recipe is not disclosed. Developer documentation
AA Intelligence Index v4.3.2 info
Total score 20.1 General 29.3 Scientific 12.7 Coding 21.0 Agents 15.4

Compare

Muse Glimmer (30B)

by Meta 🇺🇸

Meta's dense 29.6B multimodal model combines a 52-layer sandwich-norm decoder with gated GQA and a separate perception encoder; the figure focuses on the text backbone.

Scale
29.6B parameters (including perception encoder)
Context (tokens)
131,072+
Vocabulary
202,048 (~202k)
License
Apache License 2.0
Date
2026-08-10
Decoder type
Dense
Attention
Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention
Layer mix
39 sliding-window + 13 global gated GQA
KV cache / token (bf16) info
52 KiB · Low
Key detail
Applies RoPE only in 2,048-token local layers, uses NoPE in global layers, and gates the attention output before projection.
Pretraining data
Not disclosed The report describes pretraining through logit co-distillation and later mid-training, but does not report total training tokens. Muse Glimmer technical report
Post-training methods
SFT, Distillation, RL Supervised fine-tuning, on-policy distillation from Muse Spark, and reinforcement learning. Muse Glimmer technical report
AA Intelligence Index v5.0 info
Total score 6.3 Knowledge Work 3.4 Coding 1.1 Science 2.8 General 18.1
AA Intelligence Index v4.3.2 info
Total score 17.5 General 29.2 Scientific 12.3 Coding 22.7 Agents 5.9

Compare

Nemotron 3.5 Lightning (30B-A3B)

by NVIDIA 🇺🇸

NVIDIA's 30B hybrid model keeps 3B parameters active and extends the Nemotron 3 Nano backbone with a 1M-token context window and multi-token prediction.

Scale
30B total, 3B active (10% active)
Context (tokens)
1,048,576
Vocabulary
131,072
License
OpenMDW 1.1
Date
2026-08-11
Decoder type
Hybrid MoE
Attention
Mostly Mamba-2 with six GQA layers using NoPE
Layer mix
6 GQA + 23 Mamba-2 + 23 MoE; GQA + MoE MTP predictor
KV cache / token (bf16) info
6 KiB · Very low
Key detail
Adds a shared GQA-plus-MoE MTP predictor to the 52-layer Nemotron 3 Nano backbone.
Pretraining data
1.021T additional tokens Continued from Nemotron 3 Nano Base with 1T distillation tokens and 21B long-context tokens. A separate 100B-token MTP-head warmup freezes the backbone. Nemotron 3.5 technical report
Post-training methods
SFT, RLVR Single-stage text-only supervised fine-tuning followed by reinforcement learning with verifiable rewards. Nemotron 3.5 technical report
AA Intelligence Index v5.0 info
Total score 5.5 Knowledge Work 1.0 Coding 0.2 Science 0.6 General 20.6
AA Intelligence Index v4.3.2 info
Total score 12.9 General 26.3 Scientific 5.3 Coding 16.3 Agents 2.6

Related concepts

Compare

Qwen3.8 (27B)

by Alibaba Cloud 🇨🇳

Dense 27B native vision-language model with a 64-layer Gated DeltaNet and Gated Attention language backbone; the figure omits the vision encoder and MTP head.

Scale
27B parameters
Context (tokens)
262,144
Vocabulary
248,320 (~248k)
License
Apache License 2.0
Date
2026-08-14
Decoder type
Dense hybrid
Attention
3:1 Gated DeltaNet and Gated Attention
Layer mix
16 gated attention + 48 DeltaNet
KV cache / token (bf16) info
64 KiB · Low
Key detail
Uses a 64-layer 3:1 Gated DeltaNet/Gated Attention decoder with partial RoPE and one MTP layer.
Pretraining data
Not disclosed The model card does not report the pretraining token count. Model card
Post-training methods
Not disclosed The card identifies a post-trained checkpoint without naming the methods. Model card
AA Intelligence Index v5.0 info
Total score 20.1 Knowledge Work 39.7 Coding 11.2 Science 1.9 General 27.3
AA Intelligence Index v4.3.2 info
Total score 33.7 General 35.9 Scientific 19.7 Coding 26.1 Agents 45.8

Compare

Ornith 1.5 (35B-A3B)

by Ornith 🇺🇸

Uses the same sparse-hybrid decoder geometry as Qwen3.6 35B-A3B, paired with Ornith's self-improvement training recipe.

Scale
35B total, 3B active (8.6% active)
Context (tokens)
262,144
Vocabulary
248,320 (~248k)
License
MIT License
Date
2026-08-23
Decoder type
Sparse hybrid
Attention
3:1 Gated DeltaNet and Gated Attention
Layer mix
10 gated attention + 30 DeltaNet
KV cache / token (bf16) info
20 KiB · Very low
Key detail
Matches Qwen3.6 35B-A3B's 40-layer decoder geometry, with 256 routed experts, top-8 routing, and one shared expert.
Pretraining data
Not disclosed The card describes its Ornith 1.0 lineage but provides no inherited or additional pretraining-token total. Ornith 1.5 model card
Post-training methods
RL GRPO-based self-improvement trains on generated tasks, scaffolds, and agent rollouts. Rewards include correctness on verifiable tasks and broader completion criteria. Ornith 1.5 training details

Compare

GLM-5.3-Flash (320B)

by Z.ai 🇨🇳

Natively multimodal GLM refresh, previously previewed as Ox Alpha, that replaces most MLA/DSA layers with Kimi Delta Attention and adds four-stream mHC.

Scale
320B total, 18B active (5.6% active)
Context (tokens)
1,048,576
Vocabulary
154,880 (~155k)
License
MIT License
Date
2026-08-26
Decoder type
Sparse hybrid
Attention
3:1 Kimi Delta Attention (Q/K L2 normalization) and MLA with DeepSeek Sparse Attention
Layer mix
34 KDA + 11 MLA/DSA + 1 MTP layer
KV cache / token (bf16) info
11 KiB · Very low
Key detail
Mixes 34 Kimi Delta Attention layers with 11 MLA/DSA layers and routes residuals through four mHC streams.
Pretraining data
30T-token multimodal corpus Reported corpus size for a newly trained base; a separate tokens-seen total is not given. GLM-5.3 Flash release
Post-training methods
RL The release describes frontend-coding RL with environment feedback; other alignment stages are not itemized. GLM-5.3 Flash release
AA Intelligence Index v5.0 info
Total score 27.8 Knowledge Work 43.7 Coding 23.7 Science 12.5 General 30.9
AA Intelligence Index v4.3.2 info
Total score 41.8 General 39.7 Scientific 27.6 Coding 42.2 Agents 53.0

Compare

Qwen3.8-Flash-Next (125B-A6B)

by Alibaba Cloud 🇨🇳

Experimental vision-language MoE whose text backbone combines Gated DeltaNet, micro-block QSA, four-branch gated residuals, and n-gram embeddings; the figure omits the vision encoder and MTP head.

Scale
125B core, 6B active (4.8% active); plus 51B n-gram embeddings and 4B MTP
Context (tokens)
262,144
Vocabulary
248,320 (~248k)
License
Qwen Community License 1.0
Date
2026-08-26
Decoder type
Sparse hybrid
Attention
3:1 Gated DeltaNet and Qwen Sparse Attention (QSA)
Layer mix
12 QSA + 36 DeltaNet
KV cache / token (bf16) info
24 KiB · Very low
Key detail
Replaces full attention with micro-block QSA while using four gated residual branches and a 20M-entry n-gram embedding at layer 2.
Pretraining data
Not disclosed The architecture report gives proxy training budgets and relative savings, but no full-model token total. Architecture report
Post-training methods
Not disclosed The model card confirms post-training without specifying its methods. Model card
AA Intelligence Index v5.0 info
Total score N/A Knowledge Work N/A Coding N/A Science N/A General 26.6
AA Intelligence Index v4.3.2 info
Total score 39.8 General 35.8 Scientific 24.6 Coding 37.9 Agents 55.1

Compare

Tencent Hy4-preview (770B-A49B)

by Tencent 🇨🇳

Tencent's 770B sparse MoE combines gated DeepSeek Sparse Attention, cross-layer index reuse, and four-stream identity Hyper-Connections.

Scale
770B backbone, 49B active (6.4% active); plus 10B MTP
Context (tokens)
1,048,576
Vocabulary
120,832 (~121k)
License
Apache License 2.0
Date
2026-08-28
Decoder type
Sparse MoE
Attention
Gated MLA with DeepSeek Sparse Attention and IndexCache
Layer mix
78 gated MLA/DSA + 1 MTP layer
KV cache / token (bf16) info
87.8 KiB · Moderate
Key detail
Uses 256 routed experts with 8 routed plus 1 shared expert active per token, a top-2,048 DSA indexer, cross-layer index reuse, and four iHC residual streams.
Pretraining data
Not disclosed The release card says training data was scaled up but does not give a token count. Model card
Post-training methods
RL Core contributor Chenchen Zhang reports reinforcement-learning work on Hy4 preview. The full post-training recipe and reward types are not disclosed. Core contributor's project description · Model card

Compare

DeepSeek V4.1-Flash (552B)

by DeepSeek 🇨🇳

Causal encoder-decoder MoE with CSA2 cache sharing, Engram memory, and a million-token context. The figure shows the text backbone.

Scale
552B backbone + 196B Engram; 8B active prefill, 16B active decode (2.9% of backbone)
Context (tokens)
1,048,576
Vocabulary
129,280
License
MIT License
Date
2026-09-10
Decoder type
Sparse MoE (causal encoder-decoder)
Attention
MLA-style CSA2 + SWA with cross-layer KV and sparse-index reuse
Layer mix
20 encoder + 20 decoder layers; 2 SWA-only + 38 CSA2/SWA
KV cache / token (bf16) info
3.125 KiB · Very low
Key detail
Decoder global KV comes from final encoder states. Single-Pass mHC mixes four residual streams around each attention and MoE sublayer.
Pretraining data
45T tokens Multimodal pretraining from scratch, including long-context training. DeepSeek V4.1 Flash model card
Post-training methods
SFT, RL, Distillation SFT, RL, then on-policy distillation; agent-task and environment synthesis expands the training data. DeepSeek V4.1 Flash model card
AA Intelligence Index v4.3.2 info
Total score 39.5 General 34.3 Scientific 26.8 Coding 39.3 Agents 52.8

Related concepts

Compare

Step-5 Preview (600B)

by StepFun 🇨🇳

StepFun's 92-layer MoE pairs 512-token sliding-window attention with sparse GQA for a 1M-token context window; the figure shows the text decoder.

Scale
600B total, 27B active (4.5% active)
Context (tokens)
1,048,576
Vocabulary
128,896 (~129k)
License
N/A
Date
2026-09-20
Decoder type
Sparse MoE
Attention
Gated GQA with QK-Norm and 3:1 sliding-window/sparse attention
Layer mix
69 sliding-window + 23 sparse GQA + 3 MTP layers
KV cache / token (bf16) info
N/A
Key detail
Uses top-512 sparse indexing with block size 8, 352 routed experts with 8 active plus one shared expert, and MTP-3. The first three and final backbone layers use dense FFNs with intermediate size 13,824.
Pretraining data
Not disclosed The official training overview does not report a pretraining token count. Training overview
Post-training methods
SFT, RLVR The overview identifies SFT and agentic RL with verifiable environments, outcome rewards, and a learned critic. Training overview
AA Intelligence Index v5.0 info
Total score 28.5 Knowledge Work 40.9 Coding 20.5 Science 14.4 General 38.0
AA Intelligence Index v4.3.2 info
Total score 43.7 General 43.0 Scientific 33.7 Coding 46.1 Agents 49.7

Compare

Xiaomi MiMo-V2.6-Flash-RL (309B)

by Xiaomi 🇨🇳

Omnimodal sparse MoE model with the MiMo-V2.5 text backbone, mixed-domain reinforcement learning, and a five-layer DFlash speculative decoder.

Scale
309B total, 15B active (4.9% active)
Context (tokens)
1,048,576
Vocabulary
152,576 (~152k)
License
MIT License
Date
2026-09-21
Decoder type
Sparse omnimodal MoE
Attention
GQA with sliding-window/global attention
Layer mix
39 sliding-window + 9 global
KV cache / token (bf16) info
217.5 KiB · High
Key detail
Uses 256 routed experts with top-8 routing, one dense prefix layer, no shared experts, and a 128-token sliding window.
Pretraining data
48T tokens 26T text-stage tokens plus 22T omni-stage tokens. Technical report
Post-training methods
SFT, RLVR, Distillation Short SFT, mixed-task GRPO using tests and agentic graders, and MOPD2 distillation described in the series recipe. Technical report
AA Intelligence Index v5.0 info
Total score 26.7 Knowledge Work 44.2 Coding 22.1 Science 14.4 General 25.9
AA Intelligence Index v4.3.2 info
Total score 37.9 General 32.0 Scientific 23.5 Coding 37.0 Agents 53.9

Related concepts

Compare

Xiaomi MiMo-V2.6-Pro-RL (1.02T)

by Xiaomi 🇨🇳

Omnimodal sparse MoE model with the MiMo-V2.5-Pro text backbone, mixed-domain reinforcement learning, and a five-layer DFlash speculative decoder.

Scale
1.02T total, 42B active (4.1% active)
Context (tokens)
1,048,576
Vocabulary
152,576 (~152k)
License
MIT License
Date
2026-09-21
Decoder type
Sparse omnimodal MoE
Attention
GQA with 6:1 sliding-window/global attention
Layer mix
60 sliding-window + 10 global
KV cache / token (bf16) info
350 KiB · Very high
Key detail
Uses 384 routed experts with top-8 routing, one dense prefix layer, no shared experts, and a 128-token sliding window.
Pretraining data
30T tokens 27T text-stage tokens plus 3T omni-stage tokens. Technical report
Post-training methods
SFT, RLVR, Distillation Short SFT, mixed-task GRPO using tests and agentic graders, and MOPD2 distillation described in the series recipe. Technical report
AA Intelligence Index v5.0 info
Total score 29.9 Knowledge Work 45.9 Coding 28.3 Science 17.0 General 28.0
AA Intelligence Index v4.3.2 info
Total score 46.3 General 42.3 Scientific 38.0 Coding 47.9 Agents 54.9

Related concepts

Compare

Ember-1 (2.8T)

by Fireworks AI 🇺🇸

Fireworks' Kimi K3 derivative, post-trained for shorter reasoning traces. Architecture details follow the published Kimi K3 base model; the diagram focuses on its text backbone.

Scale
2.8T total, 104B active (3.7% active)
Context (tokens)
1,048,576
License
N/A
Date
2026-09-23
Decoder type
Sparse hybrid
Attention
3:1 Kimi Delta Attention and Gated MLA with NoPE
Layer mix
69 KDA + 24 Gated MLA
KV cache / token (bf16) info
27 KiB · Low
Key detail
Inherits Kimi K3's eight Block Attention Residual groups and Stable LatentMoE, with one dense layer followed by 92 MoE layers.
Pretraining data
Not disclosed Derived from Kimi K3; the release does not disclose a pretraining-token total. Ember-1 release
Post-training methods
Not disclosed The release describes additional training to shorten reasoning but does not name its training stages. Ember-1 release

Compare

Naive-N0.5-Flash (309B)

by NaiveAI 🇨🇳

MiMo-V2.5-derived sparse MoE that replaces its global attention layers with GQA-based DeepSeek Sparse Attention for a native 1M-token context.

Scale
309B total, 15.5B active (5.0% active)
Context (tokens)
1,048,576
Vocabulary
152,576 (~152k)
License
MIT License
Date
2026-09-27
Decoder type
Sparse MoE
Attention
GQA with sliding-window attention and DeepSeek Sparse Attention
Layer mix
39 sliding-window + 9 GQA/DSA
KV cache / token (bf16) info
219.75 KiB · High
Key detail
Pairs a 128-token local window with top-2,048 DSA selection, four DSA KV heads, and a 16-head indexer. Uses 256 routed experts with top-8 routing and no full-attention layers.
Pretraining data
3T additional pretraining tokens Continued from MiMo-V2.5. The reported 3.25T adaptation budget also includes 50B frozen-backbone indexer-warmup tokens and 200B SFT tokens; inherited base training is separate. NaiveAI training procedure
Post-training methods
SFT The final learning-rate decay stage uses supervised fine-tuning for 200B tokens. NaiveAI training procedure

Related concepts

Compare

Kolibri-1 (78B)

by Aleph Alpha 🇩🇪

Aleph Alpha's German-English reasoning model uses 50 MoE blocks with a 4:1 mix of sliding-window and global GQA.

Scale
78B total, 3.46B active (4.4% active)
Context (tokens)
262,144 native; 1,048,576 extrapolated
Vocabulary
128,000
License
Apache License 2.0
Date
2026-10-03
Decoder type
Sparse MoE
Attention
GQA with QK-Norm and 4:1 sliding-window/global attention
Layer mix
40 sliding-window + 10 global GQA
KV cache / token (bf16) info
100 KiB · Moderate
Key detail
Uses top-6 routing over 384 experts plus one shared expert, with sandwich RMSNorm, RoPE in local layers, and NoPE in global layers.
Pretraining data
~24T tokens Includes 20T pretraining, 3.4T midtraining, and 200B context-extension tokens; the report rounds the combined budget to 24T. Technical report
Post-training methods
SFT, RLVR, Distillation SFT is followed by RL with verifiable rewards across 38 environments, with targeted on-policy self-distillation for low-reward groups. Technical report

Related concepts

Compare

Ling 3.1 Flash (560B)

by InclusionAI 🇨🇳

Hybrid MoE that extends Ling 3.0 Flash to 72 layers, with seven Kimi Delta Attention layers for every gated MLA layer.

Scale
560B total, 25B active (4.5% active)
Context (tokens)
1M reported; 262,144 in config
Vocabulary
157,184 (~157k)
License
MIT License
Date
2026-10-13
Decoder type
Sparse hybrid
Attention
7:1 Kimi Delta Attention and gated MLA
Layer mix
63 Kimi Delta Attention + 9 gated MLA + 1 MTP layer
KV cache / token (bf16) info
10.1 KiB · Very low
Key detail
Uses four initial dense FFN layers, top-8 routing over 512 experts plus one shared expert, and MLA with a 1,536-dimensional query bottleneck and 512-dimensional KV latent.
Pretraining data
>10T tokens (training corpus) The official disclosure gives a lower bound on the preprocessed text corpus for Ling 3.1 Flash. It does not give tokens seen during pretraining. Training data summary
Post-training methods
RL The release post confirms reinforcement learning improved task performance while increasing reasoning length and interaction rounds. It does not specify the reward types or full recipe. Release post

Compare

Muse Spark Open (816B-A42B)

by Meta 🇺🇸

Meta's multimodal MoE model combines 62 latent-expert blocks with gated GQA, local-only RoPE, global NoPE, and learned residual gates; the figure focuses on the text decoder.

Scale
816B total, 42B active (5.1% active)
Context (tokens)
1,048,576
Vocabulary
202,048 (~202k)
License
Muse Model License
Date
2026-10-15
Decoder type
Sparse MoE
Attention
Gated GQA with scaleless QK-Norm, local-only RoPE, global NoPE, and 3:1 sliding-window/global attention
Layer mix
46 sliding-window + 16 global gated GQA
KV cache / token (bf16) info
248 KiB · High
Key detail
Selects 8 of 256 experts, compresses the expert path from 8,192 to 4,096 dimensions, normalizes each expert output before combining, and uses no shared expert.
Pretraining data
Not disclosed The card lists multimodal data sources but does not provide a training-set size or token total. Muse Spark Open model card
Post-training methods
Not disclosed Additional safety post-training is confirmed, but the model card does not specify SFT, preference-optimization, or RL methods. Muse Spark Open model card
AA Intelligence Index v5.0 info
Total score 24.7 Knowledge Work 37.0 Coding 7.8 Science 12.1 General 41.7
AA Intelligence Index v4.3.2 info
Total score 38.9 General 45.0 Scientific 29.4 Coding 33.3 Agents 42.8

Compare

Nemotron 3.5 Super VL (120B-A12B)

by NVIDIA 🇺🇸

NVIDIA's vision-language model extends the Nemotron 3 Super backbone with C-RADIO vision encoding and shared-weight multi-token prediction. The figure shows the language backbone, with the vision encoder and MTP omitted.

Scale
120B total, 12B active (10% active)
Context (tokens)
1,048,576
Vocabulary
131,072
License
OpenMDW 1.1
Date
2026-10-15
Decoder type
Hybrid MoE
Attention
Mostly Mamba-2 with eight GQA layers using NoPE
Layer mix
8 GQA + 40 Mamba-2 + 40 LatentMoE; GQA + MoE MTP predictor
KV cache / token (bf16) info
8 KiB · Very low
Key detail
Routes each token to 22 of 512 experts in a 1,024-dimensional latent space, alongside one shared expert at the full 4,096-dimensional width.
Pretraining data
1.2094T additional tokens Continued from Nemotron 3 Super Base with 1T distillation, 100B Indic, 35B context-extension, 65.4B agentic mid-training, and 9B context-recovery tokens. Separate 100B MTP-head warmup freezes the backbone. Nemotron 3.5 technical report
Post-training methods
SFT, RLVR, RLHF, Distillation Vision-language SFT, verifiable-reward RL, human-feedback RL, and multi-teacher on-policy distillation. Nemotron 3.5 technical report
AA Intelligence Index v5.0 info
Total score 13.6 Knowledge Work 19.5 Coding 3.7 Science 5.4 General 25.9
AA Intelligence Index v4.3.2 info
Total score 26.5 General 32.6 Scientific 20.6 Coding 23.9 Agents 26.0

Compare

Reflection Beam (501B)

by Reflection 🇺🇸

Text-only MoE decoder with three dense FFN layers followed by 49 expert layers. Each MoE layer activates 8 of 256 experts without a shared expert.

Scale
501.3B total, 23.4B active (4.7% active)
Context (tokens)
1,048,576
Vocabulary
151,936 (~152k)
License
Apache License 2.0
Date
2026-10-16
Decoder type
Sparse MoE
Attention
Gated GQA with QK-Norm and sliding-window/global attention
Layer mix
43 sliding-window + 9 global GQA
KV cache / token (bf16) info
208 KiB · High
Key detail
Combines 512-token local RoPE attention with global NoPE and query scaling. Uses sandwich RMSNorm, depth-scaled residuals, and a final logit softcap of 30.
Pretraining data
Not disclosed The card describes the text/code corpus and mid-training sources without a token count. Model card
Post-training methods
RL RL environments cover coding, agentic tasks, and STEM; the full recipe is not disclosed. Model card
AA Intelligence Index v5.0 info
Total score 17.7 Knowledge Work 26.3 Coding 4.5 Science 10.9 General 29.2

Compare

Dory1.0 (4B)

by NVIDIA 🇺🇸

NVIDIA's dense looped transformer applies a shared 20-block core twice between 8-block input and output stacks, using normalized GPT (nGPT) residual updates.

Scale
4B parameters
Context (tokens)
262,144
Vocabulary
131,072 (~131k)
License
OpenMDW 1.1
Date
2026-10-16
Decoder type
Dense
Attention
Gated GQA; 3:1 sliding-window/full, RoPE
Layer mix
8 input + 20 shared × 2 passes + 8 output (56 applications)
KV cache / token (bf16) info
224 KiB · High
Key detail
Reinjects the input-stack output before each recurrent pass. The 36 distinct blocks give 56 block applications by default, with L2-normalized residual updates and a 1,024-token attention window.
Pretraining data
6T tokens + 300B context-extension tokens The model card reports pretraining and context extension separately. Dory 1.0 model card
Post-training methods
SFT, RL, Distillation SFT uses teacher-generated data, followed by multi-environment GRPO for reasoning, coding, tool use, and instruction following. Dory 1.0 model card

Compare

Recursive Flint (40B-A2.5B)

by Recursive 🇺🇸

Recursive's coding model uses two dense FFN layers followed by 47 MoE layers. The active-parameter count includes the output head.

Scale
39.8B total, 2.5B active (6.3% active)
Context (tokens)
262,144 configured; 512k supported
Vocabulary
150,144
License
Apache License 2.0
Date
2026-10-18
Decoder type
Sparse MoE
Attention
GQA with QK-Norm, learned attention sinks, and sliding-window/global attention
Layer mix
35 sliding-window + 14 global GQA
KV cache / token (bf16) info
98 KiB · Moderate
Key detail
Uses top-6 routing over 256 experts plus one shared expert, with RoPE in local layers, NoPE in global layers, and a local window covering the current token and 512 preceding tokens.
Pretraining data
Not disclosed The card describes training from scratch but omits the token count. Model card
Post-training methods
SFT, RL The card explicitly lists supervised fine-tuning and reinforcement learning after pretraining and mid-training. Model card

Related concepts

Compare

MiniMax M3.1-Flash (428B)

by MiniMax 🇨🇳

MiniMax M3 successor with sparse attention in all 60 decoder layers, quantization-aware training, and a DSpark draft head for speculative decoding.

Scale
428B total, 23B active (5.4% active)
Context (tokens)
1,048,576
License
MiniMax Community License
Date
2026-10-19
Decoder type
Sparse MoE
Attention
GQA with QK-Norm and MiniMax Sparse Attention
Layer mix
60 MiniMax Sparse Attention; 3 dense FFN + 57 MoE
KV cache / token (bf16) info
120 KiB · Moderate
Key detail
All 60 layers use sparse attention; the first 3 retain dense FFNs, while the remaining 57 route to 4 of 128 experts plus 1 shared expert per token.
Pretraining data
Not disclosed No pretraining total is given. The release's 200T-token figure describes anonymous serving usage. MiniMax M3.1 Flash release · MiniMax M3.1 Flash model card
Post-training methods
Not disclosed The release and model card do not specify the instruction-tuning or reinforcement-learning recipe. MiniMax M3.1 Flash release · MiniMax M3.1 Flash model card
AA Intelligence Index v5.0 info
Total score 28.9 Knowledge Work 47.9 Coding 22.8 Science 14.4 General 30.5
AA Intelligence Index v4.3.2 info
Total score 42.7 General 40.1 Scientific 35.1 Coding 39.6 Agents 52.4

Related concepts

Compare

Nanbeige 4.3 (33B)

by BOSS Zhipin 🇨🇳

Dense model for coding and tool-use agents, with a 52-layer Llama-style decoder and a 524,288-token context.

Scale
33B parameters
Context (tokens)
524,288
Vocabulary
166,912 (~167k)
License
Apache License 2.0
Date
2026-10-24
Decoder type
Dense
Attention
GQA with RoPE
Layer mix
52 GQA
KV cache / token (bf16) info
208 KiB · High
Key detail
Uses 52 distinct decoder layers with 56 query heads, 8 KV heads, and untied input and output embeddings.
Pretraining data
Not disclosed The model and base-model cards do not disclose the pretraining token count for Nanbeige4.3-33B-Base. Nanbeige 4.3 base model card · Nanbeige 4.3 model card
Post-training methods
Not disclosed The released chat model is derived from Nanbeige4.3-33B-Base, but its model card does not specify the post-training methods. Nanbeige 4.3 model card
AA Intelligence Index v5.0 info
Total score 20.4 Knowledge Work 32.7 Coding 18.3 Science 6.8 General 23.8

Related concepts

Compare

LFM3 (1.7B)

by Liquid AI 🇺🇸

Liquid AI's 1.73B dense hybrid combines short convolutions with sliding-window and global GQA in a 24-layer stack.

Scale
1.73B parameters
Context (tokens)
262,144
Vocabulary
128,000
License
LFM Open License v1.0
Date
2026-10-27
Decoder type
Dense hybrid
Attention
ShortConv plus gated GQA with QK-Norm and sliding-window/global attention
Layer mix
13 ShortConv + 5 sliding-window + 6 global GQA
KV cache / token (bf16) info
22 KiB · Very low
Key detail
Uses kernel-7 ShortConv, a 256-token attention window, per-head sigmoid gates, and sandwich RMSNorm. Global attention uses 50% partial RoPE.
Pretraining data
22T tokens Includes the last 3.5T tokens of midtraining and context extension; the separate SFT budget is excluded. Model card
Post-training methods
SFT, RL, Distillation SFT is followed by mixed-domain RL, on-policy distillation from six specialist RL teachers, and a final agentic RL stage. Model card

Compare

Mistral Large 4 (1T)

by Mistral AI 🇫🇷

Mistral's trillion-parameter multimodal MoE combines MLA with sparse token selection and cross-layer KV sharing.

Scale
1T total, 52B active (5.2% active)
Context (tokens)
1,048,576
Vocabulary
204,800
License
Mistral Ecosystem Open-Weight License
Date
2026-10-29
Decoder type
Sparse MoE
Attention
MLA with modified DeepSeek Sparse Attention, index reuse, and KV sharing
Layer mix
72 MLA/DSA
KV cache / token (bf16) info
80.75 KiB · Moderate
Key detail
Selects the top 2,048 tokens, reuses sparse indices across layers, and alternates three LeRoPE layers with one NoPE layer.
Pretraining data
Not disclosed The technical report describes two multilingual multimodal pretraining phases but gives no total token count. Mistral Large 4 technical report
Post-training methods
SFT, RLVR Supervised fine-tuning followed by RL using executable task verification, alongside learned and model-judged rewards. Mistral Large 4 technical report
AA Intelligence Index v5.0 info
Total score 26.0 Knowledge Work 40.6 Coding 18.9 Science 13.0 General 31.7

Compare

LongCat-2.5 (1.6T-A48B)

by Meituan 🇨🇳

Meituan's native multimodal MoE combines Shortcut MoE, n-gram embeddings, and LongCat Sparse Attention. The figure shows the language backbone, with the vision encoder and MTP module omitted.

Scale
1.6T total, ~48B active (3% active)
Context (tokens)
524,288
Vocabulary
163,840 (~164k)
License
MIT License
Date
2026-11-02
Decoder type
Sparse MoE
Attention
MLA with LongCat Sparse Attention (LSA) and cross-layer index reuse
Layer mix
38 blocks / 76 MLA + LSA sublayers
KV cache / token (bf16) info
95 KiB · Moderate
Key detail
Reuses top-2,048 token selections across each attention pair. Shortcut MoE selects 12 of 768 learned and 128 identity experts; n-gram lookup tables hold about 135B parameters.
Pretraining data
Not disclosed The release card does not publish a pretraining-token total. Model card
Post-training methods
SFT, RL, Distillation The release describes long-sequence SFT, large-scale RL, and multi-teacher full-vocabulary KL distillation. Model card

Related concepts

Compare

Kimi K3.1 Pro (2.8T)

by Moonshot AI 🇨🇳

Post-trained from Kimi K3, with its 24 Gated MLA layers converted to Gated DeepSeek Sparse Attention. The figure shows the text backbone of this native multimodal model.

Scale
2.8T total, 104B active (3.7% active)
Context (tokens)
1,048,576
Vocabulary
163,840 (~160k)
License
Kimi K3.1 Pro License
Date
2026-11-04
Decoder type
Sparse hybrid
Attention
Kimi Delta Attention and Gated MLA with DeepSeek Sparse Attention and NoPE
Layer mix
69 KDA + 24 Gated DSA
KV cache / token (bf16) info
33 KiB · Low
Key detail
Keeps eight Block Attention Residual groups and Stable LatentMoE with 16 of 896 routed experts plus two shared experts per token. The DSA indexer selects up to 2,048 tokens using 32 heads of 128 dimensions.
Pretraining data
Not disclosed Derived from Kimi K3; the release card does not disclose a pretraining-token total for K3.1 Pro. Model card
Post-training methods
RL The K3.1 Pro update is post-trained from K3 using large-scale joint RL across hundreds of domains. The card does not specify the reward types. Model card

Compare

ERNIE 5.5 Flash Base (280B)

by Baidu 🇨🇳

Text-only base model with 37 heavily compressed attention layers and 6 MLA/DSA layers. Pretrained with up to 128K tokens of context; fine-tuning and inference support 1M tokens.

Scale
280B total, 14B active (5.0% active)
Context (tokens)
1,048,576
Vocabulary
201,216 (~201k)
License
ERNIE 5.5 Flash Base Model License Agreement v1.0
Date
2026-11-05
Decoder type
Sparse MoE
Attention
Gated HCA and MLA with DeepSeek Sparse Attention, VHA, and QK-Norm
Layer mix
37 HCA + 6 MLA/DSA; 1 dense FFN + 42 latent MoE
KV cache / token (bf16) info
8.54 KiB · Very low
Key detail
Four-stream mHC surrounds attention and latent MoE. Two sigmoid routers select 10 of 512 routed experts, alongside one always-active shared expert. ECHO reuses the final backbone layer for multi-token prediction.
Pretraining data
33.5T tokens Text-only pretraining from scratch. ERNIE 5.5 Flash Base model card
Post-training methods
None (base model) Released at the end of pretraining, before any mid-training or post-training. ERNIE 5.5 Flash Base model card

Compare

Nous Research Arete (27B)

by Nous Research 🇺🇸

Nous Research's supervised fine-tune of Qwen3.8-27B. The vision encoder was retained and frozen during text-only fine-tuning; the figure shows the language backbone.

Scale
27B parameters
Context (tokens)
262,144
Vocabulary
248,320 (~248k)
License
Apache License 2.0
Date
2026-11-06
Decoder type
Dense hybrid
Attention
3:1 Gated DeltaNet and Gated Attention
Layer mix
16 gated attention + 48 DeltaNet
KV cache / token (bf16) info
64 KiB · Low
Key detail
Retains Qwen3.8-27B's 64-layer hybrid decoder with partial RoPE. The released checkpoint has no MTP head.
Pretraining data
Not disclosed Derived from Qwen3.8-27B. The Arete card does not disclose the inherited pretraining-token total. Arete 27B model card
Post-training methods
SFT Supervised fine-tuning with Inkling-generated supervision. The card explicitly excludes RL and preference optimization. Arete 27B model card

Compare

Inkling 2 Small (594B)

by Thinking Machines Lab 🇺🇸

Multimodal sparse MoE with a 48-layer decoder, 8-of-384 expert routing, and block attention residuals.

Scale
594B total, 17B active (2.9% active)
Context (tokens)
1,048,576
Vocabulary
200,058 (~200k; padded to 201,024)
License
Apache License 2.0
Date
2026-11-12
Decoder type
Sparse MoE
Attention
Gated GQA with Q/K L2 clipping and learned relative-position bias in a 5:1 sliding-window/global pattern
Layer mix
40 sliding-window + 8 global GQA
KV cache / token (bf16) info
192 KiB · High
Key detail
Groups the decoder into eight 6-layer AttnRes blocks and adds kernel-4 short convolutions. The figure omits the vision encoder and 5-layer DSpark draft.
Pretraining data
8.4T + 3.8T tokens (H1 + H2) H1 covers natural language, code, and images; H2 uses newer, higher-quality data at 512K context. H3 uses about 320B tokens of reasoning and agent trajectories alongside pretraining replay, and H4 uses about 34B agent-trajectory tokens. Training overview
Post-training methods
RL Joint asynchronous RL with StableReinforce across 1,075 steps, over 17M rollouts, and over 3T tokens. Task-specific graders include unit tests, reference answers, LLM judges, and pairwise reward models. Training overview
AA Intelligence Index v5.0 info
Total score 30.6 Knowledge Work 45.0 Coding 20.5 Science 17.4 General 39.7

Compare

Datology XL (30B-A3B)

by DatologyAI 🇺🇸

Qwen3-style research model, included for its excellent technical report on data curation, optimization, and controlled training experiments.

Scale
30.4B total, 3.26B active (10.7% active)
Context (tokens)
65,536
Vocabulary
129,280 (~129k)
License
N/A
Date
2026-11-16
Decoder type
Sparse MoE
Attention
GQA with QK-Norm and RoPE
Layer mix
48 GQA
KV cache / token (bf16) info
96 KiB · Moderate
Key detail
Uses Qwen3-30B-A3B's 48-layer backbone with 8 of 128 experts active per token, the DeepSeek V4 tokenizer, and AdamH optimization with explicit weight-norm constraints.
Pretraining data
1T tokens XL curated model, pretrained from scratch on Datology-curated data. A further 10B tokens extend context from 8k to 64k. All token counts use the DeepSeek V4 tokenizer. Datology technical report
Post-training methods
SFT, RLVR, Model merging About 20B SFT tokens, followed by four GRPO specialists for mathematics, code, instruction following, and tool use. The specialists are merged into one model. Datology technical report

Related concepts

Compare

Marin Mochi (535B-A23B)

by Open Athena 🇺🇸

Marin's base model combines latent MoE with gated grouped-query attention. Every fourth layer attends to the full context. The other layers use a 2,048-token window.

Scale
535.48B total, 22.76B active non-embedding (4.3% active)
Context (tokens)
262,144
Vocabulary
128,256 (~128k)
License
OpenMDW 1.1
Date
2026-11-17
Decoder type
Sparse MoE
Attention
Gated GQA with exclusive self-attention (XSA). Local layers apply RoPE to half of each head, and global layers use NoPE.
Layer mix
36 sliding-window + 12 global GQA
KV cache / token (bf16) info
252 KiB · High
Key detail
Each token uses 8 of 384 routed SwiGLU experts at half width (3,072 dimensions) and 2 shared experts at full width (6,144). Kernel-4 short convolutions act on the keys, the attention output, and the MoE output.
Pretraining data
18.2T tokens The pretraining run extended context from 4k to 16k, then 65k and 262k tokens. The final checkpoint is step 393,491. Open Athena checkpoint release · Marin Mochi model card
Post-training methods
None (base model) All 57 released checkpoints are base models from the pretraining run. Open Athena checkpoint release

Compare

Meta analysis

Architecture-level comparisons beyond individual cards

The gallery cards are useful for inspecting one model at a time. The meta-analysis views aggregate selected fields across the gallery so it is easier to compare design tradeoffs across sparse and hybrid architectures.

Cite / Share

Short Description

A curated LLM Architecture Gallery with model architecture figures, compact fact sheets, source links, implementation links, and comparison tools for modern language models.

BibTeX

@misc{raschka2026llmarchitecturegallery,
  author = {Raschka, Sebastian},
  title = {LLM Architecture Gallery},
  year = {2026},
  month = {March},
  url = {https://sebastianraschka.com/llm-architecture-gallery/},
  note = {Accessed: 2026-11-18}
}

Suggested Share Text

Support this project

Your support helps me maintain and expand the gallery. The downloads below are a thank-you for paid Ahead of AI subscribers.

Architecture images

Download all diagrams and use the poster builder to make your own poster.

Download Images

Model metadata

Download model specs and info as YAML for your own analyses.

Download Metadata
Subscribe