Back to Olala v1.0 Mini 7A1B in the gallery

OVHai LLM (OVHcloud)

Olala v1.0 Mini 7A1B pairs 29 Mamba-3 layers with 7 differential tensor-product attention layers. Each of its 36 layers uses a latent MoE and GeoNorm residual connections.

Olala v1.0 Mini 7A1B architecture diagram
Olala v1.0 Mini 7A1B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
6.8B total, ~1B active (~14.7% active)
Context (tokens)
32,768
Vocabulary
151,936
License
Apache License 2.0
Date
2026-11-19
Decoder type
Hybrid MoE
Attention
Differential tensor-product attention (DiffTPA) with 36 signal heads, 12 noise heads, and a gated output
Layer mix
29 Mamba-3 + 7 DiffTPA
KV cache / token (bf16) info
15.3125 KiB (rank-4 factorized cache) · Very low
Key detail
The router selects 6 of 256 experts from the full 1,536-dimensional input. Those experts operate on 384 dimensions, while one shared expert processes the full input.

Training and benchmarks

Pretraining data
12.8T tokens Main pretraining used 10.4T tokens of multilingual text, code, mathematics, and translations. Annealing added 2.38T tokens with more mathematics, code, and instruction data. Model card and training recipe
Post-training methods
SFT, RL Supervised fine-tuning used 36.9M samples (about 162B tokens). Four RL stages covered instruction following and math, code, assistant behavior, and tool use. Model card and training recipe

Further reading and tools