Back to Tiny Aya 3.35B in the gallery

Cohere

Compact multilingual model from Cohere with a rare parallel transformer block.

Tiny Aya 3.35B architecture diagram
Tiny Aya 3.35B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
3.35B parameters
Context (tokens)
8,192
License
Creative Commons Attribution-NonCommercial 4.0
Date
2026-02-13
Decoder type
Dense
Attention
GQA with 3:1 sliding-window attention
Layer mix
27 sliding-window + 9 global
KV cache / token (bf16) info
72 KiB · Low
Key detail
Runs attention and the MLP in parallel while mixing RoPE with NoPE.

Training and benchmarks

Pretraining data
6T tokens Pretraining budget for Tiny Aya Base. Technical report
Post-training methods
SFT, Preference tuning, Model merging Global and regional variants add SFT; Global adds minimal preference tuning, and regional releases merge checkpoints. Tiny Aya Base precedes these stages. Technical report
AA Intelligence Index v4.3.2 info
Total score 4.8 General N/A Scientific 2.6 Coding N/A Agents N/A

Further reading and tools