Tiny Aya 3.35B Architecture
Back to Tiny Aya 3.35B in the gallery
Cohere
Compact multilingual model from Cohere with a rare parallel transformer block.
Architecture and specifications
- Scale
- 3.35B parameters
- Context (tokens)
- 8,192
- License
- Creative Commons Attribution-NonCommercial 4.0
- Date
- 2026-02-13
- Decoder type
- Dense
- Attention
- GQA with 3:1 sliding-window attention
- Layer mix
- 27 sliding-window + 9 global
- KV cache / token (bf16) info
- 72 KiB · Low
- Key detail
- Runs attention and the MLP in parallel while mixing RoPE with NoPE.
Training and benchmarks
- Pretraining data
- 6T tokens Pretraining budget for Tiny Aya Base. Technical report
- Post-training methods
- SFT, Preference tuning, Model merging Global and regional variants add SFT; Global adds minimal preference tuning, and regional releases merge checkpoints. Tiny Aya Base precedes these stages. Technical report