Back to xLSTM 7B in the gallery

NXAI

Recurrent 7B language model that replaces self-attention with xLSTM blocks built around matrix memory.

xLSTM 7B architecture diagram
xLSTM 7B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
7B parameters
Context (tokens)
No explicit limit
License
NXAI Community License Agreement
Date
2025-03-17
Decoder type
Recurrent
Attention
No self-attention; mLSTM recurrent layers with matrix memory
Layer mix
32 mLSTM
KV cache / token (bf16) info
0 B · No cache
Key detail
Stateful recurrent architecture aimed at fast long-context inference without an explicit context window.

Training and benchmarks

Pretraining data
~2.3T tokens DCLM and selected high-quality data. Model card
Post-training methods
None (base model) The linked xLSTM-7b release is a pretrained completion model. Model card

Further reading and tools