Back to GPT-2 XL 1.5B in the gallery

OpenAI

Late-2019 dense baseline included here as a reference point for how much decoder stacks have changed since GPT-2.

GPT-2 XL 1.5B architecture diagram
GPT-2 XL 1.5B architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
1.5B parameters
Context (tokens)
1,024
License
OpenAI "Modified MIT" license
Date
2019-11-05
Decoder type
Dense
Attention
MHA with learned absolute positional embeddings
Layer mix
48 MHA
KV cache / token (bf16) info
300 KiB · High
Key detail
Classic GPT-2 recipe with dropout, GELU, LayerNorm, and full multi-head attention.

Training and benchmarks

Pretraining data
40 GB of text; over 8M documents WebText corpus size after filtering and deduplication. The report does not provide a total training-token count. GPT-2 report
Post-training methods
None (base model) Original pretrained language model; no instruction or preference post-training in this release. GPT-2 report

Further reading and tools