GPT-2 XL 1.5B Architecture
Back to GPT-2 XL 1.5B in the gallery
OpenAI
Late-2019 dense baseline included here as a reference point for how much decoder stacks have changed since GPT-2.
Architecture and specifications
- Scale
- 1.5B parameters
- Context (tokens)
- 1,024
- License
- OpenAI "Modified MIT" license
- Date
- 2019-11-05
- Decoder type
- Dense
- Attention
- MHA with learned absolute positional embeddings
- Layer mix
- 48 MHA
- KV cache / token (bf16) info
- 300 KiB · High
- Key detail
- Classic GPT-2 recipe with dropout, GELU, LayerNorm, and full multi-head attention.
Training and benchmarks
- Pretraining data
- 40 GB of text; over 8M documents WebText corpus size after filtering and deduplication. The report does not provide a total training-token count. GPT-2 report
- Post-training methods
- None (base model) Original pretrained language model; no instruction or preference post-training in this release. GPT-2 report