Phi-4 Architecture
Microsoft
Microsoft's 14B dense Phi refresh stays close to Phi-3-medium but swaps its sliding-window attention for full-context GQA and a larger tokenizer.
Architecture and specifications
- Scale
- 14B parameters
- Context (tokens)
- 16,384
- License
- MIT License
- Date
- 2024-12-12
- Decoder type
- Dense
- Attention
- GQA with RoPE
- Layer mix
- 40 GQA
- KV cache / token (bf16) info
- 200 KiB · High
- Key detail
- Classic pre-norm RMSNorm stack with GQA, 40 heads, 10 KV heads, and a 100,352-token vocabulary.
Training and benchmarks
- Pretraining data
- 9.8T tokens The model card reports 9.8T training tokens. Phi-4 model card
- Post-training methods
- SFT, DPO Supervised fine-tuning followed by pivotal-token and judge-guided DPO stages. Phi-4 technical report