Back to Phi-4 in the gallery

Microsoft

Microsoft's 14B dense Phi refresh stays close to Phi-3-medium but swaps its sliding-window attention for full-context GQA and a larger tokenizer.

Phi-4 architecture diagram
Phi-4 architecture diagram. Select the diagram to view it at full size.

Architecture and specifications

Scale
14B parameters
Context (tokens)
16,384
License
MIT License
Date
2024-12-12
Decoder type
Dense
Attention
GQA with RoPE
Layer mix
40 GQA
KV cache / token (bf16) info
200 KiB · High
Key detail
Classic pre-norm RMSNorm stack with GQA, 40 heads, 10 KV heads, and a 100,352-token vocabulary.

Training and benchmarks

Pretraining data
9.8T tokens The model card reports 9.8T training tokens. Phi-4 model card
Post-training methods
SFT, DPO Supervised fine-tuning followed by pivotal-token and judge-guided DPO stages. Phi-4 technical report
AA Intelligence Index v4.3.2 info
Total score 5.9 General N/A Scientific 1.9 Coding N/A Agents N/A

Further reading and tools