Building an LLM from scratch means initializing the model weights randomly and creating the data and training pipeline that turns them into a language model. A small educational GPT and a competitive foundation model follow the same broad stages, although their data, compute, and distributed-systems requirements differ by orders of magnitude.

1. Define the target and budget.

Choose the intended language or domain, parameter count, training-token budget, context length, and available hardware. These choices constrain one another. A model that fits in memory may still require an impractical amount of training compute, and a long context increases activation and attention costs.

2. Build the data and tokenizer pipeline.

Collect text whose source and license permit the intended use. Cleaning, deduplication, document boundaries, quality filters, and a held-out split should be decided before the long training run. Train or select a tokenizer, then turn the token stream into fixed-length input-target examples. Inspect decoded samples because a numerically correct pipeline can still contain broken text or unexpected document mixtures.

3. Implement and test the model.

A GPT-style decoder combines embeddings, positional information, causal attention, feed-forward layers, normalization, residual paths, and a vocabulary output head. The building-block FAQ follows the tensor shapes through this stack. Unit tests should check the causal mask, parameter sharing, output dimensions, and agreement between cached and uncached generation where applicable.

Before scaling up, I would verify that the model can overfit one tiny batch. If it cannot, a larger dataset or more accelerators will usually hide the bug rather than fix it.

4. Make the training loop recoverable.

The training system needs cross-entropy loss, an optimizer, a learning-rate schedule, gradient handling, mixed precision, logging, and checkpoint save-and-resume support. For a randomly initialized model with approximately uniform token probabilities, the initial loss should be near log(V), where V is the vocabulary size. A short run should reduce both training and held-out loss without numerical overflows.

Larger runs add data, tensor, or pipeline parallelism. Those techniques distribute the same optimization problem; they do not replace the single-device correctness checks.

5. Pretrain with next-token prediction.

Pretraining repeatedly presents token sequences and updates the model to increase the probability of the observed continuation. Track validation loss, token throughput, gradient norms, and data consumption. Save enough metadata to reproduce a checkpoint, including the model configuration, tokenizer, optimizer state, scheduler state, and training step.

6. Add post-training for the intended use.

The base checkpoint can be adapted for classification or instruction following. Preference optimization, reasoning training, and tool-use traces are optional later stages. Keeping the original base checkpoint makes it possible to measure what each stage changed and to restart with a different post-training recipe.

7. Evaluate before optimizing deployment.

Evaluation should include held-out language modeling, task-specific checks, and qualitative generation. Data contamination, memorization, formatting failures, and regressions outside the finetuning domain need separate tests. Open-ended outputs may require human review or a calibrated judge, as discussed in the evaluation FAQ.

8. Build the inference path.

Generation adds token selection, stopping rules, prompt formatting, and a KV cache. Serving may also use quantized weights, batching, and memory-efficient kernels. Measure first-token latency, decoding throughput, peak memory, and output quality under the same settings expected in the application.

This order keeps failures local. Data examples can be inspected before training, model math can be tested before distributed execution, and the checkpoint can be evaluated before serving optimizations make the system harder to debug.