Most LLM projects that fail on consumer hardware start with a mismatch between the workload and the hardware budget. A checkpoint may fit on disk while loading, finetuning, or long-context inference needs much more memory. A run may also fit in memory and still take too long to be practical.

The useful first question is therefore more specific than “Can this machine run an LLM?” I would ask which model, which task, which precision, which context length, and whether the goal is inference, finetuning, or pretraining.

Start with the workload

The same checkpoint has very different requirements across workloads.

Workload Additional state beyond the weights Frequent limit
Single-request inference KV cache and temporary buffers Weight capacity or decoding bandwidth
Batched or long-context inference One growing KV cache per active sequence Device memory
LoRA or QLoRA finetuning Activations plus adapter gradients and optimizer state Activation memory
Full finetuning Activations, gradients, and optimizer state for the complete model Memory capacity and training time
Pretraining from scratch Full training state over a large token budget Compute time, memory, and data pipeline

A consumer GPU may run quantized inference for a model that it cannot finetune. LoRA can make adaptation feasible while full finetuning remains far outside the memory budget. Pretraining a small educational model can be realistic, while pretraining a competitive general-purpose model is a different scale of project.

“Consumer hardware” also covers several memory arrangements. A desktop with a discrete GPU has separate device memory and system RAM. Apple silicon and integrated GPUs use shared or unified memory, but the operating system and other applications consume part of that pool. In either case, the advertised capacity is not fully available to the model.

The checkpoint is only one part of the memory budget

The lower bound for unquantized weight memory is approximately:

parameter count x bytes per parameter

A 7-billion-parameter model requires about 14 GB when its weights are stored in bfloat16 or float16, before nonparameter buffers and runtime workspaces. A nominal 4-bit representation has a 3.5 GB raw lower bound, although scales, zero points, packing metadata, and unquantized tensors increase the real allocation.

Checkpoint size can also understate the loading peak. A naive loader may construct an initialized model, materialize a separate state dictionary, and then copy the checkpoint into the model. Dtype conversion and device transfer can create further temporary copies. The process may run out of host RAM or VRAM before reaching the smaller final footprint.

Memory-efficient loading avoids keeping a fully initialized model and a separate materialized checkpoint in memory at the same time

The model-loading FAQ works through a concrete 7B example and explains memory mapping, meta-device construction, and assigned state dictionaries.

Training adds several large allocations

Full finetuning needs much more than a forward pass. Every trainable parameter can require a gradient and optimizer state. Adam commonly maintains two 32-bit moment buffers. Some mixed-precision configurations also keep a 32-bit master copy of the weights.

For a 7B model, the two Adam buffers alone occupy roughly 56 GB. The model weights, gradients, possible master weights, activations, and temporary buffers come on top of that. A GPU that can serve the model may therefore be nowhere close to fitting full finetuning.

LoRA reduces this persistent training state by freezing the original weights and optimizing small low-rank adapters. Plain LoRA still loads the full base model. QLoRA also quantizes the frozen base, which can reduce that part of the footprint. Neither method removes the activations needed to propagate information through the network. The full-finetuning and LoRA comparison separates these costs in more detail.

Activations grow with the model dimensions, microbatch size, sequence length, and number of layers. Reducing the microbatch or context often resolves an out-of-memory error that changing the optimizer cannot fix. Gradient accumulation raises the effective batch size across several microbatches, but each individual microbatch still has to fit. Activation checkpointing saves memory by recomputing selected operations during the backward pass.

Long contexts and batches can break inference

Autoregressive inference stores the key and value tensors from earlier tokens in a KV cache. Its logical size grows with the number of retained tokens, cache-producing layers, KV heads, bytes per value, and active sequences.

This is why a model can generate normally at 2,000 tokens and run out of memory near its advertised context limit. Increasing the serving batch multiplies the cache because each sequence needs its own state. Some runtimes also reserve cache memory according to a configured maximum or memory-utilization target, so configuration can affect the allocation before a request reaches that length.

The long-context KV-cache FAQ gives a concrete calculation. For Qwen3 8B, one 32,768-token sequence requires about 4.5 GiB of logical bf16 cache. Eight such sequences require about 36 GiB before runtime overhead.

Training has a related sequence-length problem. Full self-attention has quadratic work in the sequence length, and longer sequences retain more activations for backpropagation. Cutting the microbatch may make the run fit, although a long sequence can still exceed memory at batch size one.

Fitting in memory does not guarantee useful speed

Offloading weights to system RAM can make a model load when VRAM is insufficient. The tradeoff is repeated data movement across the CPU-GPU interconnect, which can reduce generation speed substantially. Disk offload is slower again. A successful load only establishes capacity, not acceptable latency.

Quantization has a similar boundary. Smaller weights reduce capacity and memory-traffic pressure. The actual speed depends on whether the device and runtime provide an optimized kernel for that format. A quantized path can be slower when it dequantizes frequently or falls back to poorly optimized operations.

For inference, I would record time to first token and generated tokens per second separately. Prompt prefill processes many positions in parallel, while decoding repeatedly reads weights and cached state for one new token. For training, tokens per second is more informative than step time when sequence length and batch size change.

A simple wall-clock estimate can prevent an impractical run:

training time = training tokens / measured tokens per second

Use throughput from a short representative run on the target machine. Laptop cooling, power limits, background memory use, checkpoint writing, and evaluation will make a long job slower than an idealized kernel benchmark.

The advertised optimization may not apply to the machine

Hardware and software support have to match the exact model and configuration. bfloat16 is useful when the accelerator supports it efficiently. FlashAttention-style kernels have restrictions on device generation, dtype, attention layout, and software version. Quantized checkpoints need compatible loaders and kernels. A new architecture may also require a newer model library even when its tensors fit.

Fallback behavior is worth checking. An unsupported operation may move to the CPU, use a slower kernel, or fail during compilation. torch.compile can improve a stable workload, yet its initial compilation, recompilation, and temporary memory are costs of their own. I would confirm device placement and benchmark the complete loop rather than assume that an enabled flag produced the intended path.

The PyTorch optimization summary collects precision, attention, compilation, optimizer, and data-pipeline changes that should be tested against one measured baseline

A practical recovery sequence

When a project fails on a local machine, I would narrow it down in this order:

  1. Name the phase that fails: loading, prefill, decoding, forward pass, backward pass, or optimizer step.
  2. Record the exact model, dtype, quantization format, context length, microbatch, and device placement.
  3. Measure peak device memory, peak host memory, and throughput with a small representative input.
  4. Reduce one workload dimension. Start with model size, context length, microbatch size, or number of concurrent sequences.
  5. Use a memory method that addresses the measured allocation, such as streaming loads, activation checkpointing, LoRA, QLoRA, or a smaller KV cache.
  6. Test precision, optimized attention, compilation, and the input pipeline one change at a time.
  7. Estimate the complete runtime before starting the full dataset or generation job.

The phase matters because the fixes are not interchangeable. FlashAttention cannot remove duplicate checkpoint copies during loading. LoRA does not shrink an inference KV cache. Gradient accumulation cannot make one oversized training sequence fit. Offloading can solve a capacity problem while creating a latency problem.

For a shorter symptom-based diagnosis, see the limited-hardware bottleneck FAQ. The single-GPU optimization checklist covers the next steps after the workload fits and runs correctly.