Context length matters because it limits how many tokens an LLM can use in one sequence. A larger window can hold more source material, conversation history, or code. The cost grows when those tokens are actually processed, especially in full self-attention and the inference KV cache.

The limit is measured in tokens rather than words. A token may be a complete word, part of a word, punctuation, or whitespace, depending on the tokenizer. Two documents with the same word count can therefore occupy different amounts of context.

Several context-length numbers can describe one model

The advertised context window is only one part of the picture.

Quantity Meaning
Configured maximum Largest sequence the model implementation accepts
Training sequence length Lengths encountered during pretraining, continued training, or context extension
Actual request length Tokens processed for one request, including generated tokens retained in the sequence
Effective context length Range over which the model can reliably find and use information for a particular task

These quantities can differ. Changing a configuration value or rescaling rotary position embeddings may allow a forward pass at a larger position index. The model may still have poor retrieval or synthesis quality beyond the lengths it learned to handle.

A model with a 128K maximum also does not pay for 128K tokens when the current request contains 2K. The actual sequence length determines most request-level compute and cache allocation.

The prompt and output consume one token budget

For a decoder-only model, the active sequence can contain several components:

system prompt + conversation history + retrieved text + tool results + current request + generated response

Suppose a model supports 8,192 tokens and the application reserves 1,500 for the answer. The input should stay at or below 6,692 tokens. If the input already uses 7,500 tokens, the model cannot also produce a 1,500-token answer inside the same window.

Some APIs expose separate input and output limits, but the serving rules still need to be checked. A user-facing max_output_tokens option does not imply that the output sits outside the model’s positional or cache constraints.

When a request exceeds the accepted length, a system may reject it, truncate older tokens, remove retrieved passages, or summarize earlier content. Silent left truncation is risky because it can remove a system instruction or the first definition in a document. The truncation policy should be explicit and tested.

Context is also temporary. Tokens outside the current request are unavailable unless an application stores and reintroduces them. A large context window is therefore different from persistent memory.

Context length changes the training examples

During causal-language-model training, tokenized documents are split or packed into sequences. A training sequence of length \(T\) supplies next-token targets for positions inside that window. The model cannot directly condition one target on text that was placed in another independent sequence.

Tokenized text is divided into fixed-length input and target windows, which determines which preceding tokens are available to each training target

Longer training sequences expose the model to more distant relationships. This can help with long documents, code dependencies, multi-turn dialogue, and information that must be combined across sections. Exposure alone does not guarantee that the model will learn to use every position equally well. The data needs relevant long-range structure, and the positional method must work at the intended lengths.

Sequence length also changes the training system. For full attention, a sequence of length \(T\) contains on the order of \(T^2\) query-key score pairs per layer. Doubling the length from 4K to 8K produces about four times as many score pairs. Token-wise projections and MLP layers grow roughly linearly with \(T\), so the complete training step does not necessarily become exactly four times slower.

Activation memory grows as well. A longer sequence may require a smaller batch, activation checkpointing, sequence parallelism, or more accelerators. If the batch size drops too far, the optimizer setup may need adjustment to preserve the intended number of tokens per update.

FlashAttention reduces memory traffic and avoids storing the complete attention-score matrix in high-bandwidth memory. It computes exact full attention, so it improves the implementation without turning the underlying \(T^2\) score calculation into a linear one.

Inference has prefill and decoding costs

Inference begins with prefill, where the model processes the prompt. The prompt positions can run in parallel within a transformer layer, while full-attention score work grows quadratically with prompt length.

Autoregressive decoding then generates one new token at a time. A KV cache stores the keys and values from the prompt and earlier generated tokens, avoiding a complete recomputation of the prefix at each step.

During decoding, the current query attends to keys and values from earlier tokens, which are retained in the KV cache

For batch size \(B\), \(L\) cache-producing layers, retained length \(T\), \(H_{KV}\) key-value heads, head dimension \(d_h\), and \(b\) bytes per element, the logical cache size is

\[2BLTH_{KV}d_hb.\]

The cache grows linearly with retained length and batch size. Each active sequence needs its own cache, while the loaded model weights can be shared across requests. Long prompts therefore reduce the number of concurrent sequences that fit in accelerator memory.

Decoding latency also changes with context. Each new query in a full-attention layer must interact with the retained prefix. The KV cache removes repeated key and value projections, but it does not make reading a long history free. Generated tokens extend that history and make later decoding steps slightly more expensive.

Maximum length does not guarantee useful length

A model can accept a long sequence and still fail to use it well. Several capabilities need separate evaluation:

  • retrieving one exact fact at different positions
  • combining evidence from distant sections
  • following an early instruction after many later tokens
  • resolving conflicting or duplicated passages
  • maintaining generation quality near the end of the window

A simple hidden-fact test measures retrieval. It says little about summarizing a long report or reasoning across several documents. Position matters too. A model may retrieve information reliably near the beginning or end and perform worse when the relevant passage appears in the middle.

More context can also introduce irrelevant or conflicting evidence. Adding every retrieved passage to a prompt may lower answer quality while increasing cost. Retrieval, reranking, and structured summaries can often keep the actual context shorter. These components need their own evaluation because a retriever or summarizer can discard something important.

Architecture determines how the cost scales

Long-context methods reduce different terms in the cost.

  • Grouped-query attention stores fewer key and value heads per token, reducing KV-cache size and bandwidth.
  • Sliding-window attention limits the positions available in local layers, reducing score work and allowing old cache entries to be evicted.
  • KV-cache quantization lowers the bytes stored per key and value element.
  • Latent or cross-layer sharing methods compress cache representations or reuse them between layers.

Grouped-query and sliding-window attention reduce different dimensions of KV-cache growth as context length increases

These methods come with different information paths. GQA preserves full-context access in each attention layer. SWA removes direct access to sufficiently old tokens in local layers, although periodic global layers can restore full-sequence lookup. The GQA and SWA comparison works through that design choice.

Positional encodings are another part of context support. Learned absolute embeddings have a fixed table, while RoPE can compute rotations for new indices. Neither approach guarantees good length extrapolation without suitable training or scaling. The RoPE FAQ explains this distinction.

Choose the smallest context that preserves the task evidence

For a deployed system, I would start with the prompt-length distribution rather than the advertised maximum. Reserve enough tokens for the expected response, then measure task quality, time to first token, per-token decoding latency, cache memory, and concurrent batch size at several input lengths.

Use a longer actual context when important relationships would be lost through filtering, such as definitions spread across a contract or dependencies across a codebase. Use retrieval or summaries when a small subset of the source contains the evidence. The short-context and long-context comparison covers these workload tradeoffs in more detail.