Why LLMs Have Context Windows at All

A context window is not a memory limit in the way a disk is full. Nothing is being stored. The model is doing a fixed amount of arithmetic over a block of tokens, and two separate things make that block expensive to enlarge.

Understanding which is which explains why context windows grew from thousands to millions of tokens, and why a model can still get worse at using them as they grow.

Key takeaways#

• A context window is everything the model reads to produce the next token: system prompt, history, retrieved documents, and the response so far.
• The first constraint is attention costing O(n²) — doubling the input roughly quadruples the comparison work.
• The second is positional encoding, which stops behaving sensibly beyond the lengths the model was trained on.
• The model is stateless between calls. A chat feels continuous because the whole transcript is resent every turn.
• A larger window does not mean uniform recall. Retrieval accuracy commonly sags in the middle of long inputs.

What the window contains#

Every token the model considers when predicting the next one sits in the window. That means the system prompt, the entire conversation so far, any documents or tool output pasted in, and the tokens the model has generated in this response.

There is no separate place where earlier turns live. Between API calls the model holds nothing at all — it is a pure function from an input sequence to a distribution over the next token. A conversation feels continuous because the client sends the full transcript again on every turn.

This is why a long conversation slows down and costs more as it goes: each turn re-reads everything before it.

Why attention costs quadratic compute#

The transformer's mechanism is self-attention: for each token, the model computes how relevant every other token in the window is to it. That is a comparison of every position against every other position — n² comparisons for n tokens.

Double the context and the attention work goes up roughly fourfold. Ten times the context is about a hundred times the comparison work.

The memory follows the same curve during training, because the attention matrix has to be held to compute gradients. This is the reason context length was measured in thousands of tokens for years while model parameters were measured in billions: parameters scale linearly with compute, context did not.

Why positional encoding is the harder limit#

Attention is order-blind. Feed it a sentence and a shuffled version and the raw mechanism computes the same relationships — nothing in it distinguishes "the dog bit the man" from "the man bit the dog".

Position has to be added explicitly, and how it is added determines what happens past the training length. Modern models mostly use rotary position embeddings (RoPE), which encode position by rotating the query and key vectors by an angle proportional to their index. Relative distance then falls out of the dot product, which is elegant and works well.

The problem is extrapolation. A model trained on sequences up to 8,192 tokens has never seen the rotation angles that positions beyond that produce. Fed a longer sequence, the values are out of distribution and quality degrades sharply — sometimes into incoherence rather than a graceful decline.

So the window is not one number but two agreeing: how much compute you can afford, and how far the position encoding has been trained or adapted to reach.

How the window got bigger#

Both constraints were attacked separately, which is why context lengths jumped so fast.

For attention cost, FlashAttention was the significant one. It does not change the O(n²) arithmetic; it reorganises the computation so the full attention matrix is never written to slow GPU memory, computing it in tiles inside fast on-chip memory instead. The practical effect is a large constant-factor win and far less memory, which moved long-context training from impossible to expensive. Sparse and sliding-window attention take a different route, having each token attend to a subset of the others.

For position encoding, the techniques extend RoPE rather than replace it. Position interpolation scales position indices down so a longer sequence maps into the range the model already knows. NTK-aware scaling and YaRN refine that by adjusting frequencies unevenly, preserving the high-frequency detail that carries local ordering while stretching the low-frequency components that carry long-range position. These usually involve a short fine-tune at the longer length rather than working zero-shot.

Neither line of work made context free. They made it affordable.

Does a bigger window mean better recall?#

No, and this is the practical point. A model advertising a million-token window is telling you what it will accept, not that attention is distributed evenly across it.

Retrieval accuracy commonly degrades for information in the middle of a long input, with material at the beginning and end recalled more reliably — a pattern documented across models and frequently described as a "lost in the middle" effect. Needle-in-a-haystack tests, where a single fact is planted at varying depths and the model is asked to retrieve it, are the standard way to probe this, and results vary considerably between models at the same nominal length.

Three consequences for anyone building on these models:

1. Position matters. Put instructions and the most important material at the start or the end, not buried at 60% depth.
2. More context is not free accuracy. Padding a prompt with marginally relevant documents can lower answer quality as well as raising cost and latency.
3. Retrieval still earns its place. Selecting the right ten thousand tokens generally beats submitting a million and hoping, on cost, latency, and often accuracy.

What this means in practice#

Treat the context window as an attention budget rather than a container to fill.

• Put the task first. Instructions at the top, ahead of long reference material.
• Trim the transcript. Summarise old turns instead of resending them verbatim; every turn re-reads everything.
• Retrieve rather than dump. Ten relevant chunks beat a hundred loosely related ones.
• Test at your real length. A model's behaviour at 8k says little about its behaviour at 200k. Measure on your own data.
• Watch the cost curve. Long context is priced per token and re-read every turn, so a long conversation compounds.

For how the window shapes prompt design in practice, see LLM context windows explained and prompt engineering for developers.

As of August 2026#

Frontier models span roughly 100k to several million tokens of advertised context. Those figures move quickly, and the gap between advertised length and reliable-recall length is the number worth measuring for your own workload rather than reading off a specification sheet.

Measuring your own workload starts with knowing how much text you are actually sending. The Character Counter gives an exact count including multi-byte characters, which is where naive estimates drift.