The million-token context window is one of those numbers that makes an immediate impression. Put an entire product manual, a repository, or a year’s worth of support tickets into one prompt, and surely the model can finally see the whole picture. Or can it? The story is more complicated than it may seem.
It turns out that while a model can see more, that does not mean it can use more reliably. This distinction matters. A context window is a capacity limit, not a quality guarantee. A model can accept more information, but still miss relationships between documents, miss important facts, or confidently supply an answer that is absent from the material. The practical question is not, “How much can this model hold?” It is, “How much can it use correctly for this job?”
This matters in practice because the best prompt is often not the biggest one. It is the smallest, most relevant context that still supports the task. For example, if you paste hundreds of support tickets into a prompt and ask the model to identify every outage caused by one bad load balancer setting, the model may find one incident but miss the others or combine unrelated cases.
Context is not a memory guarantee
A context window is the maximum amount of input a model can accept and process in one request. It is tempting to treat that number like the capacity of a hard drive: if a model supports one million tokens, then one million tokens are available for reasoning.
Language models do not work like that. They process the request by applying attention across the prompt. In simple terms, attention is the mechanism the model uses to decide which parts of the input matter most as it generates an answer. In doing that, the model must distinguish relevant information from noise, preserve intermediate facts, and then produce a grounded answer. Any of those steps can fail. Every additional unrelated detail, every developer note, every comment, every if-else logic branch that can be ignored – all of these can add noise that consumes attention and reduces the model’s ability to focus on the relevant facts. The more data you give it, the more it has to decide what matters, and the more ways it can go wrong.
There is also a challenge for language models called context rot. This refers to a gradual decline in a model’s ability to use the provided information correctly as the prompt or conversation grows. One way it shows up is that the model may ignore or misinterpret facts that were already given. Another is recency bias, which means the model may give too much weight to content near the end of the prompt and, in some cases, the beginning, while paying less attention to the middle. Over time, the model may also become more likely to invent details instead of relying on the provided material.
To be clear, these issues are an area of heavy research. They exist with all models, and they are not unique to any one architecture or training approach. At the same time, researchers have shown that the issues become more frequent and pronounced as the result of two factors:
- The context window becomes filled with more tokens, which increases the amount of information the model must process and prioritize.
- The available context window size is increased. As the number of allowed tokens grows, the resulting model is more likely to exhibit issues.
What research shows about context
One of the clearest examples comes from a December 2025 study of hallucination in document question answering. It evaluated 35 open-weight models at 32K, 128K, and 200K tokens across more than 4,000 runs using the RIKER methodology.
A system can retrieve real facts effectively and still answer confidently about facts that do not exist. That is exactly why a retrieval-only benchmark can produce false confidence for a document question-and-answer deployment. The study separated reliable answering into two abilities:
- Grounding
- Can the model find and correctly extract information that is present in the documents?
- Fabrication resistance
- Can the model avoid inventing an answer when the requested information is not present?
The documents and questions used were generated from known truths, which let the researchers distinguish a correct answer from a fabricated one without asking another model to judge the response.
The broad result was not subtle: models became less reliable as the supplied context grew. Across the 26 models evaluated at both 32K and 128K, the median overall decline was 10.4 percentage points. Among the 11 models evaluated through 200K, the mean decline from 32K to 200K was 23.9 points. For one model, overall accuracy fell from 93.26% at 32K to 37.65% at 200K. Its fabrication rate on questions about entities absent from the documents rose from 7.04% to 71.62%.
That is not a claim that every model has the same cliff. In fact, the researchers found that each model behaved differently and showed different degradation patterns. While the overall problem was consistent and repeatable, the specific levels of degradation varied by model.
At 32K, the best model fabricates 1.19% of answers, top-tier models fabricate 5–7%, and the median model fabricates roughly 25%. At 128K, the floor nearly triples to 3.19% and only 5 of 26 tested models remain below 10% fabrication. At 200K, no model stays below 10%. In practice, fabrication rates will be higher still …
Simple retrieval is not the work you need done
An earlier RIKER methodology paper makes the next part of the story clearer. It separates single-document extraction from cross-document aggregation. The first task asks the model to find a fact. The second asks it to find all relevant facts, preserve which source each came from, and then count, compare, or combine them correctly.
That second category degraded more sharply. In the paper’s 32K-to-200K comparison, one model’s overall score dropped by 11 points, while its aggregation score fell from 92.6% to 59.6%. This does not make the model bad. It just indicates that as the context becomes larger, the model’s ability to reason over multiple related facts becomes more fragile. It is not enough to retrieve a single fact; the model must also maintain the relationships between facts, which becomes more difficult as the context grows.
Sending the full corpus feels safer than selecting a subset, because nothing was left out. As you can see, an unfiltered corpus creates a different failure mode. The model now has to reason what matters, keep all relevant details aligned, and resist plausible but unsupported completions. More information provides more opportunities to be wrong.
A million tokens is not the answer
The headline number gets even less decisive at the million-token scale. A separate 2026 exploratory study evaluated five frontier models on a classical Chinese corpus with contexts at 256K, 512K, and roughly one million tokens. It used both single-fact retrieval and three-hop relationship questions.
Several models retrieved a single planted fact perfectly at one million tokens. The harder three-hop task told a different story. One model remained stable in the test, some degraded gradually, and others showed a sharp drop between 512K and one million tokens. In other words, even models with the same advertised window did not have the same usable window for a task that required linking facts together. In fact, as the size increased from 512K to 1M tokens, one model tested went from identifying 4/5 of the facts correctly to none, while another went from 4/5 to 2/5.
This is useful corroboration, not a universal leaderboard. The authors call it an exploratory pilot with a small sample, a single classical Chinese corpus, and some non-deterministic model settings. So it is better read as evidence of where a failure cliff can appear than as a universal measure of effective context size. Still, the study demonstrates that a model’s ability to retrieve one fact from a huge prompt is not evidence that it can reliably reason over many related facts in that prompt.
That result reinforces earlier benchmark work. RULER distinguishes between a model’s claimed context length and its effective context length, meaning the longest input size where performance still stays above a chosen threshold. BABILong goes further and reports that popular language models may effectively use only 10-20% of their claimed context on reasoning-heavy tasks, though the exact usable range varies by model and task. It also shows strong positional effects, with performance often depending heavily on where the relevant facts appear in the prompt.
Build for useful context
The answer is not to avoid long-context models. A large window can still be useful when used deliberately. The key is to keep the content relevant, structured, and limited to what the model can actually use. The more focused the window, the more likely it is to succeed. It also means that, given the same model, you should prefer the smallest context size that still supports the task. Treat it like any other production dependency: measure it, understand its failure modes, and choose it for a job it has demonstrated it can do.
That is an important aspect of those studies. In those evaluations, model selection and context curation had a larger effect on reliability than decoding settings such as temperature, top-p, or top-k. This is part of why it is so important to properly construct your instructions, skills, agents, and prompts. The model’s ability to reason is not just a function of how much it can see, but how well it can use what it sees.
The million-token number is still impressive. Just do not mistake a large context for a fix-all solution or an ideal default. At the end of the day, your results are controlled by how you craft your context – not by how much you can fit into a single prompt.