Pasting a long document into a model and asking a specific question often produces an answer that is fluent, plausible and wrong about the part you cared about. The failure has identifiable mechanics rather than being random.
What a context window is
A context window is the maximum amount of text a model can consider at once, measured in tokens rather than words.
Everything the model uses must fit inside it: the system instructions, the conversation so far, the document and the space reserved for the reply.
Windows have grown enormously, and documents that once needed splitting now fit whole, which changed the shape of the problem without eliminating it.
Fitting is a necessary condition, not a sufficient one. A model can hold a long document and still fail to use the relevant part of it.
The distinction between what fits and what is actually attended to is where most confusion about long context comes from.
Why the middle receives less attention
Models attend unevenly across position. Material at the start and end of a long input has more influence on the output than material in the middle.
The pattern is consistent enough that practitioners plan around it, placing critical instructions at the beginning and repeating them at the end.
Training explains part of it. Documents in training data put important framing early and conclusions late, and the model absorbed that regularity.
The consequence is that a fact stated once, halfway through a long report, is the least reliably retrieved fact in the document.
Which is precisely where such facts tend to live in real reports.
How retrieval changes the problem
Rather than supplying a whole document, retrieval systems search for relevant passages and supply only those.
This keeps input short and focused, and it makes the model's grounding inspectable, since you can see which passages were selected.
The failure mode moves rather than disappearing. If the search returns the wrong passages, the model answers confidently from the wrong material.
Search quality now determines answer quality, and search over prose is harder than it appears when the question uses different vocabulary from the source.
Many systems described as model failures are retrieval failures with the model faithfully summarising what it was given.
Why summarising first loses the thing you needed
A common pattern is to summarise sections, then answer from the summaries. This fits within any window and is cheap.
Summarisation is lossy by design. It preserves what appeared important in isolation and discards the rest.
A caveat in a footnote, a qualifying date, an exception clause: each is minor within its section and decisive for a specific question.
The summary reads well and the answer derived from it is confidently wrong, with no signal that a discarded detail governed the outcome.
Summarise-then-answer is safe for questions about themes and dangerous for questions about specifics.
What chunking does to argument structure
Splitting a document into fixed-size pieces cuts arguments at arbitrary points.
A claim in one chunk and its qualification in the next become independent statements, and only one of them may be retrieved.
Tables suffer most. Separated from their headers and captions, rows of numbers lose the meaning that made them interpretable.
Overlapping chunks reduce the problem at the cost of duplication, and splitting on document structure rather than character counts works better where structure exists.
A document written as continuous prose without headings is genuinely hard to chunk well, which is why formatting affects answer quality.
Why longer windows did not solve it
Window growth removed the fitting problem and left the attention problem, which is a different constraint.
Cost scales with input length, so sending a hundred-page document with every question is expensive when the same answer could come from two pages.
Latency scales too. Long inputs take noticeably longer to process, which changes what kind of interface the application can offer.
Accuracy on retrieval-style questions has not improved proportionally with window size, which is the finding that surprised people most.
Larger windows are best used for genuinely holistic tasks rather than as a substitute for finding the right passage.
How document structure helps
Models use structural signals. Headings, numbered sections, tables with headers and explicit cross-references all improve retrieval and reasoning.
Converting a document to clean text with its structure preserved is worth more than most prompt refinements, and it is usually skipped.
Scanned documents processed by optical recognition are a common source of quiet failure, since column layouts interleave and tables dissolve into number sequences.
Checking what the model actually received, rather than what the file contained, resolves a surprising share of unexplained wrong answers.
The input pipeline deserves as much scrutiny as the prompt, and it receives far less.
Where long context genuinely works
Tasks that need the whole document benefit from the whole document being present.
Checking a long contract for internal contradictions, tracking whether a narrative maintains consistency, or comparing two full versions all require global view that retrieval cannot supply.
Style and tone analysis across a body of writing is similar, since the signal is distributed rather than located in specific passages.
The useful test is whether the answer depends on a few specific passages or on the document as a whole. The first case wants retrieval; the second wants the full window.
Choosing wrongly between those two is the most common architectural mistake in document-handling systems, and it produces failures that look like model weakness rather than design error.