A larger context window did not make retrieval optional

A team builds an assistant over their internal policy handbook. The handbook runs to a few hundred pages, so they paste the whole thing into every prompt. No index, no embeddings, no vector store. It answers well, it points at the right clause, and it took an afternoon. That is a good decision and it is worth saying so plainly, because most of what has been written about retrieval was written when pasting the corpus was not an option.

Some months later the same assistant is asked to cover the contract archive, the support ticket history and every superseded version of the handbook. Now there are three problems arriving together. The corpus no longer fits at any window size. A large share of the questions come from people who are not allowed to read a large share of the corpus. And somebody in finance has noticed that every question, including what time the office opens, is billing for the entire archive.

The question people actually ask at this point is whether they can delete the vector store now that windows are bigger. The useful answer is that the window changed what retrieval has to be good at. It did not touch the reasons for having it.

What the bigger window actually changed

Retrieval used to be an exercise in compression. The budget was small, so you chopped documents into passages tight enough that a handful would fit, tuned the overlap, argued about chunk size, and lived with the fact that the passage holding the answer had lost the meaning it took from the paragraph above it. Most of that work was a tax on a constraint that has loosened. With a large window you can retrieve whole sections or whole documents, and send twenty candidates rather than three. Precision matters less, because a few irrelevant documents sitting inside a hundred thousand tokens are absorbed rather than fatal. Recall matters exactly as much as it always did, because a document you never fetched cannot be reasoned over regardless of how much room there was for it.

A large window is a bigger budget for the retrieval step to spend. It is not a reason to skip the step.

The pattern, stated plainly

Whether the corpus is bounded, and who is allowed to read it

A handbook is bounded and slow moving. A contract archive, a decade of tickets, a protocol library maintained across several sites: those are not bounded, and they grow faster than context windows do. If the honest answer to how big this gets is that nobody knows, you are building retrieval eventually. Building it later costs more than building it now, because by then there are prompts, evaluation sets and user habits shaped around the paste, and all of them have to be redone.

The access question decides it more often than corpus size does, and gets discussed least. If every call carries the whole corpus, every user effectively has the whole corpus, and the only thing between a contractor and the salary review file is a line in the system prompt asking the model not to look there. That is not an access control. Filtering per user means assembling a different context for each requester, which costs more than it sounds, because the shared prefix that made pasting affordable is now unique per person. Retrieval puts the filter where it belongs. The query runs against an index that already knows who is asking, and documents the user cannot see are never candidates. The model cannot leak what was never put in front of it.

Money and seconds

Prefill is not free. Time to first token grows with the length of the input, so a corpus in the prompt sets a latency floor you cannot optimise away at the application layer. For a research tool where an analyst asks a few questions a day, several extra seconds is nothing. For an assistant that has to answer while somebody is still on a call, it is the whole product. Cost has the same shape: charged per call, against a corpus that grows, multiplied by the adoption you are hoping for. It is one of the few numbers in this work that gets worse precisely when things are going well.

The kind of question being asked

• Single fact lookup What is the notice period. Both approaches handle these, and this is the only category most demonstrations test.
• Multi-hop The answer needs two documents joined, and neither mentions the other. A long window handles it when both happen to be present. Retrieval needs a query that finds the second document, which the user's own words rarely describe.
• Aggregation How many suppliers carry a clause like this one. Neither approach is good. Top-k returns k results whatever the true count is, and a long window asks the model to count reliably across a very large input, which is not what it is built to do.
• Absence Is there anything here that contradicts this. Vector search returns nearest neighbours whether or not anything relevant exists, so the model is handed plausible near misses and often answers from them. A long window can genuinely say no, provided the corpus really does fit.
• Versioned and temporal What is the rule today. The difficulty is near duplicates: seven revisions of one policy differing in a single clause. Both approaches struggle, and this is where most production incidents come from.
• Exact identifiers Clause references, part numbers, clinical codes, ticket ids. Embedding search is poor at these and a keyword index is good at them, which is the practical reason working systems tend to run both and merge, rather than because hybrid search sounds current.

Running the comparison so that the result means something

This comparison is cheap to run and almost nobody runs it, because it requires the one artefact nobody wants to sit down and make: a graded set of real questions with agreed answers. Vendor benchmarks will not substitute for it, and neither will the needle in a haystack tests that report near perfect recall. Those plant one verbatim sentence in filler text. Your corpus is full of documents that all look like the right answer.

Then run it again when the corpus doubles, and when the question mix shifts because a new department started using it. The answer moves with corpus size and question type, not with release notes, and a comparison from last quarter against a third of today's corpus is not evidence about today.

Where retrieval is the wrong answer

If the corpus is small, stable, readable by everyone who uses the system, and busy enough to keep a cache warm, then pasting it is correct and a retrieval pipeline is machinery you will maintain for no return. We have said this to teams and watched the larger piece of work disappear with it. The pipeline is not free: an index to keep in step with the source, an embedding model that becomes a dependency with its own upgrade path, a reranker, a new failure mode where the answer is wrong because the fetch was wrong, and a second component to evaluate. That is worth paying for when the corpus is unbounded, when permissions differ by user, or when the bill scales with something outside your control. It is not worth paying for because the architecture diagram looks more serious with a vector store on it.

The test that settles the argument is whether you can state, from your own graded set, what each approach costs you per correct answer and where each one breaks. Teams that can state it change their minds calmly when the numbers move, and they usually end up somewhere in between, retrieving generously and letting the large window absorb the slack. Teams that cannot are not comparing approaches at all. They are comparing announcements.