RAG or Long Context? A Decision Guide That Doesn't Hand-Wave
Context windows grew past a million tokens and everyone declared retrieval dead. It isn't. Here is the actual decision procedure, with the trade-offs that matter in production.
Every time context windows grow, someone announces that retrieval-augmented generation is obsolete. Just put the documents in the prompt. The argument is intuitive, and for a narrow class of problems it is correct. For most production systems it is not, and the reason has less to do with capability than with economics and latency.
Here is a decision procedure you can actually apply.
Start with the size of the corpus, not the size of the window
The first question is not "does it fit?" but "does it fit every time?"
A million-token window sounds enormous until you put a real corpus next to it. A mid-sized company's internal documentation is routinely tens of millions of tokens. A single year of support tickets can exceed that. If your corpus is larger than the window, the decision is made for you: you need retrieval, and the only open question is what kind.
If the corpus genuinely fits — a contract, a codebase of moderate size, a research paper set — then long context is on the table, and the next two questions decide it.
Then look at what you pay per request
This is where the "just put it in the prompt" argument usually collapses.
Stuffing 500,000 tokens into every request means paying for 500,000 input tokens on every request. Retrieval means paying for perhaps 4,000. At the input prices prevailing in 2026, that difference is roughly two orders of magnitude per call. For a system handling a handful of queries a day, it is irrelevant. For anything with real traffic, it is the entire budget.
Prompt caching changes this calculation substantially but does not erase it. Caching works well when the large context is stable across requests — the same contract, queried many times. It works poorly when each request needs a different slice of a large corpus, which is the common case.
The rule of thumb: stable context, high query volume, fits in the window → long context with caching. Varying context → retrieval.
Then look at what you pay in latency
Long inputs are slow to process. Time-to-first-token scales with input length, and for a user-facing product the difference between a 300ms and a 4-second first token is the difference between a tool people use and one they abandon.
Retrieval adds its own latency — an embedding call plus a vector search, typically 50–150ms — but that is usually far less than the prefill cost of a very large prompt.
If your product is a batch pipeline, ignore this section entirely. If a human is waiting, it may dominate everything else.
The quality question is genuinely contested
The honest position is that neither approach dominates on quality, and which wins depends on the failure mode you care about.
Long context wins when the answer requires synthesizing information spread across a document in ways a chunk-based retriever will miss. Ask "how did this character's motivation change across the novel" and no top-k retrieval over chunks will assemble the right evidence. The model needs to see the whole thing.
Retrieval wins when precision matters more than breadth. A focused set of relevant passages produces more reliable citation and less drift than burying the relevant paragraph among hundreds of thousands of irrelevant tokens. Models have improved considerably at finding a needle in a long haystack, but performance still degrades as the haystack grows, and it degrades unevenly depending on where in the context the needle sits.
The practical consequence: if you cannot predict which parts of the corpus matter for a given query, retrieval is doing real work for you. If the relevant scope is obvious and bounded, long context removes a whole subsystem.
The hybrid that most mature systems converge on
In practice the interesting systems do not choose. They retrieve at coarse granularity and then let long context do the reading.
Instead of retrieving 400-token chunks, retrieve whole documents — or whole sections — and put five of them in a 200,000-token window. You get the precision of retrieval at the document level and the synthesis quality of long context within each document. You avoid the classic chunking failure where a passage is split down the middle and neither half makes sense alone.
This is usually the right default for document-heavy applications, and it is noticeably easier to debug than fine-grained retrieval, because you can read what the model was given.
What actually decides it
Run through this in order:
- Does the corpus fit in the window? No → retrieval. Yes → continue.
- Is the context stable across many requests? Yes → long context with prompt caching is likely cheapest. No → continue.
- Is a human waiting on the response? Yes → prefill latency probably rules out very large prompts.
- Does answering require whole-corpus synthesis? Yes → long context, or document-level retrieval feeding long context.
- Otherwise → retrieval, with the coarsest chunks you can afford.
What has genuinely changed
The claim "vector databases are dead" is wrong, but the weaker version of it is right: fine-grained chunking is much less necessary than it was. The era of splitting everything into 300-token fragments with elaborate overlap strategies was a workaround for small context windows. That workaround has largely outlived its purpose.
What has not changed is that retrieval is how you avoid paying to read your entire corpus on every question. That constraint is economic, not architectural, and larger windows do not remove it — they move it.
If you are building today and unsure, start with document-level retrieval into a large window. It is the option that is hardest to regret.