When to skip RAG — long context, fine-tuning, SQL, and prompt caching
RAG is one option, not the default. Five situations where long context, fine-tuning, a direct database query, or prompt caching beats a RAG pipeline.
The previous three articles — RAG from scratch, chunking strategies, and evaluation — assumed RAG was the right tool. This one is the counter-argument: there are five concrete cases where RAG is the wrong default, and picking it anyway adds complexity and cost without buying anything.
The honest frame: RAG is a specific solution to “the LLM doesn’t know about my data, and the data is large”. When the data isn’t large, isn’t stable, or isn’t the kind of thing a retrieval step helps with, something else wins.
Case 1 — Long context: just put everything in the prompt
The frontier LLM context windows in late 2026:
- Claude Opus 5.5 and Sonnet 5.5 — 1 million tokens
- Claude Fable 5.1 — 1 million tokens (highest reasoning tier)
- GPT-5 — 400,000 tokens
- Claude Haiku 4.5 — 200,000 tokens
One million tokens is roughly 555,000 words on the current Claude tokenizer (introduced with Opus 4.7) — about five thick technical books. For many “RAG use cases”, the entire corpus fits.
When this wins over RAG:
- Total corpus is under ~500,000 tokens (leaves comfortable room for prompt + answer within a 1M-token window)
- Content is stable (not updated more than a few times a month)
- Latency and cost per query are acceptable without optimization
Why skip RAG here: no chunking strategy to tune. No retrieval metrics to track. No embedding model to deploy. No vector database. The LLM reads the full context and composes an answer; one component to debug, not four.
The cost math: with per-token input pricing in late 2026 (varies by provider and tier — check current rates on each provider’s pricing page), a 500K-token input is a real per-query cost. For a few queries per hour, long-context is fine. For hundreds of queries per minute, the per-query cost of re-reading 500K tokens dominates and RAG (or Case 4 — prompt caching) wins.
# Pseudocode for the simplest "no RAG" setup
import anthropic
client = anthropic.Anthropic()
with open("docs/full_corpus.md") as f:
corpus = f.read() # ~400K tokens of documentation
def answer_question(question: str) -> str:
response = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=1024,
messages=[{
"role": "user",
"content": (
f"The following is the full documentation for Project X:\n\n"
f"{corpus}\n\n"
f"Based on the documentation above, answer: {question}"
),
}],
)
return response.content[0].text
That is a complete working QA system in nine lines. No chunking, no embedding, no vector DB, no retrieval evaluation. If the corpus fits and the per-query cost is bearable, this is the right answer.
Case 2 — Fine-tuning: teach the model instead of retrieving
RAG retrieves facts at query time. Fine-tuning changes the model’s weights so it knows the facts before any query. They solve different problems:
| RAG | Fine-tuning |
|---|---|
| Up-to-date (data changes → re-index) | Snapshot (data changes → retrain) |
| Teaches content | Teaches style, format, or domain behaviour |
| Facts stay auditable (chunks are visible) | Facts dissolve into weights (not inspectable) |
| Cheap to update | Expensive to retrain |
| Good at “what does the doc say about X?” | Good at “respond in our brand voice”, “always output JSON matching this schema”, “classify tickets into one of 20 categories” |
When fine-tuning wins over RAG:
- The problem is a consistent response FORMAT or STYLE, not a facts lookup. (“Always respond as a friendly customer-support rep in two paragraphs.”)
- The task is a CLASSIFICATION with 10+ categories where in-context examples aren’t enough.
- You need structured output that few-shot prompting struggles to make reliable at scale.
- The content is stable for months at a time, and inference is high-volume (so saving per-query tokens matters).
Why skip RAG here: no amount of chunking makes an LLM reliably output a specific JSON schema. Retrieval returns documents; fine-tuning teaches patterns. Different problem, different tool.
Common trap: fine-tuning to inject facts. Models forget specific facts and generalise or confabulate from fine-tuning data. Fine-tune for behaviour; use RAG (or long context) for facts.
Case 3 — SQL or direct database: structured data has better tools
If the question is “how many users signed up last week?”, “what are the top 10 products by revenue?”, or “list orders from a specific customer”, a RAG pipeline is doing the wrong job.
The data lives in a database. The LLM should write a SQL query (or call a well-defined tool), the database executes it, and the result flows back. No embeddings. No vector search. No chunk retrieval.
# Text-to-SQL via a tool call
TOOLS = [{
"name": "run_sql",
"description": "Runs a read-only SQL query against the production warehouse.",
"input_schema": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
}]
response = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=1024,
tools=TOOLS,
messages=[
{"role": "user", "content": "How many new signups in the last 7 days, grouped by day?"},
],
)
# If response.stop_reason == "tool_use", execute the SQL and send result back as tool_result
When direct queries win over RAG:
- The user’s question maps to a database aggregate or filter
- The data is structured (rows and columns), not prose
- Answers need to be exact (“1,247 signups”), not synthesised
- You need up-to-the-second freshness (database state right now)
Why skip RAG here: vector search on tabular data is a worse version of SQL. A well-indexed WHERE clause and GROUP BY is faster, cheaper, and auditably exact. RAG would reduce “1,247” to “a large number of signups” via LLM summarisation. That’s a regression.
Hybrid: many real systems need BOTH. “Why did signups drop last Tuesday?” needs SQL (to confirm the drop) PLUS RAG over internal docs/incident reports (to find the cause). Build both routes and let the LLM pick, don’t force one into the other’s shape.
Case 4 — Prompt caching: the content is small, stable, and asked many times
Both Anthropic (cache_control on message blocks) and OpenAI (automatic server-side caching) support prompt caching: when the start of a prompt is identical across requests, the model doesn’t re-process it on every call. Cached input tokens cost a fraction of fresh input tokens (specific discounts depend on provider and tier — check current docs).
For a knowledge base that fits in context and is read many times per hour, prompt caching turns Case 1’s “put everything in the prompt” into a near-free pattern:
# Anthropic prompt caching example
import anthropic
client = anthropic.Anthropic()
with open("docs/full_corpus.md") as f:
corpus = f.read()
def answer_question(question: str) -> str:
response = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=1024,
system=[{
"type": "text",
"text": corpus,
"cache_control": {"type": "ephemeral"}, # caches this block
}],
messages=[{"role": "user", "content": question}],
)
return response.content[0].text
The first call processes the full corpus. Subsequent calls within the cache window (~5 minutes by default for Anthropic ephemeral caches) reuse the cached KV state; only the per-query question is billed at full input rate.
When prompt caching wins over RAG:
- Corpus is 10K-500K tokens (fits in context, big enough that re-processing is costly)
- Query rate is high enough that cache hits are frequent (several queries per minute minimum)
- Content changes rarely (minutes to hours between updates — longer than the cache TTL just means a fresh cache miss, not a correctness problem)
Why skip RAG here: prompt caching gives you long-context quality (model sees the full corpus) at closer-to-RAG economics (per-query tokens are small). No retrieval infrastructure, no chunking tuning, no evaluation of retrieval metrics.
When this stops being enough: at ~500K+ tokens of corpus, cache cost per minute starts to approach the cost of a well-tuned RAG retrieval. At that scale RAG wins again on economics.
Case 5 — Keyword / BM25 search: exact matches don’t need semantics
Semantic search (dense vectors) is useful when the user’s phrasing differs from the document’s phrasing. For queries that reference a specific error string, product SKU, order ID, or exact phrase, keyword search is more reliable AND cheaper.
Example: a user searches for TOKEN_EXPIRED in a log corpus. A dense retriever embeds “TOKEN_EXPIRED” into a vector and asks for similar vectors — which may surface documents about “authentication timeout”, “session expiry”, “credential refresh”. All related, none exactly what the user asked for. A plain SQL LIKE '%TOKEN_EXPIRED%' or an inverted index returns the exact matches first.
When keyword / BM25 wins over RAG:
- Users frequently search for exact strings (error codes, product names, specific IDs)
- The corpus has rich metadata (titles, tags, categories) that filter cleanly
- Precision matters more than recall (“find me the ticket where X happened” vs “find me related tickets”)
Common pattern — hybrid retrieval: many production RAG systems run BOTH dense retrieval and BM25 in parallel, then merge results. Elastic (via ELSER), Weaviate, Qdrant and others support this out of the box. If “semantic + keyword” works, pure RAG was never the right default.
The decision tree
Pick based on the actual question, not the architecture reflex:
- Does the question map to a database query (filter / aggregate / join)? → SQL or tool call. Not RAG.
- Does the corpus fit in context AND change infrequently AND query volume is low? → Long context (Case 1).
- Same as above, BUT query volume is high? → Prompt caching (Case 4).
- Do users search for exact strings — error codes, IDs, product names? → Keyword/BM25 first; add dense retrieval as fallback. Not pure RAG.
- Is the task about response FORMAT or STYLE, not facts lookup? → Fine-tuning.
- Is the corpus > ~500K tokens, updates regularly, and queries need freshness? → RAG. Build the pipeline.
Most teams start at step 6 by default. The first five options cover a surprising share of real “RAG” problems with less infrastructure.
Signals that RAG is overkill
- Three sentences in a prompt would answer most questions
- The corpus is under 100K tokens and rarely updates
- Users mostly search for exact terms
- The highest-value queries resolve to a SQL
SELECT - Nobody can list five concrete queries RAG would be solving
Signals that RAG is still the right call
- Corpus is at least a few hundred thousand tokens and growing
- Content updates daily or faster (fine-tuning can’t keep up)
- Users ask open-ended, prose-shaped questions
- Freshness matters (re-training a model every day isn’t viable)
- Audit requirements make “here’s the chunk I used” valuable (compliance, legal, support)
Bottom line
RAG won the architecture-of-the-year award in 2023-2024 because long-context models were small and expensive. In late 2026, with 400K-1M-token context windows, prompt caching, cheap tool use for SQL, and mature fine-tuning APIs, “LLM over private data” is five different problems — and RAG is the right answer to only one of them.
The practical implication: pick the cheapest architecture that solves the problem. Benchmarks are easier to run against a 10-line long-context script than against a four-component RAG pipeline. If the simple version is good enough on your test set from the evaluation article, ship it. Add RAG only when the simple version demonstrably fails — on cost, on freshness, on coverage, or on retrieval precision at scale.
The full cluster
- Build a RAG pipeline from scratch — the four-step loop in 300 lines
- Chunking strategies — five splitters and when to use each
- Evaluation — retrieval and generation metrics with runnable code
- When to skip RAG entirely — this article
Read the pillar first if you landed here cold. Read “when to skip” last if you’re already shipping a RAG system — the decision tree is more useful as a design review than as a getting-started guide.