AI

When to skip RAG — long context, fine-tuning, SQL, and prompt caching

RAG is one option, not the default. Five situations where long context, fine-tuning, a direct database query, or prompt caching beats a RAG pipeline.

The previous three articles — RAG from scratch, chunking strategies, and evaluation — assumed RAG was the right tool. This one is the counter-argument: there are five concrete cases where RAG is the wrong default, and picking it anyway adds complexity and cost without buying anything.

The honest frame: RAG is a specific solution to “the LLM doesn’t know about my data, and the data is large”. When the data isn’t large, isn’t stable, or isn’t the kind of thing a retrieval step helps with, something else wins.

Case 1 — Long context: just put everything in the prompt

The frontier LLM context windows in late 2026:

  • Claude Opus 5.5 and Sonnet 5.5 — 1 million tokens
  • Claude Fable 5.1 — 1 million tokens (highest reasoning tier)
  • GPT-5 — 400,000 tokens
  • Claude Haiku 4.5 — 200,000 tokens

One million tokens is roughly 555,000 words on the current Claude tokenizer (introduced with Opus 4.7) — about five thick technical books. For many “RAG use cases”, the entire corpus fits.

When this wins over RAG:

  • Total corpus is under ~500,000 tokens (leaves comfortable room for prompt + answer within a 1M-token window)
  • Content is stable (not updated more than a few times a month)
  • Latency and cost per query are acceptable without optimization

Why skip RAG here: no chunking strategy to tune. No retrieval metrics to track. No embedding model to deploy. No vector database. The LLM reads the full context and composes an answer; one component to debug, not four.

The cost math: with per-token input pricing in late 2026 (varies by provider and tier — check current rates on each provider’s pricing page), a 500K-token input is a real per-query cost. For a few queries per hour, long-context is fine. For hundreds of queries per minute, the per-query cost of re-reading 500K tokens dominates and RAG (or Case 4 — prompt caching) wins.

# Pseudocode for the simplest "no RAG" setup
import anthropic
client = anthropic.Anthropic()

with open("docs/full_corpus.md") as f:
    corpus = f.read()  # ~400K tokens of documentation

def answer_question(question: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": (
                f"The following is the full documentation for Project X:\n\n"
                f"{corpus}\n\n"
                f"Based on the documentation above, answer: {question}"
            ),
        }],
    )
    return response.content[0].text

That is a complete working QA system in nine lines. No chunking, no embedding, no vector DB, no retrieval evaluation. If the corpus fits and the per-query cost is bearable, this is the right answer.

Case 2 — Fine-tuning: teach the model instead of retrieving

RAG retrieves facts at query time. Fine-tuning changes the model’s weights so it knows the facts before any query. They solve different problems:

RAGFine-tuning
Up-to-date (data changes → re-index)Snapshot (data changes → retrain)
Teaches contentTeaches style, format, or domain behaviour
Facts stay auditable (chunks are visible)Facts dissolve into weights (not inspectable)
Cheap to updateExpensive to retrain
Good at “what does the doc say about X?”Good at “respond in our brand voice”, “always output JSON matching this schema”, “classify tickets into one of 20 categories”

When fine-tuning wins over RAG:

  • The problem is a consistent response FORMAT or STYLE, not a facts lookup. (“Always respond as a friendly customer-support rep in two paragraphs.”)
  • The task is a CLASSIFICATION with 10+ categories where in-context examples aren’t enough.
  • You need structured output that few-shot prompting struggles to make reliable at scale.
  • The content is stable for months at a time, and inference is high-volume (so saving per-query tokens matters).

Why skip RAG here: no amount of chunking makes an LLM reliably output a specific JSON schema. Retrieval returns documents; fine-tuning teaches patterns. Different problem, different tool.

Common trap: fine-tuning to inject facts. Models forget specific facts and generalise or confabulate from fine-tuning data. Fine-tune for behaviour; use RAG (or long context) for facts.

Case 3 — SQL or direct database: structured data has better tools

If the question is “how many users signed up last week?”, “what are the top 10 products by revenue?”, or “list orders from a specific customer”, a RAG pipeline is doing the wrong job.

The data lives in a database. The LLM should write a SQL query (or call a well-defined tool), the database executes it, and the result flows back. No embeddings. No vector search. No chunk retrieval.

# Text-to-SQL via a tool call
TOOLS = [{
    "name": "run_sql",
    "description": "Runs a read-only SQL query against the production warehouse.",
    "input_schema": {
        "type": "object",
        "properties": {"query": {"type": "string"}},
        "required": ["query"],
    },
}]

response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    tools=TOOLS,
    messages=[
        {"role": "user", "content": "How many new signups in the last 7 days, grouped by day?"},
    ],
)
# If response.stop_reason == "tool_use", execute the SQL and send result back as tool_result

When direct queries win over RAG:

  • The user’s question maps to a database aggregate or filter
  • The data is structured (rows and columns), not prose
  • Answers need to be exact (“1,247 signups”), not synthesised
  • You need up-to-the-second freshness (database state right now)

Why skip RAG here: vector search on tabular data is a worse version of SQL. A well-indexed WHERE clause and GROUP BY is faster, cheaper, and auditably exact. RAG would reduce “1,247” to “a large number of signups” via LLM summarisation. That’s a regression.

Hybrid: many real systems need BOTH. “Why did signups drop last Tuesday?” needs SQL (to confirm the drop) PLUS RAG over internal docs/incident reports (to find the cause). Build both routes and let the LLM pick, don’t force one into the other’s shape.

Case 4 — Prompt caching: the content is small, stable, and asked many times

Both Anthropic (cache_control on message blocks) and OpenAI (automatic server-side caching) support prompt caching: when the start of a prompt is identical across requests, the model doesn’t re-process it on every call. Cached input tokens cost a fraction of fresh input tokens (specific discounts depend on provider and tier — check current docs).

For a knowledge base that fits in context and is read many times per hour, prompt caching turns Case 1’s “put everything in the prompt” into a near-free pattern:

# Anthropic prompt caching example
import anthropic
client = anthropic.Anthropic()

with open("docs/full_corpus.md") as f:
    corpus = f.read()

def answer_question(question: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=1024,
        system=[{
            "type": "text",
            "text": corpus,
            "cache_control": {"type": "ephemeral"},  # caches this block
        }],
        messages=[{"role": "user", "content": question}],
    )
    return response.content[0].text

The first call processes the full corpus. Subsequent calls within the cache window (~5 minutes by default for Anthropic ephemeral caches) reuse the cached KV state; only the per-query question is billed at full input rate.

When prompt caching wins over RAG:

  • Corpus is 10K-500K tokens (fits in context, big enough that re-processing is costly)
  • Query rate is high enough that cache hits are frequent (several queries per minute minimum)
  • Content changes rarely (minutes to hours between updates — longer than the cache TTL just means a fresh cache miss, not a correctness problem)

Why skip RAG here: prompt caching gives you long-context quality (model sees the full corpus) at closer-to-RAG economics (per-query tokens are small). No retrieval infrastructure, no chunking tuning, no evaluation of retrieval metrics.

When this stops being enough: at ~500K+ tokens of corpus, cache cost per minute starts to approach the cost of a well-tuned RAG retrieval. At that scale RAG wins again on economics.

Case 5 — Keyword / BM25 search: exact matches don’t need semantics

Semantic search (dense vectors) is useful when the user’s phrasing differs from the document’s phrasing. For queries that reference a specific error string, product SKU, order ID, or exact phrase, keyword search is more reliable AND cheaper.

Example: a user searches for TOKEN_EXPIRED in a log corpus. A dense retriever embeds “TOKEN_EXPIRED” into a vector and asks for similar vectors — which may surface documents about “authentication timeout”, “session expiry”, “credential refresh”. All related, none exactly what the user asked for. A plain SQL LIKE '%TOKEN_EXPIRED%' or an inverted index returns the exact matches first.

When keyword / BM25 wins over RAG:

  • Users frequently search for exact strings (error codes, product names, specific IDs)
  • The corpus has rich metadata (titles, tags, categories) that filter cleanly
  • Precision matters more than recall (“find me the ticket where X happened” vs “find me related tickets”)

Common pattern — hybrid retrieval: many production RAG systems run BOTH dense retrieval and BM25 in parallel, then merge results. Elastic (via ELSER), Weaviate, Qdrant and others support this out of the box. If “semantic + keyword” works, pure RAG was never the right default.

The decision tree

Pick based on the actual question, not the architecture reflex:

  1. Does the question map to a database query (filter / aggregate / join)? → SQL or tool call. Not RAG.
  2. Does the corpus fit in context AND change infrequently AND query volume is low? → Long context (Case 1).
  3. Same as above, BUT query volume is high? → Prompt caching (Case 4).
  4. Do users search for exact strings — error codes, IDs, product names? → Keyword/BM25 first; add dense retrieval as fallback. Not pure RAG.
  5. Is the task about response FORMAT or STYLE, not facts lookup? → Fine-tuning.
  6. Is the corpus > ~500K tokens, updates regularly, and queries need freshness? → RAG. Build the pipeline.

Most teams start at step 6 by default. The first five options cover a surprising share of real “RAG” problems with less infrastructure.

Signals that RAG is overkill

  • Three sentences in a prompt would answer most questions
  • The corpus is under 100K tokens and rarely updates
  • Users mostly search for exact terms
  • The highest-value queries resolve to a SQL SELECT
  • Nobody can list five concrete queries RAG would be solving

Signals that RAG is still the right call

  • Corpus is at least a few hundred thousand tokens and growing
  • Content updates daily or faster (fine-tuning can’t keep up)
  • Users ask open-ended, prose-shaped questions
  • Freshness matters (re-training a model every day isn’t viable)
  • Audit requirements make “here’s the chunk I used” valuable (compliance, legal, support)

Bottom line

RAG won the architecture-of-the-year award in 2023-2024 because long-context models were small and expensive. In late 2026, with 400K-1M-token context windows, prompt caching, cheap tool use for SQL, and mature fine-tuning APIs, “LLM over private data” is five different problems — and RAG is the right answer to only one of them.

The practical implication: pick the cheapest architecture that solves the problem. Benchmarks are easier to run against a 10-line long-context script than against a four-component RAG pipeline. If the simple version is good enough on your test set from the evaluation article, ship it. Add RAG only when the simple version demonstrably fails — on cost, on freshness, on coverage, or on retrieval precision at scale.

The full cluster

Read the pillar first if you landed here cold. Read “when to skip” last if you’re already shipping a RAG system — the decision tree is more useful as a design review than as a getting-started guide.