AI

Evaluating RAG — retrieval metrics, generation metrics, and tooling

A practical RAG evaluation framework. Build a test set, score retrieval + generation separately with runnable Python, use LLM-as-judge safely, log what matters.

A RAG pipeline that works on your five test questions is not a RAG pipeline that works. The pillar walkthrough built the loop; the chunking article picked a splitter. This article is how to measure whether the whole thing is any good — before shipping, and after.

The frustrating thing about RAG evaluation: a bad answer can come from four separate places, and the fix for each is different. This article covers how to tell them apart.

Why RAG evaluation is harder than it looks

A classic ML model has one failure mode — the output is wrong. RAG has four:

  1. Chunking is bad. The right content exists in the corpus but got split in a way the embedding model can’t find it.
  2. Retrieval is bad. The right chunk exists in the index, but similarity search didn’t rank it in the top K.
  3. Generation ignored the context. Retrieval returned the right chunk, but the LLM hallucinated or blended in prior training.
  4. The corpus doesn’t have the answer at all. Everything upstream is fine; the user asked something your docs don’t cover.

Debugging a bad answer means figuring out which of the four happened. That requires measuring each layer separately — not just scoring the final answer.

Step 1 — build a test set

Nothing works without this. Metrics compute scores against a known-correct answer; you need the known-correct answers first.

A usable test set has 30-100 queries. For each query, record:

  • The question text, worded the way a real user would ask
  • The IDs of the chunks that actually contain the answer (one or more)
  • A one-paragraph “ideal answer” or a bullet list of key facts the response must include

Hand-crafting beats synthetic for the first 30. Open the most-asked internal questions, docs changelog, or customer support transcripts. Pick questions where the answer unambiguously lives in one or two specific chunks.

Synthetic test sets (an LLM generates queries from your docs) scale but miss the way real users ask things — “k8s pod stuck?” vs the LLM-generated “What are the common causes for Kubernetes Pods remaining in a Pending state?”. Both are legitimate questions; only one looks like real traffic.

Minimal test-set schema:

# tests.py
TEST_SET = [
    {
        "query": "pod stuck in ImagePullBackOff",
        "golden_chunk_ids": ["chunk_imagepullbackoff_causes_1"],
        "ideal_answer_facts": [
            "image name typo is a cause",
            "private registry credentials needed",
            "kubectl describe pod shows the exact pull error",
        ],
    },
    {
        "query": "when should I use pgvector over Pinecone",
        "golden_chunk_ids": ["chunk_vector_db_tradeoffs_pgvector", "chunk_vector_db_tradeoffs_pinecone"],
        "ideal_answer_facts": [
            "pgvector keeps data in Postgres (good for existing Postgres apps)",
            "Pinecone is hosted-only (less ops, more cost)",
        ],
    },
    # ... 30-100 of these
]

The schema deliberately has nothing fancy. Store it as a Python file, JSON, or a CSV — whatever is easy to edit by hand.

Step 2 — score retrieval

Retrieval metrics ignore the LLM entirely. They ask: did the right chunks appear in the top-K retrieved results, and at what rank?

Three metrics cover 95% of cases:

Hit@K — fraction of test queries where at least one golden chunk is in the top K. Simple yes/no per query, averaged. The headline number.

MRR (Mean Reciprocal Rank) — average of 1 / rank_of_first_golden_chunk across queries. Rewards ranking the right chunk HIGH, not just finding it somewhere in top K.

Recall@K — fraction of golden chunks that appear in top K (when a query has multiple golden chunks). Matters for comparison or synthesis questions where several chunks together make the answer.

Pure Python implementation (no eval library required):

# retrieval_metrics.py
from typing import Iterable

def hit_at_k(retrieved_ids: list[str], golden_ids: Iterable[str], k: int) -> int:
    top_k = retrieved_ids[:k]
    return int(any(g in top_k for g in golden_ids))

def reciprocal_rank(retrieved_ids: list[str], golden_ids: Iterable[str]) -> float:
    for rank, chunk_id in enumerate(retrieved_ids, start=1):
        if chunk_id in golden_ids:
            return 1.0 / rank
    return 0.0

def recall_at_k(retrieved_ids: list[str], golden_ids: Iterable[str], k: int) -> float:
    golden_set = set(golden_ids)
    if not golden_set:
        return 0.0
    top_k = set(retrieved_ids[:k])
    return len(golden_set & top_k) / len(golden_set)


def evaluate_retrieval(test_set, retrieve_fn, k: int = 5):
    """retrieve_fn(query) → list of chunk_ids ordered by similarity (best first)"""
    hits, rrs, recalls = [], [], []
    for item in test_set:
        retrieved = retrieve_fn(item["query"])
        hits.append(hit_at_k(retrieved, item["golden_chunk_ids"], k))
        rrs.append(reciprocal_rank(retrieved, item["golden_chunk_ids"]))
        recalls.append(recall_at_k(retrieved, item["golden_chunk_ids"], k))
    return {
        f"hit@{k}": sum(hits) / len(hits),
        "mrr": sum(rrs) / len(rrs),
        f"recall@{k}": sum(recalls) / len(recalls),
    }

Running this against the pillar’s RAG pipeline looks like:

# evaluate_pipeline.py (continues the pillar's pipeline)
from pillar_pipeline import collection, embedder  # the Chroma collection from the pillar

def retrieve(query: str, k: int = 5) -> list[str]:
    query_embedding = embedder.encode([query])[0].tolist()
    results = collection.query(query_embeddings=[query_embedding], n_results=k)
    return results["ids"][0]  # list of chunk ids, ordered by similarity

scores = evaluate_retrieval(TEST_SET, retrieve, k=5)
print(scores)
# {'hit@5': 0.84, 'mrr': 0.62, 'recall@5': 0.71}

What good numbers look like: Hit@5 > 0.85 and MRR > 0.55 are solid baselines for a well-chunked corpus. Below that, the problem is almost always in chunking (strategy or size), not retrieval algorithm. Changing from cosine to dot-product similarity rarely moves these metrics; changing from fixed-size char to Markdown-aware chunking usually does.

Step 3 — score generation

Generation metrics evaluate the LLM’s answer given retrieved chunks. Three things to measure, each a separate dimension:

Faithfulness — did the answer use ONLY the retrieved context, or did it blend in training data? This is the hallucination metric. A faithful answer may be short or wrong, but every claim in it is supported by the retrieved chunks.

Answer relevance — does the answer actually address the question asked? Catches the case where retrieval found related-but-off-topic content, and the LLM dutifully summarized it instead of saying “I don’t know.”

Context precision — of the retrieved chunks, which ones were actually used in the final answer? Low context precision means retrieval is passing too much junk to the LLM; the LLM is doing extra work filtering context the user shouldn’t have had to pay for.

Scoring with LLM-as-judge

Faithfulness and relevance are hard to compute algorithmically. The standard approach is LLM-as-judge: a stronger model (or sometimes the same model with a careful prompt) scores the generated answer.

# generation_metrics.py
import anthropic

client = anthropic.Anthropic()

FAITHFULNESS_PROMPT = """You are a strict judge of RAG system outputs.

QUESTION: {question}

RETRIEVED CONTEXT (what the LLM was given):
{context}

LLM'S ANSWER:
{answer}

Rate the answer's FAITHFULNESS: does every factual claim in the answer appear in the retrieved context?

- Score 1.0 if all claims come from the context.
- Score 0.5 if some claims come from the context and some don't.
- Score 0.0 if claims clearly come from outside the context (hallucination or prior training).

Reply with ONLY a single number between 0.0 and 1.0. No explanation."""

def score_faithfulness(question: str, context: str, answer: str) -> float:
    response = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=10,
        messages=[{
            "role": "user",
            "content": FAITHFULNESS_PROMPT.format(
                question=question, context=context, answer=answer,
            ),
        }],
    )
    try:
        return float(response.content[0].text.strip())
    except (ValueError, IndexError):
        return 0.0  # judge failed to produce a number; count as worst case

Three practical rules for LLM-as-judge:

  1. Use a stronger judge than the generator. Scoring Haiku 4.5 output with Haiku 4.5 is noisy. Scoring with Sonnet 5.5 or Opus 5.5 is much more consistent. If the generator IS the strongest model you have, use a different model family as judge (OpenAI judging Anthropic or vice versa).

  2. Keep judge prompts short, scalar, and schema-locked. Ask for a single number, not a paragraph. Multi-criteria rubrics drift between runs; a single score with a clear anchor is reproducible.

  3. Run the same test query 3 times and average the scores. Judge temperature noise is real. For a 50-query test set this is 150 judge calls — spendable in dev, batch overnight for CI.

Full generation evaluation

# evaluate_generation.py
def evaluate_generation(test_set, rag_fn, score_fn) -> dict:
    """rag_fn(query) → (answer_text, retrieved_context_text)"""
    scores = []
    for item in test_set:
        answer, context = rag_fn(item["query"])
        s = score_fn(item["query"], context, answer)
        scores.append(s)
    return {
        "mean_faithfulness": sum(scores) / len(scores),
        "n_judged_zero": sum(1 for s in scores if s < 0.1),  # hallucinated
    }

When ground truth exists — stricter checks

If the test set has ideal_answer_facts (bullet list of required facts), do a cheap substring check as a smoke test — it catches blatant omissions without a judge call:

def fact_coverage(answer: str, required_facts: list[str]) -> float:
    """Fraction of required facts the answer MENTIONS (by distinctive term).
    Weak signal — a paraphrase that doesn't share any distinctive word is
    counted as missing. Use for cheap smoke tests only; back it up with
    LLM-as-judge for anything semantically subtle."""
    answer_lower = answer.lower()
    matches = sum(1 for fact in required_facts if _fact_present(fact, answer_lower))
    return matches / len(required_facts) if required_facts else 0.0

def _fact_present(fact: str, answer_lower: str) -> bool:
    # Pick the 3 longest words from the fact (shorter words tend to be stop words
    # or common verbs; longer ones are distinctive — library names, proper nouns,
    # domain terms). Require ANY of them in the answer, not ALL — otherwise a
    # correct paraphrase that uses a different verb fails the check.
    words = {w.strip(".,():[]!?") for w in fact.lower().split()}
    key_terms = sorted((w for w in words if len(w) > 4), key=len, reverse=True)[:3]
    if not key_terms:
        return fact.lower() in answer_lower
    return any(term in answer_lower for term in key_terms)

Treat fact_coverage > 0.9 as “answer at least mentioned the right topics” — not “answer is complete”. For complete correctness, use LLM-as-judge with the ideal answer in the prompt as reference.

Open-source tooling at a glance

If you’d rather not roll your own:

  • Ragas — Python library specifically for RAG evaluation. Implements faithfulness, answer relevance, context precision, context recall. Uses OpenAI or Anthropic models as judges under the hood. Best for teams who want a batteries-included framework.
  • DeepEval — pytest-style assertion library for LLM outputs. Lets you write assert_faithful(answer, context) and run as part of your normal test suite. Good for CI-integrated regression testing.
  • Promptfoo — more generic prompt-eval tool, but has RAG recipes. Strong at side-by-side comparing two pipelines on the same test set.

All three use LLM-as-judge for the semantic metrics; the choice is between their abstractions and the ~100 lines of Python above. For a solo project or MVP, the DIY version is less to debug.

What to log in production

The eval setup above answers “does my pipeline work on known questions”. Production has an inverse problem: a user says “the answer is wrong” and you need to figure out why, six weeks after the fact.

Log, per production query:

  • The query text (user-provided)
  • The query embedding (optional — storage cost, but lets you redo retrieval later)
  • The chunk IDs returned, with their similarity scores
  • The retrieved context text actually passed to the LLM (not just IDs — chunk content changes if you re-index)
  • The LLM model name and version
  • The full LLM response text
  • Latency per stage (embed, retrieve, generate)

With that, “the answer was wrong” becomes a 2-minute debug: look up the query, see which of the four failure modes above matches. Without it, you’re guessing from the prompt alone.

A production RAG log schema can be a single JSON row per query. In SQLite or any append-only object store it costs almost nothing until you have real traffic.

Common mistakes

  • Judging the output and not the retrieval. A 90%-faithful final answer with 40% Hit@5 means retrieval is lucky, not good. The next doc update will break it.
  • Test set too small. Fewer than 20 queries can’t distinguish a real improvement from noise. Build the test set in parallel with the pipeline — don’t wait until the end.
  • Changing chunking, embedding model, retriever, AND prompt at once. Then noting the overall score went up or down. Change ONE variable per evaluation run, otherwise the metric movement doesn’t tell you what did the work.
  • LLM-as-judge with the same model as the generator. Self-judgement is reliably optimistic. Use a different model, or at least a different model size.
  • Shipping without a logged production dataset. The next version’s test set should come partly from real production queries that failed, not just the ones you made up at the start.
  • Treating retrieval metrics as the final goal. Hit@5 of 1.0 with awful generation prompts gives bad answers from perfectly retrieved context. Both layers matter.

Bottom line

Build a 30-query test set with golden chunk IDs. Score Hit@K + MRR for retrieval, LLM-as-judge faithfulness for generation. Log full context + answer in production. That’s a defensible evaluation setup in under 200 lines of Python, which is exactly what a production RAG system should ship with.

Which chunking strategy, which embedding model, which rerank step, which prompt template — all of those decisions become A/B tests against the test set. Instead of arguing about which is better, you run the numbers.

What’s next in this series

  • When to skip RAG entirely — the problems a longer context window, fine-tuning, or a direct database query solves better than any RAG pipeline. The final article in the cluster.