RAG Evaluation Basics

Most RAG pipelines ship with no evaluation beyond "it looks right in the demo." That works until a prompt change, a chunking change, or an embedding model swap silently makes answers worse. This note is a quick reference for the two layers worth measuring separately: retrieval and generation.

Why split retrieval from generation

A RAG system fails in one of two places:

1. Retrieval failure — the right chunk was never pulled into context, so the model is answering without the information it needed.
2. Generation failure — the right chunk was retrieved, but the model ignored it, misread it, or hallucinated past it.

If you only look at final-answer quality, you can't tell which one broke. Measuring both layers separately turns "the bot gave a bad answer" into "retrieval missed the chunk" or "retrieval was fine, generation ignored it" — two very different fixes.

Retrieval metrics

• Context Precision — of the chunks retrieved, how many were actually relevant to the question. Low precision means your retriever is pulling noise alongside the useful chunk, which dilutes the context window and gives the generator more to misread.
• Context Recall — of the chunks that should have been retrieved to answer the question fully, how many actually were. Low recall means the answer is missing information no matter how good the generator is.
• Mean Reciprocal Rank (MRR) — for queries with one clearly correct chunk, how early does it appear in the ranked results. Matters a lot when top_k is small, since a correct chunk ranked 8th is useless if only the top 3 are passed to the model.

Generation metrics

• Faithfulness — does the generated answer only make claims that are actually supported by the retrieved context. This is the metric that catches hallucination specifically within a RAG system — the model inventing something not present in the chunks it was given.
• Answer Relevance — does the answer actually address the question asked, independent of whether it's grounded. A faithful but off-topic answer still fails the user.

A minimal eval loop

1. Build a small set of representative question/expected-answer pairs (20-50 is enough to start catching regressions).
2. Run retrieval in isolation and score Context Precision/Recall against a labeled "should have retrieved this chunk" set.
3. Run the full pipeline and score Faithfulness/Answer Relevance against the generated output.
4. Re-run this set on every meaningful change — chunk size, embedding model, retriever algorithm, prompt template — before merging.

Common failure patterns worth specifically testing for

• Chunking cuts mid-answer — the fact needed to answer the question is split across two chunks and only one gets retrieved.
• Near-duplicate chunks crowding out diversity — top_k fills up with slightly different phrasings of the same paragraph instead of distinct relevant content.
• Stale context ranked above correct context — happens after a knowledge base update if old and new versions of a document coexist in the vector store.

None of this requires an elaborate framework to start — a spreadsheet of question/expected-chunk pairs and a short script scoring precision/recall is enough to catch most regressions before they reach production.