RAG Evaluation Metrics: How to Measure Retrieval and Generation Quality

Read 50 answers from your RAG app, and they look fine. Then a user asks about the one product line your chunker mangled, and the bot confidently quotes a limit from the wrong section of the policy. Spot checks miss that kind of failure because RAG systems can break in two places: the retriever can hand over the wrong context, or the generator can misuse the right one.

RAG evaluation metrics score those two halves separately, so you know which one to fix. This guide covers retrieval metrics (precision@k, recall@k, MRR, NDCG, context precision, and context recall) and generation metrics (faithfulness, answer relevancy, and correctness) with formulas and worked examples. Then it shows how to use them to choose between RAG pipeline configurations.

What Is RAG Evaluation?

RAG evaluation measures how well a retrieval-augmented generation system finds the right context and how well it uses that context to answer. You score the retriever and generator separately (component-level evaluation), and the whole system end to end, against a fixed set of test cases with known-good sources or answers.

The split helps you diagnose a wrong answer. Suppose a support bot gives the wrong lost-baggage limit. If the policy section never reached the prompt, investigate retrieval: chunking, embeddings, or how many results you keep. If the right section was present but the model confused it with another plan, investigate how the generator used that context.

An end-to-end score tells you the answer was wrong. Component scores help you decide where to look next.

RAG combines retrieval with generation, and the knowledge it retrieves can change. Evaluation has to account for both: retrieval metrics need to know which chunks are relevant to each query, while generation metrics need a reference answer or at least the retrieved context to check against. Our guide to RAG pipeline stages covers the components these metrics attach to.

Why RAG Needs Its Own Metrics

General LLM evaluation metrics grade the text a model produces. RAG adds a retriever in front of the model, and these four failure modes show why retrieval and generation need separate checks:

1. The answer was never retrieved. The relevant chunk isn't in the top k, so the model guesses or refuses. Recall@k and context recall catch this.

2. It was retrieved but buried. The right chunk came back at rank 8, and the generator leaned on the seven near misses above it. Rank-aware metrics like MRR and NDCG catch this.

3. It was retrieved and misread. The context was right, but the answer contradicts it or adds details it doesn't contain. Faithfulness catches this.

4. The answer is correct but unsupported. The model answered from training data instead of your documents, which looks fine until your policy changes. Faithfulness catches this one too.

Text-quality metrics such as BLEU, ROUGE, perplexity, and toxicity can't see retrieval at all; they're covered in our guide to LLM evaluation metrics.

Retrieval Metrics

Retrieval metrics compare the ranked list your retriever returns with the relevant documents (usually chunks) that actually answer the user query. The classic ones come from information retrieval research and need labels, while newer ones let an LLM decide relevance. Every example below uses one query where the retriever returns five chunks, and the two relevant ones sit at ranks 2 and 5.

Rank-Unaware Metrics: Precision@k, Recall@k, Hit Rate, F1

Precision is the fraction of retrieved items that are relevant, and recall is the fraction of relevant items that are retrieved. Precision@k fixes the cutoff at the top k results, which matches RAG, where only a fixed number of chunks ever reach the prompt. Manning, Raghavan, and Schütze's Introduction to Information Retrieval is the standard reference for all these metrics.

In our example, two of the five retrieved chunks are relevant. That gives precision@5 = 2/5 = 0.4. Those are also the only two relevant chunks in the dataset, so recall@5 = 2/2 = 1.0: retrieval found everything it needed.

Keeping only the top three results removes the relevant chunk at rank 5. Recall@3 falls to 0.5, but hit rate@3 stays at 1 because at least one relevant chunk remains. Hit rate answers the simpler question: did retrieval find anything useful?

F1 combines precision and recall into one score using their harmonic mean:

F1 = 2 × precision × recall / (precision + recall)

F1 = 2 × precision × recall / (precision + recall)

F1 = 2 × precision × recall / (precision + recall)

With precision@5 of 0.4 and recall@5 of 1.0, F1@5 is about 0.57. Keep the separate precision and recall values too, so you can see which side needs work.

For a fixed ranked list, raising k never lowers recall, but precision can rise or fall: in this example, precision@3 is 0.33, and precision@5 is 0.4. Rerankers help manage the tradeoff: fetch a wide k for recall, then keep a short top_n for precision.

Rank-Aware Metrics: MRR, NDCG, MAP

Position matters when useful context competes with irrelevant chunks. Mean reciprocal rank (MRR) focuses on the first relevant result: score each query as 1 divided by that result's rank, then average across queries. A query with no relevant result scores zero.

Our first relevant chunk is second, so its reciprocal rank is 1/2 = 0.5. Finding another relevant chunk at rank 5 does not improve that score. Use MRR when finding one useful result early is the priority. The TREC-8 question answering track used this approach over its top five responses.

NDCG (normalized discounted cumulative gain) considers all the useful results and rewards placing them early. It can also distinguish a highly relevant chunk from a somewhat relevant one, rather than treating relevance as a yes-or-no label.

The calculation discounts each chunk's gain by log2(rank + 1), then compares the total with the score of the ideal ordering. With binary labels, our example's NDCG@5 is about 0.62. A perfect ordering scores 1.

This graded-relevance approach comes from Järvelin and Kekäläinen's 2002 work on cumulated gain. The BEIR benchmark uses nDCG@10 as its headline metric across 18 retrieval datasets, making it a useful reference point when comparing retrievers.

Mean average precision (MAP) is useful when several chunks matter. For each query, calculate precision at every rank where a relevant chunk appears, add those values, and divide by the total number of relevant chunks, including any that were missed. Then average the query scores.

In our example, the relevant chunks appear at ranks 2 and 5. Precision at those positions is 1/2 and 2/5, so average precision is (1/2 + 2/5) / 2 = 0.45.

Our FiQA tutorial notebook includes a metric function that computes precision, recall, F1, NDCG@5, and MRR, returning each one as a named value per batch. Here are recall@k and reciprocal rank as plain functions you can drop into any eval loop.

Run this with Python 3.9+; no third-party packages are needed.

The example prints 0.5 twice: one of two relevant chunks is in the top three, and the first relevant chunk is second. Average reciprocal rank across queries to get MRR.

This example assigns 0 when no relevant IDs are supplied; choose and document that convention for your dataset:

def recall_at_k(ranked_ids: list[str], relevant_ids: set[str], k: int) -> float:    """Share of the relevant chunks that show up in the top k."""    if k < 0:        raise ValueError("k must be non-negative")    if not relevant_ids:        return 0.0    return len(set(ranked_ids[:k]) & relevant_ids) / len(relevant_ids)def reciprocal_rank(ranked_ids: list[str], relevant_ids: set[str]) -> float:    """1 / rank of the first relevant chunk, or 0 if none was retrieved."""    for rank, chunk_id in enumerate(ranked_ids, start=1):        if chunk_id in relevant_ids:            return 1.0 / rank    return 0.0# One query: the two chunks that answer it are c7 and c2ranked = ["c4", "c7", "c9", "c1", "c2"]  # keep the list: order is the whole pointrelevant = {"c7", "c2"}print(recall_at_k(ranked, relevant, k=3))  # 0.5print(reciprocal_rank(ranked, relevant))  # 0.5

def recall_at_k(ranked_ids: list[str], relevant_ids: set[str], k: int) -> float:    """Share of the relevant chunks that show up in the top k."""    if k < 0:        raise ValueError("k must be non-negative")    if not relevant_ids:        return 0.0    return len(set(ranked_ids[:k]) & relevant_ids) / len(relevant_ids)def reciprocal_rank(ranked_ids: list[str], relevant_ids: set[str]) -> float:    """1 / rank of the first relevant chunk, or 0 if none was retrieved."""    for rank, chunk_id in enumerate(ranked_ids, start=1):        if chunk_id in relevant_ids:            return 1.0 / rank    return 0.0# One query: the two chunks that answer it are c7 and c2ranked = ["c4", "c7", "c9", "c1", "c2"]  # keep the list: order is the whole pointrelevant = {"c7", "c2"}print(recall_at_k(ranked, relevant, k=3))  # 0.5print(reciprocal_rank(ranked, relevant))  # 0.5

def recall_at_k(ranked_ids: list[str], relevant_ids: set[str], k: int) -> float:    """Share of the relevant chunks that show up in the top k."""    if k < 0:        raise ValueError("k must be non-negative")    if not relevant_ids:        return 0.0    return len(set(ranked_ids[:k]) & relevant_ids) / len(relevant_ids)def reciprocal_rank(ranked_ids: list[str], relevant_ids: set[str]) -> float:    """1 / rank of the first relevant chunk, or 0 if none was retrieved."""    for rank, chunk_id in enumerate(ranked_ids, start=1):        if chunk_id in relevant_ids:            return 1.0 / rank    return 0.0# One query: the two chunks that answer it are c7 and c2ranked = ["c4", "c7", "c9", "c1", "c2"]  # keep the list: order is the whole pointrelevant = {"c7", "c2"}print(recall_at_k(ranked, relevant, k=3))  # 0.5print(reciprocal_rank(ranked, relevant))  # 0.5

An easy bug to ship: converting retrieved IDs to a set before computing MRR or NDCG, which throws away the order those metrics depend on.

LLM-Judged Context Metrics: Context Precision, Context Recall, Context Relevance

Chunk labels are expensive and break every time you re-chunk, so LLM-judged context metrics swap them for a model's judgment.

Ragas' context precision rewards placing useful chunks early in the retrieved list. It has the same shape as average precision, except an LLM decides relevance and the denominator only counts relevant retrieved chunks, so a missed chunk doesn't lower it. In the documented example, placing an irrelevant chunk second leaves the score near 1.0; putting it first lowers the score to about 0.5.

Context recall covers what context precision ignores: the share of the reference answer's claims that the retrieved context supports. It needs a reference answer but no chunk labels.

Context relevance asks how much of the retrieved material helps answer the question. The Ragas paper defined it as the fraction of context sentences an LLM judges crucial to the answer. TruLens checks each chunk against the query.

Treat this as a judge's assessment, not a definitive label. In the Ragas paper's experiments, context relevance agreed with human annotators in 0.70 of pairwise comparisons, compared with 0.78 for answer relevance and 0.95 for faithfulness. Selecting the crucial sentences was particularly difficult for ChatGPT in longer contexts.

For graph RAG, the retrieval unit changes but the metrics carry over, though for corpus-wide questions the GraphRAG paper used an LLM evaluator in head-to-head comparisons on comprehensiveness, diversity, and empowerment.

Generation Metrics

Generation metrics take the retrieved context as given and ask what the model did with it.

Grounding Metrics: Faithfulness and Context Adherence

Faithfulness splits the answer into claims, checks each against the retrieved context, and divides supported claims by total claims. An answer with four claims, three backed by the context, scores 0.75. TruLens uses groundedness for this check in its RAG triad; context adherence is another name you'll see. Amazon Bedrock also evaluates whether the answer stays supported by retrieved text rather than introducing hallucinations.

Faithfulness is not correctness. If retrieval returned last year's policy, a faithful answer is faithfully wrong, which is why grounding metrics sit next to retrieval metrics rather than replacing them. Track the share of answers with at least one unsupported claim too, since that's closer to how a user experiences a hallucination than an average is.

Answer Quality Metrics: Answer Relevancy, Answer Correctness, Completeness

Answer relevancy in Ragas asks whether the response addresses the user's question. It works backward: an LLM generates questions that the answer could respond to, three by default, then compares them with the original question.

The score averages their cosine similarity, a measure of how closely the embedded questions align. This measures intent match, so a relevant answer can still be factually wrong. Cosine similarity can also be negative; the score is not guaranteed to stay between 0 and 1.

Ragas' answer correctness metric compares the response with a reference answer. It separates matching claims (true positives), claims in the response not supported by the reference (false positives), and missing reference claims (false negatives).

Suppose the reference says "born in 1879 in Germany" and the response says "born in Spain in 1879." The year matches, the country is wrong, and the correct country is missing. That gives one true positive, one false positive, and one false negative.

The factual score uses F1:

F1 = TP / (TP + 0.5 × (FP + FN))

F1 = TP / (TP + 0.5 × (FP + FN))

F1 = TP / (TP + 0.5 × (FP + FN))

Here, the result is 0.5. Ragas then blends that factual score with semantic similarity using configurable weights.

Completeness is the recall side of correctness: did the answer cover every part of the question? Bedrock exposes it as its own metric, and Ragas' factual correctness approximates it in recall mode. For part numbers or dollar limits, skip the judge and use exact match.

Reference-Based vs. Reference-Free Scoring

Reference-free metrics check grounding and relevance using the question, retrieved context, and answer. They do not establish that the underlying evidence is correct. Reference-based metrics such as context recall and answer correctness add an expected answer, so the evaluation can check coverage and factual agreement against that reference. Use both, with expert-reviewed references for the facts your application must get right.

How RAG Metrics Are Computed: Labels, Heuristics, and LLM Judges

In RAG evaluation, the same metric name can hide very different computations. "Context precision" might be an exact ID match, an embedding similarity, or an LLM call, and those three won't agree.

Labels and Golden Datasets

Labeled relevant chunks or pages plus reference answers make up your golden dataset. Labels can come from domain experts, from support tickets with known resolutions, or from synthetic data: generate questions from a document and record which page each came from. Synthetic labels are cheap and consistent, but they measure agreement with an answer key a model wrote, so trust relative comparisons more than absolute scores.

Heuristics and Code Checks

Exact match, regex checks for citation format, and ID-based context precision are deterministic, fast, and inexpensive to rerun. Similarity over cached embeddings is also inexpensive to recompute; generating those vectors costs API tokens or local compute. Pin the embedding model and cache the vectors for repeatability. The limit of all these checks is meaning: a word-overlap score used as a groundedness proxy measures shared tokens, so an answer that reuses the context's words while flipping a number can score well.

LLM Judges

In Ragas, and in DeepEval's default mode, faithfulness, relevancy, and most context metrics are LLM calls under the hood: a first step extracts claims or generates questions, and a second step scores them. ARES takes a different route, training smaller language models as judges on synthetic data and correcting their scores with a small set of human-annotated examples through prediction-powered inference. Judges handle meaning that heuristics can't, but they bring their own biases, cost tokens on every call, and change when you change the judge model or prompt. Our guide to LLM as a judge covers how to validate one.

Freeze the judge while you compare RAG pipelines. Two judge implementations can produce scores that aren't directly comparable even on the same model, as the travel-insurance case study in the configuration-comparison section below shows.

Human Evaluation

Human evaluation sets the standard everything else is checked against. People don't grade every run. They label a sample so you can check that your judges agree with them before automated evaluation runs at scale. Once the judges track those labels, you can evaluate multiple metrics side by side and use them to decide which configuration to promote.

Operational Metrics: Latency, Cost, and Token Spend

Quality scores alone don't show what a configuration costs to run. Track these alongside them on every RAG evaluation run:

• Latency per query, at p50 and p95.

• Cost per query: query embedding, input tokens for the retrieved context, output tokens, and reranker compute. k and top_n set most of the input tokens.

• Token spend for the evaluation itself. With one separate judge call per metric per answer, 200 test cases × 20 configs × 3 judged metrics means 12,000 calls; claim-based judges often use multiple calls. A single rubric prompt scoring all three would use 4,000 calls, before retries or caching.

A config that buys a sliver of faithfulness by stuffing ten chunks into the prompt can lose to a three-chunk config once you price it. The best retrieval config in that travel-insurance study kept only two. And offline evaluation isn't production monitoring: you need the first to choose a config before you ship and the second to catch drift on live traffic.

How to Use RAG Metrics to Improve Your Pipeline

Here's a workflow for choosing between dozens or hundreds of RAG pipeline configurations, with numbers from two published sweeps.

Build a Representative Eval Set (and Keep It Fixed)

Start with real user queries, label sources and reference answers, and tag each case with a slice such as product line or domain. Then freeze it, or you won't know whether the pipeline improved or the test got easier.

Small eval sets help find failures, but their scores can shift substantially when a few answers change. For an illustrative 70% pass rate on 28 independent questions, a simple normal approximation gives a 95% margin of about 17 percentage points. That uncertainty is too large to treat a small score gap as decisive.

The calculation is standard error = √(0.7 × 0.3 / 28) ≈ 0.087, with an approximate 95% margin of 1.96 × 0.087 ≈ 0.17. Evan Miller's "Adding Error Bars to Evals" discusses reporting uncertainty, accounting for related groups of questions, and comparing systems using their question-level differences.

Tune on the development set, then confirm the selected configuration on held-out questions. Keep those final checks separate from the examples used to choose it.

Choose 3-5 Metrics That Match Your Failure Modes

Pick one metric per failure mode you actually see, not everything your framework offers. The FAQ below has a default set of five; for short factual answers, swap correctness for exact match and save the judge calls.

Expect metrics to disagree about which knob matters. In the 432-config automotive sweep discussed below, swapping only the embedding model around the winning config moved faithfulness by about 0.02 but swung answer correctness by more than 0.2, so watching faithfulness alone would have hidden the embedder's effect.

Compare Configurations Side by Side, Not One Change at a Time

RAG knobs interact: chunk size changes what the embedder sees, the embedder changes what retrieval returns, and k changes what the reranker gets. Tuning one knob at a time can miss the combination that wins, so compare full configurations on the same eval set. Start with data preparation and retrieval (chunking strategy, embedding model, search type, k, and reranking), then models and prompts.

Cross-encoder reranking shows up in both winning configs of one of our published case studies. It covers two sweeps, including one that scored 432 RAG configurations on seven Ragas metrics. The other was a travel-insurance chatbot from a global leader in enterprise data and analytics:

• We pinned the generator and embedder and swept 40 retrieval configurations against 76 question-and-answer pairs synthesized from the policy.

• The winner used 512-token chunks, retrieved the top 10, and reranked down to 2. It lifted the retrieval composite from 0.439 to 0.627, about 43 percent on that benchmark, and improved every retrieval metric.

• Removing the reranker traded ranking precision for more grounding context, and the composite dropped.

This is a controlled demonstration with a synthesized answer key. The answer-quality gain is reported qualitatively because the baseline used a multi-call RAGAS judge and the sweep used a single-call rubric judge, both on gpt-4o-mini; their absolute scores are not directly comparable.

We built RapidFire, our open-source experiment engine, for these retrieval and context engineering comparisons. You supply the metric functions, including any judge calls, and run many RAG configurations on the same eval set.

For RAG evaluations, it shows running metric estimates with confidence intervals as data shards finish, provided each metric has a supported aggregation type and value range. You can stop weak configurations, clone promising ones, and change their search, reranker, prompt, or generator settings. With API models, this lets you direct more of the token budget toward candidates worth exploring.

Shards follow dataset order. Shuffle a set sorted by domain or difficulty before running it, so early results are less likely to overrepresent one part of the workload.

Decide With a Weighted Score, Then Check Worst-Case Slices

Often, no configuration is best on every metric. The Pareto frontier is the set of options for which improving one score requires giving something else up: no other tested configuration is at least as good on every score and better on one. Choosing among them requires priorities.

Our 432-config automotive sweep from the case study linked in the previous section illustrates this. A global automotive manufacturer's factory-floor assistant covered welding, assembly, electronics, and powertrain. Its team spent a couple of months benchmarking existing stacks and curating a 28-question eval set. After a handful of days learning the data, we ran the grid in two days, scoring each configuration on seven Ragas metrics and three LLM-judge metrics.

The composite weighted answer correctness was 50 percent. The winner ranked first of all 432 runs on answer correctness and only about 282nd on faithfulness. Around the winning config, swapping just the model to gemini-2.5-flash, or the search to MMR, pushed faithfulness past 0.90. Re-weighting toward faithfulness and a different config wins, and because every config was scored on every metric, you can re-pick without re-running anything.

Check each domain as well as the overall score. The selected configuration kept faithfulness at or above 0.70 in all four domains, ranging from 0.72 on powertrain to 0.83 on welding. One incumbent stack scored 0.95 on electronics but 0.00 on faithfulness, context recall, and context precision for powertrain. Its strong result in one area concealed a serious failure in another.

The selected configuration beat the stronger of the two incumbents, metric by metric, on five of seven shared Ragas metrics, but it still trailed on answer relevancy, 0.45 to 0.59. No system cleared every internal pass target. These results describe the measured eval set and chosen weighting, not a guarantee for production.

When generation is stochastic, repeat the finalists before relying on a small difference. Review the individual failures too; a better average is not enough if an important use case still fails.

Conclusion

Evaluate RAG systems in two layers: retrieval metrics tell you whether the right context arrived and how high it ranked, and generation metrics tell you whether the model used it faithfully. Put latency and cost next to both, keep the eval set fixed, and pick a config with a weighting you can defend, after checking its worst slice. For agents that retrieve across several steps, see our guide to AI agent evaluation.

To try this on real data, start with a RAG tutorial with retrieval metrics built in. Our FiQA example compares 8 configs on a BEIR dataset, and its metric function already computes precision, recall, F1, NDCG@5, and MRR. Open the RAG notebook on Colab to run a simplified 4-config version with no local setup, or pip install rapidfireai to point it at your own corpus.

If you're planning this for a larger retrieval stack, talk to our team.

FAQ

What Are the Most Important RAG Evaluation Metrics?

Start with five: recall@k or context recall to confirm the answer is retrievable, NDCG or MRR for ranking, faithfulness for grounding, answer correctness against reference answers, and latency or cost per query. Add others only when you see a failure those five miss.

How Do You Evaluate a RAG System Without Ground Truth Labels?

Use reference-free metrics: faithfulness, answer relevancy, and context relevance or reference-free context precision. They run on an LLM judge, so validate the judge against a small human-labeled sample first. Then write a few dozen reference answers anyway, because reference-free scores can't tell you an answer is correct.

What Is the Difference Between Faithfulness and Answer Relevancy?

Faithfulness checks the answer against the retrieved context: what share of its claims the context supports. Answer relevancy checks the answer against the question, ignoring factual accuracy. An answer can be faithful but irrelevant (accurately quoting the wrong section) or relevant but unfaithful (on topic, with an invented number).

What Is Ragas, and Do I Need It?

Ragas is an open-source, Apache-2.0 RAG evaluation framework for Python that implements LLM-judged RAG metrics such as faithfulness, answer relevancy, context precision, and context recall. You can write the same checks yourself, but it gives you consistent definitions fast. Pin its version and the judge model so scores stay comparable.