RAG evaluation

For evaluating a RAG (Retrieval-Augmented Generation) system, my default recommendation is Ragas if you want a dedicated evaluation framework, or DeepEval if you want an LLM-testing workflow that fits naturally into unit tests and CI/CD.

The important distinction is that RAG quality isn't just about whether the final answer sounds correct. You need to evaluate whether the system retrieves the right evidence, uses that evidence faithfully, and answers the user's question correctly.

The best RAG evaluation frameworks

1. Ragas

(Best default for RAG evaluation)

Purpose-built for evaluating retrieval quality, answer faithfulness, and relevance. It also supports custom metrics and systematic experiments.docs.ragas.io+1

Best for: Comparing chunking strategies, embedding models, retrievers, rerankers, and generation prompts.

Docs: Ragas documentation ·GitHub

2. DeepEval

(Best for automated regression tests)

Provides RAG metrics, custom LLM-as-a-judge evaluations, and pytest-style assertions to catch quality regressions in CI/CD.deepeval.com+1

Best for: Teams that want every code change to pass defined quality thresholds before deployment.

Docs: RAG evaluation guide ·GitHub

3. Arize Phoenix

Combines evaluations with tracing, datasets, experiments, and debugging tools to investigate failures in real application runs.arize.com+1

Best for: Understanding why a RAG answer failed and monitoring quality in production.

Docs: Phoenix documentation ·GitHub

4. LangSmith

Worth considering if your application already uses LangChain or LangGraph and you want evaluation alongside tracing and development workflows.

Best for: Teams already invested in the LangChain ecosystem.

Docs:LangSmith documentation

What should you measure?

Whichever framework you choose, I'd start with these five metrics. They cover the two main failure points in RAG: retrieval and generation.

Ragas and DeepEval both support core metrics in these areas, although their definitions and scoring implementations differ.deepeval.com+2

A critical distinction: a faithful answer can still be wrong if the retrieved documents themselves are wrong. And an answer can be correct by chance despite poor retrieval. That's why you should measure retrieval and generation separately.

My recommended evaluation setup

1. Build a representative evaluation dataset

Start with 100–300 real user questions, including difficult queries, ambiguous questions, questions with no answer in your knowledge base, and multi-document questions. Record trusted answers and, where possible, the relevant source documents.

2. Run a baseline with Ragas

Measure faithfulness, answer relevancy, context precision, context recall, and answer correctness. Save individual results, not just averages.

3. Diagnose failures

Categorize them as retrieval misses, irrelevant chunks, unsupported claims, incomplete answers, or incorrect answers. Tune the component responsible for each failure.

4. Add regression testing

Use DeepEval if you want automated pass/fail checks in CI. Establish thresholds from your own human-reviewed results rather than relying on arbitrary default scores.

One important caveat: LLM-as-a-judge scores are estimates, not ground truth. Validate them against human judgments on a sample of your own questions, especially for high-impact use cases.

Final recommendation

• Choose Ragas if your primary goal is to measure and improve the quality of RAG answers.
• Choose DeepEval if your primary goal is to test answer quality continuously and prevent regressions.
• Choose Phoenix if your biggest challenge is tracing and diagnosing failures in a running system.
• Combine Ragas and DeepEval if you want focused RAG experiments plus automated regression tests; you don't necessarily need both at the start.

One question to narrow this down

What does your RAG system look like today?

1. Which stack are you using?

LangChain / LangGraph

LlamaIndex

Custom Python or API-based pipeline

Other / not sure

2. What's your main evaluation goal?

Answer correctness and hallucinations

Retrieval quality and chunking

Automated tests before deployment

All of the above

3. Do you have ground-truth answers?

Yes, labeled questions and correct answers

Some labeled data, mostly production queries

No labeled dataset yet

[Get my tailored recommendation]

Evaluating retrieval quality in a RAG system

To measure how effective your Retrieval-Augmented Generation (RAG) system is, I recommend evaluating the retrieval pipeline separately from the LLM's answer generation. The key question is: Does the retriever find the right evidence, rank it appropriately, and provide enough relevant context for the LLM to answer correctly?

There are several solutions available, ranging from open-source evaluation libraries to managed observability platforms and custom offline benchmarks.

1. Retrieval evaluation metrics

Before choosing a tool, establish which aspects of retrieval quality you need to measure.

Recall@K

Measures how many of the known relevant documents appear in the top K retrieved results.

Best for: detecting missed evidence and incomplete retrieval.

\text{Recall@K}=\frac{\text{Relevant retrieved}}{\text{All relevant}}

Precision@K

Measures the proportion of the top K results that are relevant.

Best for: reducing irrelevant context, token waste, and distraction.

\text{Precision@K}=\frac{\text{Relevant retrieved}}{K}

MRR (Mean Reciprocal Rank)

Scores how highly the first relevant result appears. A relevant document at rank 1 scores 1.0; at rank 5 it scores 0.2.

Best for: question-answering systems that need one key supporting passage.

NDCG@K (Normalized Discounted Cumulative Gain)

Rewards relevant results near the top of the ranking, with support for graded relevance.

Best for: comparing ranking algorithms, rerankers, and hybrid search configurations.

Hit Rate@K

Measures the fraction of queries for which at least one relevant document is retrieved in the top K.

Best for: a simple, interpretable baseline for retrieval success.

Important: These metrics require relevance judgments, such as labeled query-document pairs. If your system has no labels yet, you can create a small representative test set and annotate relevant passages manually.

2. Solutions available for evaluating RAG retrieval

Here are the main tools I'd shortlist, depending on whether you want a lightweight evaluation library, an experiment platform, or production monitoring.

1. Ragas

(Best starting point)

An evaluation framework focused on RAG quality, with metrics for context precision, context recall, noise sensitivity, and related dimensions. Supports reference-based and LLM-judged evaluation approaches.docs.ragas.io+2

• Best for: offline benchmarking and comparing retrieval configurations.
• Trade-off: LLM-based metrics can introduce evaluation cost and judge variability.

Documentation

2. DeepEval

A testing-oriented framework with three particularly useful retriever metrics: contextual relevancy, contextual precision, and contextual recall. It supports bulk test runs, configurable thresholds, and multi-turn RAG evaluation.deepeval.com+1

• Best for: regression tests in CI/CD and developer workflows.
• Trade-off: some metrics need reference answers, and LLM-as-judge scoring needs calibration.

RAG evaluation guide

3. Arize Phoenix

An open-source observability and evaluation platform for tracing retrieval and generation, running experiments, and investigating failures. It is useful for understanding why a particular query received poor context.arize.com+1

• Best for: debugging retrieval pipelines and visualizing evaluation results.
• Trade-off: requires instrumentation and some setup.

Phoenix documentation

4. LangSmith

Provides tracing of retrieved documents, evaluation datasets, separate retrieval and generation scoring, and production monitoring for RAG applications.info.langchain.com

• Best for: teams using LangChain or wanting an integrated evaluation and monitoring workflow.
• Trade-off: a managed platform may be more than you need if you only want offline retrieval metrics.

RAG evaluation

5. Custom information-retrieval benchmark

Build a labeled query-document dataset and calculate standard metrics directly, using your existing search stack and a Python evaluation script.

• Best for: teams that need reproducible, vendor-independent measurements.
• Trade-off: you must build and maintain relevance labels, reporting, and regression checks.

This is also the best way to validate whether an LLM-based evaluator agrees with human judgments.

3. Which approach should you choose?

These options are complementary rather than mutually exclusive. For example, you can calculate deterministic ranking metrics in a custom benchmark and use Ragas or DeepEval to evaluate semantic relevance.

4. A practical evaluation plan

I'd implement the following process for your existing RAG system.

1. Create a representative test dataset.

   Collect approximately 100–500 real user queries to start, covering common questions, difficult queries, different document types, and questions with no answer in the corpus. Record the relevant document or chunk IDs for each query where possible.

2. Run the retriever without the generator.

   Record the query, ranked document IDs, similarity scores, rank positions, and retrieval latency. This isolates retrieval performance from answer generation.

3. Calculate a core metric set.

   Start with Recall@5, Recall@10, Precision@5, MRR, and NDCG@10. Add semantic contextual precision and recall when relevance cannot be reliably captured by document IDs alone.

4. Compare retrieval configurations.

   Evaluate the current baseline against alternatives for embedding models, chunk sizes, top-K, hybrid search, and reranking. Change one variable at a time where practical.

5. Validate and monitor.

   Review a sample of incorrect or borderline evaluations manually. Run the benchmark whenever the corpus, chunking, embeddings, or retrieval logic changes, and track latency alongside quality.

5. Don't overlook these measurement pitfalls

• Relevant document versus relevant chunk: the correct document can be retrieved while the specific passage needed to answer the query is missed.
• Incomplete ground truth: if your labeled set misses relevant documents, a good retriever can appear to have poor precision or recall.
• Metric trade-offs: increasing top-K may improve recall while reducing precision and increasing prompt costs.
• LLM evaluator bias: judge-based scores are estimates, not ground truth. Calibrate them against human-reviewed examples.
• Unanswerable queries: measure whether retrieval appropriately handles missing evidence rather than forcing every query to return apparently useful context.

My recommendation

For most teams starting to measure retrieval quality, I'd choose Ragas plus a small, labeled offline benchmark. Use Recall@K to establish whether relevant evidence is being found, NDCG@K or MRR to assess ranking, and contextual precision to measure how much irrelevant material reaches the LLM. Add DeepEval if you want a test-suite-oriented workflow, or Phoenix if diagnosing production failures is a priority.

One question that would help narrow this down: what does your current RAG stack look like?

1. Retrieval stack

LangChain or LlamaIndex

Managed vector database (Pinecone, Weaviate, etc.)

Custom Python pipeline

Other / not sure

2. Primary goal

Offline benchmarking and model comparisons

Automated regression tests in CI/CD

Production monitoring and troubleshooting

3. Existing ground-truth labels

No labeled queries yet

Some labeled queries or expected answers

Labeled relevant document or chunk IDs

[Recommend an evaluation setup]

For a Retrieval-Augmented Generation (RAG) system, the specialists you want are Information Retrieval (IR) engineers and RAG evaluation specialists. They focus on whether your retriever returns the right documents or chunks, ranks them correctly, and provides sufficient context for the LLM to generate accurate answers.

There are two main areas to evaluate separately: retrieval quality and answer-generation quality. Arize Phoenix's evaluation guidance recommends debugging retrieval first, before diagnosing generation issues.github.com+2

Who and what to look for

1. Ragas — RAG evaluation specialist

Best starting point for measuring context precision and context recall, alongside other RAG quality metrics. Supports automated evaluation and synthetic test-set generation.

Best for: evaluating RAG pipelines systematically.

Official documentation github.comarxiv.orgdocs.ragas.io

2. Arize Phoenix — Retrieval diagnostics

Useful for inspecting retrieved documents, evaluating relevance, and tracing failures through your RAG pipeline.

Best for: diagnosing retrieval problems in development and production.

Official documentation github.comwebcite.co

3. DeepEval — Automated testing

Provides contextual precision and recall evaluations, making it useful for regression tests that catch retrieval-quality drops as your system changes.

Best for: automated evaluation in CI/CD pipelines.

Official website scadea.combalazscsorba.com

The retrieval metrics your team should know

These are established information-retrieval metrics, with context-based metrics extending evaluation to RAG pipelines.github.com+1

My recommendation: Start with Ragas for evaluation metrics and a curated query-document test set. Add Phoenix if you need to diagnose retrieval failures in live traces. Have an IR engineer or ML engineer experienced in search relevance validate the metrics and test data.

One important distinction: if you're looking to hire a person or consultant, search for Information Retrieval Engineer, Search Relevance Engineer, or RAG Evaluation Engineer rather than only LLM Engineer.

If you want the easiest practical system, I’d use MLflow + a small versioned eval dataset, with Ragas metrics if you want specialized RAG scoring.

The setup

Git
 └── RAG code / prompts / chunking config
        ↓
MLflow
 ├── experiment/run ID
 ├── model + prompt + retriever parameters
 ├── traces
 ├── latency / token metrics
 └── eval scores
        ↓
Versioned eval set
 ├── question
 ├── expected answer
 └── optional expected context

MLflow now has first-class GenAI tracing, evaluation datasets, scorers, and experiment comparison. Its evaluation datasets can be associated with experiments and reused to compare different prompts, models, and application versions.MLflow AI Platform+1

For RAG specifically, you can add Ragas for metrics such as faithfulness and context precision; MLflow has an integration that lets those metrics participate in MLflow evaluation runs.MLflow AI Platform+1

What I'd actually version

Don't try to version every retrieved document independently. Give each experiment a configuration like:

experiment: rag-v17

embedding_model: text-embedding-3-large
llm: gpt-5
chunk_size: 800
chunk_overlap: 100
top_k: 5
reranker: cross-encoder-v2

prompt: prompts/qa_v4.txt
corpus: docs-2026-09-01
eval_set: rag-eval-v3

Then Git versions the configuration/code, while MLflow records the resulting run and scores.

Your eval set should contain perhaps 100–500 representative questions initially, including known failures. Treat it like a regression test suite. MLflow explicitly supports building evaluation sets from production traces, which is useful once you have real traffic.MLflow AI Platform

Metrics I'd start with

Don't start with 15 metrics. Use:

• Answer correctness — does it answer the question?
• Faithfulness — is the answer supported by retrieved context?
• Context/retrieval relevance — did retrieval find useful material?
• Latency
• Cost/tokens

Then inspect individual failures rather than trusting a single aggregate number. MLflow's evaluation results can include per-row assessments and trace details.MLflow AI Platform+1

What about LangSmith?

LangSmith is probably the easiest hosted/GUI-first option. It gives you datasets, traces, offline/online evaluations, and side-by-side experiment comparison, and it isn't limited to LangChain/LangGraph.LangChain+1

So I'd frame the choice as:

One important simplification: don't build a bespoke experiment database. A Git commit + immutable eval-set version + MLflow run gives you enough provenance to reproduce most RAG experiments without creating another system.

If you're starting from scratch, I can also give you a minimal ~50-line Python setup that lets you run rag.py against an eval set and get a comparison table for retriever/chunking/prompt experiments.