RAG evaluation

There is no single "best" framework , because the right choice depends on where you are in your development cycle and how you want to run your tests.[machinelearningmastery.com](https://machinelearningmastery.com/llm-evaluation-frameworks-compared-how-to-actually-measure-what-your-model-does/)

Instead, the ecosystem has converged on a few standout frameworks, each tailored for a specific workflow. The top options and when to use them include:

• Ragas (Retrieval Augmented Generation Assessment)
  • Best for: Deep, component-level RAG metrics.
  • Why choose it: It treats retrieval and generation as separate pipelines. It evaluates the "RAG Triad" concepts—faithfulness, answer relevance, and context precision/recall—with or without ground truth references.[machinelearningmastery.com](https://www.lyzr.ai/blog/best-tools-for-rag-evaluation/)[[2](https://www.lyzr.ai/blog/best-tools-for-rag-evaluation)](https://www.prompts.ai/blog/best-llm-evaluation-tools-machine-learning-2026.html)
• DeepEval by Confident AI
  • Best for: CI/CD pipelines and unit testing.
  • Why choose it: It is pytest -native, meaning you can write RAG quality gates right alongside your code tests and run evaluations automatically on every GitHub commit. It features over 14 metrics covering bias, toxicity, and RAG correctness.[machinelearningmastery.com](https://medium.com/@future_agi/llm-evaluation-frameworks-metrics-and-best-practices-2026-edition-162790f831f4)[[2](https://www.lyzr.ai/blog/best-tools-for-rag-evaluation)](https://machinelearningmastery.com/llm-evaluation-frameworks-compared-how-to-actually-measure-what-your-model-does/)[[3]()](]()]()
• Arize Phoenix
  • Best for: Visualizing traces and embedding space evaluations.
  • Why choose it: Excellent for local or self-hosted tracing. It excels at diagnosing why a retrieval step failed by letting you inspect clusters, embeddings, and localized evaluation metrics visually.[machinelearningmastery.com](https://www.lyzr.ai/blog/best-tools-for-rag-evaluation/)
• LangSmith
  • Best for: End-to-end tracing, monitoring, and team collaboration.
  • Why choose it: If your stack is built on LangChain (or even if it isn't), LangSmith offers industry-standard logging, production monitoring, dataset management, and human-in-the-loop evaluation queues.[machinelearningmastery.com](https://www.prompts.ai/blog/best-llm-evaluation-tools-machine-learning-2026.html)
• Promptfoo
  • Best for: Fast prompt iteration and security red-teaming.
  • Why choose it: YAML-driven, incredibly lightweight, and great for comparing how different models or prompt templates handle a test suite before you write heavy application code.[machinelearningmastery.com](https://machinelearningmastery.com/llm-evaluation-frameworks-compared-how-to-actually-measure-what-your-model-does/)

A Practical Recommendation

• Early development/experimentation: Start with Ragas or Promptfoo to nail down your core metrics and prompt/retrieval setup.
• Engineering automation: Integrate DeepEval so regressions in RAG accuracy fail your builds automatically.
• Production/Observability: Use Arize Phoenix or LangSmith to keep an eye on live traffic. Mature teams often pair an offline test framework (like DeepEval or Ragas) with an online observability platform.[machinelearningmastery.com](https://www.lyzr.ai/blog/best-tools-for-rag-evaluation/)[[2](https://www.lyzr.ai/blog/best-tools-for-rag-evaluation)](https://machinelearningmastery.com/llm-evaluation-frameworks-compared-how-to-actually-measure-what-your-model-does/)[[3]()]()

To help narrow down the ideal choice, tell me:

• Are you looking to test offline (during development) or monitor live traffic (in production)?
• What orchestration framework or stack are you using (e.g., LangChain, LlamaIndex, custom Python)?

Evaluating retrieval quality in a RAG system

To measure how effective your Retrieval-Augmented Generation (RAG) system is, I recommend evaluating the retrieval pipeline separately from the LLM's answer generation. The key question is: Does the retriever find the right evidence, rank it appropriately, and provide enough relevant context for the LLM to answer correctly?

There are several solutions available, ranging from open-source evaluation libraries to managed observability platforms and custom offline benchmarks.

1. Retrieval evaluation metrics

Before choosing a tool, establish which aspects of retrieval quality you need to measure.

Recall@K

Measures how many of the known relevant documents appear in the top K retrieved results.

Best for: detecting missed evidence and incomplete retrieval.

\text{Recall@K}=\frac{\text{Relevant retrieved}}{\text{All relevant}}

Precision@K

Measures the proportion of the top K results that are relevant.

Best for: reducing irrelevant context, token waste, and distraction.

\text{Precision@K}=\frac{\text{Relevant retrieved}}{K}

MRR (Mean Reciprocal Rank)

Scores how highly the first relevant result appears. A relevant document at rank 1 scores 1.0; at rank 5 it scores 0.2.

Best for: question-answering systems that need one key supporting passage.

NDCG@K (Normalized Discounted Cumulative Gain)

Rewards relevant results near the top of the ranking, with support for graded relevance.

Best for: comparing ranking algorithms, rerankers, and hybrid search configurations.

Hit Rate@K

Measures the fraction of queries for which at least one relevant document is retrieved in the top K.

Best for: a simple, interpretable baseline for retrieval success.

Important: These metrics require relevance judgments, such as labeled query-document pairs. If your system has no labels yet, you can create a small representative test set and annotate relevant passages manually.

2. Solutions available for evaluating RAG retrieval

Here are the main tools I'd shortlist, depending on whether you want a lightweight evaluation library, an experiment platform, or production monitoring.

1. Ragas

(Best starting point)

An evaluation framework focused on RAG quality, with metrics for context precision, context recall, noise sensitivity, and related dimensions. Supports reference-based and LLM-judged evaluation approaches.docs.ragas.io+2

• Best for: offline benchmarking and comparing retrieval configurations.
• Trade-off: LLM-based metrics can introduce evaluation cost and judge variability.

Documentation

2. DeepEval

A testing-oriented framework with three particularly useful retriever metrics: contextual relevancy, contextual precision, and contextual recall. It supports bulk test runs, configurable thresholds, and multi-turn RAG evaluation.deepeval.com+1

• Best for: regression tests in CI/CD and developer workflows.
• Trade-off: some metrics need reference answers, and LLM-as-judge scoring needs calibration.

RAG evaluation guide

3. Arize Phoenix

An open-source observability and evaluation platform for tracing retrieval and generation, running experiments, and investigating failures. It is useful for understanding why a particular query received poor context.arize.com+1

• Best for: debugging retrieval pipelines and visualizing evaluation results.
• Trade-off: requires instrumentation and some setup.

Phoenix documentation

4. LangSmith

Provides tracing of retrieved documents, evaluation datasets, separate retrieval and generation scoring, and production monitoring for RAG applications.info.langchain.com

• Best for: teams using LangChain or wanting an integrated evaluation and monitoring workflow.
• Trade-off: a managed platform may be more than you need if you only want offline retrieval metrics.

RAG evaluation

5. Custom information-retrieval benchmark

Build a labeled query-document dataset and calculate standard metrics directly, using your existing search stack and a Python evaluation script.

• Best for: teams that need reproducible, vendor-independent measurements.
• Trade-off: you must build and maintain relevance labels, reporting, and regression checks.

This is also the best way to validate whether an LLM-based evaluator agrees with human judgments.

3. Which approach should you choose?

These options are complementary rather than mutually exclusive. For example, you can calculate deterministic ranking metrics in a custom benchmark and use Ragas or DeepEval to evaluate semantic relevance.

4. A practical evaluation plan

I'd implement the following process for your existing RAG system.

1. Create a representative test dataset.

   Collect approximately 100–500 real user queries to start, covering common questions, difficult queries, different document types, and questions with no answer in the corpus. Record the relevant document or chunk IDs for each query where possible.

2. Run the retriever without the generator.

   Record the query, ranked document IDs, similarity scores, rank positions, and retrieval latency. This isolates retrieval performance from answer generation.

3. Calculate a core metric set.

   Start with Recall@5, Recall@10, Precision@5, MRR, and NDCG@10. Add semantic contextual precision and recall when relevance cannot be reliably captured by document IDs alone.

4. Compare retrieval configurations.

   Evaluate the current baseline against alternatives for embedding models, chunk sizes, top-K, hybrid search, and reranking. Change one variable at a time where practical.

5. Validate and monitor.

   Review a sample of incorrect or borderline evaluations manually. Run the benchmark whenever the corpus, chunking, embeddings, or retrieval logic changes, and track latency alongside quality.

5. Don't overlook these measurement pitfalls

• Relevant document versus relevant chunk: the correct document can be retrieved while the specific passage needed to answer the query is missed.
• Incomplete ground truth: if your labeled set misses relevant documents, a good retriever can appear to have poor precision or recall.
• Metric trade-offs: increasing top-K may improve recall while reducing precision and increasing prompt costs.
• LLM evaluator bias: judge-based scores are estimates, not ground truth. Calibrate them against human-reviewed examples.
• Unanswerable queries: measure whether retrieval appropriately handles missing evidence rather than forcing every query to return apparently useful context.

My recommendation

For most teams starting to measure retrieval quality, I'd choose Ragas plus a small, labeled offline benchmark. Use Recall@K to establish whether relevant evidence is being found, NDCG@K or MRR to assess ranking, and contextual precision to measure how much irrelevant material reaches the LLM. Add DeepEval if you want a test-suite-oriented workflow, or Phoenix if diagnosing production failures is a priority.

One question that would help narrow this down: what does your current RAG stack look like?

1. Retrieval stack

LangChain or LlamaIndex

Managed vector database (Pinecone, Weaviate, etc.)

Custom Python pipeline

Other / not sure

2. Primary goal

Offline benchmarking and model comparisons

Automated regression tests in CI/CD

Production monitoring and troubleshooting

3. Existing ground-truth labels

No labeled queries yet

Some labeled queries or expected answers

Labeled relevant document or chunk IDs

[Recommend an evaluation setup]

If you want the easiest practical system, I’d use MLflow + a small versioned eval dataset, with Ragas metrics if you want specialized RAG scoring.

The setup

Git
 └── RAG code / prompts / chunking config
        ↓
MLflow
 ├── experiment/run ID
 ├── model + prompt + retriever parameters
 ├── traces
 ├── latency / token metrics
 └── eval scores
        ↓
Versioned eval set
 ├── question
 ├── expected answer
 └── optional expected context

MLflow now has first-class GenAI tracing, evaluation datasets, scorers, and experiment comparison. Its evaluation datasets can be associated with experiments and reused to compare different prompts, models, and application versions.MLflow AI Platform+1

For RAG specifically, you can add Ragas for metrics such as faithfulness and context precision; MLflow has an integration that lets those metrics participate in MLflow evaluation runs.MLflow AI Platform+1

What I'd actually version

Don't try to version every retrieved document independently. Give each experiment a configuration like:

experiment: rag-v17

embedding_model: text-embedding-3-large
llm: gpt-5
chunk_size: 800
chunk_overlap: 100
top_k: 5
reranker: cross-encoder-v2

prompt: prompts/qa_v4.txt
corpus: docs-2026-09-01
eval_set: rag-eval-v3

Then Git versions the configuration/code, while MLflow records the resulting run and scores.

Your eval set should contain perhaps 100–500 representative questions initially, including known failures. Treat it like a regression test suite. MLflow explicitly supports building evaluation sets from production traces, which is useful once you have real traffic.MLflow AI Platform

Metrics I'd start with

Don't start with 15 metrics. Use:

• Answer correctness — does it answer the question?
• Faithfulness — is the answer supported by retrieved context?
• Context/retrieval relevance — did retrieval find useful material?
• Latency
• Cost/tokens

Then inspect individual failures rather than trusting a single aggregate number. MLflow's evaluation results can include per-row assessments and trace details.MLflow AI Platform+1

What about LangSmith?

LangSmith is probably the easiest hosted/GUI-first option. It gives you datasets, traces, offline/online evaluations, and side-by-side experiment comparison, and it isn't limited to LangChain/LangGraph.LangChain+1

So I'd frame the choice as:

One important simplification: don't build a bespoke experiment database. A Git commit + immutable eval-set version + MLflow run gives you enough provenance to reproduce most RAG experiments without creating another system.

If you're starting from scratch, I can also give you a minimal ~50-line Python setup that lets you run rag.py against an eval set and get a comparison table for retriever/chunking/prompt experiments.