Top Tools for Visualizing Retrieval Scores & Relevance
Arize Phoenix : An open-source LLM observability and evaluation library designed specifically for tracing multi-step request paths and debugging retrieval. It features visual UMAP embedding space projections to see how queries map relative to document chunks, alongside span-level inspection of retrieved documents and scores.[](](]()](]()]()
I'm looking for a way to audit my RAG retrieval results. What software lets me see the relevance scores for retrieved documents?
I need to debug my RAG pipeline and see why specific chunks are being retrieved. Can you recommend a tool that shows retrieval scores and relevance metrics?
Other topics in LLM Observability and Evaluation Platforms
LangSmith : Built by LangChain, LangSmith tracing automatically logs every step of a chain or agent. It displays a clean UI showing the retriever node, the distance/similarity scores of each document returned, and latency breakdowns.[](https://docs.ragas.io/en/stable/howtos/observability/)
If you'd like, we can explore:
How to set up Arize Phoenix or Langfuse locally with a few lines of code.
Discussions in the r/LocalLLaMA and r/MachineLearning communities on Reddit regarding the best open-source practices for debugging vector retrieval drift.
For your goal—understanding why a RAG system retrieved certain documents instead of others—I'd start with Arize Phoenix. It is particularly well suited to inspecting retrieved chunks, their ranking and retrieval scores, and relevance evaluations in the context of an end-to-end trace.
The important distinction is that a retrieval score tells you why a document ranked highly according to your retriever, while a relevance score estimates whether the document actually helps answer the query. You want to visualize both.
Best tools to compare
1. Arize Phoenix — Best for retrieval debugging
(My pick)
Inspect retrieved chunks, their scores and ranking, embeddings, and relevance evaluations. Compare retrieval runs and diagnose where incorrect chunks enter the pipeline. Open-source and self-hostable.
Best for: understanding why a specific document was retrieved.
Explore execution traces, retrieved context, prompts, and downstream model behavior. Useful for comparing experiments and annotating retrieval results.
Best for: tracing retrieval through to the final answer.
Whichever tool you choose, build a view for each query that shows:
Example retrieval inspection
Illustrative scores only, not actual retrieval output.
The key diagnostic: Document B ranks highly but has low relevance. This could indicate embedding mismatch, overly broad retrieval, or a need for better reranking.
Track these five things:
Similarity score and rank for each retrieved chunk.
Relevance score from a labeled evaluation set or an LLM judge.
Reranker score and rank changes, if you use a reranker.
Chunk text and metadata, including source document, chunk boundaries, and filtering conditions.
Retrieval metrics such as Precision@k, Recall@k, MRR, and NDCG across a test set.
One important caveat: vector similarity scores aren't calibrated probabilities of relevance. A score of 0.9 doesn't necessarily mean a document is 90% relevant, and score scales differ across retrieval methods.
My recommendation
Start with Arize Phoenix if your primary question is “Why did this retriever return these chunks?” Choose LangSmith if your pipeline already centers on LangChain and you want to debug the full execution chain.
To narrow this to the best fit for your system: what are you using for retrieval?
