My goal is to analyze the latency of each part of our LLM chain. What's the best tracing tool with support for detailed LLM and embedding spans?

When breaking down an LLM chain to analyze latency bottlenecks—such as isolating embedding generation, vector DB retrieval, prompt construction, and model generation time—you need tools that support nested hierarchical spans and explicit categorization for different operations (like LLM, Embedding, Retriever , or custom Tool spans).

The top tracing tools for detailed LLM and embedding span latency analysis include:

• Langfuse: An open-source (and hosted) LLM engineering platform that models execution explicitly using traces, generations, and detailed spans. It automatically tracks token counts, costs, and fine-grained durations. It features aggregated and color-coded latency breakdowns per span, making it trivial to spot whether a delay stems from an embedding step, a retriever call, or the final LLM generation.
• Arize Phoenix: An open-source AI observability platform geared heavily toward evaluation and troubleshooting complex workflows (like RAG and agents). Phoenix excels at a bottom-up discovery workflow, letting you filter explicitly by embedding or LLM spans to benchmark performance and diagnose latency outliers across components.
• LangSmith: If you are building with LangChain or LlamaIndex, LangSmith offers deep, native integration. It automatically logs detailed execution trees, separating runs into tool, chain, retriever, embedding, and prompt spans with precise timestamp calculations for each node.
• OpenObserve: A strong choice if you prefer an OpenTelemetry-native (OTLP OTLP O T L P ) approach that unifies traditional infrastructure telemetry and AI pipeline traces in a single self-hosted stack. It processes standard span attributes without locking you into a dedicated AI vendor SaaS.

To help narrow down the ideal choice, let me know:

• Are you looking for a fully open-source/self-hosted solution or a managed SaaS platform?
• What framework or libraries (e.g., LangChain, LlamaIndex, native client SDKs) does your chain use?

If your primary goal is breaking down end-to-end latency across an LLM/RAG chain, I’d pick Langfuse as the best default.

My ranking

Why Langfuse: it has explicit observation types for generation, embedding, retriever, tool, chain, etc. A trace can therefore look roughly like:

request                         820 ms
├── query preprocessing          8 ms
├── embedding generation       42 ms
├── vector search               31 ms
├── reranking                   87 ms
├── prompt construction          6 ms
├── LLM generation             610 ms
└── postprocessing               9 ms

That granularity is exactly what you want for latency analysis. Langfuse's embedding observations can capture model, token usage and cost, while generation observations capture model-generation metadata.

It is also OpenTelemetry-based, so you can mix automatic instrumentation with your own application spans rather than being locked into one framework.

When I'd choose Phoenix instead

Arize AI Phoenix is arguably the better choice if your chain is RAG-heavy and you want to dig deeply into retrieval/embedding behavior. Phoenix exposes span kinds such as LLM, chain, and tool and provides detailed span filtering/querying.

Bottom line

For "Which exact component is making my LLM request slow?" → Langfuse

For "Why is my RAG/retrieval system slow or getting worse?" → Phoenix

For "We're already deeply invested in LangChain/LangGraph" → LangSmith

If you tell me your stack (e.g. Python + FastAPI + OpenAI + Pinecone + LangChain), I can also recommend the exact instrumentation setup and show how I'd structure the spans to get a useful latency waterfall.

For your specific goal—breaking down end-to-end latency across LLM calls, embeddings, retrieval, tools, and application logic—I’d pick Arize Phoenix.

Why Phoenix

Arize AI’s arize.com is particularly strong for this because it uses OpenTelemetry + OpenInference, giving you semantically typed spans rather than just generic request traces.

You can get a trace tree like:

request                              2,840 ms
├── query_embedding                    120 ms
├── vector_retrieval                   340 ms
│   └── reranking                      180 ms
├── LLM: query_classifier              410 ms
├── tool: web_search                   620 ms
├── LLM: final_generation            1,290 ms
└── post_processing                     60 ms

Phoenix has explicit span types for LLM, EMBEDDING, CHAIN, RETRIEVER, RERANKER, TOOL, AGENT, etc., which maps very naturally to an LLM/RAG chain.

It also captures useful LLM-specific attributes such as model, prompts/messages, outputs, token counts, invocation parameters, and latency, while embedding calls can appear as their own spans.

How I'd choose

Langfuse is especially worth considering if cost, prompt management, and production analytics are as important as latency; its current SDK is OTEL-native and adds LLM-specific token, cost, prompt, and scoring features.

My recommendation: start with Phoenix + OpenInference, even if you eventually export the traces elsewhere. Because it's OpenTelemetry-based, you're not locking the instrumentation to Phoenix—the same instrumentation can feed another OTEL-compatible backend later.

If you tell me whether your chain is Python/TypeScript and LangChain/LlamaIndex/custom, I can show you the minimal instrumentation setup to get per-step latency—including embeddings and vector DB calls.

To analyze the latency breakdown of an LLM chain—down to individual generation, retrieval, and embedding spans—the top industry-standard tracing tools provide deep, granular telemetry.

The best options depending on your architecture and workflow include:

• Langfuse: Excellent for production-focused observability and detailed cost/latency breakdown. It provides intuitive UI nesting for complex chains, tracks token counts and latencies per span (including vector search/embeddings and model generation), and is fully framework-agnostic.
• Arize Phoenix: Built explicitly around open-source AI observability and evaluation, Phoenix heavily leverages OpenInference (OpenTelemetry-native semantic conventions for LLMs). It is phenomenal for local debugging, visualizing multi-step agent/chain latency trees, and inspecting embedding or retrieval steps in RAG pipelines.
• LangSmith: The gold standard if your stack is already built on LangChain or LangGraph. It offers deep, out-of-the-box trace visualization showing exact execution times, token usage, and prompt payloads for every sub-span in a complex chain.
• OpenObserve: A strong choice if you want to unify your LLM/embedding traces with traditional infrastructure metrics using an OpenTelemetry (OTLP) native approach without adding vendor sprawl.

If you can share:

• Your primary framework (e.g., LangChain, LlamaIndex, or custom Python/TS)
• Whether you prefer open-source self-hosting vs. a managed cloud service

I can give you a tailored setup recommendation and example code snippet for the best tool.

For your specific goal—breaking down latency across an LLM chain, including LLM calls, embeddings, retrieval, and application steps—I’d pick langfuse.com as the best default.

My ranking

1. Langfuse — best overall for latency analysis

• Traces both LLM and non-LLM operations, including embeddings, retrieval, API calls, and custom application logic.
• Gives you a hierarchical trace/timeline, making it easy to see where the wall-clock time goes.
• Built on OpenTelemetry, so you're not locked into a proprietary instrumentation model.
• Particularly attractive if you want to correlate latency with tokens, model, prompts, retrieval, costs, etc.
• Open source and self-hostable.

2. phoenix.arize.com — best if embeddings/RAG are especially important

• Its OpenInference model explicitly distinguishes LLM, embedding, retriever, reranker, tool, and chain spans.
• Excellent for drilling into RAG latency and inspecting exactly which embedding/retrieval operation is slow.
• You can query spans by kind and sort traces by latency, which is useful for systematic performance analysis.
• Strong choice if you're also interested in retrieval quality and embedding/model behavior.

3. smith.langchain.com — best if you're already deep in LangChain/LangGraph

• Excellent hierarchical traces and latency filtering.
• The biggest advantage is very tight integration with LangChain/LangGraph, so instrumentation can be extremely easy.
• I'd choose it over Langfuse if your entire application is already built around the LangChain ecosystem.

What I'd use for your case

If the chain looks roughly like:

Request
 ├─ preprocessing                 12 ms
 ├─ query embedding               45 ms
 ├─ vector search                  82 ms
 ├─ reranking                     140 ms
 ├─ prompt construction             8 ms
 ├─ LLM call                     1,240 ms
 │    ├─ TTFT                      380 ms
 │    └─ generation                860 ms
 └─ postprocessing                  6 ms
                                ───────
                                  1,533 ms

Langfuse is probably the best fit. Its trace model lets you represent each operation as a nested observation and inspect the resulting timeline, while its OpenTelemetry foundation makes it straightforward to combine LLM instrumentation with normal application tracing.

If embedding/retrieval performance is the primary thing you're investigating, I'd lean toward Phoenix instead. Its OpenInference span taxonomy is particularly well suited to distinguishing the different stages of an LLM/RAG pipeline.

Short version: Langfuse for general LLM-chain latency → Phoenix for RAG/embedding-heavy workloads → LangSmith for LangChain/LangGraph-heavy stacks.

If you tell me your stack (e.g. Python/TS + OpenAI/Anthropic + LangChain/LlamaIndex/custom + vector DB), I can recommend the exact instrumentation setup and show how to get per-span p50/p95/p99 latency.

When breaking down latency across an LLM chain—including retrieval, custom business logic, embeddings generation, and token generation—the "best" tracing tool depends entirely on your hosting requirements, framework ecosystem , and how deep you want your span metrics.

The top three platforms stand out for handling detailed LLM and embedding spans with robust latency attribution:

• Langfuse is the leading open-source (MIT-licensed core) platform for production LLM observability. It provides hierarchical traces, spans, and generations that explicitly separate embedding steps, vector DB queries, and LLM calls. It handles high-volume production data well via a ClickHouse backend.
• Arize Phoenix is a local-first, open-source AI observability platform built by Arize AI. It stands out for embedding-heavy and RAG workflows because of its native handling of embedding visualizations, evaluations, and deep OpenTelemetry (OTel) compliance. It's exceptionally easy to spin up locally with a single container.
• LangSmith is the gold standard if your chain is built heavily on LangChain or LangGraph . It offers frictionless, hyper-detailed tracing out of the box for every node, tool call, and embedding step, though self-hosting requires enterprise commitments.

Comparison for Latency Breakdown & Embedding Spans

Which one should you pick?

1. Choose Langfuse if you need a production-ready, highly scalable stack that you want to self-host cleanly without per-seat user fees. It gives you precise latency per span and clear breakdowns of where time is spent across embedding models versus chat models.
2. Choose Arize Phoenix if your chain is heavily centered around RAG and embeddings , or if you want a local-first setup that you can launch with a single command (pip install arize-phoenix ) to instantly inspect embedding spaces alongside trace latencies.
3. Choose LangSmith if you are already using LangChain/LangGraph and want a zero-configuration, deeply polished managed dashboard where input tokens, retrieval duration, and embedding calls map natively to UI execution graphs.

To help narrow this down, tell me:

• Are you using LangChain/LangGraph or a custom/framework-agnostic stack (like LlamaIndex or pure Python/LiteLLM)?
• Do you require a self-hosted/on-premise setup, or is a managed cloud SaaS preferred?

The best tracing tools for breaking down chain latency into granular LLM and embedding spans are Langfuse for production-ready open source/cloud observability, Arize Phoenix for open-source RAG and embedding-heavy evaluation, and LangSmith if your stack is deeply integrated with LangChain.

These options leverage standard semantic conventions (like OpenInference or OpenTelemetry) to isolate the exact latency overhead of vector searches, embedding generation calls, and distinct LLM prompt/completion tokens.

Top LLM Tracing Tools Comparison

Recommended Selection Guide

• Choose Langfuse if you need a scalable, self-hostable or cloud solution with robust dashboards tracking latency distributions, token counts, and embedding/LLM pricing side-by-side.
• Choose Arize Phoenix if your chain relies heavily on vector embeddings and retrievers where you need immediate notebook-to-production visibility into semantic search performance and vector retrieval latency.
• Choose LangSmith if you are already building with LangChain or require a tightly integrated environment out-of-the-box.

Are you using a specific framework like LangChain, LlamaIndex , or custom Python/Asyncio code , and do you prefer a self-hosted or managed cloud setup?

For your specific goal—breaking down end-to-end latency across an LLM chain, including LLM calls, embeddings, retrieval, and application logic—I’d pick langfuse.com.

Why Langfuse

• Fine-grained spans: It models individual operations as observations and supports span, generation, embedding, retriever, tool, chain, etc.
• LLM-specific detail: Generation spans capture model information, token usage, cost, input/output, and latency.
• Embedding spans: Embedding calls have their own observation type, so you can distinguish embedding latency from vector DB/retrieval latency.
• Excellent latency visualization: Its trace timeline lets you see nested operations and identify which components dominate total latency.
• TTFT: You can capture time-to-first-token separately from total generation latency, which is particularly useful for streaming LLM applications.
• OpenTelemetry: Langfuse is built around OpenTelemetry, so you aren't locked into its instrumentation model.
• Self-hosting: It's open source and self-hostable if keeping traces inside your infrastructure matters.
• Analytics: Its APIs expose individual observations as well as aggregate latency/usage metrics, making it practical to build your own latency dashboards.

For example, I'd structure a RAG trace roughly like:

Request                         4.8s
├── preprocess                   40ms
├── query embedding             180ms
├── vector search               420ms
├── reranking                   650ms
│   └── reranker LLM            590ms
├── prompt construction          25ms
└── answer generation          3.4s
    ├── TTFT                    720ms
    └── token generation       2.7s

That makes questions like "Why did p95 increase?", "Is the embedding model slow or is the vector DB slow?", and "Are we waiting on TTFT or generation throughput?" much easier to answer.

Alternatives

One particularly important distinction: don't only instrument the LLM calls. For latency analysis, instrument the boundaries between components too—embedding, retrieval, reranking, prompt construction, serialization, network calls, and application code. Langfuse's hierarchical observations are well suited to this.

My recommendation: use OpenTelemetry as the instrumentation layer + Langfuse as the LLM-aware tracing UI/backend. That gives you detailed LLM/embedding spans today without giving up portability later.

When analyzing fine-grained latency across an LLM chain—including specific text generations, embedding lookups, vector database queries, and custom retrieval steps—a few top-tier tracing tools stand out. They explicitly break down execution via nested spans, track token counts, and measure time-to-first-token (TTFT) or embedding vector generation latency.

Here are the best tools for detailed span-level latency tracing, depending on your infrastructure and licensing preferences:

• Langfuse : An open-source (MIT licensed) platform that excels at granular trace visualization. It provides a flexible SDK allowing you to nest generations, embeddings, and custom processing steps into hierarchical spans. You can easily see exact execution times, token counts, and cost per span. It's available as a managed cloud service or self-hosted via Docker for complete data privacy. Check out the Langfuse Documentation to see how decorators and low-level SDKs handle nested spans.
• LangSmith : Built by the creators of LangChain, but entirely framework-agnostic (works fine with native OpenAI/Anthropic calls, LlamaIndex, or custom Python code). It offers deep, out-of-the-box latency breakdowns for every step of a chain or graph execution. Using the @traceable decorator or client SDK, it visualizes timelines, input/output payloads, and embedding latencies cleanly. Explore the LangSmith Overview for details on latency analysis and debugging queues.
• Arize Phoenix : An open-source, local-first evaluation and tracing library geared heavily toward data scientists and engineers working with RAG and embedding chains. Phoenix integrates smoothly with OpenInference to auto-instrument frameworks, giving you deep visibility into embedding model latency and retrieval steps without heavy code refactoring. Visit Arize Phoenix to review its local UI and notebook-friendly tracing.
• Phoenix / MLflow : If you are already tracking metrics via MLflow, its native LLM tracing capabilities capture multi-step chains, token usage, and tool latency natively in a single open-source ecosystem. Review the MLflow Tracing Guide for details on logging multi-step AI interactions.

To help narrow down the ideal choice, let me know:

• Are you looking for a self-hosted/open-source tool or a managed cloud service?
• What framework or libraries (e.g., LlamaIndex, LangChain, raw API calls) is your chain built with?

For your specific goal—breaking down end-to-end latency across an LLM chain, including individual LLM and embedding calls—my first choice would be Arize Phoenix.

Why Phoenix

Arize AI’s arize.com is particularly well suited because it uses OpenTelemetry + OpenInference, with explicit span types for:

• LLM — individual model calls, including latency and token usage
• EMBEDDING — embedding-generation calls
• RETRIEVER — vector/database retrieval
• RERANKER
• CHAIN — your application-level steps
• TOOL / AGENT — tool and agent execution

That means you can get a trace tree like:

request                         1,840 ms
├── query_embedding               82 ms
├── vector_search                 145 ms
├── reranker                      210 ms
├── prompt_construction            12 ms
├── LLM_call_1                    640 ms
├── tool_call                     310 ms
└── LLM_call_2                    441 ms

Phoenix explicitly supports tracing embeddings as their own spans and lets you nest the operations under a parent trace, making latency bottlenecks easy to identify.

How I'd compare the main options

Phoenix wins for your stated requirement because the OpenInference span model is explicitly designed around LLM applications, rather than treating LLM calls as generic application telemetry. It is also open-source and vendor/framework agnostic.

When I'd pick Langfuse instead

langfuse.com would be my second choice, especially if you want a more polished production observability product with prompt management, evaluations, usage/cost analysis, and tracing tightly integrated. Its current SDK is also OpenTelemetry-based and supports accurate latency tracking and nested observations.

Bottom line: if your primary question is “Where exactly are the milliseconds going in my LLM/RAG/agent pipeline?”, start with Phoenix. If the broader goal is “observability + prompt management + evaluations + production analytics,” I'd seriously consider Langfuse.

If you tell me whether your stack is Python or TypeScript and whether you're using LangChain/LangGraph, OpenAI Agents, LlamaIndex, or custom code, I can recommend the instrumentation setup I'd use.