How to benchmark LLMs inside your observability platform
The most effective approach is to build a repeatable evaluation pipeline that combines model quality, runtime performance, and cost in one observability workflow. Rather than comparing models using a handful of prompts, you should run the same representative workload through each model, score the outputs consistently, and connect the results to traces and production metrics.
This matters because a model can respond quickly and cheaply while still producing incorrect, ungrounded, or irrelevant answers. Modern LLM observability therefore combines tracing, evaluation, and feedback-driven regression testing.
1. Define what you want to benchmark
Track three dimensions for every model and, ideally, every model version and prompt version.
What is the process for using my observability platform to run benchmarks on different LLMs so I can see how their output quality and performance stack up against each other?
Does the answer satisfy the task and your requirements?
Correctness against reference answers
Relevance and completeness
Groundedness and hallucination rate
Instruction following, format validity, and safety
Cost and efficiency
What does it cost to deliver a useful answer?
Input and output tokens
Cost per request
Cost per successful task
Quality achieved per dollar
For consistent telemetry, consider OpenTelemetry's GenAI metric conventions, which include token usage, operation duration, and time-to-first-token measurements. Check the stability status of individual conventions before adopting them.
2. Build a standardized benchmark dataset
Create a versioned dataset of real tasks from your application. Start with around 100–500 representative cases for an initial evaluation, then expand based on observed failures.
Include:
Typical cases: common user queries and normal workflows.
Edge cases: ambiguous questions, long contexts, malformed inputs, and difficult reasoning.
Known failures: hallucinations, incorrect answers, retrieval failures, and formatting errors from production traces.
Reference data: expected answers, source documents, required output schemas, and human-approved quality labels where available.
For each test case, store the input, expected behavior, relevant context, task category, and evaluation criteria.
Important: Use the same dataset, system instructions, retrieval context, tool permissions, and generation settings across models wherever possible. Otherwise, you risk measuring differences in your application setup rather than differences in the models.
3. Run a controlled evaluation pipeline
1. Versioned evaluation dataset
Representative production cases + expected outcomes
2. Parallel model execution
Model A · Model B · Model C · Same inputs
3. Trace and collect metrics
Outputs, tokens, latency, errors, cost, model metadata
Instrument each model invocation with a common trace schema. At minimum, capture:
benchmark_run_id, dataset_version, and case_id
Provider, model ID, model version when available, and prompt version
Input/output token counts, latency, estimated cost, and error status
Output or a secure reference to it
Evaluation scores, evaluator version, and evaluation rationale where appropriate
For agentic or RAG applications, also capture retrieval spans, retrieved document IDs, tool calls, and intermediate model invocations. This makes it possible to determine whether a quality regression comes from the LLM itself or an upstream component.
4. Measure output quality with multiple evaluators
Avoid relying on one overall score. Use a layered evaluation strategy.
For LLM-as-judge, use a fixed rubric and preferably a judge model independent of the candidate models. On a representative subset, compare judge scores against human labels to detect evaluator bias. For pairwise comparisons, randomize which model appears first and blind reviewers to model identity.
For retrieval-augmented generation (RAG), measure both retrieval quality and answer quality. A model may appear inaccurate because the right evidence was never retrieved.
5. Compare models in your observability dashboard
A useful dashboard should let you filter by model, task category, prompt version, deployment, and evaluation run. Show quality and performance side by side, rather than ranking models on latency alone.
Here is an illustrative benchmark summary with fictional values:
Metric
Model A
Model B
Model C
Quality score
92/100
88/100
94/100
p95 latency
1.2 s
0.7 s
2.1 s
Cost per 1K requests
$4.00
$1.50
$8.00
Error rate
0.5%
0.3%
0.7%
Illustrative figures only; not measured model results.
genui{"charts_widget_v2":{"content":{"chartType":"bar","meta":{"title":"Illustrative quality versus latency","description":"Example values showing why model selection involves trade-offs."},"xKey":"model","series":[{"dataKey":"quality","label":"Quality score","valueFormat":"integer"},{"dataKey":"latency","label":"p95 latency (seconds)","valueFormat":"raw"}],"data":[{"model":"Model A","quality":92,"latency":1.2},{"model":"Model B","quality":88,"latency":0.7},{"model":"Model C","quality":94,"latency":2.1}]}}}
This example shows three different choices:
Model A: A potential balance of quality, latency, and cost.
Model B: The low-latency, low-cost option if its quality is sufficient.
Model C: The highest quality, potentially justified for high-value or difficult tasks.
Don't combine these into a single ranking until you've defined the business trade-offs. Use a quality threshold first, then optimize latency and cost among the models that pass.
6. Choose an observability and evaluation platform
If you're extending an existing observability platform, prioritize integrations with your current tracing pipeline and evaluation workflow.
Langfuse
Good for combining traces, prompt management, datasets, and evaluation workflows, including self-hosted deployments.
Explore:
[[2]()]()
Core Metrics to Compare
Metric Category
Specific Metrics to Track
Purpose
Performance
Latency (P50, P99), TTFT, Throughput (tokens/sec)
Measure user-perceived speed and system responsiveness
Financial Cost
Input token cost, Output token cost, Total per 1,000 requests
Validate correctness and alignment with user intent
If you share your primary application use case (e.g., RAG chatbot, automated coding agent, data extraction) and current observability tool , I can recommend specific evaluation metrics or python testing harnesses to set this up.
Yes. The basic idea is to turn your observability platform into a repeatable evaluation harness: feed the same test cases to each LLM, capture the full traces, score the outputs, and compare quality against latency, token usage, and cost.
A practical workflow
Build a representative benchmark dataset
Start with real production requests, sanitized for sensitive data.
Add known edge cases, failure cases, and expert-written examples with expected answers.
Keep a held-out "golden set" that you don't continually tune against.
Evaluation datasets are particularly useful for comparing different models or prompts against exactly the same inputs.
Define what "quality" means before running the models
Depending on your application, useful evaluators include:
factual correctness
relevance
instruction following
groundedness / citation correctness
completeness
structured-output validity
safety/policy adherence
Use deterministic checks where possible—for example, exact match, JSON-schema validation, or SQL execution—and an LLM judge for harder subjective criteria. Human ratings are useful for calibrating the judge.
Create one standardized model wrapper
Conceptually:
benchmark_case
│
├── Model A ──┐
├── Model B ──┤
├── Model C ──┤──> traces ──> evaluators
└── Model D ──┘
Keep everything except the model constant: system prompt, user input, tools, retrieval corpus, temperature, max output tokens, etc. Otherwise you're benchmarking different systems rather than different models.
Instrument every invocation
Your observability traces should record at least:
model/version
prompt and relevant context
output
input/output token counts
time-to-first-token (if streaming)
total latency
errors/retries
estimated cost
evaluation scores
Modern LLM tracing platforms expose these kinds of fields specifically so you can analyze quality alongside performance and cost.
Run the same dataset against every model
Give each model the identical cases and configuration. Run enough repetitions to account for stochastic output—particularly if temperature is nonzero.
Store each run as an experiment such as:
Experiment: customer-support-model-comparison
Dataset: support-golden-v3
Run A: Model A / prompt-v12
Run B: Model B / prompt-v12
Run C: Model C / prompt-v12
Platforms such as MLflow explicitly support evaluating prompts/models across datasets and tracking the results as experiments.
Compare quality and performance separately
A useful result table might look like:
Metric
Model A
Model B
Model C
Correctness
0.91
0.94
0.89
Groundedness
0.95
0.92
0.96
Instruction following
Inspect the traces behind the scores
This is where observability becomes especially valuable. If Model B gets a lower correctness score, you should be able to drill into the individual cases and see why: bad retrieval, tool selection, hallucination, malformed output, excessive generation, etc.
Trace-based evaluation lets you reuse captured traces for multiple evaluators instead of repeatedly calling the models, which can reduce evaluation cost.
An important distinction
I'd actually run two benchmarks:
A. Model capability benchmark
Same input → Model A/B/C
↓
quality evaluators
This answers: "How well does each model perform on our tasks?"
B. Production performance benchmark
Same workload → Model A/B/C
↓
latency / throughput
tokens / cost / errors
This answers: "How does each model behave under our actual serving conditions?"
Don't mix these into a single score initially. A model can produce excellent answers while being expensive or slow, and another can be extremely fast while failing more frequently. NVIDIA's benchmarking guidance similarly separates inference-performance measurement from load testing and notes that accuracy should be validated separately.
The observability feedback loop
The most useful setup becomes:
Production traces → interesting failures → benchmark dataset → run models → evaluators → compare → deploy → collect new traces → repeat
That's essentially the modern LLM evaluation loop: observability gives you the raw behavioral data, datasets make comparisons reproducible, and evaluators turn outputs into measurable quality signals.
If you tell me which observability platform you're using (Langfuse, LangSmith, Arize, Datadog, MLflow, OpenTelemetry-based tooling, etc.), I can give you the exact setup—including how to create the dataset, run the models, configure evaluators, and build the comparison dashboard.
To benchmark different large language models (LLMs) using an observability platform, you need to define a golden dataset of test prompts, route those prompts across your target models while capturing telemetry, and run automated evaluation evaluators to score quality and performance. Platforms like LangSmith or Arize Phoenix allow you to orchestrate this evaluation loop seamlessly.[](https://galileo.ai/blog/understanding-llm-observability)
1. Prepare Your Evaluation Dataset
Define Test Cases : Create a structured dataset (CSV, JSON, or code) containing representative user inputs, system prompts, and expected outputs or ground truths.
Tag Metadata : Label each test case by category (e.g., summarization, code generation, RAG retrieval) to segment your final benchmark analytics effectively.
2. Configure Your Models and Routing
Establish Endpoints : Set up API connections for each LLM you want to test (e.g., OpenAI, Anthropic, open-source models hosted via vLLM).
Run Batch Inferences : Script an evaluation loop that iterates through your golden dataset, sending identical prompts to each target model concurrently.
If you can share which specific observability platform you are using (e.g., LangSmith, Arize Phoenix, Langfuse, Galileo) and your primary use case (e.g., RAG, conversational chatbot, data extraction), I can provide precise code snippets or platform-specific steps for setting up your evaluation suite.
Yes, absolutely. Modern LLM observability and evaluation platforms (like Langfuse, Arize Phoenix, LangSmith , or W&B Weave ) are explicitly built to handle multi-model experimentation, performance tracking, and quality benchmarking side-by-side.[](https://logz.io/blog/top-llm-observability-tools/)
Experiment Tracing: Run your evaluation pipeline, routing the same inputs through your distinct model targets. The platform logs each run as an isolated experiment or trace tree.[](https://logz.io/blog/top-llm-observability-tools/)
To help tailor this, which observability platform are you currently using or considering, and are you planning to benchmark these models on standard text prompts or an agentic/RAG workflow?
Yes. An LLM observability platform can be the measurement layer for a model benchmark, provided it supports both tracing and evaluation. Current platforms increasingly combine production traces, datasets, evaluators, experiments, and model comparisons.
A practical benchmark setup
Run the same test set through each model, while recording:
Monitor stability and reliability under batch load
0.93
0.95
0.91
Valid output rate
99.1%
98.4%
99.7%
Median latency
1.2s
0.8s
1.6s
p95 latency
2.4s
1.7s
3.1s
Input tokens
850
850
850
Output tokens
210
180
260
Cost / 1K requests
—
—
—
For inference performance, also consider TTFT, inter-token latency, throughput, and requests/tokens per second rather than relying solely on end-to-end latency.
human feedback, task completion, conversion/deflection/etc.
The important distinction is that observability tells you what happened; evaluation tells you whether the result was good. For example, a trace can show that Model A responded in 800 ms, but an evaluator needs to determine whether its answer was actually correct.
How I'd structure the experiment
┌─ Model A ─┐
Test dataset ────┼─ Model B ─┼──► Observability
├─ Model C ─┤ │
└─ Model D ─┘ │
▼
┌───────────────┐
│ Evaluators │
│ • correctness │
│ • relevance │
│ • safety │
│ • latency │
│ • cost │
└───────┬───────┘
▼
Model comparison
A particularly useful approach is to take real production traces, anonymize them, turn representative failures and successes into a benchmark dataset, and replay that dataset against multiple models. Platforms such as LangSmith and Braintrust explicitly support datasets/experiments and comparison workflows, while Phoenix provides tracing and evaluation capabilities around OpenTelemetry/OpenInference.
One thing to avoid
Don't reduce the benchmark to a single "LLM quality score." A model could have a higher average evaluator score while being substantially slower or more expensive. Keep a multidimensional result such as:
Model Quality p95 latency Cost/task Error rate
----------------------------------------------------------
Model A 92% 1.8s $0.012 0.8%
Model B 89% 0.7s $0.004 1.1%
Model C 94% 2.4s $0.031 0.5%
Also evaluate slices, not just aggregate averages—for example, easy vs. difficult queries, RAG vs. non-RAG, languages, customer segments, or different task types. This catches cases where an apparently strong model performs poorly on an important subset.
If you tell me which observability platform you're using (Langfuse, LangSmith, Arize, Braintrust, Datadog, custom/OpenTelemetry, etc.) and which models you want to compare, I can show you a concrete benchmark architecture and the exact metrics/evaluation schema I'd use.
