Prompt A/B testing in production

Production A/B testing for LLM prompts differs from traditional software A/B testing because "success" isn’t just a simple click or conversion; it requires evaluating output quality , semantic correctness, token cost, and latency.traceloop.com+1

A select group of LLM gateway, observability, and experimentation platforms natively support live, production-traffic prompt A/B testing and quality evaluation:

• Portkey AI Gateway
  • How it works: Portkey allows you to manage versioned prompts in its registry and set up weighted load-balancing configs via its gateway. You can split production traffic (e.g., 50/50) between two distinct prompt IDs or underlying models.
  • Quality Evaluation: It captures user feedback (thumbs up/down, scores) and operational metrics mapped back to specific trace and prompt IDs to let you review which prompt performs better on live data. Explore the setup guide in the Portkey Documentation.docs.portkey.ai
• Braintrust
  • How it works: Built heavily around the "experiment" primitive, Braintrust handles both offline evaluations and production logging/experimentation. It treats datasets, prompts, and scoring functions as first-class versioned artifacts.
  • Quality Evaluation: It allows you to mix deterministic code assertions, human feedback, and LLM-as-a-judge scorers simultaneously to compare prompt variations seamlessly. Read more about their approach on the Braintrust vs LangSmith Comparison.thegeocommunity.com
• Pezzo
  • How it works: Pezzo is a cloud-native LLM ops platform featuring centralized prompt management, remote prompt deployment, and native A/B testing capabilities.
  • Quality Evaluation: It allows updating and A/B testing prompts centrally without redeploying application code, tracking metrics like latency, token consumption, and error rates per prompt variant.
• LangSmith (by LangChain)
  • How it works: While LangSmith is primarily an observability and evaluation platform, it excels at prompt lifecycle management, logging production traces, and running comparative evaluations.
  • Quality Evaluation: You can commit and tag prompt variations, push them to production, and use online/offline evaluators (including custom LLM-as-a-judge feedback loops) to grade live production outputs. Check out the workflows in the LangSmith Prompt Concepts Guide.docs.langchain.com+1
• Arize Phoenix / W&B Weave
  • How it works: These are primarily ML/LLM observability and tracing tools. While they don't always do the traffic-splitting at the gateway level, they ingest production logs with detailed evaluations.
  • Quality Evaluation: They allow you to route variant traffic via your own application logic and stream both variants into Phoenix or Weave to evaluate embedding drift, hallucinations, and quality metrics side-by-side using evaluators.thegeocommunity.com

If you'd like to dive deeper, let me know:

• Are you looking for a proxy/gateway-based splitter (like Portkey) or an eval/observability platform (like Braintrust or LangSmith)?
• Do you plan to use LLM-as-a-judge or collected user feedback to score quality?

I can help recommend the best integration architecture for your stack.

Several platforms support production experimentation for LLM prompts, but there's an important distinction:

1. True online A/B testing: Randomly route production users or requests to prompt A and prompt B, then compare results.
2. LLM output-quality evaluation: Score actual model responses using LLM-as-a-Judge, custom evaluators, human feedback, or deterministic metrics.
3. Offline prompt experiments: Run competing prompts against the same dataset before deployment.

If your goal is to determine whether a new prompt produces better answers for real users, ideally you want all three.

Platforms worth evaluating (2026)

These are the strongest candidates, grouped by their primary capabilities.

1. LLM-native evaluation and experimentation platforms

These are particularly relevant when output quality—not just clicks or conversions—is your primary metric.

• 

1. Braintrust (Top recommendation)

• Offline experiments comparing prompt versions.
• Custom scorers and LLM-as-a-Judge.
• Production trace scoring and monitoring.
• Dataset management, human review, regression testing.
• Experiment comparisons and CI/CD integration.

Best for: Teams wanting an evaluation-first workflow that connects experiments to production quality.

Website ·arize.com+2

---

• 

2. LangSmith

• Prompt versioning and experiment tracking.
• Dataset-based prompt comparisons.
• Online evaluators against production traffic.
• LLM judges, custom evaluators, and human annotation.
• Production tracing and regression detection.

Best for: Teams using LangChain or LangGraph.

Website ·arize.com+1

---

• 

3. Langfuse

• Prompt versioning and prompt labels.
• Dataset-based prompt experiments.
• Production tracing and quality scores.
• LLM-as-a-Judge and custom evaluators.
• Open-source and self-hosting options.

Best for: Teams wanting an open-source platform with prompt experimentation and production observability.

Website ·arize.com+5

---

• 

4. Arize AX / Arize Phoenix

• Prompt and model experiments.
• Production evaluation and monitoring.
• Hallucination and relevance evaluation.
• Human feedback and annotation.
• Trace-level and session-level quality analysis.

Best for: Enterprise teams needing production observability alongside evaluation.

Website ·arize.com+2

---

• 

5. Future AGI

• Prompt variant experiments.
• Built-in evaluation metrics.
• Prompt optimization.
• Experiment comparisons and CI/CD quality gates.

Best for: Teams focused on systematic prompt optimization and evaluation.

Website ·futureagi.com+4

---

• 

6. Vellum

• Prompt and workflow versioning.
• Test suites and custom evaluators.
• Online evaluations.
• Release testing and deployment controls.

Best for: Teams looking for a more integrated visual prompt development and testing environment.

Website ·futureagi.com+4

2. Traditional experimentation platforms

These are especially useful if you need statistically rigorous randomized experiments with real production traffic.

These platforms generally need to be integrated with an LLM evaluation layer to measure subjective response quality, such as correctness, relevance, helpfulness, or hallucination rate.

Sources: www.growthbook.ioatlan.comironclad.academy.

3. Capability comparison

This is a practical shortlist based on the distinction between offline evaluation and production experimentation.

Important: Production evaluation does not automatically mean randomized prompt assignment. Verify native traffic allocation, experiment persistence, and statistical analysis with each vendor before purchasing.

arize.comwww.growthbook.iofutureagi.comatlan.com

4. Recommended architecture

For a production application, I would consider combining an experimentation platform with an LLM evaluation platform.

Production user requests

Incoming LLM requests

Experiment assignment

Statsig / GrowthBook / custom router

Group A

Current prompt

Group B

New prompt

LLM output evaluation

Braintrust / LangSmith / Arize

Correctness · Relevance · Helpfulness · Hallucinations

Experiment results

Quality scores · Confidence intervals · Latency · Cost

This architecture lets you answer two different questions:

• Does prompt B produce objectively better answers?
• Does prompt B improve the actual user experience?

5. My recommendation

I'd shortlist these three combinations:

One important consideration: Don't rely exclusively on an LLM judge. Combine automated scoring with human-reviewed samples and task-specific ground truth. Also track latency, token cost, and failure rates so a quality improvement doesn't come at an unacceptable operational cost.

---

One question to narrow the recommendation

Are you primarily looking for:

• A. True online A/B testing: split live traffic 50/50 between two prompt versions.
• B. Automated quality evaluation: continuously score production outputs and compare prompt versions.
• C. Both: randomized live experiments with automated LLM-as-a-Judge scoring and statistical significance.

I'd recommend C if you're building a production-grade prompt experimentation system.