What's the best way to simulate thousands of conversations to test a voice agent?

The best approach is to build a synthetic-caller evaluation harness that runs real audio conversations against your deployed voice agent, rather than generating thousands of text transcripts.

The key is to test three things separately: conversation behavior, voice behavior, and system/load behavior.

Recommended architecture

Scenario generator
       │
       ▼
Synthetic caller ──► TTS ──► real voice transport ──► Your voice agent
       ▲                                                │
       │                                                ▼
 Persona + goal                                      STT/LLM/TTS
       │                                                │
       └────────────── conversation state ◄─────────────┘
                              │
                              ▼
                    Trace + audio + tool calls
                              │
              ┌───────────────┴──────────────┐
              ▼                              ▼
        Deterministic checks             LLM judges
        (did task happen?)          (natural? concise? safe?)

This is increasingly the direction of serious voice-agent evaluation: bot-to-bot audio, with the simulator actually speaking and listening rather than simply feeding transcripts to the agent.

1. Define scenarios, not conversations

Instead of writing 10,000 scripts, create perhaps 100–500 scenario templates:

• Goal: "Reschedule my appointment"
• Persona: impatient, elderly, confused, hurried, etc.
• Context/state: existing appointment at 3 PM
• Constraints: "I can only do mornings"
• Expected outcome: appointment moved successfully
• Adversarial behavior: changes mind, interrupts, gives incomplete information
• Failure conditions: agent invents availability, exposes private data, fails to complete the booking

Then generate many variations from each scenario.

This gives you 10,000 different conversations without needing 10,000 hand-authored scripts. Microsoft's current simulation tooling similarly recommends scenario descriptions plus desired turn counts rather than manually authoring every conversation.

2. Make the simulated user stateful

Don't use an LLM that simply says "the next thing a user might say."

Give it a private state:

{
  "goal": "reschedule_appointment",
  "known_facts": ["appointment_id=123"],
  "constraints": ["morning_only"],
  "personality": "impatient",
  "hidden_intent": "will accept Tuesday if offered",
  "behavior": [
    "occasionally interrupt",
    "ask unrelated question once",
    "correct agent if misunderstood"
  ]
}

The simulator should decide what to say based on what the agent actually just said.

This matters because research is finding that synthetic users can be overly cooperative and miss the friction of real users.

3. Test audio—not just text

For a voice agent, I'd run at least two tiers.

Tier A: cheap/high-volume

Run thousands of simulations with text or lightweight audio to catch:

• incorrect tool calls
• broken workflows
• hallucinations
• failure to follow policy
• forgotten constraints
• premature termination
• bad state management

Tier B: expensive/realistic

Run hundreds/thousands of actual audio calls through the production-equivalent voice stack to test:

• STT errors
• accents
• background noise
• interruptions/barge-in
• silence detection
• turn-taking
• latency
• TTS problems
• caller talking over the agent
• agent talking over the caller

Modern voice-agent evaluation frameworks explicitly use this bot-to-bot audio architecture because text replay misses these failure modes.

4. Score every call automatically

Don't have an LLM give the conversation one vague "7/10."

Use deterministic assertions wherever possible.

For example:

Task completed?                 YES/NO
Correct tool called?            YES/NO
Correct arguments?              YES/NO
Unauthorized disclosure?       YES/NO
Required confirmation obtained? YES/NO
Correct final state?            YES/NO

First-response latency          420 ms
p95 turn latency                1.8 s
Interruptions handled           4/5
Conversation length              11 turns

Then use an LLM judge for subjective dimensions:

• Was the agent understandable?
• Was it unnecessarily verbose?
• Did it recover naturally after misunderstanding?
• Did it sound conversational?
• Did it appropriately acknowledge interruptions?
• Did it follow the intended persona/task?

The emerging EVA benchmark, for example, separates task accuracy from conversation experience, including progression, conciseness, and turn-taking.

5. Deliberately inject difficult callers

Your distribution shouldn't be 10,000 polite users.

I'd roughly create buckets like:

• 40% normal users
• 15% confused users
• 10% impatient users
• 10% verbose/rambling users
• 10% interruption-heavy users
• 5% noisy audio
• 5% accents/dialect variation
• 5% adversarial/safety cases

And vary things like:

• speech rate
• pauses
• volume
• background noise
• latency
• interruptions
• corrections
• repeated questions
• contradictory information
• unexpected requests

EVA's open-source benchmark takes a similar approach by perturbing conversations with noise, accents, connection degradation, and other conditions.

Then add real load testing

This is a separate problem from conversation quality.

For example:

             10 concurrent calls   → baseline
             100 concurrent calls  → normal load
           1,000 concurrent calls  → expected peak
           2,000 concurrent calls  → stress
           5,000 concurrent calls  → spike

Measure:

• call connection success
• p50/p95/p99 latency
• STT latency
• LLM latency
• TTS latency
• concurrent sessions
• provider rate limits
• WebSocket failures
• tool latency
• dropped calls
• task-completion rate
• cost per completed task

A conventional HTTP load test isn't sufficient because voice calls are long-lived, stateful, streaming sessions.

The most important trick: replay the same scenarios

Give every scenario a stable seed:

scenario_id = appointment_reschedule_0472
seed        = 918273
agent       = v1.8.2

Then you can run:

v1.8.2 → 10,000 calls
v1.8.3 → same 10,000 calls

and compare them directly.

That's much more useful than simply generating another 10,000 random conversations after every release. Conversation-simulation systems such as MLflow explicitly support reproducible scenario-based evaluation for this reason.

What I'd build

If you're building this yourself, my stack would look roughly like:

Scenario engine → synthetic caller → real-time audio transport → voice agent → trace collector → deterministic assertions + LLM judge → results database/dashboard

And I'd store the audio, transcript, tool calls, timing events, scenario state, and scores for every call.

For a starting point rather than building everything yourself, there are now several useful references/projects: ServiceNow's EVA is open source and specifically designed around bot-to-bot voice evaluation, while DeepEval provides conversation simulation and voice-mode evaluation.

If I had to pick one principle: don't optimize for "10,000 conversations." Optimize for 10,000 independently generated attempts to break the agent, with enough evidence to explain exactly why each failure happened. That's what turns synthetic testing from a demo into an actual QA system.

Simulating thousands of conversations to stress-test a voice agent requires a bot-to-bot architecture—pairing a programmatic "Simulated User" against your voice agent, orchestrating it via parallel execution scripts, and passing the resulting transcripts through an LLM judge.

The industry-standard blueprint to pull this off efficiently relies on a structured, 5-stage process:

1. Build a Persona-Driven "Simulated User"

Don't just use a generic LLM prompt for your caller. Create a parameterized simulated user framework (using tools or custom scripts with frameworks like DeepEval's Conversation Simulator or open architectures like ServiceNow's EVA).

• Give them a persona & goal: Inject randomness (e.g., impatient customer, confused elderly user, high background noise simulation, or someone who interrupts mid-sentence).
• Define constraints: Give the simulated user a specific hidden piece of information they must reveal or a task they must successfully complete (e.g., "book a flight for next Tuesday, but change your mind on the seat selection halfway through").

2. Choose Your Simulation Medium (Text vs. Audio-Path)

• Text-in-the-loop (Fast & Cheap): Mock the Speech-to-Text (STT) and Text-to-Speech (TTS) layers, feeding text straight into your agent's LLM core. Use this for running your first 9,900 out of 10,000 simulations because it runs in minutes and costs pennies.
• Audio-in-the-loop (True-to-Life): Route the simulated user's text through a TTS engine, pass it over a live WebRTC/SIP/telephony connection into your voice agent, and capture the audio back. Reserve this for a representative subset (100 - 500 runs) to realistically benchmark audio latency (P50/P95), interruptions, and Word Error Rate (WER).

3. Orchestrate via Parallel Batch Execution

To get through thousands of runs without taking days, you need asynchronous execution:

• Containerize your simulation script or use testing platforms designed for voice QA (such as Cekura or custom Python scripts running via asyncio).
• Spin up parallel worker threads targeting your voice agent's staging webhook or telephony endpoints.
• Implement strict retry budgets and timeout logic so a hanging connection or a slow LLM response doesn't stall the entire batch queue.

4. Deploy an Automated LLM Judge

Manually listening to thousands of audio files is impossible. Instead, capture full call transcripts and run them through an automated evaluation judge. Score each conversation on quantifiable axes:

• Task Completion: Did the agent achieve the goal defined in the user's persona?
• Conciseness & Conversational Progression: Did the agent ramble, or was it optimized for human ear-span? (Voice agents fail often by outputting huge paragraphs that sound unnatural when spoken aloud).
• Interruption Handling: Did the agent cleanly stop talking when the simulated user interrupted?

5. Turn Failures into Regression Suites

• Automatically siphon off any simulated conversation that fails the judge's threshold into a permanent regression test suite.
• Run this mini-suite every time you tweak your voice agent's system prompt or swap your STT/TTS providers to ensure zero performance regression.

If you're setting this up, let me know:

• Are you testing an audio-native model or a cascade pipeline (STT -> LLM -> TTS)?
• What is your primary channel (WebRTC browser calls vs. telephony/SIP phone numbers)?

I can help you map out the specific tech stack or script architecture to build this out.

The best approach is to treat a voice agent like a production system: simulate realistic callers, run conversations at scale, and score outcomes automatically. Don't start by generating thousands of random calls; build a test system that creates meaningful failure coverage.

A strong architecture looks like this:

1. Build a synthetic caller population

Create thousands of caller personas, not just scripts.

Examples:

• "Impatient customer who interrupts frequently"
• "Older caller who speaks slowly and repeats information"
• "Non-native speaker with unusual phrasing"
• "Customer who changes their mind mid-call"
• "Caller who refuses to provide information"
• "Caller who tries to bypass policy"
• "Caller with background noise and poor audio"

Each simulated caller should have:

• Goal: what they want
• Context: account state, history, constraints
• Personality: patience, verbosity, cooperativeness
• Failure behavior: how they create pressure on the agent

Persona-driven simulation tends to expose more issues than fixed scripts because real users rarely follow one path.

2. Separate conversation testing from audio testing

Use two layers:

Layer A: Dialogue simulation (high volume)

Run millions of turns cheaply using text simulation:

• Did the agent identify intent?
• Did it ask the right questions?
• Did it use tools correctly?
• Did it follow policies?
• Did it reach the desired outcome?

This is where you find most logic bugs.

Layer B: Voice pipeline testing (lower volume, higher realism)

Run fewer but realistic audio calls:

• Speech-to-text errors
• Accents
• Background noise
• Speaking speed
• Interruptions/barge-in
• Silence handling
• Latency
• Text-to-speech quality

A voice agent that works perfectly in text can still fail when audio enters the loop.

3. Generate scenarios from real conversations

The highest-value test cases usually come from:

• Production transcripts
• Support tickets
• Human agent call recordings
• Known incidents
• Compliance requirements

Convert failures into regression tests:

"Caller disputed a charge, refused verification twice, then requested cancellation." Then generate variations:

• polite caller
• angry caller
• confused caller
• caller with missing information
• caller using slang

Some agent testing systems explicitly support turning real conversations into reusable simulations.

4. Use outcome-based grading

Avoid checking exact wording. Grade the result.

Bad:

"Agent must say: 'I understand your concern.'" Better:

• Was the caller's goal achieved?
• Was the correct tool called?
• Were required disclosures made?
• Did the agent avoid prohibited actions?
• Did it escalate appropriately?

For example:

Scenario:
Customer wants to cancel subscription.

Pass criteria:
✓ Identity verified
✓ Cancellation completed
✓ Retention offer followed policy
✓ Confirmation provided
✓ No unauthorized refund promised

Full-conversation evaluation is more useful than judging isolated replies.

5. Add adversarial testing

Have synthetic callers intentionally stress the agent:

• Interrupt repeatedly
• Provide contradictory details
• Ask unrelated questions
• Give partial information
• Try prompt injection attacks
• Use ambiguous language
• Pretend not to understand

These users find the failures that happy-path testing misses.

6. Measure the right metrics

For thousands of simulated calls, track:

Business metrics

• Task completion rate
• Containment rate
• Correct escalation rate
• Tool success rate

Conversation metrics

• Number of turns
• Abandonment rate
• Clarification loops
• Hallucination rate

Voice metrics

• Time to first response
• End-to-end latency
• Interruption handling
• ASR error rate
• Audio quality

Safety/compliance

• Policy violations
• Missing disclosures
• Data leakage

7. Run it continuously

A mature setup looks like CI/CD:

New agent version
        |
        v
Run 10,000 synthetic calls
        |
        v
Score outcomes
        |
        v
Compare against baseline
        |
        v
Block deployment if regression detected

Practical stack design

A common implementation:

• Scenario generator: LLM creates caller personas and variations
• Caller simulator: LLM + optional TTS/audio generation
• Voice gateway: your real telephony stack
• Agent under test: production configuration
• Evaluator: separate model + deterministic checks
• Dashboard: failure clustering and regression tracking

The biggest mistake is using only scripted conversations. The better pattern is scenario → simulated caller → real agent → automated evaluator → regression suite. That gives you thousands of realistic conversations without requiring humans to manually place calls.

If you're building this internally, the key design choice is whether you want to test the conversation brain, the full audio/telephony stack, or both—the architecture changes quite a bit depending on that.

The biggest mistake is using only scripted conversations. The better pattern is scenario → simulated caller → real agent → automated evaluator → regression suite. That gives you thousands of realistic conversations without requiring humans to manually place calls.

Simulating thousands of conversations to test a voice agent requires a shift away from brittle, hardcoded unit tests toward LLM-driven multi-agent simulation . Because voice interactions are non-deterministic and run through a multi-stage pipeline (Telephony → STT → LLM/Core Logic → TTS), you need a decoupled architecture that can run massively parallel, programmatic evaluations.

The industry-standard architecture to achieve this at scale involves four core components:

1. The Architecture Blueprint

• The User Simulator Agent: An LLM-powered agent configured with a specific persona (e.g., an impatient customer, a confused elderly user , or someone in a noisy room ). It reads the voice agent's output and generates the next conversational turn.
• The Agent Under Test (AUT): Your actual voice application, ideally hooked up via a headless API wrapper or a programmatic telephony/SIP client rather than raw audio streams to save compute time and cost.
• The Orchestrator / Runner: A parallel execution engine (using Python asyncio , Celery, or a dedicated testing framework) that spins up hundreds of concurrent simulation threads.
• The LLM Judge (Evaluator): A separate LLM call or deterministic evaluation script that scores the completed conversation transcript against a pass/fail rubric.

2. Step-by-Step Implementation Strategy

1. Define Test Scenarios (The Seed Matrix)
   • Don't just generate random prompts. Create a matrix of 50–100 core intents, edge cases, and adversarial paths (e.g., user interrupts, changes their mind mid-sentence, goes off-topic).
   • Use programmatic parameterization (combinatorial generation) to expand these seeds into thousands of distinct test cases (e.g., mixing 10 user personas × 20 distinct booking dates × 5 background noise profiles/sentiment levels).
2. Decouple the Voice Layer for Bulk Runs
   • Testing thousands of calls via actual PSTN (Public Switched Telephone Network) phone numbers is slow and expensive.
   • Instead, run bulk simulations directly against your agent's text/websocket/chat completion endpoint if the core logic is what you're stress-testing.
   • Reserve actual audio-to-audio pipeline testing (evaluating Speech-to-Text Word Error Rate and Text-to-Speech latency) for a smaller subset (e.g., 5% of your simulated run) using specialized tools.
3. Implement the Evaluation Rubric (LLM-as-a-Judge)
   • After each simulated conversation concludes, pass the full transcript to an evaluator LLM.
   • Grade across quantifiable dimensions:
     • Goal Completion Rate (GCR): Did the user achieve what they set out to do?
     • Constraint Adherence: Did the agent break character, leak system prompts, or offer unauthorized discounts?
     • Interruption Handling / Flow: Did the agent gracefully handle being cut off?
4. Track Latency and Technical Metrics Concurrently
   • While the conversation flows, log infrastructure telemetry: time-to-first-token (TTFT), STT confidence scores, and total turn-taking latency. Track P50 and P95 percentiles rather than averages.

3. Tooling Options

• Dedicated Platforms: Specialized voice evaluation and simulation platforms like Hamming AI or Cekura AI provide end-to-end pipelines, telephony injection, and pre-built metrics for voice QA. Developer tools like Koval or observability platforms like LangWatch offer scenario-testing frameworks explicitly designed for LLM and voice agent simulations.
• Custom Open-Source Build: If building in-house, orchestrate a Python script using asyncio combined with an LLM framework (like LangChain/LlamaIndex or raw instructor-backed OpenAI/Anthropic APIs) to play the dual roles of User Simulator and Judge, pushing results into a data warehouse for batch analysis.

Are you looking to test pure conversational logic and tool-calling via text simulation, or do you need to stress-test the actual audio pipeline (STT/TTS latency and telephony) at scale?

The best way to simulate thousands of conversations for a voice agent is to deploy automated LLM-to-LLM simulation testing combined with real-world audio injection . Instead of relying on slow, expensive manual QA or simple static text testing, you use a secondary "User Simulator LLM" to interact dynamically with your voice agent at scale.

🛠️ The Ideal 4-Step Testing Architecture

To test thousands of conversations efficiently, you must construct a specialized pre-production pipeline:

1. Define the "User Simulator" Personas

Configure a secondary LLM to act as the customer. Do not use generic prompts. Give the simulator highly specific persona guidelines:

• Demographics & Traits: "An impatient, elderly caller who speaks slowly" or "A frustrated customer trying to cancel a subscription".
• Intent & Inputs: Provide the simulator with data parameters to test specific logic nodes (e.g., trying to book an appointment on a date that is already full).

2. Run Headless Multi-Turn Simulations First

Do not stream full voice-to-voice audio for all thousands of initial test runs; the compounding latency of STT → LLM → TTS will make the simulation too slow and expensive.

• Run parallelized text-based API interactions between your simulator and your voice agent.
• Tools like Retell AI (Conductor) or LangWatch Scenario can drive thousands of multi-turn text conversations in parallel within minutes to stress-test your agent's reasoning, prompts, and guardrails.

3. Inject Audio Realism & Telephony Stress (The Voice Stack)

Once your agent handles text logic perfectly, you must test the actual voice pipeline (STT/TTS). An agent that succeeds in pristine text can fail miserably over the phone.

• Use specialized voice-load testing tools like Hamming.ai or Cekura to simulate real calls.
• Programmatically inject real-world variables: background noise, heavy accents, network latency, mid-sentence user interruptions, and telephony compression (8 kHz audio).

4. Automated LLM "Judge" Evaluation

Manually reviewing 10,000 transcripts is impossible. Implement an automated Judge Agent.

• The Judge reviews the final conversation transcript and audio telemetry.
• It automatically scores each call on predefined assertions (e.g., "Did the agent authenticate the user?", "Did it hallucinate a price?", "What was the end-to-end response latency?").

🚀 Leading Platforms for Scale Testing (2026)

If you don't want to build this infrastructure from scratch using custom Python scripts, several platforms specialize exactly in high-volume voice agent simulation:

• Hamming.ai: Simulates thousands of calls in minutes across 65+ languages, altering persona styles and accents.
• Cekura: Excellent for rigorous pre-production regression testing, CI/CD pipeline integration, and high-concurrency voice load testing.
• Retell AI (Conductor): Features a specialized built-in simulator that generates test cases automatically based on your agent’s structure.
• LangWatch: Connects directly into your development workflow to trigger simulations via GitHub actions on every single code commit.

💡 Core Strategy: Surgical Testing over Broad Testing

When scale testing, avoid the temptation to make every simulation a long, open-ended conversation. The most effective method is to deploy "surgical" test scenarios . Create thousands of test cases that focus on a single, isolated decision point (e.g., testing only how the agent handles a specific curse word, or how it acts when a user says "Wait, hold on" mid-sentence). If a test fails, your automated judge can pinpoint exactly which prompt or line of code caused the break.

If you are currently building an agent, tell me a bit more about your tech stack (e.g., Retell, Vapi, Custom WebSockets) and your primary use case (e.g., customer support, booking appointments). I can provide a more tailored framework or script outline for your setup!

The best approach is not to generate 10,000 fixed scripts. Build a synthetic-caller simulation harness that conducts real audio conversations with your agent, varies the caller behavior, and automatically evaluates the outcome.

Current voice-agent testing systems are converging on this architecture: an LLM-driven user simulator generates the caller's behavior, TTS produces actual speech, the voice agent receives it over its normal transport, and the agent's audio is recorded/transcribed and evaluated.

A good architecture

Scenario generator
       │
       ▼
┌──────────────────┐
│ Synthetic caller │
│ goal + persona   │
│ behavior + state │
└────────┬─────────┘
         │ text
         ▼
        TTS
         │ audio
         ▼
┌──────────────────┐
│   Voice Agent    │
│ STT → LLM → TTS  │
└────────┬─────────┘
         │ audio
         ▼
       STT
         │
         ▼
┌──────────────────┐
│ Evaluation layer │
│ outcome + audio  │
│ latency + tools  │
└──────────────────┘

The important part is that the simulator talks to the agent through the same audio path as a real caller. Testing only with transcripts won't expose things like STT mistakes, barge-ins, silence thresholds, awkward turn-taking, or TTS latency.

1. Generate scenarios, not conversations

Define scenarios at a higher level:

goal: "Change a flight reservation"
persona:
  patience: low
  familiarity: novice
  speaking_style: rushed
  emotional_state: frustrated

constraints:
  has_booking: true
  flight_is_changeable: true

behaviors:
  - interrupt_agent
  - provide incomplete_information
  - change_topic_once
  - ask_for_confirmation

success:
  - correct_flight_changed
  - correct_fee_explained
  - no_unauthorized_action

Then generate hundreds/thousands of variations from those dimensions.

I'd create dimensions for:

• Intent: the thing they're trying to accomplish
• Persona: patient, confused, angry, elderly, expert, terse, chatty
• Information quality: complete, incomplete, contradictory, incorrect
• Conversation behavior: interruptions, corrections, topic changes, silence
• Speech characteristics: fast/slow, accents, background noise, disfluencies
• Emotional state: calm → frustrated → angry
• Tool/data state: successful API, timeout, stale data, missing record
• Adversarial behavior: prompt injection, attempts to bypass policy, social engineering

This gives you combinatorial coverage without manually writing thousands of scripts. Microsoft similarly recommends generating simulation seeds/scenarios rather than authoring every conversation by hand.

2. Make the simulated user stateful

This is probably the most important design decision.

Don't use:

"Generate the next user message." Use:

"You are a caller with goal X. Maintain this private state. React naturally to what the agent says. Do not reveal your goal unless it would be natural. Stop when your goal is satisfied or you decide the agent has failed." The simulator should maintain things like:

Goal: cancel reservation
Known information:
  reservation_id = 83921
  cancellation_reason = "price"

Hidden behavior:
  impatient = true
  will interrupt after long pauses
  will accept alternative only if savings > $50

Termination:
  success / failure / abandonment

That produces much more realistic conversations than a scripted dialogue tree.

3. Validate the simulator before scoring the agent

This is an easy thing to overlook.

Suppose the caller simulator was supposed to say that it had a valid reservation, but it accidentally claims it doesn't have one. If your agent then fails, you don't have an agent failure—you have a bad test.

Modern voice-eval frameworks explicitly validate simulated conversations and regenerate invalid ones before scoring them.

So have a validator ask:

Did the simulated caller:
✓ maintain its assigned state?
✓ pursue the assigned goal?
✓ obey the scenario constraints?
✓ respond coherently to the agent?
✓ terminate for a valid reason?

Only then score the agent.

4. Use several kinds of evaluators

Don't reduce the result to one "LLM score."

I'd track at least:

Voice-specific evaluation needs to include interaction experience—not merely whether the final textual answer was correct. EVA, for example, separates task accuracy from conversational experience such as progression, conciseness and turn-taking.

5. Combine deterministic checks with LLM judges

Use deterministic assertions wherever possible:

assert booking.status == "cancelled"
assert refund.amount == expected_refund
assert tool_calls.count("cancel_booking") == 1
assert unauthorized_tools == []

Then use an LLM judge for things that are inherently fuzzy:

Was the agent's explanation understandable?
Did it acknowledge the user's frustration?
Did it unnecessarily repeat information?
Did the conversation feel natural?
Did the agent appropriately handle the interruption?

And periodically compare your automated judges against human judgments. Recent research specifically finds that LLM-judge reliability and calibration are important issues for voice-agent evaluation.

6. Don't run 10,000 conversations identically

I'd use a distribution such as:

                 10,000 simulations

Normal cases       4,000  ████████████████████
Edge cases         2,000  ██████████
Adversarial        1,500  ███████
Noisy/voice        1,000  █████
Tool failures        750  ████
Rare scenarios       500  ██
Regression suite     250  █

And seed the randomness so a failed test can be reproduced.

Every conversation should produce an immutable test record:

{
  "scenario_id": "cancel_042",
  "seed": 193847,
  "agent_version": "2026-09-07.14",
  "user_persona": "impatient",
  "outcome": "failure",
  "failure_type": "wrong_refund",
  "latency_p95_ms": 1840,
  "tool_errors": [],
  "audio": "...",
  "transcript": "...",
  "trace": "..."
}

7. Make the simulation progressively more realistic

I'd build the system in three levels:

Level 1 — Cheap: Thousands of text simulations to test agent logic and tool behavior.

Level 2 — Audio: Hundreds/thousands of TTS → agent → STT conversations to test the actual voice pipeline.

Level 3 — Realistic stress: Audio + interruptions + background noise + varied speaking rates + latency + API failures + concurrent calls.

This saves enormous amounts of money. You don't need to spend GPU/TTS/telephony resources testing a simple business-rule regression.

Tools such as LiveKit's agent testing and DeepEval's voice simulation already follow variants of this model, including full multi-turn simulated conversations and actual audio capture.

8. Turn it into CI

The killer feature is making every agent change trigger:

git push
   ↓
500 cheap simulations
   ↓
Regression suite
   ↓
100 audio simulations
   ↓
Evaluation
   ↓
Compare against previous version
   ↓
FAIL PR if:
   task success ↓ > 2%
   policy violations ↑
   P95 latency ↑ > 15%
   critical regression detected

Then run the 10k+ overnight suite rather than blocking every developer commit.

What I'd build

If you're building this yourself, my preferred stack would be:

• Scenario generator: LLM + structured scenario schema
• User simulator: strong LLM with persistent hidden state
• TTS: multiple voices rather than one synthetic voice
• Transport: the actual WebSocket/SIP/WebRTC interface your production agent uses
• Agent: completely unchanged production stack
• STT: production STT, not a perfect transcript
• Trace collector: audio + transcript + timestamps + tool calls + state
• Validators: deterministic Python checks
• LLM judge: separate model from the agent where practical
• Human calibration set: ~100–500 conversations continually re-scored by humans
• Dashboard: failure clustering + regression comparison

There are now open-source frameworks specifically aimed at this bot-to-bot audio architecture; EVA is one notable example, while DeepEval and LiveKit provide simulation/testing infrastructure you can adapt rather than building everything from scratch.

The key insight: don't think of this as "generate 10,000 conversations." Think of it as generate 10,000 independent test environments with goals, users, world states, and failure modes—and let the conversations emerge from those conditions. That's what makes the results useful rather than 10,000 variations of the same happy-path demo.

Use:

"You are a caller with goal X. Maintain this private state. React naturally to what the agent says. Do not reveal your goal unless it would be natural. Stop when your goal is satisfied or you decide the agent has failed." The simulator should maintain things like:

Goal: cancel reservation
Known information:
  reservation_id = 83921
  cancellation_reason = "price"

Hidden behavior:
  impatient = true
  will interrupt after long pauses
  will accept alternative only if savings > $50

Termination:
  success / failure / abandonment

That produces much more realistic conversations than a scripted dialogue tree.

3. Validate the simulator before scoring the agent

This is an easy thing to overlook.

Suppose the caller simulator was supposed to say that it had a valid reservation, but it accidentally claims it doesn't have one. If your agent then fails, you don't have an agent failure—you have a bad test.

Modern voice-eval frameworks explicitly validate simulated conversations and regenerate invalid ones before scoring them.

So have a validator ask:

Did the simulated caller:
✓ maintain its assigned state?
✓ pursue the assigned goal?
✓ obey the scenario constraints?
✓ respond coherently to the agent?
✓ terminate for a valid reason?

Only then score the agent.

4. Use several kinds of evaluators

Don't reduce the result to one "LLM score."

I'd track at least:

Voice-specific evaluation needs to include interaction experience—not merely whether the final textual answer was correct. EVA, for example, separates task accuracy from conversational experience such as progression, conciseness and turn-taking.

5. Combine deterministic checks with LLM judges

Use deterministic assertions wherever possible:

assert booking.status == "cancelled"
assert refund.amount == expected_refund
assert tool_calls.count("cancel_booking") == 1
assert unauthorized_tools == []

Then use an LLM judge for things that are inherently fuzzy:

Was the agent's explanation understandable?
Did it acknowledge the user's frustration?
Did it unnecessarily repeat information?
Did the conversation feel natural?
Did the agent appropriately handle the interruption?

And periodically compare your automated judges against human judgments. Recent research specifically finds that LLM-judge reliability and calibration are important issues for voice-agent evaluation.

6. Don't run 10,000 conversations identically

I'd use a distribution such as:

                 10,000 simulations

Normal cases       4,000  ████████████████████
Edge cases         2,000  ██████████
Adversarial        1,500  ███████
Noisy/voice        1,000  █████
Tool failures        750  ████
Rare scenarios       500  ██
Regression suite     250  █

And seed the randomness so a failed test can be reproduced.

Every conversation should produce an immutable test record:

{
  "scenario_id": "cancel_042",
  "seed": 193847,
  "agent_version": "2026-09-07.14",
  "user_persona": "impatient",
  "outcome": "failure",
  "failure_type": "wrong_refund",
  "latency_p95_ms": 1840,
  "tool_errors": [],
  "audio": "...",
  "transcript": "...",
  "trace": "..."
}

7. Make the simulation progressively more realistic

I'd build the system in three levels:

Level 1 — Cheap: Thousands of text simulations to test agent logic and tool behavior.

Level 2 — Audio: Hundreds/thousands of TTS → agent → STT conversations to test the actual voice pipeline.

Level 3 — Realistic stress: Audio + interruptions + background noise + varied speaking rates + latency + API failures + concurrent calls.

This saves enormous amounts of money. You don't need to spend GPU/TTS/telephony resources testing a simple business-rule regression.

Tools such as LiveKit's agent testing and DeepEval's voice simulation already follow variants of this model, including full multi-turn simulated conversations and actual audio capture.

8. Turn it into CI

The killer feature is making every agent change trigger:

git push
   ↓
500 cheap simulations
   ↓
Regression suite
   ↓
100 audio simulations
   ↓
Evaluation
   ↓
Compare against previous version
   ↓
FAIL PR if:
   task success ↓ > 2%
   policy violations ↑
   P95 latency ↑ > 15%
   critical regression detected

Then run the 10k+ overnight suite rather than blocking every developer commit.

What I'd build

If you're building this yourself, my preferred stack would be:

This is probably the most important design decision.

Don't use:

"Generate the next user message." Use:

The best way to simulate thousands of conversations for a voice agent is to deploy automated LLM-to-LLM simulation testing combined with real-world audio injection . Instead of relying on slow, expensive manual QA or simple static text testing, you use a secondary "User Simulator LLM" to interact dynamically with your voice agent at scale.

🛠️ The Ideal 4-Step Testing Architecture

To test thousands of conversations efficiently, you must construct a specialized pre-production pipeline:

1. Define the "User Simulator" Personas

Configure a secondary LLM to act as the customer. Do not use generic prompts. Give the simulator highly specific persona guidelines:

• Demographics & Traits: "An impatient, elderly caller who speaks slowly" or "A frustrated customer trying to cancel a subscription".
• Intent & Inputs: Provide the simulator with data parameters to test specific logic nodes (e.g., trying to book an appointment on a date that is already full).

2. Run Headless Multi-Turn Simulations First

Do not stream full voice-to-voice audio for all thousands of initial test runs; the compounding latency of STT → LLM → TTS will make the simulation too slow and expensive.

• Run parallelized text-based API interactions between your simulator and your voice agent.
• Tools like Retell AI (Conductor) or LangWatch Scenario can drive thousands of multi-turn text conversations in parallel within minutes to stress-test your agent's reasoning, prompts, and guardrails.

3. Inject Audio Realism & Telephony Stress (The Voice Stack)

Once your agent handles text logic perfectly, you must test the actual voice pipeline (STT/TTS). An agent that succeeds in pristine text can fail miserably over the phone.

• Use specialized voice-load testing tools like Hamming.ai or Cekura to simulate real calls.
• Programmatically inject real-world variables: background noise, heavy accents, network latency, mid-sentence user interruptions, and telephony compression (8 kHz audio).

4. Automated LLM "Judge" Evaluation

Manually reviewing 10,000 transcripts is impossible. Implement an automated Judge Agent.

• The Judge reviews the final conversation transcript and audio telemetry.
• It automatically scores each call on predefined assertions (e.g., "Did the agent authenticate the user?", "Did it hallucinate a price?", "What was the end-to-end response latency?").

🚀 Leading Platforms for Scale Testing (2026)

If you don't want to build this infrastructure from scratch using custom Python scripts, several platforms specialize exactly in high-volume voice agent simulation:

• Hamming.ai: Simulates thousands of calls in minutes across 65+ languages, altering persona styles and accents.
• Cekura: Excellent for rigorous pre-production regression testing, CI/CD pipeline integration, and high-concurrency voice load testing.
• Retell AI (Conductor): Features a specialized built-in simulator that generates test cases automatically based on your agent’s structure.
• LangWatch: Connects directly into your development workflow to trigger simulations via GitHub actions on every single code commit.

💡 Core Strategy: Surgical Testing over Broad Testing

When scale testing, avoid the temptation to make every simulation a long, open-ended conversation. The most effective method is to deploy "surgical" test scenarios . Create thousands of test cases that focus on a single, isolated decision point (e.g., testing only how the agent handles a specific curse word, or how it acts when a user says "Wait, hold on" mid-sentence). If a test fails, your automated judge can pinpoint exactly which prompt or line of code caused the break.

If you are currently building an agent, tell me a bit more about your tech stack (e.g., Retell, Vapi, Custom WebSockets) and your primary use case (e.g., customer support, booking appointments). I can provide a more tailored framework or script outline for your setup!

The best approach is to build a synthetic-caller simulation harness that exercises your actual voice stack end-to-end, rather than generating thousands of text transcripts.

A good architecture looks like this:

Scenario generator
      ↓
Synthetic caller / persona
      ↓
   Real-time audio
      ↓
Your actual voice agent
(STT → LLM → tools → TTS)
      ↓
Conversation trace + audio
      ↓
Independent evaluator
      ↓
Metrics / failures / regression report

This is increasingly the standard approach because voice failures often come from things text-only tests miss: interruptions, silence, accents, background noise, latency, turn-taking, and tool failures.

1. Don't generate 10,000 random conversations

Instead, create a scenario × persona × behavior matrix.

For example:

• Scenarios: book appointment, cancel appointment, change address, request refund
• Personas: cooperative, confused, impatient, elderly, highly verbose, technically sophisticated
• Speech: fast, slow, mumbling, accent variation
• Conversation behavior: interrupt agent, change mind, answer partially, go off-topic, remain silent
• Environment: quiet room, car, restaurant, bad phone connection
• Adversarial: contradictory information, repeated requests, attempts to bypass policy

You can then generate thousands of combinations from perhaps a few hundred carefully designed scenarios.

2. Give the simulated caller a goal, not a script

This is important.

Instead of:

Say "I'd like to cancel my appointment." Give the simulator:

goal:
  primary: cancel appointment
  appointment: March 14 at 3pm

persona:
  mood: mildly frustrated
  verbosity: high

constraints:
  - Don't volunteer the appointment ID unless asked.
  - If the agent asks an unnecessary question, complain.
  - If the agent misunderstands you, correct it once.
  - If cancellation succeeds, end the call.

The caller LLM should decide what to say next based on the agent's actual response.

That produces much more useful conversations than replaying predetermined scripts. Recent voice-agent benchmarks similarly emphasize dynamic multi-turn simulated callers rather than fixed dialogues.

3. Test the real audio path

If your production system is:

microphone → telephony → VAD → STT → LLM → tools → TTS → telephony

your simulator should ideally hit that same path.

Don't evaluate only:

LLM prompt → text response

because you'll miss things such as:

• barge-in
• endpointing/VAD errors
• long pauses
• TTS latency
• STT mistakes
• overlapping speech
• telephone compression
• background noise
• accents

Realistic audio and turn-taking are particularly important for voice systems.

4. Separate the simulator from the judge

Don't have the same model decide:

"Did I, the caller, behave correctly and did the agent succeed?" Use an independent evaluator.

For every call, collect structured results such as:

{
  "task_completed": true,
  "policy_violation": false,
  "hallucination": false,
  "correct_tool_usage": true,
  "unnecessary_escalation": false,
  "turns": 17,
  "latency_p95_ms": 1240,
  "caller_hangup": false
}

And ideally evaluate against ground truth state, not merely the transcript.

For example, if the agent says:

"I've cancelled your appointment." the evaluator should check the backend:

appointment.status == "cancelled"

rather than trusting what the agent said. This kind of trace-level evaluation catches agents that sound successful while failing to perform the underlying action.

5. Use three layers of evaluation

I'd use:

Deterministic checks

Best for things that must be objectively true:

• Did the correct API get called?
• Were the correct arguments used?
• Was the appointment actually cancelled?
• Was a required disclosure made?
• Was P95 latency < X?
• Did the agent transfer when required?

LLM judge

Useful for:

• Did the agent understand the caller?
• Was the response appropriate?
• Did it maintain context?
• Did it recover from misunderstanding?
• Was the conversation natural?

Human review

Don't manually review thousands of calls.

Instead, review:

• failures
• borderline scores
• new failure clusters
• randomly sampled passes
• high-risk scenarios

This hybrid approach is recommended in current voice-agent evaluation work.

6. Make every failure reproducible

This is one of the most important engineering details.

Store:

test_id
scenario_id
persona_id
random_seed
agent_version
prompt_version
model_version
STT version
TTS version
tool/backend version
full audio
transcript
tool trace
evaluation results

Then when test #7,438 fails, you can run:

replay 7438

and reproduce essentially the same interaction.

Otherwise "we had a weird failure somewhere in 10,000 calls" isn't very useful.

7. Run it like a regression test

For every agent change:

                    ┌─ critical scenarios
New agent version ──┼─ broad regression set
                    └─ adversarial/random set
                              ↓
                         1,000–10,000 calls
                              ↓
                       compare to baseline
                              ↓
                    PASS / FAIL / investigate

I'd maintain three suites:

• Smoke: ~50–100 calls, every commit
• Regression: ~500–2,000 calls, every release
• Stress: 10,000+ calls, periodically or before major launches

The exact numbers depend on cost and complexity; what's more important is that the scenario library grows whenever production exposes a new failure.

8. Separately test concurrency

There are actually two different questions:

Does one conversation work? and

Does the system still work when 2,000 conversations happen simultaneously? Don't confuse them.

For the second test, progressively increase concurrency:

10 → 50 → 100 → 250 → 500 → 1,000 → 2,000

Track:

• P50/P95/P99 response latency
• dropped calls
• STT failures
• TTS failures
• WebSocket exhaustion
• tool/API errors
• CPU/memory
• provider rate limits
• task-completion degradation

A voice agent can have excellent single-call performance while falling apart under concurrency.

The stack I'd build

If you're implementing this yourself, I'd make the core abstraction something like:

scenario = Scenario(
    goal="cancel appointment",
    initial_state=backend_state,
    persona=persona,
    behaviors=behaviors,
)

result = await simulator.run(
    caller=scenario,
    agent=production_voice_agent,
)

evaluation = await evaluator.score(
    scenario=scenario,
    result=result,
    final_state=backend.snapshot(),
)

Then run thousands of these asynchronously with a queue/worker architecture.

The key insight: don't optimize for "10,000 conversations." Optimize for 10,000 independently generated attempts to break the agent, with every failure tied to ground truth and reproducible evidence.

If you're building this today, I'd strongly favor synthetic callers + real audio + stateful backend + independent evaluator + parallel execution. That's substantially more powerful than scripted calls or transcript-only LLM evals.

For the second test, progressively increase concurrency:

10 → 50 → 100 → 250 → 500 → 1,000 → 2,000

Track:

• P50/P95/P99 response latency
• dropped calls
• STT failures
• TTS failures
• WebSocket exhaustion
• tool/API errors
• CPU/memory
• provider rate limits
• task-completion degradation

A voice agent can have excellent single-call performance while falling apart under concurrency.

The stack I'd build

If you're implementing this yourself, I'd make the core abstraction something like:

Does one conversation work? and

Simulating thousands of conversations to test a voice agent requires moving away from manual QA and adopting an automated, agent-to-agent simulation pipeline . Borrowing a page from autonomous vehicle testing, the modern approach uses a "Simulated User" powered by an LLM to talk to your "Agent Under Test" at scale, supervised by an automated "Judge".

The standard architecture and step-by-step framework to execute this effectively involve several key components:

1. The Core Architecture

• The User Simulator: An LLM agent prompted with specific personas (e.g., angry customer, hurried caller, non-native speaker, elderly user) and given a specific goal (e.g., "book a flight to Chicago").
• The Agent Under Test (AUT): Your actual voice pipeline (integrating orchestration layers like LiveKit, Pipecat, Vapi, or Retell).
• The Telephony/Audio Bridge: A programmatic way to route audio or text streams. For pure logic/latency testing, text-in/text-out or direct API websocket mocking is cheapest. For full pipeline testing, you need programmatic SIP/WebRTC injection.
• The LLM Judge (Evaluator): An independent evaluation model that reviews the transcript/audio post-call against pass/fail criteria (e.g., Did it stick to the refund policy? Did it interrupt politely? Was latency acceptable?).

2. Step-by-Step Implementation Strategy

   • Happy Path: Direct, cooperative user completing a transaction.

   • Adversarial/Edge Case: User changes their mind mid-sentence, speaks in fragments, or goes off-topic.

   • Acoustic Variance: Inject background noise profiles or different accents into the simulation configuration.

   • Text-level Simulation (Fast & Cheap): Strip the STT (Speech-to-Text) and TTS (Text-to-Speech) layers and test the core conversational logic via text APIs. This lets you run 10,000 simulations in minutes for pennies to catch logical loops, prompt injections, or goal failures.

   • End-to-End Audio Simulation (Realistic & Costly): Route synthetic audio through text-to-speech generators for the user simulator, hitting your actual telephony endpoints. Use platforms designed for this scale.

   • Explore dedicated platforms like Cekura AI or Roark which specialize in running thousands of persona-driven voice simulations, handling telephony, and tracking aggregate success metrics.

   • Utilize evaluation and scenario orchestration libraries like the LangWatch Scenario framework to code custom multi-turn audio-to-audio agent tests.

   • Whenever a live user call fails in production, convert that exact conversation path into a programmatic test case.

   • Add it to your nightly batch simulation suite so that bug is permanently locked out from regressing.

To help narrow this down, are you looking to test pure conversational logic/prompts (text-based) at high volume, or do you need full audio-to-audio pipeline testing including latency and telephony integration?

The best approach is LLM-driven user simulation + real audio + deterministic/LLM evaluation, rather than simply generating thousands of text transcripts.

A good architecture looks like this:

Scenario generator
      ↓
Synthetic caller personas
      ↓
   TTS audio
      ↓
┌─────────────────┐
│   Voice Agent   │
│ STT → LLM → TTS │
└─────────────────┘
      ↓
Transcript + audio + tool calls + timings
      ↓
     Graders
      ↓
Pass/fail + metrics + failure clusters

1. Start with scenarios, not conversations

Create perhaps 100–500 scenario templates, each specifying:

• User goal
• Relevant facts/data
• Expected successful outcome
• Constraints/policies
• Persona
• Possible failure/edge cases

For example:

Goal: Reschedule a doctor's appointment Persona: Busy, impatient caller Facts: Appointment is Tuesday at 3 PM Behavior: Interrupts twice, initially gives the wrong date, asks an unrelated question Success: Appointment moved to an available slot and confirmation provided Then generate 10–100 variations of each scenario. Conversation simulation is increasingly being used specifically this way: scenario descriptions become simulated multi-turn conversations, allowing large-scale and repeatable testing.

2. Make the simulated caller non-cooperative

This is probably the most important part.

Don't tell the simulator:

"Have a conversation that successfully completes the task." Instead give it an objective and behavioral characteristics:

• impatient
• confused
• verbose
• terse
• forgetful
• skeptical
• changes their mind
• interrupts
• talks over the agent
• gives information out of order
• misunderstands questions
• asks unrelated questions
• refuses to answer
• uses ambiguous language

Research on synthetic users suggests that naive simulators can be substantially more cooperative and predictable than real users, producing overly optimistic evaluations.

3. Actually test the voice layer

Don't run everything as text if the product is a voice agent.

For a representative subset—or ideally the entire suite—run:

simulated user → TTS → phone/WebRTC → agent → audio → simulated user

Vary:

• speaking rate
• pauses
• accent/dialect
• disfluencies ("uh", "um")
• pronunciation
• background noise
• barge-ins
• overlapping speech
• long pauses
• ASR errors
• network/latency conditions

This catches failures that a transcript-only simulator completely misses. Recent voice-agent evaluation work specifically emphasizes bot-to-bot audio conversations and validation of the simulated user's behavior.

4. Run multiple stochastic trials

Don't run one conversation per scenario.

If you have:

• 300 scenarios
• 10 personas
• 10 repetitions

that's 30,000 conversations.

The repetitions matter because the agent itself is stochastic. You want:

"This scenario succeeds 97% of the time" rather than:

"This scenario worked."

5. Score outcomes, not just "conversation quality"

I'd use several independent graders.

Hard/deterministic checks

• Did the correct tool get called?
• Were required fields collected?
• Was the appointment actually changed?
• Was an unauthorized action taken?
• Did the call terminate correctly?
• Did latency exceed your threshold?
• Did the agent violate a policy?

LLM graders

• Did the agent understand the user's goal?
• Was the response appropriate?
• Did it recover from confusion?
• Did it unnecessarily repeat itself?
• Was the escalation appropriate?
• Did it maintain context?

Voice metrics

• time to first response
• interruption rate
• barge-in handling
• turn latency
• ASR error rate
• silence duration
• talk/listen ratio

Then produce something like:

30,000 conversations

Task success          96.8%
Correct escalation    98.1%
Policy compliance     99.4%
Tool correctness      97.9%
Median latency        720 ms
P95 latency           1.8 s

Failures:
  412  missed user intent
  287  interruption failures
  193  incorrect tool arguments
  106  context loss
   74  inappropriate escalation

6. Turn every production failure into a regression test

This is where the system becomes genuinely powerful.

Production call
      ↓
Failure detected
      ↓
Anonymize + classify
      ↓
Convert into scenario
      ↓
Generate 20–100 variants
      ↓
Run against new agent
      ↓
Compare versions

That gives you a continuously growing test suite instead of repeatedly inventing hypothetical tests. OpenAI describes a related deployment-simulation approach in which realistic historical conversations are replayed against candidate systems to make pre-deployment evaluation more representative of actual traffic.

7. Use a pyramid rather than making every test a full phone call

I'd structure a 30,000-test run approximately like:

That gives you scale without making every iteration expensive.

What I'd build

If you're building this yourself, I'd make the core test case a structured object:

{
  "scenario": "reschedule_appointment",
  "goal": "Move appointment to another available time",
  "persona": {
    "patience": 0.2,
    "verbosity": 0.7,
    "interruptions": 0.4,
    "confusion": 0.3
  },
  "facts": {
    "current_time": "Tuesday 3 PM"
  },
  "constraints": [
    "Do not cancel without confirmation",
    "Must verify identity"
  ],
  "success_criteria": [
    "identity_verified",
    "new_slot_booked",
    "confirmation_given"
  ]
}

Then your simulator can generate thousands of distinct callers from the same underlying scenario, while your evaluator checks the same objective ground truth.

If I were starting today, I'd prioritize this order:

1. Scenario library
2. LLM user simulator with diverse personas
3. 10–100 stochastic runs per scenario
4. Deterministic outcome graders
5. Audio/interruptions/latency testing
6. Production failures automatically becoming regression scenarios
7. Human review of a statistically meaningful sample of simulator failures

The key insight is that "10,000 synthetic conversations" isn't inherently useful. Ten thousand nearly identical, cooperative LLM callers can give you a beautifully misleading 99.8% success rate. The value comes from distributional coverage, behavioral realism, audio realism, and reliable grading. Recent research specifically finds a gap between synthetic and real-user behavior, so I'd validate your simulator against a sample of real conversations before trusting its scores.

If you tell me what your stack is (e.g. Vapi, Retell, LiveKit, Twilio, ElevenLabs, custom WebRTC), I can sketch the exact architecture and a 10k-conversation test harness.