How do I benchmark whether my customer-support agent actually resolves tickets?

3. How to Set and Compare Benchmarks

• Establish a Baseline: Track the agent's performance over their first 30–60 days or take a 90-day team average to understand your current operational baseline.
• Segment by Complexity: Do not benchmark a Tier 1 generalist against a Tier 2 technical specialist using the same resolution time metrics. Group tickets by tier, category, or tagging difficulty.
• Compare Against Team Medians: Look at the median performance of the entire support team rather than just the top performer, which can skew expectations.

If you'd like, let me know:

• What help desk software you use (e.g., Zendesk, Intercom, Salesforce)
• What tier of support these agents handle (Tier 1 vs. technical escalation)

I can help you set up a specific formula or dashboard structure to track these metrics.

2. Track a small “resolution scorecard”

I'd measure these together:

Reopen rate is particularly important because a closed ticket isn't necessarily a resolved ticket.SurveyMonkey+1

3. Build a labeled evaluation set

Don't benchmark only against aggregate production metrics. Create, say, 500–1,000 historical tickets covering your major categories:

• Password/account issues
• Billing/refunds
• Order/shipping issues
• Product questions
• Technical problems
• Cancellation requests
• Edge cases / ambiguous requests

Have a human reviewer establish the ground-truth outcome for each ticket:

resolved / partially resolved / unresolved / should have escalated

Then run your agent against the same tickets under controlled conditions.

This gives you a much cleaner number:

Resolution accuracy = correctly resolved tickets / tickets the agent attempted

You can separately measure whether its escalations were appropriate.

4. Test for “fake resolution”

This is where many support agents look better than they actually are.

For every supposedly resolved ticket, ask:

1. Did the agent understand the customer's actual goal?
2. Did it have the authority/data required to act?
3. Did it actually perform the required action?
4. Was the action correct?
5. Would the customer reasonably consider the problem finished?
6. Would they need to contact support again?

For example:

Customer: “My package says delivered but I don't have it.” Bad resolution: “Your tracking says the package was delivered. Please check with your neighbors.”

Potentially genuine resolution: The agent verifies delivery details, follows the appropriate lost-package procedure, creates the claim/replacement/refund, and tells the customer exactly what happens next.

The difference is outcome vs. response quality.

5. Measure delayed failures

Don't immediately declare success when the agent sends “resolved.”

Use a follow-up window appropriate to your business—for example, 72 hours for straightforward support, and potentially longer for issues where customers naturally take more time to discover whether something worked. Different sources recommend different windows, which is why consistency within your own benchmark matters more than blindly adopting an industry number.OpenAgent+1

A useful production calculation is:

True resolution rate = resolved tickets with no same-issue repeat contact ÷ eligible tickets

Also separately track customers who never respond again. Silence isn't proof of resolution; some customers simply give up.The Experience Desk

6. Segment the benchmark

Don't let one blended number hide failures.

For example:

This tells you where the agent actually works rather than merely whether its overall number looks good.

And don't compare an agent handling mostly password resets with one handling API failures—the attainable FCR is inherently different by complexity.Pylon

7. Establish a human baseline

The most useful benchmark isn't necessarily “industry average.”

Run the same evaluation set through:

• Your AI agent
• Your existing human support process
• Ideally, your best-performing human agents

Then compare:

AI vs. human resolution rate AI vs. human repeat-contact rate AI vs. human escalation rate AI vs. human time-to-resolution AI vs. human error rate

That tells you whether the agent is actually replacing support work or simply producing convincing-looking replies.

A particularly useful north-star metric

I'd ultimately put this on the dashboard:

% of customer issues that the agent resolves correctly without requiring another customer contact or human intervention. Then pair it with customer-confirmed resolution and repeat-contact rate.

That combination is much harder to game than “tickets closed,” response time, or even FCR alone.CorePiper+1

If you tell me what kind of support agent you're benchmarking (e-commerce, SaaS, fintech, etc.), I can give you a concrete evaluation rubric + SQL/metrics schema + 100-point scoring methodology for it.

Don't let one blended number hide failures.

For example:

This tells you where the agent actually works rather than merely whether its overall number looks good.

And don't compare an agent handling mostly password resets with one handling API failures—the attainable FCR is inherently different by complexity.

7. Establish a human baseline

The most useful benchmark isn't necessarily “industry average.”

Run the same evaluation set through:

1. Define “resolved” operationally

For each ticket, label it Resolved = Yes only if:

• The customer's underlying issue was actually addressed.
• Any required action was completed—not merely explained.
• The answer/action was correct.
• The customer didn't need to contact support again for the same issue within a defined window.
• It wasn't silently escalated or abandoned.

For example, “Here's how to request a refund” is not resolved if the customer's request was “Please refund my order” and the agent had the ability to do it.

This distinction is particularly important for agentic systems that can call APIs and modify customer records.Intercom+1

2. Build a gold-standard test set

Take perhaps 500–2,000 historical tickets and stratify them by:

• Intent/type
• Complexity
• Customer segment
• Channel
• Language
• Severity
• Whether an external action is required
• Whether the ticket historically required escalation

Have experienced support people independently label each ticket:

Outcome

• Resolved
• Partially resolved
• Not resolved
• Should escalate

Quality

• Correct
• Incorrect
• Missing important information
• Policy violation
• Wrong action

Customer experience

• Clear
• Appropriate tone
• Excessive effort
• Created additional work

Don't make the benchmark disproportionately easy. Real production queues contain ambiguous, messy and emotionally charged cases; evaluating only curated happy paths can give misleading results.Intercom+1

3. Measure these metrics

I'd use a scorecard like this:

FCR is conventionally calculated as first-contact resolutions divided by total issues, while modern AI-support measurement increasingly emphasizes resolution, accuracy, CSAT, customer effort and cost together.Intercom+1

4. Add a “did it really resolve?” follow-up

This is probably the most valuable measurement.

For a random sample of supposedly resolved tickets, look 7–14 days downstream and ask:

Did this customer contact us again about the same underlying issue? Then calculate:

Durable resolution rate = resolutions with no related repeat contact ÷ purported resolutions

You can also detect repeats algorithmically using customer ID + intent/topic similarity.

This catches the classic failure mode:

Agent gives plausible answer → ticket closes → customer comes back three days later → support solves it manually. Your ticketing system might call the first interaction a success. Your customer doesn't.

5. Separate “answer quality” from “resolution”

An agent can be factually accurate while failing the customer.

For example:

Customer: “Cancel my subscription and refund the last charge.” Possible outcomes:

• A: Explains cancellation policy → not resolved.
• B: Cancels subscription but doesn't refund → partial.
• C: Correctly cancels + refunds → resolved.
• D: Says it was cancelled/refunded, but backend state didn't change → incorrect/not resolved.

This is why I would not use LLM-judged answer quality as your primary KPI. Use observable outcomes wherever possible: database state, transaction success, repeat contact, escalation, etc.

6. Compare against humans

Run the exact same benchmark against:

• Your current human-support process
• Your agent
• Optionally, a simpler automation/bot baseline

Then compare by ticket type, not just overall.

For example:

The aggregate number can hide important failures if your agent gets disproportionately easy tickets.

7. Use production experiments

Once offline evaluation looks good, run a controlled deployment.

For example:

• 10% of eligible tickets → agent
• 10% → human/control
• Keep the remaining traffic unchanged.

Measure incremental true resolution, not just agent automation.

A useful top-line metric is:

Autonomous resolution rate = tickets genuinely resolved by the agent / all eligible tickets

This differs from “the agent answered” or “the ticket didn't reach a human.” Intercom, for example, explicitly distinguishes automation rate from involvement and resolution rates.Intercom

8. Set up an evaluation flywheel

I'd run this continuously:

Historical benchmark → offline eval → limited production → outcome measurement → failure analysis → benchmark update

Every serious failure becomes a new regression test.

For example:

Agent incorrectly refunds annual subscription → add that scenario + variants to benchmark → fix workflow → verify old cases still pass. That eventually gives you a living support-agent benchmark, rather than a one-time demo score.

A practical north-star dashboard

If I were building this from scratch, I'd put these six numbers at the top:

1. True resolution rate
2. Durable resolution rate
3. Correct resolution rate
4. Repeat-contact rate
5. Escalation rate
6. Cost per durable resolution

Then segment all six by intent, complexity, channel and customer segment.

The most important principle is: don't let “closed ticket,” “no human involved,” “positive LLM evaluation,” or “customer didn't reply” substitute for evidence that the customer's problem was actually solved.Zendesk+1

I'd run this continuously:

Historical benchmark → offline eval → limited production → outcome measurement → failure analysis → benchmark update

Every serious failure becomes a new regression test.

For example:

Agent incorrectly refunds annual subscription → add that scenario + variants to benchmark → fix workflow → verify old cases still pass. That eventually gives you a living support-agent benchmark, rather than a one-time demo score.

A practical north-star dashboard

If I were building this from scratch, I'd put these six numbers at the top:

1. True resolution rate
2. Durable resolution rate
3. Correct resolution rate
4. Repeat-contact rate
5. Escalation rate
6. Cost per durable resolution

Then segment all six by intent, complexity, channel and customer segment.

For example:

The aggregate number can hide important failures if your agent gets disproportionately easy tickets.

7. Use production experiments

Once offline evaluation looks good, run a controlled deployment.

For example:

• 10% of eligible tickets → agent
• 10% → human/control
• Keep the remaining traffic unchanged.

Measure incremental true resolution, not just agent automation.

A useful top-line metric is:

Autonomous resolution rate = tickets genuinely resolved by the agent / all eligible tickets

This differs from “the agent answered” or “the ticket didn't reach a human.” Intercom, for example, explicitly distinguishes automation rate from involvement and resolution rates.

8. Set up an evaluation flywheel

I'd run this continuously:

Would you like me to help you set specific baseline targets for these metrics based on your industry/support channel , or outline a step-by-step QA scorecard template?

3. Account for Complexity (Weight Your Tickets)

• Not all tickets are created equal. A password reset takes 2 minutes; a deep technical bug investigation takes 3 days.
• Categorize Ticket Complexity: Group tickets into tiers (Tier 1: Transactional/Easy, Tier 2: Moderate, Tier 3: Complex/Technical).
• Adjust Resolution Time Benchmarks: Benchmark an agent's resolution rate relative to the tier of complexity they handle, rather than raw volume.

4. Establish a Baseline and Cohort Comparison

• Compare Against Team Averages: Look at standard deviations from the team median rather than absolute numbers. If the team average FCR is 72%, an agent sitting at 55% needs targeted coaching.
• Tenure-Based Benchmarking: Ramp-up matters. Compare a new hire's resolution metrics against other agents in their 30/60/90-day cohort, not your veteran tier-3 specialists.

If you want, tell me:

• What support channel are you primarily using (email, live chat, phone, or ticketing system like Zendesk/Intercom)?
• What is your current average ticket volume or team size?

I can help you tailor specific target benchmark percentages or draft a QA scorecard template for your team.

A practical evaluation framework:

1. Define what “resolved” means

Create a resolution label independent of the agent’s “Solved/Closed” status.

A ticket counts as resolved only if:

• The customer’s stated goal was achieved.
• The answer/action was correct.
• No follow-up is required for the same issue within a defined window (for example, 7–30 days).
• The customer did not need to repeat information or contact another channel.

This matters because agents (human or AI) can close tickets without fixing the underlying problem.

2. Track resolution outcome metrics

A. True Resolution Rate

Formula:

Example:

• 10,000 tickets closed
• 1,500 reopened or generated duplicate contacts

True resolution rate = 85%

Segment it by:

• issue type
• customer tier
• product area
• difficulty
• agent version/model

B. First Contact Resolution (FCR)

Measures whether the issue was solved in the first interaction.

FCR is commonly used as a core support effectiveness metric, though it should be interpreted alongside ticket complexity because easy tickets naturally have higher FCR.Zendesk Support+1

C. Reopen rate

This is one of the strongest “did you really solve it?” checks.

Watch for:

• customers opening a new ticket instead of reopening
• agents closing tickets prematurely

D. Escalation rate

Measure:

Break this down:

• necessary escalations (complex billing dispute, security issue)
• avoidable escalations (agent lacked knowledge or failed to diagnose)

3. Evaluate answer quality, not just outcome

Sample tickets and score them with a rubric:

A 10-point quality score gives you a way to compare agents even when outcomes are noisy.

4. Use customer signals

Useful customer-side metrics:

• CSAT after resolution
• Customer effort score (“How easy was it to solve your issue?”)
• Repeat-contact rate
• Churn/retention impact for high-value customers

CSAT is useful, but don’t use it alone: customers may like a friendly answer that still fails to fix the issue.Salesforce

5. Build a gold-standard test set

Create a benchmark dataset:

• 500–2,000 historical tickets
• known correct resolutions
• tricky edge cases
• ambiguous requests
• policy-sensitive cases
• tickets requiring tool use

For each ticket, score:

1. Did it identify the problem?
2. Did it choose the correct action?
3. Did the customer need another interaction?
4. Did it avoid unsafe/wrong actions?

This gives you an offline benchmark before deploying changes.

6. Compare against a baseline

Run an A/B test:

Control

• current support process

Treatment

• new AI agent, workflow, prompt, or knowledge base

Compare:

Resolution time is worth tracking too, but speed alone can hide poor outcomes. Support analytics commonly pair resolution time with measures like first-contact resolution and customer satisfaction for that reason.Zendesk Support+1

A simple “agent resolves tickets” scorecard

I’d start with:

• 40% True Resolution Rate
• 20% Reopen Rate (inverse)
• 15% FCR
• 15% Human escalation quality
• 10% Customer satisfaction

Then drill down by ticket category. A support agent that resolves 90% of password resets but fails 40% of billing issues is not actually performing at 90%.

The key metric is: “After this agent responds, does the customer still have the same problem?” Everything else is a proxy.