Partnership

QA Scoring with AI: How to Score Every Conversation

How AI QA scoring works, how to build a scorecard, calibrate against human reviewers and handle recording consent, so you can score every call and chat.

Analyst reviewing a customer service call

AI QA scoring evaluates every call and chat against a scorecard you define, then sends only the conversations that need a human to your reviewers. It replaces the sample-and-listen routine with full coverage. It works well when the scorecard is precise, the scores are calibrated against human judgment, and call-monitoring consent is handled before the first conversation is analyzed.

This guide covers what to score, how to build the scorecard, how to check the AI’s accuracy, and the legal point that catches US teams off guard.

Monobot
Gain clearer insight into customer conversations
Monobot offers real-time analytics and AI voice and chat assistants to help teams understand interactions and automate routine customer service tasks.

Explore Monobot

TL;DR

  • Manual QA typically reviews fewer than 5% of interactions. AI scoring can cover 100% of calls and chats.
  • Treat AI as a consistent first pass, not a final judge. Reliability depends on the metric and must be calibrated against human reviewers.
  • Score in several dimensions (compliance, resolution, empathy, process) instead of one blended number.
  • In all-party consent states, AI analysis of a live call counts as monitoring. Update your disclosures before you switch scoring on.
  • Start with one queue, one scorecard and a weekly calibration session, then expand.

Table of Contents

What AI QA scoring does

Traditional QA means a analyst pulls a handful of recordings per agent each month, listens, and fills in a form. The process is slow, and it only sees a sliver of your volume. A 2026 framework paper on enterprise service quality puts manual coverage at under 5% of interactions and reports that scores from different human evaluators can vary by 15 to 22 percentage points on the same work.

AI scoring changes three things:

  1. Coverage. Every conversation is transcribed, evaluated against your criteria and given a score, including chats, not only calls.
  2. Consistency. The same rubric is applied the same way at 9 a.m. and at 9 p.m., to the new hire and the 10-year veteran.
  3. Speed. Scores appear shortly after the conversation ends, so coaching happens this week and not next quarter.

The idea is not new. Researchers described an automated system for scoring call center conversations using speech recognition and classification years before large language models. What changed is that an LLM can read a transcript against a plain-language criterion such as “did the agent verify identity before discussing the account?” and answer with evidence.

The same research describes a shift from episodic audits to what its authors call continuous quality intelligence: quality becomes an always-on operating capability, not a periodic check.

How the scoring pipeline works

Whatever platform you use, the flow is the same:

  1. Capture. Call audio or chat text is ingested from your phone system or chat channels.
  2. Transcribe. Speech-to-text turns audio into a transcript with speaker labels and timestamps.
  3. Evaluate. An LLM checks the transcript against each scorecard item and returns a result plus the evidence it relied on.
  4. Aggregate. Item results roll up into category scores, agent scores and team trends.
  5. Route. Low scores, compliance misses, disputed results and borderline cases go to a human reviewer.

Step 5 is what separates a useful program from a dashboard nobody trusts. Without a path from score to human action, you only have a bigger pile of numbers.

Build a scorecard an AI can actually apply

Most AI scoring problems are scorecard problems. A criterion that two human reviewers read differently will be scored inconsistently by a model too. Use these rules.

Write criteria as observable behaviors

Compare these two items:

  • Weak: “Agent showed empathy.”
  • Strong: “When the customer reports a problem, the agent acknowledges the issue in their own words before moving to troubleshooting.”

The second can be answered yes or no from a transcript, and a reviewer can check the evidence in seconds.

Score in dimensions, not one number

“Quality” is not a single quantity. The same framework paper notes that strong teams use configurable, multidimensional scoring that reflects different call types, and avoids a unified model that squashes a billing dispute and a password reset into one metric.

A practical structure:

Category Example items Typical treatment
Compliance Required disclosures, identity verification Pass/fail, critical
Resolution Issue identified, correct answer given, next step confirmed Weighted points
Communication Acknowledgment, clarity, no dead air Weighted points
Process Correct tools used, accurate wrap-up notes Weighted points
Outcome signals Customer sentiment at close, repeat contact Tracked, not scored per agent

Define critical items and exceptions

Decide which items fail the whole interaction if missed (a skipped legal disclosure, for example) and which only subtract points. Then write down the exceptions. If a customer hangs up before the agent can verify identity, that item should read “not applicable,” not “failed.” Exceptions are where automated scoring most often generates disputes.

Use different scorecards for different interaction types

Sales calls, collections, technical support and appointment scheduling reward different behaviors. One scorecard per interaction family keeps scores meaningful. If you start with templates for your industry, adapt them before going live; our ready-to-use templates are built for that kind of starting point across healthcare, banking, retail and logistics.

How to calculate a QA score

The arithmetic is simple. The decisions behind it are not.

Score = points earned ÷ points possible × 100, after removing items marked not applicable.

A worked example for a support call with 10 weighted items worth 100 points in total. The agent earns 82 points, and one 5-point item is not applicable. The score is 82 ÷ 95 × 100, or 86.3%. If the call also missed a critical compliance item, it is flagged and scored zero regardless of the points, which is a policy choice you make in advance and document.

Two cautions:

  • Weights are policy. Setting resolution at 40 points and tone at 10 tells agents what you value. Review the weights against customer satisfaction and repeat-contact data, not opinion.
  • Averages hide risk. An agent with an 88% average and one compliance failure a week needs a different conversation than one with a steady 88%. Report critical failures separately.

Keep humans in the loop: calibration

AI scoring is not equally reliable on every question. A benchmark of LLM judges for voice-agent evaluation found that agreement with human annotators varies across attributes such as safety, goal achievement and efficiency, and concludes that automated evaluation needs calibration against human ground truth, with people adjudicating the metrics that demand contextual judgment.

The framework paper reports that structured human calibration, where supervisors periodically review and adjudicate AI scores, improves defect detection by 34% to 47% compared with automation alone.

Here is a calibration routine that fits in a normal week:

  1. Sample. Each week, pull 20 to 30 conversations across score bands, including some the AI scored very high and very low.
  2. Blind review. Two reviewers score them without seeing the AI result.
  3. Compare. For each scorecard item, note where the AI and humans disagree.
  4. Diagnose. Is the criterion ambiguous, is the transcript wrong, or is the model missing context?
  5. Fix. Rewrite the criterion, add an exception or a worked example, or move the item to human-only review.
  6. Track agreement by item. Not overall. A 95% overall match can hide a 60% match on the one item that matters.

The right target is reliable alignment, not perfection. Human reviewers disagree with each other too, which is exactly the variance AI is meant to reduce.

Also give agents a way to dispute a score. A dispute queue is a free calibration signal, and it builds the trust that makes agents accept the program.

Coverage is the real prize

Sampling 2% of calls means you learn about the other 98% from complaints. Scoring everything lets you find patterns a sample can never show: a policy change that confuses customers on Tuesday afternoons, one product line that drives most escalations, a script phrase that correlates with repeat calls. For more on why broad monitoring beats small samples, see this look at how customer feedback drives revenue.

Call sampling compared with complete QA coverage

Full coverage also changes how supervisors spend their time. Instead of listening to random recordings, they review the 5% of conversations the system surfaces as risky or ambiguous, then coach. Our guide on quality assurance automation walks through how a team can review thousands of conversations this way, and our AI call analytics buyer guide covers the analytics layer in more depth.

This is the part of AI QA scoring that most vendor pages skip. Several US states require all parties to consent to call recording or monitoring. A law-firm analysis of AI-driven real-time call monitoring argues that replacing a human supervisor with an AI engine does not change the core rule: contemporaneous listening is monitoring, and what triggers the issue is access, not storage. On that view, an AI tool that only keeps deidentified metrics can still require consent if it analyzes the call live.

There is a second concern, which is disclosure. Forrester’s analysts note that the familiar line “calls may be recorded for quality purposes” is no longer enough when third-party AI is involved, particularly if data may be shared or used to train a vendor’s models.

What to do about it:

  • Ask your counsel which states your callers are in and what each requires. Do not rely on a vendor’s summary.
  • Update IVR prompts, chat notices and scripts to say that calls and chats may be analyzed by automated tools and third-party providers.
  • Ask every vendor in writing whether your data is used for model training, where it is stored and how long it is retained.
  • If you score after the call from recordings, you still need a lawful recording basis. Treat that as a separate question from live analysis.

This is a legal area where facts and statutes differ by state, and cases are still developing. Your general counsel or privacy lawyer should sign off on the wording before launch.

Our perspective

We build voice and chat agents, so we see both sides of the conversation: the automated agent handling a call and the quality program judging it. Three convictions come out of that work.

Score the AI agent the way you score a human. If a virtual agent handles up to 80% of routine calls and chats, its conversations deserve the same scorecard, with the same critical items. Regression-test changes before they go live; our practical playbook for testing AI behavior shows how.

Quality data should feed action, not sit in a report. Sentiment, conversation intelligence and interaction dashboards matter only if a supervisor sees a flagged conversation while there is still time to coach, or a knowledge base article is fixed because scoring exposed a wrong answer. That is why we pair analytics with real-time agent assistance and not only after-the-fact reports.

Start narrow. One queue, one scorecard, one weekly calibration meeting. Teams that try to score every channel with a 40-item form on day one end up with scores nobody believes.

— Alex

Put it into practice with Monobot

If you want scoring, analytics and automation in one place, Monobot gives you no-code AI voice and chat agents, dashboard insights and reporting, and live transcription of every interaction. Plans start at 200 USD per month for Starter, with Growth at 500 USD per month and Business at 1000 USD per month. Enterprise is available for larger deployments, and the HIPAA option is 1000 USD per month. See current plans and fees or schedule a demo to see how your own scorecard would look.

A 30-day rollout plan

Week 1: Scope. Choose one high-volume queue. Write 10 to 15 observable criteria, mark the critical ones and define exceptions. Get legal sign-off on disclosures.

Week 2: Pilot in shadow mode. Run AI scoring alongside your existing QA. Do not show agents the scores yet. Compare results with your analysts on the same conversations.

Week 3: Calibrate. Run the weekly routine above. Fix the five items with the lowest agreement. Decide which items stay human-only.

Week 4: Go live. Share scorecards with agents, open the dispute channel and set a threshold for what routes to supervisors. Review results against customer satisfaction and repeat-contact rates after a month.

Watch for these failure modes:

  • Score inflation. If nearly everyone scores 95%, your criteria are too easy to pass.
  • Gaming. Agents learn the phrases that trigger a pass. Combine behavior items with outcome metrics.
  • Drift. New products or policies make old criteria stale. Recalibrate after every significant change.
  • Bias blind spots. Check score distributions by shift, language, accent and channel, and have a person with authority to reverse a result.

Choosing a tool: what to ask

Whether you buy a standalone QA product, use a contact center suite’s built-in scoring or adopt an AI agent platform with analytics, ask:

  • Can I write my own criteria in plain language, and can I see the evidence behind each score?
  • Can I run different scorecards by queue, channel and interaction type?
  • How does the tool handle exceptions and “not applicable”?
  • Is there a dispute and review workflow, and does it feed calibration?
  • How are privacy, retention and model-training terms stated in the contract?
  • What does it cost at my volume, with all fees counted?

Our guide to choosing an AI virtual assistant for customer service covers evaluation criteria for the automation side of the same decision.

FAQ

What does QA mean in AI scoring?

In this context, QA means quality assurance: checking that customer interactions meet your standards for accuracy, compliance and service. AI QA scoring automates that check by evaluating each call or chat against a scorecard. It is separate from QA testing in software development.

Will AI replace human QA analysts?

Not entirely. Research on LLM judges recommends a hybrid approach: AI handles high-volume assessment, while people adjudicate complex or high-stakes judgments. The analyst’s job shifts from listening to samples toward calibrating the model, reviewing flagged conversations and coaching.

How accurate is AI call scoring?

It depends on the criterion. Objective items such as a required disclosure are easier to score than subjective ones such as tone. Measure agreement with your own reviewers item by item before trusting any overall figure.

Often yes. In all-party consent states, live AI analysis can count as monitoring, and a vague “recorded for quality” notice may not be enough. Ask a privacy lawyer to review your disclosures for the states where your callers are located.

Sources