Last updated: September 12, 2026
What AI call scoring does to a recording
AI call scoring assigns points to a recorded call by matching transcript lines against the criteria on your scorecard, then flags critical fails and totals the result. Each scored line should carry a timestamp and a confidence value so a supervisor can hear the moment behind it. Rule based criteria score reliably, while tone criteria need a human check.
Walk one call through it. The recording lands after the caller hangs up, text comes out of the audio with a timestamp on every line, and the transcript gets split so the agent's words sit apart from the caller's. Each criterion on your scorecard then gets checked on its own, auto fails get applied, and a number comes out.
Five steps, and each one can go sideways in a way that lands in the number.
Ingest sounds like the boring step and still breaks things, since a recording that caught one leg of the call is worse input than the conversation was. Transcription then fights accents, crosstalk, and the vocabulary of whatever business the call was about, and a criterion looking for an exact phrase will not find one transcribed sideways.
Speaker separation is where a failure does the most damage, and it is the step most sensitive to what your phone system hands us.
Mono audio, meaning any recording where the agent and the caller land on the same channel, makes the two voices hard to tell apart, and multi party calls and background noise make it worse. That is a live issue in this product today rather than a theoretical one. It shows up across every client and early adopter whose phone system never split the two sides, and the fix sits on our side.
Every criterion that depends on knowing who spoke breaks along with it, so an agent can pick up credit for an empathy line the caller said. Check whether your own recordings arrive as two channels or one before anything else.
Criteria evaluation is the most literal part of the whole thing. Each criterion gets read against the transcript by itself, so one written as an observable behavior gets a defensible answer and one written as "showed empathy" gets a coin flip. The points each criterion carries and the auto fails then turn that into a number. The category and the deployment question live on the category explained end to end.
One call, scored line by line
An invented 47 second call, scored against an invented 33 point scorecard. Nothing below came from a customer.
| Time | What the transcript has | Criterion | Points | Confidence |
|---|---|---|---|---|
| 00:02 | "Thanks for calling, this is Dana, what can I do for you?" | Opened with the account name and the agent's name | 4 of 8 | High |
| 00:07 | "Quick note, this call is recorded for quality." | Recording disclosure delivered before the caller gives details | 5 of 5 | High |
| 00:21 | "Yeah, coming home to that is no fun at all." | Acknowledged the caller's situation before moving to the fix | 6 of 6 | Low |
| 00:39 | "Let me read that back to you. Five one two, five five five, zero one four four." | Callback number read back digit by digit | 8 of 8 | High |
| 00:45 | No matching line found | Confirmed the next step and who would call the caller back | 0 of 6 | High |
| Total | 23 of 33 |
Two of those five lines deserve an argument. The greeting took half credit because Dana gave her own name and skipped the account name, a rule the transcript settles by itself. Line three scored full points at low confidence, meaning the phrase matched a vocabulary set and the system is unsure, so a supervisor who cares about that criterion should listen rather than accept six points.
The 00:45 miss is the useful kind. Nothing matched, nothing scored, and somebody still has to decide whether Dana skipped the close or used words the criterion does not recognize.
Points and why the total moves
Move the same call onto a 100 point scorecard where the greeting carries 20 points and the callback capture carries 6, and the score drops even though the call did not change. Point values encode what a mistake costs you, so two teams scoring identical calls to different totals is normal.
The greeting's points are where the argument usually lands. Push it high and agents nail the first ten seconds and hurry the rest. Although that is too tidy, because in most centers the greeting got its high point value after one client complained years ago and nobody revisited it. Check that before blaming the floor, and point values and rubric design has the template.
Critical fails and the zeroed call
A critical fail zeroes the whole call no matter what the other criteria did. A missed recording disclosure in a two-party consent state does it, and so does a TPMO disclaimer that never arrived before the benefits discussion or an emergency that never got dispatched under the account's on-call schedule. That is by design, because a 91 with a compliance failure inside it is a worse outcome than a 60.
The failure mode is a long auto fail list. Once a third of your calls come back as zeroes you cannot rank bad calls against each other. Five or six auto fails per scorecard is usually the ceiling, and critical fail rate and calibration variance both have formulas.
Criteria a machine scores reliably
Anything binary, observable, and written in words two people would agree on scores well. Four examples from the scorecards on this site:
- Whether the recording disclosure was delivered before the caller gave any details
- Whether the callback number was read back digit by digit
- Whether a required script line appeared, and where in the call it landed
- Whether a term on the account's do-not-say list showed up
Scope of Appointment confirmation sits in the same family: either the words are there or they are not, and the timestamp proves which.
Criteria it gets wrong
Tone, empathy, and judgment are the weak ground, which Aircall and StackAI both report about automated scoring generally. A caller who says "great, just great" while furious reads as satisfied. Sarcasm and a long pause that meant the agent was thinking do not survive the trip through a transcript intact. We are not going to tell you the scoring matches a human reviewer on tone, because it does not.
Speaker attribution failures are a second kind of error needing a different fix. A tone miss is the system reading a real line the wrong way, so you rewrite the criterion or route it to a person. An attribution failure means the line got assigned to the wrong mouth, and that fix is upstream: channelized audio, or waiting on us.
Mixing the two together is how QA programs end up distrusting every number on the dashboard instead of the handful that deserve it.
Calibrating machine scores against your reviewers
Calibration converts a skeptical supervisor, and it takes an afternoon a month.
- Pull 20 calls across shifts, agents, and call types, including a few you already argued about.
- Have two supervisors score them by hand, separately, without seeing the machine's score.
- Compare criterion by criterion rather than total by total, since two scores can match at 84 while disagreeing on half the lines underneath.
- Where the humans disagree with each other, rewrite the criterion wording. If they agree and the machine does not, check the evidence link before touching anything.
- Run it again next month and track the spread.
Adjusting a confidence threshold to make the disagreement go away is tempting and wrong, since it buries the criterion that needed rewriting. Running a calibration session has the full procedure.
What to check on your own recordings first
Before you buy anything here, take ten calls where you already know the answer: five your team scored and agreed on, five that generated a complaint. Score them with whatever tool you are evaluating, then read the evidence behind every line that disagrees with your read.
What you are testing is whether the disagreement is traceable. A number with no moment behind it cannot be argued with, so it cannot be coached from either. Ask about channelized audio too.
Ten calls is a vendor test rather than a program. Call Centre Helper puts a statistically defensible read at roughly 65 evaluations per agent per month, and that figure assumes an agent taking 1,400 calls a month at 90% confidence. Nobody hand scores 65.
When an agent disputes a score
A QA program lives or dies here. The first disputed score is the moment an agent decides whether the whole thing is fair, and the answer they land on crosses the floor by lunch.
Disputes work at the line, never at the total. An agent who says "I disagree with my 78" has given a supervisor nothing to check. Compare that with "criterion four, the callback readback, I did that at 39 seconds." That hands over a timestamp, so the supervisor opens the call at 00:39, listens either side of it, and rules. Two minutes. Arguing about a total takes a week.
The supervisor needs four things on screen: the transcript line, the timestamp into the audio, the criterion wording exactly as it read that month, and the confidence value. That third one matters more than people expect, since scorecards get edited and an agent scored in March against April's wording is right to be annoyed. In CallSniper that runs off transcripts with playback beside them, so the disputed moment is one click from the disputed line.
Then there is the case nobody prepares for, where the machine is right and the score is still unfair. The agent followed the account script exactly and the script is bad. Uphold the score, say out loud that it is not the agent's fault, and route the fix to whoever owns the script.
Watch the shape of the disputes rather than the count. A spike right after a new scorecard launches is normal. Where they cluster is the signal: five agents disputing the same criterion means the criterion is wrong, while one agent disputing everything is a management conversation with nothing to do with QA. A dispute rate sitting at zero usually means agents stopped bothering, which is worse than a noisy month.
The call review scores six months of your own historical audio, so you see your disputes on your own calls before committing to anything, and it costs a fixed pilot price.
Frequently asked questions
Is AI call scoring accurate?
Accuracy depends on the criterion. Rule based criteria such as a recording disclosure or a callback number read back score reliably, because the transcript settles them. Tone and empathy criteria are weaker and need a human reviewer, per Aircall and StackAI. A single accuracy percentage for a whole scorecard would be meaningless.
How does AI score a call?
The recording gets transcribed with timestamps, the transcript gets split between the agent and the caller, and each criterion gets checked against what the transcript says. Points and auto fails total into a score, with every scored line carrying the moment it came from.
Can an agent dispute an AI score?
Yes, and the dispute should name a criterion and a timestamp rather than the total. A supervisor pulls the transcript line, the criterion wording as it stood that month, and the audio at that moment, then upholds or overturns it. Disputes rising after a new scorecard launches is normal.
Do you still need human QA reviewers?
Yes, and the work changes shape. Reviewers stop hand scoring a sample and spend the time on calibration, disputes, the subjective criteria flagged at low confidence, and the coaching conversation afterward.
What is a confidence score on a scored line?
It is the system saying how sure it is that the transcript line satisfies the criterion. High confidence on a rule based criterion can be taken at face value. A low confidence flag is an instruction to go listen, and a supervisor with limited time should spend it there first.
Sources
- Aircall and StackAI on rule based criteria scoring more reliably than tone and empathy. Third party, unverified.
- Call Centre Helper on roughly 65 evaluations per agent per month at 90% confidence for an agent taking 1,400 calls. Third party, unverified.
- Transcript timestamps, per line confidence values, scorecards with points per criterion, and critical fail flags. Trinsic feature sheet and call notes, 2026-08-27 and 2026-09-03. Verified internally.
- Mono and single channel speaker attribution as a current known issue. Trinsic call notes, 2026-09-10. Internal, fix status unverified.
- The 33 point example scorecard and the 47 second call. Invented for this article.