Last updated: September 12, 2026
What automated quality assurance changes
Automated quality assurance scores every recorded call against your scorecard instead of the two or three per agent a supervisor reaches by hand. The reviewing work shifts toward triaging flagged calls, tuning criteria, and coaching. Expect the first month to surface more failures than your old sample ever did, because the sample was never representative.
The part that stays off the vendor pages is the queue. Say a 40 seat floor, each agent taking about 1,400 calls a month, so the month closes near 56,000 calls that two reviewers used to sample 200 of. Score all of them and, if one call in twelve trips a criterion, Monday opens with roughly 4,600 flags. Call it 210 a working day, for two people.
One in twelve is a shape rather than a benchmark, since your real rate falls out of how the criteria are worded.
So this transition fails in a way most teams never brace for. A program starved of information for years hits week three with more than anyone can read, the queue ages, and flags stop getting opened.
The sampling week
The old week has a shape anyone who has run one can recite. Two reviewers block Tuesday and Wednesday afternoons, pull five calls per agent, score them in a spreadsheet copied forward since 2019, and email the results at month end. Supervisors who coach from those scores coach from them, and the ones who do not, do not.
How the five got chosen decides everything downstream and almost never gets decided. Calls come off the top of the folder, out of a complaint, or because they ran long.
Verint, Gistly, and ViciStack put manual review between 1% and 3% of calls, and SQM Group reports that about 60% of centers evaluate five or more calls per agent per month. Every vendor here opens with those numbers, ours included, so read them as a description of the room rather than an argument.
Why the sample was never representative
Call Centre Helper puts a statistically defensible read at about 65 evaluations per agent per month, and the assumptions are the whole story: 90% confidence, for an agent taking 1,400 calls that month. Change the volume or the confidence level and the sample size moves.
Sixty five per agent across 40 agents is 2,600 scored calls, and at twelve minutes each to pull, listen, score, and write the note, that is 520 hours a month. Nobody staffs that. Almost nobody picked five on purpose either, since five is what fits in the afternoon that was left over.
The coverage week
Same two people, same 40 seats, nothing about the calls has changed. Work arrives now instead of getting fetched, and that one difference reorganizes the week.
Triage comes first and takes the most time. The queue gets sorted by consequence rather than by score, so critical fails sit at the top: a recording disclosure that never got read, an emergency never dispatched on an account with an on-call schedule. Those get opened the same day, and they are a small slice of 4,600.
Somewhere in the same week, two criteria start an argument. A criterion firing on calls where the agent did the right thing is the usual cause, and the fix is a rewrite rather than a conversation with an agent.
The rest of the week is work the old one never reached. Two supervisors score the same ten calls without seeing each other's numbers, then compare against each other and against the machine, which is calibrating machine scores against your reviewers. Four agents get coached from a flagged call with the moment queued up rather than an average in an email. A client report goes out Friday covering every call taken in that client's name.
The hours moved rather than vanished. Nobody pulls recordings or maintains the spreadsheet, and that part of the old job sucks, so nobody misses it. Reading evidence somebody else assembled and ruling on it is harder work.
The first month looks worse
Your failure count goes up and your average score comes down, and both are measurement changes that get read as performance changes by anyone you have not warned.
The old average rested on 200 calls somebody chose, and the new one rests on 56,000 nobody chose. Critical fail rate jumps hardest, since auto fails that used to sit inside the 99.6% of calls nobody opened are now counted. Coverage rate and critical fail rate move first, for arithmetic reasons rather than because the floor got worse.
Brief the owner the week before that report, in one written sentence: scores now come from every call instead of a sample, so month one is a new baseline and last quarter is not a comparison. Delivered after a bad number, it reads as an excuse.
The backlog of audio you already have
Most teams start with a few months of recordings in a platform export, and the obvious move is to score all of it the first night and put the trend in front of the owner.
We would reach for that too, right up to the part where the rubric changes in week three. Then the whole baseline sits against wording that no longer exists, and scoring six months of history again every time somebody fixes a criterion is a habit no team sustains. Score a slice instead: one account, one month, enough calls that every criterion fired a few dozen times.
The harder rule covers what history is for. Historical scores show account level patterns and they tune the rubric. They make poor material for coaching an agent about a call from April, when the account script has been revised twice since. Coach forward from the day the scorecard settles, and use the backlog to argue with a bad script instead of with the floor.
The rubric tightens once every call runs through it
A scorecard written for human reviewers carries assumed context, and a machine brings none of it.
Take a criterion like "used the caller's name". A reviewer applies judgment without noticing, skipping it where the caller refused to give one. The machine reads what the sentence says and runs it across all 56,000. That criterion held up for three years because it was only ever asked 200 questions a month.
So the rubric gets rewritten, usually twice inside the first eight weeks. Wording picks up an exception, scope picks up a condition such as calls over 30 seconds, and a criterion gets cut because nobody could phrase it so two supervisors score it the same. How a score is produced covers what point values do to the total.
The auto fail list takes the roughest treatment. Nine absolutes look responsible on paper and, at full coverage, zero out a quarter of your calls, leaving you unable to rank bad calls against each other. Cut back to what genuinely voids a call, such as a disclosure that was required and never read.
New failure modes you are buying
Alert fatigue arrives first. A rule written the week everyone is excited fires on every critical fail and every keyword in the vocabulary library, and by day ten the supervisor has a filter routing all of it to a folder. Alert on what somebody will act on before end of day.
Queue size is the wrong thing to watch next, since 4,600 is fine if the top 40 get handled daily. Age tells the truth, and an oldest unread flag of three weeks means the program has stopped working however the dashboard looks.
Criteria that fire on the wrong thing are the most demoralizing. A do-not-say term sitting in the client's own approved script fails every operator on that account, and the instinct is to coach the floor. When everybody fails one criterion, the criterion or the script is what to fix.
A rollout that does not break the team
- Pick one queue or one client account, ideally a busy one you already field complaints about.
- Shadow score for two to four weeks: the machine scores every call, reviewers keep their usual sample, and no agent sees either number.
- Compare criterion by criterion, since two scores can both land on 84 and disagree on half the lines underneath.
- Rewrite the wording wherever your two reviewers disagree with each other, which matters more than their gap with the machine.
- Cut the auto fail list to the absolutes.
- Publish to supervisors only for a month before any agent sees a score.
- Widen to the next queue, and expect its account script to break two criteria that worked fine on the first one.
Two to four weeks is a range because the calendar is not what ends shadow scoring. You are waiting for every criterion to fire enough times to judge it, so a queue taking 3,000 calls a week gets there quickly and a small account with two operators takes longer. Stop when the criteria stop changing.
What the QA analyst does now
Underneath the budget conversation sits a question about two specific people. The role gets harder rather than smaller.
Four things fill the hours listening used to take. Calibration becomes monthly instead of aspirational. Criteria authorship turns into a real job, since somebody writes and defends the wording. Disputes arrive with evidence attached and need a ruling. Coaching load goes up, because problems the old sample never surfaced now carry a timestamp.
Writing a criterion that two supervisors score identically across 500 calls is a skill. Listening was the easy part.
Triage: what goes in a queue and what goes to a person
Coverage only pays off if flags get sorted before anyone looks at them. A team that reviews every flag has recreated the original problem at a larger scale, with a nicer dashboard on top.
| Route | What lands there | When it gets opened |
|---|---|---|
| Straight to a named person | High confidence critical fails, such as a missing disclosure or an emergency that never dispatched | Same day, ahead of any monthly average |
| Weekly review queue | Low confidence lines on any criterion, plus the criteria the team is coaching this quarter | In the weekly pass, oldest first |
| Scored and left alone | High confidence passes, and everything on the ignore list | Nobody opens these, and that is the decision |
Confidence cuts the queue in the useful direction. A rule based criterion matching at high confidence, a callback number read back digit by digit at 0:39, settles itself. Low confidence lines are where a person adds something, and raising the threshold to shrink the queue feels like tuning while working like deletion, since the calls it drops are the ambiguous ones. Tone and empathy criteria stay in that lane permanently, because machine scoring is weaker there than a decent reviewer, which Aircall and StackAI both describe.
Write the ignore list down and treat it as a decision rather than a backlog. Filler word counts, flags on 15 second calls where the caller hung up, a soft criterion nobody has rewritten yet: those keep getting scored and stay out of the queue until the coaching focus changes. A team that cannot name what it is deliberately not reviewing reviews nothing on purpose and everything by accident.
Alerting is that decision made in advance, so keep alert rules pointed at failures somebody will act on before going home. A scored sample of your own calls runs six months of your own audio through the scorecard you use today, which is the cheapest way to see your real flag volume, and it costs a fixed pilot price. Count the flags that month produces, then staff to that number rather than to the call count.
Frequently asked questions
What is automated QA in a call center?
Automated QA transcribes and scores every recorded call against your own scorecard, instead of the handful per agent a supervisor can reach by hand. The QA team stops pulling and listening to a sample and starts triaging flags, rewriting criteria, calibrating, and coaching from specific moments.
Does automated QA mean you can score 100% of calls?
Yes, every ingested call gets scored. Coverage is table stakes here and every vendor claims it, so the question worth asking is what sits behind each scored line. A score carrying a transcript timestamp and a confidence value can be checked, and one without either cannot be coached from or disputed.
Will automated QA replace our QA analysts?
The role changes shape and usually gets harder. Calibration, writing and defending criteria, handling disputes, and coaching fill the hours listening used to take. Headcount is your decision, though a team of two sized for 200 calls a month is not obviously sized for several thousand flags.
How long does it take to roll out?
Plan on two to four weeks of shadow scoring on one queue before anything reaches agents. That period ends when every criterion has fired often enough to judge, so a busy queue finishes faster than a small account.
Why did our scores drop after we automated?
The denominator changed. An average built on a chosen sample of 200 calls sits higher than one built on every call, and critical fail rate jumps because failures inside unopened calls are now counted. Tell leadership before the first report that month one is a new baseline, since the same explanation lands badly afterward.
Sources
- Manual QA review covering 1 to 3% of calls. Verint, Gistly, and ViciStack. Third party, unverified, setup only.
- About 60% of call centers evaluating five or more calls per agent per month. SQM Group. Third party, unverified.
- About 65 evaluations per agent per month at 90% confidence for an agent taking 1,400 calls. Call Centre Helper. Third party, unverified, assumptions printed beside the figure.
- Rule based criteria scoring more reliably than tone and empathy criteria. Aircall and StackAI. Third party, unverified.
- Scoring every ingested call, critical fails, alert rules, vocabulary library, and per account dashboards. Trinsic feature sheet and call notes, 2026-08-27 and 2026-09-03. Verified internally.
- The 40 seat floor, the 56,000 call month, and the one in twelve flag rate. Hypothetical figures used for arithmetic.
- Twelve minutes to pull, listen, score, and write up one call. Editorial estimate for this article.