Last updated: September 9, 2026
What QA calibration is
QA calibration is a scheduled session where everyone who scores calls grades the same call independently, then compares results and agrees on one reading of the scorecard. Programs run calibration to keep scoring consistent between reviewers, settle disputes with evidence, and fix criteria that turn out to be ambiguous once real calls hit them.
Say two reviewers score the same recorded call against the same 100 point scorecard, an hour apart, without talking first.
| Criterion | Points | Reviewer A | Reviewer B |
|---|---|---|---|
| Greeting included the business name | 10 | 10 | 10 |
| Callback number read back digit by digit | 20 | 20 | 0 |
| Caller name and reason on the account script | 15 | 12 | 15 |
| Empathy on a distressed caller | 15 | 13 | 10 |
| Correct dispatch against the on call schedule | 25 | 25 | 25 |
| Hold taken with permission and a check back | 15 | 9 | 15 |
| Total | 100 | 89 | 75 |
Fourteen points apart, and the totals are the least interesting part. Reviewer B scored the callback number at zero and flagged a critical fail, because the criterion says the number gets read back digit by digit and the agent read it once at normal speed. The other reviewer heard it repeated correctly and gave full credit. Nothing in the criterion said which reading counts.
Then the coaching happens. Whichever sheet reaches the one on one is the sheet the agent hears about, so this agent either gets a nudge about hold handling or takes a critical fail for a readback they performed. What nobody discusses is what read back means on this account.
A session produces a scorecard everyone reads the same way, with a note on every criterion that had to be argued about.
Why reviewer variance happens
Criteria get written in language that sounds obvious in a planning meeting and falls apart on a live call. "Showed empathy" means a named feeling to one reviewer and a slower pace to another.
Tenure puts a different script in every reviewer's head, so somebody who worked the overnight dispatch desk grades the on call lookup against what they used to do at 3am.
Most programs never picked a reference call. A new reviewer then calibrates against whoever trained them, and the drift compounds quietly for a year.
Weights go stale. A scorecard built when hold time was the loudest complaint still spends 20 points there long after dispatch accuracy took over.
Calibration will not rescue a scorecard built on criteria nobody can point to in a transcript. When two reviewers cannot find words that settle a criterion either way, the criterion needs rewriting.
How to run a calibration session
Six steps, about 45 minutes, same order every time.
- Pick a call that is genuinely ambiguous rather than an obvious pass or fail, because a clean call teaches nobody anything.
- Score it blind. Everyone scores independently before anyone speaks, so nobody anchors on the loudest voice.
- Compare criterion by criterion instead of on the total, since two identical totals can hide four disagreements.
- Open the transcript at the timestamp for every criterion where scores diverged.
- Agree on one reading and write it into the scorecard as a note.
- Change the criterion when the disagreement was about the wording rather than the call.
A sample calibration agenda
| Minutes | Segment | Output |
|---|---|---|
| 0 to 5 | Set up, restate the scorecard version in use | Same version for everyone |
| 5 to 15 | Blind scoring of the selected call | One score sheet per participant |
| 15 to 20 | Post scores side by side, criterion by criterion | The variance table |
| 20 to 35 | Work the criteria where scores diverged, transcript open | An agreed reading per criterion |
| 35 to 42 | Decide which criteria need rewriting | A change list |
| 42 to 45 | Assign owners and set the next session | Named owners and a date |
Run it against a calibration scoring form so everyone fills in the same sheet.
How to measure calibration variance
Calibration variance is the spread between the highest and lowest score on the same call, tracked per session. Start there, because it takes ten seconds to compute and gives the program a number to watch.
Then stop leaning on it. Those two reviewers could have landed a point apart while disagreeing on four criteria in opposite directions, so per criterion variance is the number that tells you which line is broken.
As of September 2026 there is no published threshold for acceptable variance that we could verify. Set your target from your own first three sessions and hold the program to improvement against it.
Calibrating against automated scores
Calibration used to be humans arguing with humans. Most sessions now include a score a machine produced, and the supervisor who disagrees with it has nowhere to take it. A score nobody can trace to the words that produced it will not survive an agent asking why.
What machine scoring gets right
Some criteria have exactly one right answer sitting in the transcript:
- Whether the greeting included the business name
- A callback number read back digit by digit
- The TPMO disclaimer landing before any benefit came up on a Medicare sales call
- Scope of Appointment collected before the appointment started
On each of those the trigger is inspectable, so a supervisor can see what fired the score. We built the vocabulary library so a criterion points at a list of phrases your team can open and edit, covering greetings, probing questions, filler words, and caller ID capture. Aircall and StackAI land in the same place: automated scoring is accurate on rule based criteria and less reliable once a criterion asks for judgment.
Where a human still has to read
Tone and empathy are where machine scoring gets soft, and pretending otherwise is how a QA program loses the floor. A caller who says "that's fine" in a flat voice after a four minute hold has not been handled well, and no keyword says so. Sarcasm reads as agreement.
So the position we take is narrow. A machine score on a subjective criterion is a first pass that a human confirms, and the scorecard should mark which criteria those are.
How to audit a machine score in the session
Spending part of a calibration hour arguing with a machine score sounds like a poor trade. It is still the cheapest way to learn whether the criterion or the model needs work.
Open the call at the timestamp behind the disputed criterion. Read the words that triggered it out loud. Check the confidence on that line, because a criterion firing at low confidence is worth rewriting whether or not the score turned out correct. Then decide which one was wrong, the call or the criterion.
The audit works only if every score carries its evidence, which is why CallSniper attaches transcript timestamps and confidence scores to each scored line, and why critical fails get reviewed before the point criteria. Scrubbing 45 minutes of audio to settle one criterion sucks, so the timestamp earns its keep. None of that makes the machine right about tone, though it does keep the argument short.
The page on how a recorded call gets scored covers the path from audio to a finished sheet, and we wrote a longer piece on where automated call scoring holds up.
The dispute workflow
An agent who thinks a score is wrong needs a route that ends somewhere other than an argument with their supervisor.
- The agent disputes a specific criterion rather than the total.
- QA attaches the evidence: transcript timestamp, the words that drove the score, and the confidence if it was automated.
- A second reviewer scores that criterion blind, without seeing the original result.
- Readings that still disagree go to the next calibration session with the call attached.
- The decision goes back to the agent inside the same week, against the criterion they named.
Three outcomes end a dispute. The score stands with the evidence attached, or the score changes, or the criterion gets rewritten and the rewrite goes out to everyone scoring against it. That last one is worth wanting, because a rewritten criterion fixes every call after it.
How often to calibrate
Weekly while a scorecard is new, through roughly its first month, then monthly once variance settles. Run one immediately after any criterion changes, which is the trigger that matters more than the calendar.
That cadence is our recommendation rather than a published standard, and we did not find a source that sets one.
Download the calibration form
Two files, ungated. The blind scoring sheet gives one row per criterion and a column per participant, and the variance tracker holds one row per criterion per session.
Frequently asked questions
How often should you run QA calibration? Weekly while a scorecard is new, then monthly once variance settles. Add a session whenever a criterion changes or a dispute shows two reviewers reading the same line differently.
Who should attend a calibration session? Everyone who scores calls attends, plus the supervisor who delivers the feedback, and past about eight people a session starts producing opinions where you wanted decisions.
What is a good calibration variance? No published threshold turned up that we could verify as of September 2026, so set the target from your own first three sessions. Track variance per criterion, since matching totals can hide disagreements that cancel out.
How do you calibrate AI call scoring? Score the call blind first. Then open the transcript at the timestamp behind each disputed criterion and read the words that triggered it out loud. Automated scoring holds up on binary criteria and needs a human on tone and empathy, which is where Aircall and StackAI both put the line.
What is a QA dispute process? A dispute process gives an agent a route to challenge a score and gives the program a route to catch a bad criterion. Every dispute ends with the score standing, the score changing, or the criterion being rewritten.
One session tells you whether your reviewers agree
Forty five minutes buys a full calibration session, and half that settles whether your reviewers agree at all. A demo takes about 30 minutes on Zoom: a supervisor scores one of your calls blind, the machine result comes up beside it, and the two get compared criterion by criterion. Use the call your team already argued about. Six months of your historical audio through the paid pilot is the longer version, at a fixed pilot price.
Sources
- Aircall and StackAI, on automated scoring accuracy for rule based versus subjective criteria. Third party, unverified by us.
- Acceptable variance threshold: no verified published source found as of 2026-09-08.
- Calibration cadence: internal recommendation, not an industry standard.
- Transcript timestamps, per line confidence scores, critical fail flags, and the vocabulary library: CallSniper product documentation. Verified.