An AI judge scores a conversation 92 out of 100. The team lead who opens it later would have failed it outright: in the third message the agent promised a refund the policy does not allow, and the judge’s rationale praises the warm, well-structured close. It takes one incident like that for the QA dashboard to stop being a source of truth and start being a thing people argue about.
Coverage is the easy part of AI QA. Where manual review samples a few percent of conversations, an AI evaluator can score every eligible one. Whether anyone trusts those scores depends on calibration: testing the judge against your experienced reviewers, measuring exactly where it diverges, and fixing the rubric or the prompt until the remaining disagreements are ones you understand.
Supportman lets you calibrate AI evaluations against conversations your team has already graded, so the judge scores the way your best reviewer would.
Generic prompts produce generic judgments
Ask a model “Was this a good support conversation?” and it has to invent the standard. It may reward a confident answer that violates your refund policy, miss a product-specific error, or grade a terse-but-correct reply below a warm, wrong one.
The fix is a structured rubric: weighted attributes, observable criteria, real examples and counterexamples, and explicit critical-failure rules. Building one is its own project, and we cover it in detail in how to build a customer support QA rubric. This article assumes the rubric exists and covers the next problem: getting an AI judge to apply it the way your best reviewer would.
One thing matters doubly for an AI judge: the rubric must encode your domain context — current policy, product capabilities, plan limits, escalation rules, required disclosures, approved troubleshooting. A human reviewer carries that context in their head. The judge only knows what you give it, and generic dialogue-quality standards miss exactly the failures that matter: the wrong plan limit stated confidently, the unapproved workaround, the missing disclosure.
The LLM-Rubric research (ACL 2024) is direct about why calibration is a separate step: the model’s raw rubric answers did not line up with human judgments on their own — the accuracy gains came from fitting the model’s outputs to individual human reviewers’ historical scores. Model judgments do not automatically agree with human ones, and human reviewers do not fully agree with each other. Calibration is the work of making those differences visible and manageable.
Seed traps for known judge biases
LLM judges have documented, repeatable failure modes. The MT-Bench judge research names three: verbosity bias (longer answers score higher), position bias (order of presentation shifts pairwise judgments), and self-enhancement bias (a judge favors text written by its own model family). Support QA adds a fourth from everyday experience with these systems: politeness reads as quality, so a courteous, apologetic, factually wrong reply can outscore a blunt, correct one.
Do not just know the list — seed your calibration set with a trap for each bias and check that the judge fails the right conversations:
- A long, warm, well-formatted conversation with a factual error in the middle. It must fail on technical accuracy.
- A short, terse, completely correct resolution. It must pass.
- A polite, apologetic close on an issue that was never actually resolved. The tone score must not rescue the resolution score.
- If the same model that drafts agent replies also judges them, include AI-drafted replies with known flaws and check for inflated scores. If they score systematically high, use a different model or configuration for judging.
A judge that passes the traps has earned some trust. A judge that fails them tells you, before launch, exactly which criterion needs a counterexample or a harder rule.
Create a human-scored calibration set
Pull 50–100 real conversations, with a floor of roughly 10 per channel and language you intend to report on — below that, the agreement number for a slice is noise. Include:
- Common and uncommon issues
- Strong and weak examples, plus the bias traps above
- Short and long conversations
- Resolved and escalated interactions
- Positive and negative customer ratings
- Edge cases reviewers find difficult
Have at least two experienced reviewers score them independently using the rubric.
Do not begin with AI. First test whether humans interpret the standard consistently. If your two best reviewers agree on the tone attribute only 60% of the time, an AI judge cannot do better than 60% agreement with either of them in any meaningful sense — the standard itself is ambiguous.
Where reviewers disagree:
- Identify the criterion involved.
- Discuss the evidence each reviewer used.
- Rewrite ambiguous language.
- Add an example.
- Rescore the case.
The resulting set becomes the benchmark for AI evaluation. It also decays: refresh it quarterly, and immediately after any policy or product change large enough to invalidate examples. A calibration set that references a retired feature or last year’s refund policy is quietly testing the wrong standard.
Compare scores at the attribute level
Overall-score agreement can hide serious problems. An AI and a human reviewer may both assign 82 out of 100 while disagreeing about technical accuracy and tone in opposite directions.
For each attribute, measure exact-match agreement and within-one-band agreement (one point on a 5-point scale, roughly 10 points on a 100-point scale), plus the average signed difference — does the judge run hot or cold on this attribute? Then apply two operational bars:
- The yardstick is your human-human agreement on the same set. If two reviewers land within one band of each other 85% of the time on communication, the judge should meet or beat 85% against them. Demanding more of the judge than of people is a bar no reviewer on your team clears.
- Hold back any attribute where the AI diverges from both humans by more than about 10 points on average. Ship the attributes that pass into reporting; keep human review on the ones that don’t until a rubric or prompt fix closes the gap. Partial launch beats a dashboard nobody trusts.
Pay special attention to the high-risk criteria — security, compliance, refund policy, technical correctness — where a single serious false pass costs more than fifty small tone disagreements.
When a gap appears, work it as a case, not a statistic. A typical disagreement looks like this:
AI: 90. Human: 62. The judge’s rationale cited the polite, well-structured close. The reviewer cited message 4, where the agent suggested an unverified registry edit as a workaround. Diagnosis: verbosity and politeness outweighing accuracy. Fix: add the conversation as a counterexample under technical accuracy — “an untested workaround presented as a fix caps this attribute regardless of tone” — then rescore the set and confirm the accuracy agreement number moved.
Six disagreements worked this way will improve the judge more than any amount of prompt-tuning in the abstract.
Test the explanation, not only the number
A useful evaluation points to conversation evidence. For a sample of scored conversations, review whether the rationale:
- Cites the correct part of the interaction
- Applies the stated criterion rather than a vibe
- Avoids inventing facts absent from the conversation
- Distinguishes product failure from agent behaviour
- Suggests an action that follows from the evidence
An eloquent explanation can still be wrong. A score whose rationale cites message 4 can be checked in seconds; a score with a fluent, unanchored rationale cannot be checked at all. Evidence must be traceable to the conversation — this is also what makes the confidence signals in the next section possible.
Define confidence and review paths
Not every AI score deserves equal treatment — but first, be honest about where a confidence number comes from. Asking the model “how confident are you?” is not it; self-reported confidence is unreliable and models will happily claim near-certainty about wrong answers. Usable signals are structural:
- Repeat-run variance: score the conversation three times; a wide spread means low confidence.
- Cross-config disagreement: run a second judge configuration (different model or prompt) on a sample; divergence flags the case.
- Evidence failure: the judge is required to cite conversation evidence for each attribute; when it cannot, that attribute is unscorable.
Route on those signals:
| Tier | Trigger | What happens | Who owns it |
|---|---|---|---|
| High confidence, low risk | Stable across repeat runs; no high-risk criterion fired | Score enters trend reporting automatically | Support ops |
| Medium confidence | Spread wider than one band across repeat runs | Score visible but flagged; 10–20% sampled weekly for human review | QA lead |
| Low confidence or high risk | Judge configs disagree, or a compliance, security, or refund criterion fired | Held from reporting until a human reviews it | Team lead, within an SLA (e.g. two business days) |
| Insufficient evidence | Rationale cannot cite conversation evidence | Not scored | — |
The high-risk row is where AI QA must stay an input, never a verdict. Walk through what actually happens when the judge flags an agent for a compliance miss: the score is withheld from the agent’s record, the team lead reviews the conversation within the SLA, and the agent sees the flagged conversation and the rationale — and can respond — before anything counts. AI evaluation surfaces the case and prepares the evidence; a human makes any decision that touches coaching records, compensation, or a compliance report, and repeated patterns matter more than any single score.
The response path should be open in both directions. Agents should be able to appeal any score by flagging:
- Missing context
- Incorrect product assumptions
- Misapplied criteria
- Attribution for something outside their control
- Evidence the evaluator overlooked
Track appeal outcomes: an appeal that stands is a calibration finding, and it should feed a rubric fix, not just a corrected score.
Monitor drift
AI QA changes even when the dashboard looks stable. Some causes announce themselves — a new model version, an edited prompt, a policy change. Others creep: product releases, new channels or languages, shifting customer behaviour, changed routing.
Two rules cover both kinds:
- On every model or prompt change, rescore the full calibration set and compare per-attribute agreement against the previous run before the new scores enter reporting. No exception for “minor” prompt edits — minor edits move scores.
- On a schedule — monthly works for most teams — have a human reviewer blind-score a spot-sample of ~20 recent AI-scored conversations. This is the only mechanism that catches drift from the causes that never announce themselves.
To make either comparison possible, keep versioned records of:
- Rubric, weights, and examples
- Model and prompt
- Calibration set
- Agreement results
- Launch date of each version
When agreement degrades, treat it like the pre-launch process: find the criterion, work the disagreements as cases, fix, rescore.
Scale the standard, not just the scoring
There is a payoff loop hiding in all of this. Every worked disagreement, every appeal that stands, every drift check that catches a gap ends the same way: a criterion gets rewritten or a counterexample gets added. The rubric — the standard itself — improves. Manual QA never had the volume to expose its own ambiguities; a calibrated judge surfaces them every week, and a team that closes that loop ends up with a sharper definition of good support than it started with.
Supportman trains AI evaluation around the weighted attributes and criteria your team cares about, links each result back to conversation evidence, and lets you calibrate against conversations your reviewers have already graded.
Start calibrating your AI reviewer →
Calibration checklist
- Start from a tested rubric with domain context encoded.
- Test human–human agreement before involving AI.
- Build a 50–100 conversation calibration set, seeded with bias traps; refresh quarterly and after major changes.
- Compare AI and human scores per attribute; use human–human agreement as the bar.
- Hold back any attribute diverging more than ~10 points from both humans.
- Verify rationales cite real conversation evidence.
- Derive confidence from repeat-run variance and cross-config disagreement, never self-report.
- Route low-confidence and high-risk scores to humans with an owner and an SLA.
- Give agents an appeal path that feeds rubric updates.
- Rescore the calibration set on every model or prompt change; blind spot-check monthly.