Five monitored calls a month is enough to coach an agent and nowhere near enough to rank one. That’s the call monitoring sample size problem in a sentence, and most QA programmes run straight into it because the number five came from what the team leaders had time for, not from any maths. Score five calls from an agent whose true pass rate is 80%, and the honest range on what you’ve measured runs from about 38% to 96%. That’s not a measurement. A monthly QA league table built on it is mostly noise with names attached, and agents can usually tell.
I used to think the fix for a QA programme nobody trusted was a better scorecard. It isn’t. It’s knowing what the number underneath can carry. This post is about that one decision: how many calls to score, per agent and per team, and what each number can and can’t support. Scorecard design and calibration matter too, and I’ll get to both, but none of it helps if the sample can’t hold the weight you’re putting on it. Everything below uses a simple pass or fail per call: did the call meet the standard on every must-have item or not. Percentage scores behave the same way, just with more arithmetic.
Why is five calls a month not enough to rank agents?
Each scored call is a coin toss weighted by how good the agent really is. Toss it five times and luck dominates. The table below shows what happens to the range as the sample grows, using the interval method NIST recommends for small samples of pass/fail data (the Wilson interval), because the textbook normal approximation goes silly below about ten failures. The last column takes two agents, one who passes 80% of calls and one who passes 90%, scores the same number of calls for each, and asks how often the weaker one comes out level or ahead.
| Calls scored per agent | Typical result | 95% range on the true pass rate | Weaker agent ties or beats the stronger one |
|---|---|---|---|
| 5 | 4 of 5 | 38% to 96% | 51% of months |
| 10 | 8 of 10 | 49% to 94% | 38% |
| 20 | 16 of 20 | 58% to 92% | 25% |
| 40 | 32 of 40 | 65% to 90% | 13% |
| 62 | 50 of 62 | 69% to 89% | 7% |
Here’s how to read a row. Take the top one: an agent passed 4 of the 5 calls you scored, so their measured rate is 4 / 5 = 0.80, or 80%. The third column says that, given only those five calls, their real pass rate could sensibly be anywhere from 38% to 96%. The last column comes from working through every possible score the two agents could get out of five and adding up the chances of the 80% agent matching or beating the 90% one. It lands at 51%, which is a coin flip.
Read that top row again. At five calls each, the better agent fails to finish ahead about half the time. Even an agent who genuinely passes 80% of calls will score a perfect five out of five in a third of months: 0.8 multiplied by itself five times is 0.328, or 32.8%. They’ll score three or fewer in a quarter of them (26.3%). So the same person can be top of the table in March and in a performance conversation in April without changing a thing they do. Nobody trusts a QA score that jumps like that, and they’re right not to.
How do you work out the sample size in Excel?
Two formulas cover it, and they work as written in Excel 2007 (and in any later Excel or Google Sheets). Open a new workbook and name the first sheet Inputs. Column A holds labels and column B holds the numbers. Put “Expected pass rate” in A2 and 0.80 in B2. Put “Precision wanted” in A3 and 0.10 in B3, meaning plus or minus ten points. Leave row 4 empty. Put “Calls scored” in A5 and your current monthly sample, say 5, in B5. Results go in B7 and B8, with labels beside them in A7 and A8.
In B7 type =ROUNDUP(1.96^2*B2*(1-B2)/B3^2,0). In words: 1.96 is the multiplier for 95% confidence, and squaring it gives 3.8416. B2*(1-B2) is how much a pass/fail result naturally wobbles, 0.80 x 0.20 = 0.16. B3^2 is the precision squared, 0.10 x 0.10 = 0.01. So it’s 3.8416 x 0.16 / 0.01 = 61.47, and ROUNDUP(…,0) rounds up to a whole call, giving 62. You round up, never down, because 61 calls would miss the target.
B8 flips it round and tells you what your current sample buys you: =1.96*SQRT(B2*(1-B2)/B5). Same 0.16 on top, divided by the 5 calls in B5, gives 0.032. SQRT takes its square root, 0.179, and 1.96 x 0.179 = 0.35, or 35 points either way. If you’ve no idea what the pass rate is, put 0.5 in B2. That’s the worst case, because 0.5 x 0.5 = 0.25 is the biggest wobble a pass/fail result can have, and B7 climbs to 3.8416 x 0.25 / 0.01 = 96.04, rounded up to 97 calls. Change B3 to 0.05 and B7 shows 3.8416 x 0.16 / 0.0025 = 245.9, so 246 calls per agent for plus or minus five points.
What does scoring 62 calls an agent actually cost?
That’s where the maths meets the rota. As an illustration, say listening to and scoring a call takes 15 minutes, including the replay you always end up doing on the tricky bit. 62 calls x 15 minutes = 930 minutes, and 930 / 60 = 15.5 hours of evaluator time per agent per month. For a 15-agent team that’s 15.5 x 15 = 232.5 hours a month. A full-time evaluator on a 37.5-hour week works 37.5 x 52 / 12 = about 162 hours a month, before holidays and meetings, so 232.5 hours is more than one person doing nothing but QA, all month, for one team.
And that’s only the evaluator’s side. Every score is supposed to end in a feedback conversation, and that time comes off the phones. If each agent gets a 30-minute coaching session a week, that’s roughly two hours a month per agent that belongs in your shrinkage figures alongside training and team meetings. Leave it out and the rota looks fully staffed on paper while the queue tells a different story every Tuesday afternoon.
Almost nobody funds 62 calls an agent. I’ve never seen a QA budget that did, and I don’t think chasing it is the right goal anyway. The useful question isn’t “how do we score 62 calls each” but “what can we honestly claim from the calls we do score”. Answer that and you can keep the five-call sample, stop pretending it’s something it isn’t, and spend the saved hours on coaching. The sample size isn’t the failure. The claim you hang on it is.
What can a small sample actually tell you?
More than the table suggests, as long as you ask it the right question. Five calls are excellent for coaching, because coaching is about behaviours, not scores. If an agent skips the security check on two of five calls, you don’t need a confidence interval to know there’s something to talk about. A missed must-have item is a fact on that call, whatever the sample size. The mistake is turning those five calls into a percentage and filing it as a rating.
Team-level numbers are where small samples earn their keep. Fifteen agents at five calls each is 15 x 5 = 75 calls a month. Put 75 in B5 on the Inputs sheet and B8 returns 1.96 x SQRT(0.16 / 75) = 1.96 x 0.046 = 0.09, so the team pass rate is pinned to roughly plus or minus 9 points. Pool a quarter and it’s 75 x 3 = 225 calls and about plus or minus 5, which is good enough to see whether a training change moved anything. This is the same discipline as tracking any KPI: capture it the same way every time so March still compares with October.
Individual ratings are the hard case. Even a full quarter of five calls a month is only 5 x 3 = 15 calls per agent. Put 15 in B5 and B8 gives 1.96 x SQRT(0.16 / 15) = 0.20, still plus or minus 20 points. Two quarters gets you 30 calls and about plus or minus 14. If QA has to feed an appraisal, pool at least those two quarters, or treat the score as one input alongside the coaching notes rather than the verdict. Show me an appraisal that hangs on one month of QA and I’ll show you an agent who’s been rated on luck.
How should you pick which calls to score?
Randomly, and by a rule the agents know in advance. Left to choose, team leaders pick the long calls, the complaints and the ones somebody mentioned in the kitchen. Those calls are worth listening to, and the angry ones are often the best coaching material you’ll get, but they belong in a separate coaching pile, not in the score. A score built from hand-picked hard calls measures how agents handle hard calls. That’s a different number.
The random pick is easy to do in the same workbook. Add a second sheet called Calls and paste last month’s call list from the phone system: call ID in column A, agent name in B, shift (Early, Late or Weekend) in C, starting at row 2 with headings in row 1. In D2 type =RAND() and fill it down to the last call. RAND gives every row a random number between 0 and 1. Copy column D and paste it back as values so the numbers stop changing, then sort by agent (column B) and, within each agent, by D. The first five rows under each name are that agent’s sample. Glance down column C, and if all five came from one shift, swap the fifth for the next row from a shift they’re missing.
Then tell agents the selection rule before you start recording against it. “Five calls a month, picked at random by call ID, at least one from each shift you work” takes the suspicion out of the process. Nothing damages a QA programme faster than agents believing someone’s being targeted, and a published rule answers that before it’s asked. It also protects the team leader. When an agent asks “why that call?”, the answer is “the spreadsheet picked it”, and the conversation moves on.
How do you know your evaluators agree?
You test it, every month. Have two or three evaluators score the same recorded call independently, without talking first, and compare. Running regular calibration sessions is one of the core skills ICMI teaches in its quality training, and it’s the step most programmes skip once the scorecard is printed. If evaluators disagree by 15 points on one call, no sample size will rescue the averages, because the noise is in the scoring, not the sampling.
Keep it on a third sheet called Calibration. Row 1 holds headings: call ID in A, then one column per evaluator in B, C and D. Each month adds a row. As an illustration, if the three evaluators give one call 85, 70 and 80, put those in B2, C2 and D2. In E2 type =MAX(B2:D2)-MIN(B2:D2): MAX finds the highest score (85), MIN finds the lowest (70), and the subtraction gives the spread, 85 – 70 = 15 points. In F2, =AVERAGE(B2:D2) gives (85 + 70 + 80) / 3 = 78.3, which is a starting point for the agreed score once the three of you have talked it through. A spread that stays wide month after month is a scorecard problem in disguise: the wording of an item means different things to different people, and that’s the line to rewrite.
One change costs nothing and fixes most of the arguments. Print the number of calls next to every QA score you publish, every time, including on the slide that goes into the weekly metrics meeting. Once a team leader sees “60% (5 calls)” beside “80% (5 calls)”, the debate about who’s better usually settles itself, and the conversation moves to what happened on the calls, which is where it should have been all along.
How many calls a month does your team score per agent, and has anyone ever checked what that number can actually support? Tell me in the comments.
Hutch Morzaria is a Director-level CX and Support Leadership professional with 19 years of experience building global support organizations across SaaS, Fintech, and enterprise technology. He has hired dozens of support and CX leaders across his career and holds ITIL Expert certification across V3 and V4.
