When Your KPI Passes the Test and Fails the Customer

Gaming a KPI is what happens when the number becomes the goal and the thing it was meant to represent quietly drops out of the picture. The most expensive example of the past year was not in a call centre. It was in a car factory, and it shows exactly what a KPI program looks like when nobody asks whether the measurement still means what it used to.

If you read my piece on fixing a KPI with DMAIC, this is the other side of it. DMAIC assumes the number you are improving is the right number. This post is about what happens when it is not.

What did Volkswagen actually do with its emissions numbers?

In September 2015 the US Environmental Protection Agency issued a notice of violation alleging that Volkswagen had fitted certain diesel cars with software that could tell when a car was being emissions-tested. During the test, the emissions controls ran at full strength. On the road, they were reduced, and the EPA said nitrogen oxide emissions could be up to 40 times the standard. Volkswagen later said the software was in roughly 11 million vehicles worldwide.

The facts are still being worked through by regulators and courts, and I am not going to speculate beyond what has been reported. What matters for this post is the structure, which is simple. There was a standard. There was a test that measured the standard. And there was a gap between the test and the real world that someone decided to exploit.

That is not unusual as a pattern. It is just unusually well documented. Every KPI is a test. It is a proxy for something you care about, taken under certain conditions. The moment people are rewarded for the proxy and not for the thing, the proxy and the thing start to drift apart.

Where do support teams do the same thing, in a small way?

Almost nobody in support is writing defeat software. The versions I see are smaller, and most are not even deliberate. They are the natural result of a target meeting a person who wants to hit it.

• Average handle time. Set a target for it and calls get shorter. Some of that is efficiency. Some of it is agents ending calls before the problem is solved, because the number rewards speed, not outcome.
• Tickets closed. Measure closures and tickets get closed. A customer replies a day later to say it is not fixed and a second ticket opens. The first one still counts as a win.
• First-contact resolution. If it is tracked by whether the agent marked the issue resolved, rather than whether the customer called back, it will look excellent.
• Survey scores. If agents can ask customers for a good rating, or choose who receives the survey, the score measures how well people ask, not how well they help.

In every case the number is real. The behaviour that produced it is the problem. And the dashboard will not tell you, because the dashboard is only showing the test.

How does DMAIC help you catch this?

DMAIC is useful here because it forces a question the dashboard never asks: is this the right thing to measure at all? The stages each carry a check for it.

Define. Write down what the KPI is supposed to represent in plain language. Not “average handle time under seven minutes” but “customers get their problem solved without wasting their time or ours.” If you cannot state the purpose without the number, you do not have a purpose yet.

Measure. Measure the number, and measure something that should move with it if the number is honest. If handle time falls, repeat contacts should not rise. If resolution rates climb, callbacks should not. A KPI with no companion check is a test with no real-world comparison.

Analyze. Look for the gap. Pull a sample of calls where the target was hit and listen to them. Pull a sample where it was missed. Ask which ones you would be happy to have a customer hear. If the missed calls sound better than the hit ones, the KPI is steering you the wrong way.

Improve. Fix the measure before you fix the number. Sometimes that means a second metric. Sometimes it means removing a target entirely, which is uncomfortable, and less uncomfortable than a number everyone knows is gamed.

Control. Review the KPI itself on a schedule, not only its value. Ask each quarter whether it still represents what it did when you chose it. People change, products change, and the test slowly stops matching the world.

What does an honest KPI review look like?

Take handle time as an illustration. Suppose the target is seven minutes and the team is averaging six and a half. On the dashboard, that is green. An honest review asks three more questions before anyone celebrates.

• What happened to repeat contacts over the same period? If customers are calling back about the same issue, the short calls were not short, they were unfinished.
• What do the longest calls have in common? Often they are the hard problems that most need an experienced person, and the target is quietly pushing agents to rush them.
• What would the team do if the target disappeared tomorrow? If the answer is “take longer on the difficult ones,” the target is working against the purpose.

This is not an argument against targets. Targets give a team something to aim at, and without them a lot of operations drift. It is an argument for being clear that a target is a hypothesis: if we hit this number, customers will be better served. Like any hypothesis, it can be wrong, and you only find out by checking.

Who should own the question?

The person who owns the KPI should also own its honesty. In practice that means the manager responsible for the number is the one who has to explain, in plain language, what it measures and what it does not. If that explanation is uncomfortable, good. It is the conversation you want to have before an outsider has it for you.

It also helps to have someone outside the team look at the number occasionally. People who live with a metric stop seeing its blind spots. A colleague from another function, or a quality reviewer who listens to a handful of calls, will notice things the team has stopped noticing.

How do you measure the thing and not the test?

Think about the customer, not the process. The test is always something your own team can see. The thing is what the customer experienced, and they are usually the last to be asked.

• Pair every speed metric with a quality metric. Handle time with repeat contact. Time to close with reopen rate.
• Use outcomes measured from the customer’s side where you can: did they call back, did they complain, did they renew.
• Sample the work, not just the data. Listening to ten real calls a week tells you things no report will.
• Keep the people being measured involved in designing the measure. They know exactly where it can be gamed, and they will tell you if you ask.

None of this is complicated. What is hard is being willing to look. A good number is comfortable. A bad number is useful. A number that looks good and is wrong is the one that costs you.

What should you do on Monday?

Take your three most-watched KPIs. For each, write one sentence describing what it is supposed to stand for. Then write one way a person could hit the number without delivering that thing. If you can write the second sentence in under a minute, you have found the gap.

Then ask whether anything on your dashboard would reveal it. If not, add the companion measure this week. It does not have to be elegant. It just has to be there, so that the next person to look at the number is looking at more than the test.

For the foundations, the original posts on choosing a KPI and on tracking it are still where I would start. And if you want the primary source for the event this post refers to, the EPA’s announcement of the notice of violation is public.

The dashboard does not automatically win. A measure only deserves your trust for as long as it still means what you think it means.