A service can be up and still be failing the people who use it. That’s the whole lesson of the MobileMe launch in the summer of 2008, and it’s the clearest case I know for why you should choose the right KPI from the customer’s side of the table instead of from the server room’s.
Apple launched MobileMe in July 2008, alongside the new iPhone 3G, and it went badly. Mail, contacts and calendar sync misbehaved for weeks. By the end of July, TidBITS reported that Apple’s own product team was acknowledging that about 1 percent of users had been stranded without their archived mail since July 18. One percent sounds small. If you’re running a service for millions of paying people, it’s a lot of individual customers, each of whom had lost something personal.
On August 4, Steve Jobs wrote to Apple’s staff, and AppleInsider published his note. He said the service needed more time and testing, and that it had been a mistake to launch it at the same time as the iPhone 3G. I’m not here to relitigate Apple’s launch. I’m interested in a narrower question that every support leader should be able to answer: when a number says things are mostly fine and the customers say they aren’t, which one do you believe?
Why can a healthy dashboard hide an unhappy customer?
Because most dashboards measure the system, and customers experience the outcome. Uptime tells you whether the servers answered. It can’t tell you whether the email a customer needs is actually in the inbox. A mail platform can sit at a very respectable availability figure while 1 in 100 of its users has lost the history of their correspondence. The uptime graph stays green and the support queue fills up.
I wrote the first version of this argument in 2007, in choosing the right KPI. A metric chosen for internal convenience tends to measure effort. A metric chosen from the customer’s side measures outcome. Those are correlated on a good day and completely disconnected on a bad one, and a launch is the bad day you plan for.
Here’s the uncomfortable part. Nobody inside a company is lying when the dashboard is green. Each number is defensible on its own terms. The failure is that nobody chose a number whose job was to say “the customer has a problem,” so the first signal arrives from the customer, in public, and in whatever tone they feel like using.
I’ve watched an NPS score swing 20 points in a quarter while every internal efficiency metric stayed flat or improved. The team was optimizing for the wrong side of the table. A launch makes that gap visible in days instead of quarters, which is the only good thing about launches that go wrong.
Which KPI would have caught the problem first?
Work backward from the customer. Ask what a person is actually trying to do with the service, then ask how you’d know within an hour if they couldn’t. For a mail and sync product, the answer isn’t server availability. It’s something closer to “can this person find their last week of messages and see the same calendar on both devices?”
That gives you a few candidates worth measuring. The share of accounts where sync completes cleanly. The number of contacts to support per thousand customers in the first week after launch, split by reason. The share of those contacts that arrive as repeats from the same customer, which is the clearest sign a fix didn’t hold. None of these come out of a monitoring tool by default. Somebody has to decide they matter and build the tag or the report.
Notice that all three are cheap if you set them up before launch and nearly impossible to reconstruct afterwards. That’s the trap I flagged in the original post: most ticketing systems ship with a handful of metrics switched on, and it’s tempting to treat that list as a decision already made. It was made by a vendor trying to suit everyone, not by you trying to protect your customers.
First call resolution and uptime are the two starting points I usually name, and the MobileMe case shows why neither is automatically right. First call resolution is a fine measure for a help desk and a poor one for an incident where thousands of customers have the same defect that no agent can fix on the phone. Uptime is a fine measure for infrastructure and a poor one for data loss. Pick the one that matches what your specific customers are judging you on.
What do you do about the customers already affected?
You measure that too, and you do it separately from the fix. Apple extended subscriptions for all MobileMe users as compensation, a thirty-day extension first and a further two months later in August. Whatever you think of the gesture, it’s a decision that needs a number behind it: how many customers were affected, for how long, and what did it cost us in goodwill?
A support team can build that number if it tagged incidents properly while the problem was live. This is where the incident and problem split I described in what a help desk is earns its keep. Every contact about the defect is an incident, and each one is logged. The underlying defect is one problem. If the incidents are all tagged to that problem, you can tell leadership exactly how many people it hit and how many contacts it generated. If they aren’t, you’re guessing in a meeting.
The other number worth watching during a live incident is time since last customer update. Customers will tolerate a defect for longer than you’d think if they believe somebody is working on it, and far less than you’d like if they believe nobody is. A stale status page is a KPI failure that no uptime figure will ever show.
How do you choose a KPI before the launch, not after?
Use the three questions I always start with. Which problem are you trying to solve? Who does it actually impact? What outcome do you want to see once you’ve started measuring it? If you can’t answer all three for a number you’re about to put on the launch dashboard, you haven’t chosen a KPI yet. You’ve picked something that was easy to pull.
Then add a rule I wish more teams followed: every launch dashboard needs at least one measure that the customer would recognise. Not “service availability,” but “customers who completed their first sync.” A person outside your company should be able to read it and say, yes, that’s what I care about.
You also need to decide in advance what number makes you stop. Jobs’s note said the launch could have been delayed without consequence. That’s an easy observation in August and a hard decision in July, when the date is public and everyone wants to ship. A pre-agreed threshold, such as a limit on contacts per thousand customers, turns a political argument into a reading from a gauge. Write it down before the launch, and give someone the authority to act on it.
Plan to revise. Your first set of numbers won’t be right, and that isn’t a failure of the plan. It’s the plan working. Measure, analyze, act, and then go back and check that the number still predicts what customers are telling you. The metrics that earn a permanent place on the dashboard are the ones that keep passing that check. Everything else is noise that happened to correlate for a quarter. If you want the mechanics of tracking the number once you’ve picked it, I cover them in part two of that series.
A launch-week scorecard doesn’t need to be clever. I’d put four lines on it and review them twice a day. First, contacts per thousand customers, split by reason. Second, the share of contacts that are repeats. Third, the age of the oldest open customer-facing problem. Fourth, the time since the last update went out to customers. Each line has an owner, a threshold and an agreed action if it’s crossed. That’s it. If a fifth number can’t name the decision it would change, leave it off.
The discipline that makes it work is boring. Whoever owns the line reads it out loud at the same time every day, even when it’s fine. A number that only gets discussed when it’s red teaches everyone to look away from it the rest of the time.
For an outside view on building a metrics program around customer outcomes, ICMI publishes plenty of practitioner guidance on contact centre measurement.
Green dashboards don’t call customer service. Customers do.
Hutch Morzaria is a Director-level CX and Support Leadership professional with 19 years of experience building global support organizations across SaaS, Fintech, and enterprise technology. He has hired dozens of support and CX leaders across his career and holds ITIL Expert certification across V3 and V4.
