When a Supplier Fails: Escalating Past the Manager

The last step of any escalation plan is the one people avoid writing down: what happens when the problem is bigger than anyone on the support team can fix, and bigger than the manager’s authority too. That step needs a senior person, a decision, and a plain message to the customer. In October 2009, T-Mobile’s Sidekick customers got an example of what happens when the plan stops short of it.

In my post on the curse of the berry I argued that a manager should never be the final point of escalation, and that senior management belongs in the plan. Here is why.

What happened to Sidekick customers in October 2009?

On Friday, October 2, Sidekick owners began to lose their data connection. The Sidekick was a messaging phone made by Danger, which Microsoft had acquired, and its contacts, calendar and photos lived on servers rather than on the handset. T-Mobile and Microsoft first expected service to return around October 4. It didn’t.

By the 10th, T-Mobile was telling customers that personal information stored on the device and no longer on it had almost certainly been lost, and that the likelihood of a successful recovery was extremely low. It advised owners not to reset their phones or let the battery run flat, because that would erase what remained locally. By October 10 the data centre trade press was reporting that Microsoft had lost the Sidekick backups, and the failure was at Microsoft’s Danger subsidiary, not at T-Mobile.

For a support organisation, the technical story matters less than the shape of it. The customer’s relationship was with T-Mobile. The fault was in a supplier’s data centre. Every call to a T-Mobile agent was a call about something the agent couldn’t see and couldn’t fix.

Why does a vendor failure break the normal escalation path?

Because the normal path assumes the fix lives inside your company. Tier 1 escalates to Tier 2, Tier 2 to the manager, the manager to an engineer. That chain works when the engineer is yours. When the fault sits in a supplier, the chain ends at a wall, and every level below it just relays the wait.

There are three typical symptoms:

• The same script, over and over. Agents repeat “we’re working on it” because that’s the only thing they know.
• Repeat contacts climb. People who hear nothing new call again. Volume rises exactly while the team has the least to say.
• The manager becomes the final point by default. They get the calls at night because there’s nobody above them who can say anything different.

The way out is to put a supplier-failure branch into the escalation matrix before you need it. I described the escalation matrix as a list of who gets called and in what order. Add one more row for supplier failure: “the supplier is the cause.” That row should name the person on your side who owns the supplier relationship, the person at the supplier you reach, and the senior person on your side who can decide what to tell customers.

What does senior management actually decide?

Three things that nobody lower down can decide, and all of them are time-sensitive.

• What to say, and how soon. In the Sidekick case, the hardest message was that the data was probably gone. Someone senior has to approve that sentence, and the delay between knowing it and saying it is where trust is lost.
• What to offer. Refunds, credits, replacement service, a way to recover whatever can be recovered. Agents can’t make these up. They need to be approved and written down so everyone says the same thing.
• Who owns the customer while the supplier works. One named person, with a status update on a fixed schedule, even if the update is “no change.”

Look at what the T-Mobile instructions to customers had in common: they were specific. Don’t reset the phone. Don’t let the battery die. Those two sentences protect something real. That’s the standard to aim for in a message to customers: the most useful thing they can do right now, in one line, even when the news is bad.

What should a manager do while waiting for the decision?

Stop being the message. A manager who relays every update by phone at 11pm is using up the one resource a long incident depends on, which is their judgment at hour six. Instead:

• Write one holding statement and put it where every agent can find it.
• Set a fixed update time, for example every two hours, and keep it even when there’s nothing new.
• Rotate the people on the incident. A long outage is a relay, and people who’ve been on it for ten hours make worse calls.
• Log the contacts. The log shows what the incident cost, and it feeds the review afterwards.

It’s the same rota logic I described for ordinary after-hours cover. Long incidents just make the need for it obvious.

What does a good supplier-failure message sound like?

It says what happened, what the customer can do, and when they’ll hear next, in that order and in plain language. Not a paragraph of apology, not a technical explanation, not a hope. Three short statements.

A workable template, written for a hosted tool outage, looks like this: “Our ticketing supplier is down and we can’t see your open requests. If your issue is urgent, phone this number and give us your name and the problem. We’ll post an update at the top of every hour until it’s resolved.” Every sentence in it is something an agent can say, and none of it requires access to the system that is broken.

The honest version of bad news is also the cheapest. In the Sidekick case, the hardest sentence in T-Mobile’s message was the one saying that data had almost certainly been lost. It was also the one that let customers stop waiting and start deciding what to do. Customers can cope with a loss. What they can’t cope with is not knowing.

How do you prepare before it happens?

With a one-hour exercise. Sit down with your senior contact and pick one supplier. Ask the question: what do we do if this supplier loses our data or goes down for two days? Write the answer on one page. It should name three people, one phone number for each, the first message to customers, and the first decision someone will have to make.

You won’t get it perfect, and you don’t need to. The aim is that on the day, nobody has to invent the escalation path under pressure. Most of the stress in a long incident comes from not knowing who is allowed to decide, and a page written in advance removes most of it.

Who should be in the room when the decision is made?

Fewer people than you think. A long incident attracts everyone who has an opinion, and a call with twelve people on it rarely produces a decision. Keep a small decision group: the senior owner who can approve spend and messaging, the person who owns the supplier relationship, the support manager, and one person taking notes. Everyone else gets updates after the fact.

Write the group down in advance, with a backup for each seat. When the decision needs to be made at midnight, you don’t want to be deciding who gets to decide.

What is the lesson for a team nowhere near a mobile network?

Most of us don’t run a phone service. We run a help desk that relies on a hosted ticketing tool, an outsourced payment provider, a cloud phone system. Each of those is a Sidekick waiting to happen on a smaller scale. The question to ask this week is simple: if our supplier lost our data tonight, who would we call, who would decide what to tell customers, and who would be allowed to say it?

If the answer is “the manager, probably,” then the plan is unfinished.

A good escalation plan ends with someone who can make the decision, not someone who can only absorb the call. Build that last row now, while nothing is on fire.

When you run that exercise, keep the output where the on-call person can reach it from a phone. A plan stored on a shared drive nobody can open during an outage is not a plan. It helps to print one copy as well, and to check once a quarter that the phone numbers still ring.