Vendor Outage SLA: What the 2008 BlackBerry Outage Taught Support Teams

When a vendor goes down, your customer’s clock keeps running, and it’s your name on the SLA. The February 2008 BlackBerry outage is the cleanest example I know of what that looks like from the inside of a support team: millions of people, one cause nobody in the building controlled, and a help desk that still had to answer the phone. A vendor outage SLA that ignores that reality isn’t worth the paper it’s printed on.

In my post on internal and external SLAs, I argued that every link behind a customer promise needs a commitment faster than the promise itself. This post pushes that one step further, to the link you don’t own at all.

What happened during the BlackBerry outage?

On Monday 11 February 2008, the BlackBerry email network run by Research In Motion went down. According to Data Center Knowledge’s report on the day, service was out for about four hours, and most users were back at roughly 7:30pm Eastern. RIM’s own emergency notice to customers, sent around 4pm, said the outage affected enterprise clients and users of the Americas network. CNBC reported, per the same article, that email was affected for all of an estimated 12 million BlackBerry users. Phone service kept working.

The wording changed as the evening went on. Early reports described a “critical severity outage.” The company later described the problem to media as “intermittent service delays.” Anyone who has written a status update knows how that happens. Nobody is lying. The first message is written when you know the least and fear the most, and the second is written by someone who has been asked to make it sound smaller.

It was also the second large RIM outage in under a year. The first was in April 2007. When the same vendor fails twice in twelve months, the third time is no longer a surprise, and any plan that treats it as one is a plan with a hole in it.

Whose problem is a vendor’s outage?

Yours. Not legally, perhaps, and not morally. Operationally. Think about who picked up the phone on the afternoon of 11 February. The BlackBerry user in the field called their company’s help desk. The help desk had no way to fix it. The help desk’s customer, the user’s manager, wanted to know when it would be fixed and had no interest in the supplier’s org chart.

That’s the uncomfortable part. Your SLA to your customer is a promise about their experience, and a vendor failure is inside that experience. You can’t hand the problem across the table and say “that was RIM.”

There’s a longer chain behind it. The device maker runs the network. The carrier sells the plan. The company’s IT team runs the BlackBerry server. The help desk answers the user. Each of those has its own clock and its own definition of “resolved.” The user holds all of them to one: their email works, or it doesn’t.

What can a support team control when the vendor goes down?

More than most teams think. You can’t fix the outage, but you can control four things, and they decide whether the next day’s conversation is about the vendor or about you.

• How fast you know. Did you find out from your monitoring, from the vendor or from a user? If the answer is the user, you’re already late. Even a simple check, such as a test device that sends a message every few minutes, buys you time.
• How fast you tell people. One short, honest message to the affected users and their managers within the first 30 minutes does more than a perfect report four hours later.
• What you say. Say what you know, what you don’t and when you’ll next update. Don’t repeat the vendor’s softest wording. If they call it an “intermittent delay” and your users can’t send email, say email isn’t working.
• What the team does with the calls. Put a recorded message or a banner at the front. Give agents a single script. Move people off other work. A customer who gets a straight answer in 20 seconds doesn’t need to stay on the line for ten minutes.

I used to think a vendor outage was a reason for patience. It’s a reason for preparation. The patience comes later.

Why do vendor SLAs rarely cover what you promised?

Most vendor SLAs are written around the vendor’s service, not your customer’s outcome. They promise a percentage of availability across a period and, if they miss it, a credit. They don’t promise that your customer’s email will work at 3pm on a Monday, and they rarely compensate you for the cost of the calls you take.

Do the arithmetic, using round numbers for illustration. A service that promises 99.9% availability over a 30-day month can be unavailable for about 43 minutes and still be in compliance. Four hours in a single day, as on 11 February, is far beyond that, but the SLA still only offers whatever credit is in the contract. The credit goes to whoever signed it. Your customer, who lost the afternoon, doesn’t see a cent.

So the question isn’t “is the vendor SLA good?” It’s “does the vendor SLA plus my own commitment still add up to what I told the customer?” If you sold a 4-hour response and the vendor’s own contract gives them 24 hours to fix a major fault, you’ve promised something you can’t deliver, and you’ve promised it on a clock you don’t control.

How should the customer-facing SLA be written?

With the vendor in mind from the start. A few approaches work.

• Carve out clearly, not sneakily. If the customer’s service depends on a named third party, say so in the agreement and say what your commitment is when that party fails. Usually it’s a commitment to communicate, escalate and restore on your side, not to guarantee the third party’s uptime.
• Promise what you control. Time to acknowledge, time to first update, time to workaround. These are fully yours, and they’re what customers remember.
• Have a plan B for the critical dependency. For a business that relies on mobile email, that might be webmail, SMS alerts or a conference bridge. It doesn’t need to be as good. It needs to exist.
• Put the vendor in the escalation matrix. Name the contact at the vendor, the number to ring and the person on your side who rings it. My tiered SLA matrix post shows how to lay this out by customer tier.
• Review the vendor’s track record. Two outages in a year is information. Put it in the next renewal conversation and in your risk register.

None of these removes the risk. They change who’s surprised when it happens.

What does a good message look like in the first 30 minutes?

Here’s a template, in plain language, for a vendor outage affecting your customers.

• What’s happening: “Email on mobile devices is not working. We’ve confirmed this is a problem with the service provider’s network, not with your device or our server.”
• Who’s affected: “This affects all users who receive email on their handheld. Phone calls and text messages are unaffected.”
• What we’re doing: “We’ve escalated to the provider and we’re in contact with them. We’ll update you again by 5pm.”
• What you can do: “If you need urgent access, use webmail from a computer. Call us only if you can’t reach your email any other way, so we can keep the line free.”

That last line matters. Asking customers not to call is a legitimate staffing tool, but only if you give them another route. Without one it sounds like a brush-off.

How do you learn from it afterwards?

The review isn’t about the vendor. It’s about your own part of the chain. After any third-party failure, sit down with the team for 30 minutes and answer five questions.

• When did we first know, and how?
• How long until the first message to customers?
• Which calls could a better message have avoided?
• Did the vendor’s updates arrive before or after our users’ questions?
• What would we change in the SLA, the escalation list or the plan B?

Write down the answers and attach a name and a date to each change. Without that, the review becomes a conversation. With it, the review is how you get to be a better supplier.

What would I do on Monday?

List every third party your customers depend on through you. Not the whole vendor list, just the ones where a failure becomes your customer’s problem. Next to each, write the vendor’s response commitment, your promise to the customer and the gap between them.

Then pick the biggest gap and decide what you would say, in the first 30 minutes, if that vendor went dark at 3pm on a Monday. Write the first message now.

A chain is only as fast as its slowest link, and the slowest link is often the one you don’t own. You can’t speed it up. You can make sure your customer hears about it from you first.