Vendor Outage SLA: What the 2008 BlackBerry Outage Taught Support Teams

When a vendor goes down, your customer’s clock keeps running, and it’s your name on the SLA. A vendor outage SLA that ignores that isn’t worth the paper it’s printed on. The February 2008 BlackBerry outage is the cleanest example I know of what this looks like from inside a support team: millions of people, one cause nobody in the building controlled, and a help desk that still had to answer the phone. Nobody on that help desk could fix a thing. Every one of them still owned the customer’s afternoon.

In my post on internal and external SLAs, I argued that every link behind a customer promise needs a commitment faster than the promise itself. This post pushes that one step further, to the link you don’t own at all. That’s the one most service agreements quietly skip. It’s also the one that causes the worst afternoons.

What happened during the BlackBerry outage?

On Monday 11 February 2008, the BlackBerry email network run by Research In Motion went down. According to Data Center Knowledge’s report on the day, service was out for about four hours, and most users were back at roughly 7:30pm Eastern. RIM’s own emergency notice to customers, sent around 4pm, said the outage affected enterprise clients and users of the Americas network. CNBC reported, per the same article, that email was affected for all of an estimated 12 million BlackBerry users. Phone service kept working.

The wording changed as the evening went on. Early reports described a “critical severity outage.” The company later described the problem to media as “intermittent service delays.” Anyone who has written a status update knows how that happens. Nobody is lying. The first message is written when you know the least and fear the most, and the second is written by someone who has been asked to make it sound smaller.

It was also the second large RIM outage in under a year. The first was in April 2007. When the same vendor fails twice in twelve months, the third time isn’t a surprise, and any plan that treats it as one has a hole in it. I’d go further. Two failures is a pattern you can plan against, and that’s good news, because a pattern is the one kind of risk you can actually staff for.

Whose problem is a vendor’s outage?

Yours. Not legally, maybe, and not morally. Operationally. Think about who picked up the phone on the afternoon of 11 February. The BlackBerry user in the field called their company’s help desk. The help desk had no way to fix it. The help desk’s customer, the user’s manager, wanted to know when it would be fixed and had no interest in the supplier’s org chart.

That’s the uncomfortable part. Your SLA to your customer is a promise about their experience, and a vendor failure sits inside that experience. You can’t hand the problem across the table and say “that was RIM.” The customer hears an excuse, and they’re right to. They bought a working service from you, not a list of your suppliers.

There’s a longer chain behind it. The device maker runs the network. The carrier sells the plan. The company’s IT team runs the BlackBerry server. The help desk answers the user. Each of those has its own clock and its own definition of “resolved.” The user holds all of them to one: their email works, or it doesn’t.

ITIL has treated this as its own discipline since version 3 came out in 2007, under supplier management, and the current ITIL 4 supplier management practice guide from Axelos still frames it the same way: the supplier’s performance is part of the service you deliver, so you manage it like part of the service. I find that framing more useful than any contract clause. It moves the vendor from “someone else’s problem” to “a dependency we run.” Once it’s a dependency, it gets monitoring, an owner and a plan.

What can a support team control when the vendor goes down?

More than most teams think. You can’t fix the outage, but you control four things, and they decide whether the next day’s conversation is about the vendor or about you. The first is how fast you know. Did you find out from your monitoring, from the vendor or from a user? If the answer is the user, you’re already late, and even a simple check, such as a test device that sends a message every few minutes, buys you time.

The second is how fast you tell people. One short, honest message to the affected users and their managers within the first 30 minutes does more than a perfect report four hours later. The third is what you say. Say what you know, what you don’t and when you’ll next update. Don’t repeat the vendor’s softest wording: if they call it an “intermittent delay” and your users can’t send email, say email isn’t working.

The fourth is what the team does with the calls. Put a recorded message or a banner at the front. Give agents a single script. Move people off other work for the duration. A customer who gets a straight answer in 20 seconds doesn’t need to stay on the line for ten minutes, and every call that ends in 20 seconds frees a line for someone who has a different problem entirely.

I used to think a vendor outage was a reason for patience. It’s a reason for preparation. The patience comes later.

Why do vendor SLAs rarely cover what you promised?

Most vendor SLAs are written around the vendor’s service, not your customer’s outcome. They promise a percentage of availability across a period and, if they miss it, a credit. They don’t promise that your customer’s email will work at 3pm on a Monday, and they rarely compensate you for the cost of the calls you take. I made the case in why an SLA credit isn’t a remedy, using Amazon S3’s outage the same month, and BlackBerry is the same lesson from the other side of the desk.

Here’s the arithmetic in plain numbers, using a 99.9% monthly target as the example. A 30-day month is 30 x 24 x 60 = 43,200 minutes. The 0.1% the vendor is allowed to miss is 43,200 x 0.001 = 43.2 minutes, so roughly 43 minutes of downtime keeps them in compliance. Four hours is 240 minutes, and 240 / 43,200 = 0.0056, or 0.56%, which leaves the month at 99.44% available. That’s a clear breach, and the remedy is still only whatever credit the contract names.

The credit goes to whoever signed the contract. Your customer, who lost the afternoon, doesn’t see a cent. So the question isn’t “is the vendor SLA good?” It’s “does the vendor SLA plus my own commitment still add up to what I told the customer?” If you sold a 4-hour restore and the vendor’s own contract gives them 24 hours to fix a major fault, you’ve promised something you can’t deliver, on a clock you don’t control.

You can find that gap before the vendor finds it for you. Put it in a spreadsheet. I’d call the workbook “Vendor gap” and keep it to one sheet with a header row and one row per dependency. Column A holds the dependency’s name. Column B holds the vendor’s restore commitment in hours, taken from their contract. Column C holds the restore or workaround time you promised your own customer, also in hours. Column D works out the gap and column E flags it.

Take a made-up example in row 2: mobile email, with a vendor commitment of 24 hours in B2 and a customer promise of 4 hours in C2. In D2, type =C2-B2. That subtracts the vendor’s time from your promise, so 4 – 24 = -20. A negative number means your promise is shorter than the vendor’s, by 20 hours. In E2, type =IF(D2<0,”GAP”,”OK”). The IF checks whether D2 is below zero, and shows “GAP” if it is and “OK” if it isn’t, so row 2 shows GAP.

Copy those two formulas down for every row. Then sort by column D, smallest first, and the worst gap sits at the top. A gap doesn’t always mean you change the contract. Sometimes it means you change your promise to “workaround within 4 hours” and make sure the workaround exists. Either way, you found it on a quiet Tuesday instead of on the day it broke.

How should the customer-facing SLA be written?

With the vendor in mind from the start. Carve the dependency out clearly, not sneakily. If the customer’s service depends on a named third party, say so in the agreement and say what your commitment is when that party fails. Usually it’s a commitment to communicate, escalate and restore on your side, not to guarantee the third party’s uptime.

Then promise what you control. Time to acknowledge, time to first update and time to workaround are fully yours, and they’re what customers remember. Nobody recalls the availability percentage a year later. They recall whether you told them before they had to ask.

Have a plan B for the critical dependency. For a business that relies on mobile email, that might be webmail, SMS alerts or a conference bridge. It doesn’t need to be as good. It needs to exist. And put the vendor in the escalation matrix: the named contact, the number to ring and the person on your side who rings it. My tiered SLA matrix post shows how to lay this out by customer tier.

Finally, review the vendor’s track record. Two outages in a year is information. Put it in the next renewal conversation and in your risk register, with a date next to it. None of this removes the risk. It changes who’s surprised when it happens.

What does a good message look like in the first 30 minutes?

The first message decides how many calls you take, so it’s worth writing before you need it. I covered the internal side of those first minutes in escalation triggers. This is the external side. Here’s a template, in plain language, for a vendor outage affecting your customers.

Part of the message	Example wording
What’s happening	“Email on mobile devices is not working. We’ve confirmed this is a problem with the service provider’s network, not with your device or our server.”
Who’s affected	“This affects all users who receive email on their handheld. Phone calls and text messages are unaffected.”
What we’re doing	“We’ve escalated to the provider and we’re in contact with them. We’ll update you again by 5pm.”
What you can do	“If you need urgent access, use webmail from a computer. Call us only if you can’t reach your email any other way, so we can keep the line free.”

That last line matters. Asking customers not to call is a legitimate staffing tool, but only if you give them another route. Without one it sounds like a brush-off. With one, it’s the cheapest capacity you’ll ever add.

Here’s why, with made-up round numbers. Say 2,000 people in a company use the handhelds, and 15% of them call during the outage: 2,000 x 0.15 = 300 calls. At 6 minutes each, that’s 300 x 6 = 1,800 minutes, and 1,800 / 60 = 30 agent hours in a single afternoon. If a clear message in the first 30 minutes halves those calls, you take 150 calls instead of 300, and 150 x 6 / 60 = 15 hours of work disappear. That’s two people’s afternoons back for one paragraph of text.

How do you learn from it afterwards?

The review isn’t about the vendor. It’s about your own part of the chain. After any third-party failure, sit down with the team for 30 minutes, while it’s fresh, and answer five questions. Keep the vendor’s post-incident note to hand, if they sent one, but don’t let it set the agenda. Their report explains their failure. Yours explains how your customers felt it, and only your team can write that one.

• When did we first know, and how?
• How long until the first message to customers?
• Which calls could a better message have avoided?
• Did the vendor’s updates arrive before or after our users’ questions?
• What would we change in the SLA, the escalation list or the plan B?

Write down the answers and attach a name and a date to each change. Without that, the review becomes a conversation. With it, the review is how you become a better supplier yourself. The fourth question is the one I’d watch closest. If your users keep asking before the vendor tells you anything, the vendor’s communication is the gap, and that’s a renewal conversation, not a help desk one.

Then go back to the Vendor gap sheet and update it. If the outage ran longer than the vendor’s contract allows, note the real duration next to the promised one. Show me a vendor gap sheet that never changes after an outage and I’ll show you a review that didn’t happen.

What would I do on Monday?

List every third party your customers depend on through you. Not the whole vendor list, just the ones where a failure becomes your customer’s problem. Put each one in a row of the sheet, with the vendor’s restore commitment, your promise to the customer and the gap between them. For most teams it’s fewer than ten rows, and it takes an afternoon.

Then pick the biggest gap and decide what you would say, in the first 30 minutes, if that vendor went dark at 3pm on a Monday. Write the first message now, and save it where the person on shift can find it without asking you.

Last, send the sheet to whoever owns the vendor contracts and book 30 minutes with them before the next renewal. Bring the worst row and the calls-avoided sum. Procurement teams negotiate on price because price is what they’re shown. Show them that a 20-hour gap costs your team 30 agent hours every time it fires, and the conversation moves to restore times. That’s the clause that protects your customer.

A chain is only as fast as its slowest link, and the slowest link is often the one you don’t own. You can’t speed it up. You can make sure your customer hears about it from you first.