Partial Outage Communication: What Google’s 14 Percent Morning Teaches Support Teams

Partial outage communication is harder than full outage communication, because half your customers are telling you it’s broken and the other half are telling you it’s fine, and both are right. Google’s rolling outage on 14 May 2009 shows the problem clearly: by Google’s own account, about 14 percent of its users had slow service or interruptions, which means about 86 percent didn’t.

A full outage is a clean story. Everything is down, everyone knows, and the message writes itself. A partial one is messy. It’s the kind of morning where your status page says green, your engineers say they can’t reproduce it, and your phone queue says otherwise.

What happened at Google on 14 May?

According to Data Center Knowledge’s account of the rolling outage, Urs Hölzle, who oversees Google’s data centre operations, explained that an error in one of Google’s systems sent some of its web traffic through Asia and created a traffic jam. About 14 percent of users saw slow services or interruptions. Google News was the most visible casualty, and Google’s Apps status page listed a service disruption for Gmail. Hölzle’s post said Google was very sorry it happened.

The post itself, on the official Google blog, was titled “This is your pilot speaking. Now, about that holding pattern.” It’s a good title. It’s plain, it’s a little human, and it tells you the writer knows you were stuck waiting.

InformationWeek reported that users took to Twitter in large numbers while it was happening. That detail matters more than the cause. The customers who were affected found each other faster than Google’s own channels found them.

Why is a partial outage harder to communicate?

Because every signal you normally rely on is mixed. Your monitoring may show most requests succeeding. Your engineers, sitting on a different network path, may see everything working. Your first-line agents test it themselves and it loads. Meanwhile a customer in one region, or on one provider, or using one feature, can’t get anything done.

The danger is the “works for me” reply. It’s the most natural thing an agent can say, and during a partial outage it’s the most damaging. It tells the customer that the problem is theirs, that they should go and check their own setup, and that you’re not going to look further. When it later turns out you were the problem, that reply is what they’ll quote.

I said in my post on the Gmail outage apology that the apology is judged against what customers lived through, not against the uptime figure. Partial outages make that gap wider. The 14 percent didn’t experience 86 percent availability. They experienced 0 percent.

What should your agents say when they can’t reproduce it?

Something that keeps the door open. Train this as a standard line, so nobody has to invent it under pressure: “I can’t reproduce it from here, but that doesn’t mean it isn’t happening. Tell me where you are and what you’re seeing, and I’ll check it against other reports.” It’s longer than “works for me.” It’s also true.

Then make sure there’s somewhere for those details to go. During a partial outage, the pattern is the diagnosis. Ten tickets that each look like an individual problem become an incident the moment someone notices they share a region, an ISP, a browser, or a time window. Your agents are the ones holding that data, often before monitoring sees it.

• Capture location and network details on every “it’s slow” ticket during unusual volume.
• Tag those tickets so a supervisor can spot a cluster within minutes, not hours.
• Set a threshold. Five similar reports in fifteen minutes is enough to call it in, even if the dashboards look fine.

The numbers in that last line are an example, not a standard. Pick your own, but pick something. A threshold that’s never written down is a threshold that depends on who happens to be on shift.

Where should the public message go?

Where your customers are already talking. In 2009 that increasingly means Twitter, and I wrote in December about who gets listened to on Twitter. The short version: if your affected customers are comparing notes in public, your silence is part of the conversation too.

The message for a partial outage needs one thing a full outage message doesn’t: a description of who is affected. “Some users are experiencing slow service” is better than nothing. “Some users, mainly in Asia and on certain networks, are seeing slow service; most users are unaffected” is far better, because it lets the unaffected stop worrying and the affected stop doubting themselves. If you don’t yet know who’s affected, say that, and say when you’ll update.

Should a partial outage count against your SLA?

It depends on how you wrote it, and most SLAs weren’t written with this in mind. A promise of 99.9 percent availability usually means the service as a whole. A customer in the affected 14 percent had a terrible morning that may not register in your monthly number at all.

That’s the point I made in why an SLA credit isn’t a remedy: the contract measures what’s convenient to measure, and the customer measures what happened to them. For your top-tier customers, consider whether “available” should be measured from their side, not yours. At minimum, make sure your account managers know which of their customers were in the affected group before those customers ring them.

What does the post-mortem need to say?

Three things that customers can use, and the technical cause is only one of them.

• What happened, in plain language. Hölzle’s “traffic jam” image is a good example: no jargon, and still accurate.
• Who was affected. Not just a percentage. Which services, which regions, which window of time.
• What will change. A cause without a change is a description, not an explanation.

Hölzle also said, in an earlier interview quoted in the same coverage, that configuration issues and rate of change play a significant role in many of Google’s outages. That’s an honest sentence, and it’s useful for your own team. A partial outage caused by a routine change is the most common kind, and the hardest to see from inside.

What does a partial outage look like from the queue?

Usually like a bad morning rather than an incident. Handle times creep up because agents are troubleshooting things that aren’t broken on the customer’s side. Repeat contacts rise because the first answer didn’t work. The ticket categories scatter, because each agent files the problem under whatever the customer happened to describe: login, slowness, email, search.

That scatter is the reason partial outages get missed. Nobody sees one big category spike. They see a dozen small ones, each inside normal variation. A supervisor looking at the queue by category will think it’s a busy day. A supervisor looking at it by time and location might see that all the odd calls started at the same minute.

So give your floor leads one extra view for unusual days: contacts in the last hour, grouped by where the customer is and what they’re using, not by what they said was wrong. It doesn’t need special software. A spreadsheet pivot will do it if the fields are captured. The point is to look at the queue the way the fault would see it, not the way your categories do.

And when the pattern shows up, act on it before engineering confirms it. Telling customers “we’re seeing reports from your area and investigating” costs nothing if it turns out to be nothing. Waiting for confirmation costs you every customer who calls in the meantime and hears “works for me.”

How do you prepare for the next one?

Write the partial-outage version of your incident message now, before you need it. Most teams have a template for “everything is down.” Few have one for “something is wrong for some people.” The second is the one you’ll use more often.

Then rehearse the “works for me” moment with your agents. Give them a scenario where the system works perfectly on their screen and a customer insists it doesn’t. See what they say. If the first instinct is to send the customer off to clear their cache, you’ve found your training gap.

Finally, look at how fast you’d notice. If the first signal of a partial outage at your company would be a customer on Twitter, your monitoring is measuring the wrong thing, or your agents’ reports aren’t reaching anyone who can act on them.

What’s the real lesson from the 14 percent?

That averages hide people. A morning where 86 percent of users had no problem is a good morning on a chart and a terrible one for everyone else. Support teams are the part of the company that hears from the 14 percent first.

Make sure that when they do, someone believes them. The outage was Google’s problem to fix. Believing the customer is yours.