Customer-Reported Outages: What the Hudson Landing Photo Teaches Support Teams

Your customers will often see an incident before you do. Plan for it. Customer-reported outages aren’t a failure of monitoring, they’re a source of monitoring, and the support teams that handle them well are the ones that treat the first outside report as a trigger, not as noise.

The clearest picture of that this month didn’t come from a data centre at all. On January 15, US Airways Flight 1549 came down in the Hudson River, and one of the first images the world saw was taken by a passenger on a ferry heading out to help. Janis Krums took it on his iPhone and sent it to Twitter with a short message: there’s a plane in the Hudson. A UK mobile blog described the photo going round the world within minutes of being posted.

I’m not comparing a support queue to a plane landing in a river. Everyone walked off that plane, and that’s the story. The part that matters for a support team is narrower: the first public report of a major event came from someone standing next to it, with a phone, before any official channel had said a word. That’s now normal, and it’s going to be normal for your service too.

Why do customers spot problems before you do?

Because there are more of them than there are of you, and they’re using the product in ways your monitoring doesn’t cover. Your checks test the login page from one location every few minutes. Your customers are logging in from hundreds of places, on different browsers, doing things in an order nobody tested.

When something breaks for a slice of them, they notice in seconds. Your dashboard may stay green for another twenty minutes, because the check that would catch it doesn’t exist yet. The gap isn’t laziness. It’s arithmetic: a handful of synthetic checks against thousands of real sessions.

That gap used to be hidden, because customers told you by phone and email, one at a time, and the first few calls looked like isolated problems. What’s changing is that they now tell each other first, in public.

What changes when the first report is public?

The clock starts earlier, and you don’t control when. In my post on escalation triggers and the first twenty minutes, the timeline began with the first agent noticing a pattern. If the first report is a post on Twitter or a forum, the first twenty minutes may already be gone before an agent sees anything.

That has two consequences. First, other customers are reading that post and drawing conclusions while you’re still silent. Second, your agents start getting contacts that say “I saw online that you’re down.” If the agent has nothing to say back, the customer learns that the internet knows more than your support team does.

Neither of those is fatal. Both are avoidable if outside reports are part of how you detect incidents, rather than something you find out about afterwards.

How do you turn outside reports into a trigger?

Treat them like tickets. Decide which public places your customers actually use to complain, and give one person on each shift the job of watching them, the same way someone watches the queue. In early 2009 that’s probably a search on Twitter for your product name, the main user forum, and your own community pages. It isn’t every corner of the web.

Then write a rule, the same shape as any other trigger:

• Two or more public reports of the same symptom inside fifteen minutes means the watcher posts it to the duty lead, with links.
• One public report that matches tickets already in the queue counts toward the volume trigger, not separately.
• A public report from a named high-value customer escalates one level higher, the same way their ticket would.
• A public report with no matching tickets gets checked by someone who can reproduce it, within ten minutes.

The numbers are a starting point, not a standard. Set them against your own normal volume. The point is that the watcher doesn’t have to decide whether a complaint “counts.” The rule decides, and the watcher follows it.

Who should answer in public?

Not whoever happens to see it first. Pick one or two people who are allowed to reply on behalf of the company, and give them a short set of approved holding lines. “We’re aware some customers can’t sign in and we’re looking into it now. We’ll post an update here by 3pm.” That’s the whole message.

The same approval rule I described for incident updates applies. A holding line goes out without sign-off. Anything that admits fault, promises a remedy or mentions money goes up a level. Get that agreed in a quiet week, because during an incident nobody will agree to anything in less than half an hour.

Late last year I wrote about who gets listened to on Twitter, and the uncomfortable answer was that loud customers in public sometimes get faster service than patient ones in the queue. Fold your public replies into the same incident so that doesn’t happen. The customer who emailed calmly should get the same update at the same moment as the one who posted angrily.

What do you tell agents when the internet knows first?

Exactly what you’re posting in public, word for word, as soon as it’s posted. The worst moment in a support call is a customer reading out a status message the agent hasn’t seen. It tells the customer the agent is out of the loop, and it’s usually true.

Put the current holding line at the top of whatever screen agents already look at. Change it every time the public message changes. If you can’t do that technically, a person reading it out across the floor every thirty minutes is better than nothing. It’s crude. It works.

Agents also need permission to say “I don’t know yet, and here’s when we’ll know more.” An agent who invents a cause to sound helpful creates a second problem for the incident team to clean up later.

How do you tell a real pattern from one loud customer?

By checking it against something you can see. One angry post isn’t an incident, and treating every complaint as an outage will burn out the watcher and the duty lead inside a week. The rule above handles most of it: two reports of the same symptom, or one that matches tickets already in the queue. The rest comes down to a quick reproduction.

Give the watcher a short checklist. Can someone on the team reproduce the problem in under five minutes? Is it one customer’s setup, or does it happen on a clean account? Are the reports all from one region, one browser, one type of plan? Three yes-or-no answers usually tell you whether you’re looking at a person having a bad day or the first edge of something bigger.

When it turns out to be one customer, reply to them anyway, privately if you can, and log it as a ticket. They told you about a problem. Even if the problem was theirs, the public reply that says “we looked, here’s what we found” is worth more than silence.

Isn’t this just for big companies?

No. It’s more important for small ones. A large company has a monitoring team, a status page and a press office. A twelve-person support team has none of those, and the first sign of trouble is almost always a customer. Watching three public places for an hour a day costs very little. Finding out from your biggest client’s angry account manager costs a lot more.

The size changes the rota, not the principle. On a small team the watcher may also be taking calls. That’s fine, as long as watching is written into someone’s shift instead of being everyone’s job, which in practice means nobody’s.

What should you change this week?

Three things, none of them expensive:

• List the public places your customers actually complain, and stop pretending it’s “the internet.” Usually it’s three or four.
• Name a watcher per shift and add the outside-report rule to your escalation matrix, next to the volume and severity triggers.
• Write five holding lines for the failures you can predict: sign-in, slow pages, email not arriving, billing errors, a full outage. Have them approved now.

Then test it. Next time you run an incident drill, start it with a fake public post instead of a ticket, and see how long it takes for the duty lead to hear about it.

The photo from the Hudson was a reminder of something support teams keep relearning: the people closest to a problem see it first, and they now have a way to tell everyone at once. Build that into your process, and the customer with the phone stops being a surprise. They become your earliest sensor.