Capacity Planning for a Known Event: Inauguration Day Traffic

A demand spike you can put in the calendar is still a spike you can fail. Capacity planning for a known event means working out what the peak looks like, who you depend on to serve it and what you do when that dependency slows down. On 20 January 2009, the Obama inauguration showed what that looks like when millions of people do the same thing at the same moment.

In my earlier post on staffing for a product launch, the spike was a surprise. This one is the harder lesson in a way. Everyone knew the date. Everyone knew roughly when the speech would start. A lot of well-run websites still struggled, and the reasons apply to a support team.

What did Inauguration Day do to the internet?

The numbers are large, and they come with the caveat that sources differ on the exact figures. According to PC World’s account of the day, Akamai, the content delivery network, saw a peak of about 7 million simultaneous streams, mostly live, at roughly 12:15pm Eastern, with total traffic passing 2 terabits per second. The Associated Press, as reported by ABC News, put the figure at 7.7 million people watching Akamai streams at the same time, and noted that on ordinary days simultaneous viewers rarely go above 2 million.

CNN said it served about 1.3 million concurrent live streams just before the inaugural address, breaking the record of 5.3 million total live streams it set on election night, 4 November. According to informitv, that was the largest live online video event to date. CNN also put some viewers on a wait list for a while before they could see the stream.

And it wasn’t only video. PC World reported that a group of 40 large sites tracked by the performance firm Keynote slowed on average during the swearing-in and speech. Several well-known news sites struggled, along with some government sites.

I don’t have any company’s support volumes from that day. What I can say is that if you ran a help desk for any business whose staff were meant to be working, the building was full of people watching a stream, and the network was slow for everyone.

Why did a known event still cause trouble?

Knowing the date isn’t the same as knowing the shape. Three things were hard to see in advance.

• The size of the peak. Online video had never been asked to do this. There was no last year’s inauguration to forecast from, only election night and a guess.
• The width of the peak. Millions of people started at once, over minutes, not hours. A spike that sharp is hard to absorb by adding capacity gradually.
• The dependency chain. A news site doesn’t serve video alone. It leans on a CDN, a network provider, a database and an advertising system, and the weakest of them sets the ceiling.

I used to think a known date was most of the battle. It’s maybe a third of it. The other two thirds are knowing the shape and knowing the chain.

What is the support-team version of an inauguration?

Every business has one or two dates when many customers do the same thing together. A billing run. A tax deadline. A product launch. A sale. A statement date. A rollout of a mandatory update. Think of your own calendar and list them.

For each, ask the same four questions.

• What is the peak, and how fast does it build? Not the daily total. The busiest 15 minutes.
• What do customers do when it goes wrong? They retry. They call. They email. They go to social media. Each of those is a second wave you have to plan for.
• Who do we depend on to serve the peak? Payment processor, delivery partner, hosting provider, a telecoms carrier. Name them.
• What do we tell people before it happens? The cheapest way to cut a peak is to move part of it. “Log in before Friday” can remove a third of your Friday.

That last point is underrated. CNN’s wait list is the same idea in the other direction: when you can’t serve everyone at once, a visible queue beats a silent failure.

How do you plan around a third party you don’t control?

Start by writing down what the third party has actually promised. Most have promised an average, not a peak. Ask them directly: what’s the largest load you’ve served, and what have you changed for our date? You will learn a surprising amount from how they answer, including whether they’ve thought about it.

Then plan your side of the failure.

• A degraded mode. Decide what the lighter version of your service looks like. Text instead of video, a static page instead of a live one, a short form instead of a long one. Having it ready beats designing it at noon.
• A message you can switch on. A banner, an IVR announcement and a social post, drafted and approved. The test is whether you can publish it in five minutes.
• A named person at the vendor. A phone number that gets a human, held by someone on your side who has agreed to ring it.
• A watch. Someone whose only job on the day is to look at the queue and the error rate, and to say “it’s going wrong” before the customers do.

This is the same logic as the escalation matrix: decide in advance who does what, so no one is deciding in a panic.

What does it do to the contact centre?

When a site slows, some of the traffic becomes calls. Not all, and often not at the moment the problem starts. People first try again, then try a different route, then pick up the phone. That lag matters. It means the contact centre peak can trail the web peak by several minutes, and it can last longer.

Staff for the lag. If you have the data, look at the last time something slowed and measure how many minutes passed before the phones picked up. If you don’t have it, start collecting it, because next time you’ll want the number.

Also remember that your own agents are people. On a day like 20 January a lot of them will want to watch. Agree in advance how that works: staggered breaks, a screen in the break room or a rule that the floor is covered. Pretending it won’t matter just means it happens quietly.

What about the people who are not at their desks?

An event like this splits your customers into two groups: those who are watching and those who are working. The watchers are busy, distracted and slow to reply, so your outbound work, such as callbacks and chasing documents, gets a worse response than usual. The workers are on a slower network than normal, which means your site feels broken to them even if it’s fine.

Neither group is a surprise once you’ve said it out loud. So avoid scheduling anything important to land in the window. No big email campaign at 11:45am, no deadline at noon and no planned maintenance on the same afternoon. Moving your own activity is the cheapest capacity you’ll ever buy.

How do you tell leading signals from lagging ones?

The best protection on the day is a signal that moves before the queue does. In my post on leading indicators for support teams, the idea was to watch something that moves ahead of the problem. For an event, that might be the rate of logins in the ten minutes before the start, the number of people on the wait list or the volume of searches on your help site.

Pick two and put them on one screen. A leading signal turns a surprise into a warning of a few minutes, which is often enough to switch on the message and call the flexible pool.

What would I do on Monday?

Open your calendar for the next quarter and mark every date on which a large share of your customers will do the same thing at the same time. For the biggest one, write one page: the expected peak, the third parties involved, the degraded mode, the message and the person who watches.

Then ask the vendor what they have changed for that date. If the answer is “nothing,” you’ve learned something.

A date on the calendar gives you time. It doesn’t give you a plan. The inauguration was the most scheduled event of the year, and a lot of capable teams still found out where their limits were in front of a very large audience.