Staffing for a Product Launch You Cannot Forecast: Erlang C Meets iPhone Day

Erlang C staffing for a product launch only works if you can forecast the demand, and a launch is the one day you cannot. The formula is exact about the maths. It knows nothing about the day half a continent decides to do the same thing at the same moment, which is what happened on July 11, 2008, when the iPhone 3G went on sale in 21 countries.

If you read my earlier post on Erlang C scheduling for call centres, you know the model needs three honest inputs: arrival rate, handle time and a service target. A launch breaks the first one, and then the other two follow. This post is about what to do when that happens.

What went wrong on iPhone launch day?

The iPhone 3G went on sale in 21 countries on July 11, 2008. Buyers were meant to walk out of the store with an activated phone. Instead, the activation step ran through Apple’s iTunes servers, which could not cope with the load. Reports from the day described stores telling customers to go home and finish activation on their own computers, where the same servers were just as hard to reach. A software update released the same day for the earlier iPhone also required reactivation, which added existing owners to the queue.

The result was a lot of people holding a phone they couldn’t use. And when something doesn’t work, people call. Every one of those customers had a reason to contact a store, a carrier or a support line, and most of them did it at once.

I don’t have any company’s call volumes from that day, and I’m not going to guess. What the public reports show is enough for the lesson: a single dependency, an activation server, turned a normal launch into a flood of contacts that no schedule built on last month’s pattern could have absorbed. I used to think a better forecast was the answer. It isn’t. A better forecast of the wrong day is still wrong.

Why can Erlang C not see a launch coming?

Erlang C is a queueing model. You tell it how many calls arrive in an interval, how long each takes and how quickly you want to answer, and it tells you how many agents you need. It’s excellent at that, as long as the arrivals look roughly like a steady random stream. A launch isn’t a stream. It’s a wall.

A launch violates three of its assumptions at the same time.

• Arrivals aren’t steady. They arrive in a spike with a long tail, driven by one event, not by many independent customers.
• Handle time isn’t normal. A broken activation call is longer than a routine one. The agent is troubleshooting something they may be seeing for the first time, and handle time rises exactly when volume does.
• Callers don’t give up at a steady rate. People who cannot get through call back, and the retries inflate the arrival rate beyond the number of real customers.

The formula still works. The inputs you give it are simply wrong. Garbage in, perfectly calculated garbage out. Show me a launch-day schedule built on the average of the last four Tuesdays and I’ll show you a team that spends the morning apologising.

What can you plan before the event?

You can’t forecast the spike precisely. You can still prepare for its shape. A few habits are worth building before the date is on the calendar, because the week of the launch is too late to start.

• Treat every launch as a scenario, not a forecast. Work out the staffing for a low, expected and high case, and decide in advance what triggers each one. Write the triggers down, for example: queue over 10 minutes, abandon rate above 20 percent, or a single error code showing up in a third of calls. Numbers beat gut feel at 11am.
• Ask what the customer needs to do after buying. Every extra step, such as an activation, a registration or a download, is a place for volume to appear. List them and ask what happens if each one fails.
• Build a short script for the dependency that can fail. If the thing that breaks is outside your control, agents need an honest message ready, not a different answer from every person. Ten agents improvising ten explanations is how a known issue becomes a rumour.
• Pre-agree your escalation route. The escalation matrix is the part of the plan people write once and never test. Test it before the day. Call the numbers on it and see who picks up.
• Think about the whole queue, not just the phones. A launch day lands on email, chat, stores and social channels at once. Decide in advance which one you protect first.

What should you do while the spike is happening?

Once it starts, the job changes. You’re no longer scheduling for a forecast. You’re managing a queue that’s already too long, and the useful questions are different.

First, what is the cause, and can it be fixed upstream? If one dependency is generating most of the contacts, the best use of your time is to get that fixed or at least explained. An accurate message on the website or the IVR can remove more calls than extra agents can answer. Ten seconds of recorded honesty is cheaper than ten more seats.

Second, can you shorten the common call? If most contacts are about the same problem, a standard answer, a fast-path and permission for agents to skip the usual steps will do more than another ten seats.

Third, what do you tell the people who are waiting? A customer who knows the problem is known and being worked on is less likely to call twice. Silence produces redials, and redials make the spike bigger. Tell people early, even when the news is “we don’t have a fix yet.”

Finally, look after the team. A launch day is long, repetitive and often unpleasant, because the customer is annoyed about something the person on the phone didn’t break. Breaks, rotation and a manager who is visibly in the room matter more than they do on a normal day. Build in a change of task after a couple of hours of angry calls.

Should the service target change on launch day?

Yes, and say so in advance. An 80 percent in 20 seconds target is a sensible everyday promise. On a launch day with a broken dependency, chasing it can mean burning your people out to answer calls that shouldn’t exist.

Agree a different target with the business before the day, written down, so nobody argues about it at noon. Something like: answer everything eventually, protect the oldest waiting customers, and put the saved effort into removing the cause. A lower target held honestly beats a normal one missed in silence.

It also helps to decide who is allowed to change the plan mid-day. Someone has to be able to say, now, move people off email and onto phones. If that decision needs three approvals, the spike will be over before it’s made.

How do you build a surge team without breaking the rest of the operation?

There are only so many places extra capacity can come from. Overtime and shift changes are the quickest, and the most expensive in fatigue. Cross-training is slower and cheaper, which is why it has to start months ahead. Cross-trained staff from other teams are the next. Outside help is the slowest to arrive and the hardest to train in a day.

Whichever you choose, the lesson is the same: the preparation has to happen before the event. People who have never taken a call can’t start on launch day. A short list of staff who have been briefed in advance, who know the product basics and the escalation route, is worth more than a larger list you try to assemble at nine in the morning. Even 10 or 15 briefed people changes the first hour.

The same applies to your tools. Make sure the people you pull in have the access they need. A surge agent who cannot see the customer record is another person adding to the queue, not removing from it.

How do you learn from it afterwards?

The week after is where the real value is. You now have a clean record of what a bad day looks like. Keep it, because nobody will believe the numbers in six months and you’ll need them.

• Record the arrival pattern by interval, including the retry spike, so the next scenario starts from data and not from guesswork.
• Record the real handle time on the problem calls, not the average for the day.
• Write down which message or fix removed the most volume, and how long it took to get it live.
• Ask the team what they needed and did not have. They will tell you if you ask soon enough.

Then feed it back into the model. The next launch still won’t be predictable, but your scenarios will be better and your first hour will be calmer. That’s the whole return on the exercise.

What would I do on Monday?

Pick the next event your business has on the calendar: a release, a promotion, a price change, a seasonal peak. Write down the one step after the customer says yes that could fail and send them to you. Then ask what your plan is for that step if it does.

Erlang C will tell you the number of agents for the calls you expect. Planning is the work you do for the calls you don’t. The formula is a tool, not a plan.

For the background reporting on the day itself, coverage of the iTunes activation problems gives a useful snapshot of what customers actually ran into.