On February 3 we switched on AI ticket triage at Q4, and from the first day it handled 100% of incoming tickets. Two models, one launch date, no soft launch. The model was the easy part. The work that made it possible was cleaning years of inconsistently tagged tickets, and the work that came after was deciding what to do for the people whose job it had been.
The full numbers are in the AI ticket triage case study, with a shorter account of what I learned implementing it. Here I want to tell it as part of the plan I set out in building a three-to-five-year support plan, where AI came last for a reason, and talk about what it meant for the team.
What problem was AI triage meant to solve?
Every incoming ticket had to be classified by a person before it could be routed to the right team. Somebody read the request, decided what category it belonged to and tagged it. Added up across the support organisation, the work was consuming the equivalent of six full-time people. It was repetitive and low in value, and it was error-prone, because misclassified tickets arrive in the wrong queue, wait there and get rework from agents who weren’t the right people to handle them.
It also fed the problem I’d found in my first weeks: a long wait for a first response. Hand sorting is a delay at the front of every request. Remove it and you shorten the time to first response for every customer, not just for a subset.
The case for acting was stronger than I’d have guessed a few years earlier, because the technology had matured. Ticket classification was exactly the kind of task machine learning does well: a lot of historical examples, a limited number of categories and an objective way to check the answer.
How did the two models work?
We chose Forethought as the platform and designed two models to run from launch. The first was the category triage model, which classified each incoming ticket and routed it to the right team automatically. It replaced the main manual workload and accounted for $160K of the savings. The second was an approve-or-decline model, which handled a specific type of request that previously needed a person’s review every time, and it added $40K. Together that made $200K in the first year.
Both were trained on historical ticket data before launch. That training data is where most of the effort went. Tickets that had been tagged by hand over several years had been tagged inconsistently, and a model trained on inconsistent labels learns the inconsistency. We had to clean, standardise and re-categorise a large amount of history. The model took weeks. The data took months.
We also built a human-in-the-loop layer. When the model’s confidence in a prediction fell below a threshold, the ticket went to a person instead of being forced into a category. I’d argue that decision mattered more than the accuracy figure. It kept the system trustworthy. The tickets sent to humans were disproportionately the ambiguous ones, which is exactly where you want human judgement, and they gave us a stream of hard examples to learn from.
Why launch to 100% on day one?
Because we had enough confidence in the data and in the models, and because a partial rollout creates its own problems. Running two systems in parallel means two sets of rules, two places for things to go wrong and an ongoing argument about which to trust. Going to full coverage meant everyone learned the new process at once, and any problem showed up quickly, when we were watching.
The first week gave us 89% accuracy across both models, with the confidence threshold keeping the queue for humans manageable. Over the first year, the models made 54,821 correct predictions. I won’t pretend that approach is right for everyone. If your data is poor or your stakes are high, a phased launch is wiser. But half-measures aren’t free, and in our case the readiness work had already been done.
One thing I’d do differently is start the retraining cadence earlier. We had that large volume of correct predictions to learn from in year one, and a structured retraining schedule from around month three would have lifted accuracy sooner. A model isn’t something you finish. It’s something you maintain.
What happens to the people whose work is automated?
This is the part vendor decks leave out, and for me it was the hardest. Six people’s worth of work was being done by software. The temptation, and the pressure from a savings figure, is to treat that as a headcount number. It was people, and they’d been doing work they’d been asked to do. Redeploying that capacity meant honest conversations, redesigned roles and a lot of leadership time. It took more of my attention than the implementation itself.
I’d been through the other side of that a couple of years earlier, as I wrote in reducing a support team by 60%. I wasn’t willing to run a second version of it as a by-product of a technology project. The principle I held to was that people should hear it from me, early, with a clear picture of what the work was changing into and what they’d be doing. The classification work went away. The need for people who understand the product, the customer’s deadlines and the exceptions the model gets wrong didn’t.
That’s an argument for doing the structural work first, as in my plan. A team with defined levels, training and a visible path has somewhere for the displaced capacity to go. A team without one has nowhere, and the savings come at the cost of trust. If you’re planning AI triage, build the destination for the people before you switch the model on.
What did we learn that the sales pitch didn’t say?
Three things. Data quality is the real project, and it’s invisible in demos. Organisational change is harder than technology, and needs more leadership time than you’ll budget. And confidence thresholds matter more than accuracy targets. “89% accurate” sounds fine until you ask what happens to the other 11%. Designing the edge-case route is what turns a model into a service you can rely on.
I’d add a fourth: involve the people who understand the work. Anyone who has tagged tickets by hand knows where the categories are ambiguous, which ones overlap and which requests don’t fit anywhere. That knowledge should shape the category definitions you train on, and it makes the models better. Treating those people as the experts, not as the cost being removed, is both the right thing to do and the practical one.
A word of caution about enthusiasm for this technology. AI is excellent at the repetitive, well-defined parts of support and poor at judgement, empathy and novelty. Use it where the first applies and keep people where the second does.
I’ll add a note on communication, because how you announce a change like this affects how it lands. Tell the team what’s coming and why before the date, explain what the models will and won’t do and give people a way to raise problems with the output. Where a model gets something wrong, you want it reported quickly and treated as useful information, not as a failure of the technology or of the person who spotted it.
Watch the downstream effects, too. Once triage is automatic, the first response time depends less on sorting and more on capacity and skill at the next step. If you remove one bottleneck, the next one becomes visible, and you should be ready to look for it. In our case that pointed back to the structure and training work already under way, which was a good reminder that these pieces depend on each other.
Finally, keep the business case honest. A savings figure is easy to inflate. Count the manual classification work that no longer needs doing and the review workload that goes away, and keep the cost of the platform and the people who maintain the models in view. The figure that survives that scrutiny is the one worth putting in front of leadership, because it will still be true a year later.
It’s also worth saying that the first year of a model teaches you what to ask for in the second. Review the categories where it’s weakest, check whether those categories are really distinct, and be willing to merge or split them. Sometimes a model’s confusion tells you that people were confused too, and the fix is a clearer definition, not a better algorithm.
AI didn’t replace six people. It replaced six people’s worth of sorting, and gave us the chance to put that capacity somewhere that mattered more.
Hutch Morzaria is a Director-level CX and Support Leadership professional with 19 years of experience building global support organizations across SaaS, Fintech, and enterprise technology. He has hired dozens of support and CX leaders across his career and holds ITIL Expert certification across V3 and V4.
