Skip to content
Insights/Article

When AI Classifies Support Tickets, What Happens When It Is Wrong or Unsure?

Key takeaway

Write category definitions as operational rules with boundary examples and counterexamples. Send low-confidence or multi-label tickets to a human review queue. Reclassification keeps the original category, editor, and reason; errors return to evaluation. Urgency and sentiment need separate judgment.

An AI classifying a refund ticket as a product inquiry, a person moving it to refund handling with a reason, and the error returning to evaluation

Consider a hypothetical case. A customer writes in the after-sales form: "The machine I bought last week broke. I am requesting a refund." AI classifies it as a product inquiry. That category routes to pre-sales, which handles hundreds of inquiries a day. Two days later someone opens it, sees a refund request, and notices the customer has already sent a second, more urgent email.

The cause may lie in category definitions, examples or model judgment; one output does not establish the cause. In this example, there is also no review path for uncertain results, so a classification error becomes a routing error.

If a model supplies a confidence score, first test whether it distinguishes real errors. A high self-reported score is not automatically a calibrated probability of correctness. Set thresholds and routing scope from labeled examples, consequences, and available human review.

Classification is for routing, not labeling

Ticket classification is not about giving each record a tidy label. It is about sending it to someone who can act. The category decides the team, the deadline, the permissions, and the next step. A wrong category does not mean an ugly label; it means the right person never saw it.

Design categories backward from who handles them, how fast, and with what permissions, not from what topics the company finds interesting. A category that maps to no team or action may not be needed.

Category definitions must be operational

A name is not enough. "Refunds and returns" means different things: a customer requesting a refund, or goods already shipped back.

Write at least four things per category:

  1. Definition: when a ticket belongs here, in business language.
  2. Boundary examples: three typical tickets and why they belong.
  3. Counterexamples: two easily confused cases and where they belong instead.
  4. Routing: the default team, whether human confirmation is required, and the deadline.

Boundary examples and counterexamples are more useful than the definition alone. They align AI and people on the same judgment and help new staff learn quickly.

Low confidence and multiple labels

When AI cannot decide, do not force a choice. It can output two candidate categories with reasons and send the ticket to a human review queue.

Multiple labels help too: a ticket may involve both a refund and shipping damage. It can carry both labels, but one primary category must decide routing. AI suggests the primary, a person confirms, or a rule decides.

Do not invent a confidence threshold. Test it on historical tickets: how many high-confidence tickets were wrong, and how many low-confidence tickets were actually right? The threshold's job is not to make AI do more work; it is to keep errors where a person can catch them.

Where human confirmation sits

Three placements:

Before routing: every ticket is looked at before assignment. High accuracy, high labor, suited to high-risk or low-volume work.

Low-confidence review: only low-confidence, multi-label, and rule-conflict tickets go to a person. Most route automatically, a few go to review. This is the common compromise.

Post-hoc sampling: tickets route by AI and are sampled later. Suited to high volume, reversible errors, and teams that can transfer work among themselves.

In every case, the review screen shows the customer's words, AI's candidate categories, and reasons, not just one label. When a person changes the category, they see before and after at a glance.

Reclassification and feedback

Guideline G9 recommends making AI errors easy to correct. A reclassification control applies that principle to tickets.

Moving a ticket from "product inquiry" to "refunds and returns" records the original category, the new one, the editor, the time, and the reason.

Do not leave the reason as free text only. Offer options—keyword misread, unclear boundary, ambiguous customer wording, rule conflict—plus an optional note. Options make statistics possible.

Error samples return to the evaluation set. Every week or month, take misclassified and human-corrected tickets and retest the rules or prompts. Without feedback, the model repeats the same errors while the team concludes "AI is not useful."

Urgency is not the category

Refunds, complaints, and safety issues often need faster response. But urgency is not fully determined by category: within one category, a customer who has chased three times differs from one who just submitted.

Judge category and urgency separately. Category decides who handles; urgency decides how fast. Do not make "complaint" the only high-priority category, or a genuine safety incident may sit in a normal inquiry.

Handle sentiment carefully. AI may flag "customer expresses strong dissatisfaction," but it should not automatically upgrade to a complaint or promise compensation. Sentiment is a hint; a person decides.

A hypothetical category table

Category

Definition

Boundary example

Counterexample

Default team

Human confirmation

Product inquiry

Pre-sale questions about features, price, fit

Asking whether a model supports an interface

A fault after purchase

Pre-sales

On low confidence

Order and shipping

Status, dispatch, address changes

Asking ship date or changing address

Damaged goods with refund request

Order team

On low confidence

Refund and return

Refund, return, exchange requests

Machine fault with refund request

Only asking about the policy

After-sales

Recommend all

Complaint

Dissatisfaction with service, staff, or process

Complaining about installation delay

Only asking for progress

Service manager

Recommend all

The table is not training data; it is shared judgment for people and AI. Update it when categories change. Historical tickets need not be rewritten, but record when the rules changed.

Review Misclassified and Uncertain Tickets

[AI ticket classification checklist]

1. Categories
- Does each category have a definition, boundary examples, and counterexamples:
- Does each map to a team and a deadline:
- Are categories without routing necessary:

2. Uncertainty
- Do low-confidence tickets enter a human queue:
- Does a primary category decide routing when multiple labels apply:
- Does the review screen show the original words and candidate reasons:

3. Correction and feedback
- Does reclassification record old category, new category, editor, and reason:
- Are error samples returned to evaluation regularly:
- Are changes to categories or rules versioned:

4. Priority
- Are urgency and category judged separately:
- Do safety, complaint, and refund cases have their own escalation rules:
- Is sentiment only a hint, never an automatic promise:

Take last week's human-reclassified tickets and see where errors cluster at category boundaries. If the same boundary fails repeatedly, change the definition before changing the model. When to hand a conversation to a person is a separate decision; this article covers classification and routing only. Routing rules and response times are covered in routing inbound inquiries to the right person. Automated FAQ responses are covered in the customer service FAQ workflow. The scope of ticket classification and human review can be discussed within AI applications and workflows.

After choosing a category, define human handoff triggers and use failure-path acceptance questions to check that mistakes reach a workable queue.

Sources

Let's talk about your project

Websites / Business systems / AI applications

Chat on WhatsApp