When AI inquiry routing loses part of a customer request

One message can contain more than one piece of work

An AI inquiry router should be tested against every actionable request in a message, not just whether it selects a plausible department. A scheduling label can look correct while the workflow loses a request for a phone response. The useful test is whether each request remains visible, has an accountable owner, and reaches a documented resolution.

Consider a synthetic message: “Please change my appointment. I cannot use the online form. Please call me.” Sending it to a scheduling queue addresses only part of the message. A reply directing the sender back to the same form would also miss the stated difficulty. This is a proposed evaluation example, not a report about a customer or a deployed system.

Write the expected request record before testing

Have a person familiar with the process read the synthetic inquiry and record its separate requests before seeing the router output. For this example, the record should preserve the appointment-change request, the difficulty with the form, and the request for a call. These may remain in one task if a single owner can handle them. Splitting a message into several tickets is not automatically an improvement.

For each item, record the supporting phrase, responsible role, current status, and next action. Mark missing information as missing. A statement that the sender cannot use a form does not establish a reason, a diagnosis, or a preferred contact time. The system should not infer those facts.

NIST’s AI RMF Measure Playbook recommends choosing evaluations suited to the intended use and documenting limitations. Apply that principle here by defining request-level expectations before running the test. The record described in this article is our proposed evaluation method, not a checklist prescribed by NIST.

Why the New Jersey context deserves attention

The New Jersey Division on Civil Rights’ January 2025 guidance explains that automated decision tools can impede reasonable accommodations, and discusses accessibility concerns. That is a relevant concern when a business introduces automated handling around access to a service.

Testing whether a request for another contact method survives routing is an operational inference from that concern. The guidance does not prescribe this routing design, and passing this example does not establish legal compliance. Staff should retain requests that need their attention, while qualified advisers determine any legal obligations in the business’s actual setting.

Change the overlooked clause, not just the wording

Create paired synthetic messages that differ in one meaningful detail. Compare an appointment request with the same request plus a request for a call. Then replace the call request with “please reply by email.” Include a negative instruction such as “do not call.” The expected record should change with that detail. A router that returns identical work for every pair may be ignoring part of the message.

Add cases where the second request appears at the end of a long message, repeats an earlier request, conflicts with a previous instruction, or arrives as a follow-up. Test what happens when the responsible queue is unavailable. Preserve uncertainty for a reviewer rather than silently discarding a clause or inventing an alternative instruction.

Keep source text and expected outcomes together during review. NIST’s Generative AI Profile recommends verifying sources in generated outputs and cautions against generalizing performance from narrow assessments. A successful result on a few synthetic messages supports only those tested conditions.

Check what happens after the first task is completed

A routing result is only the beginning of this test. Follow the synthetic record through reassignment and closure. Suppose one team changes the appointment while another must arrange contact. Completing the schedule change must not automatically mark the separate contact request as complete.

If work is split, retain a shared inquiry identifier and a visible unresolved-request list. Define who owns a request while reassignment is waiting, who notices an unanswered item, and who can close the inquiry. If one person owns the whole inquiry, make the same outstanding work visible in that person’s task.

Use the business’s existing process as the authority for whether a requested action actually happened. A drafted response or a routing acknowledgment is not evidence that a call occurred. Keep customer replies and record changes outside the first test until the responsible team has approved those actions.

Measure omissions and premature closure

Count requests that disappear, requests assigned to the wrong role, tasks created without support in the message, unresolved items left without an owner, and inquiries closed before all required work is resolved. Record how much reviewer effort is needed to find and correct each error. Keep these measures separate from message-level classification accuracy.

Agree acceptance and stop conditions with the process owner before the trial. For example, an observed request disappearing from the record should trigger investigation before expanding the workflow. That is a proposed operating boundary, not an industry benchmark. Do not report time savings without measuring the full handling and correction effort.

Begin with a small, accountable evaluation

Start with synthetic messages and no connection that sends replies or changes customer records. Review the expected request list, router output, task ownership, and closure behavior together. If the design cannot preserve requests in that setting, revise it before considering live data.

Before a later pilot, agree approved inputs, access, retention, exception handling, monitoring, and the person responsible for stopping the system. Staff remain responsible for service decisions and unresolved customer needs. Review the evaluation again when instructions, intake formats, or connected systems change.

AI Gaur builds AI systems, agents, and workflow automation. Bring one inquiry workflow and a description with confidential details removed to discuss a bounded assessment. A paid next step can define the test set, acceptance criteria, deliverables, usage limits, and support responsibilities before implementation is scoped.