Testing a first AI automation pilot in New Jersey

Begin with a task you can compare

For a first AI automation pilot, choose one internal task whose result a person can check against an existing record. An end-of-shift handoff summary is a useful design example: the assistant drafts unresolved work from approved notes, while the current process remains responsible for assigning and completing tasks. The pilot asks whether the draft preserves the facts and reduces total handling time, including review and corrections.

This article uses an illustrative service-office workflow, not a client result. NIST AI RMF 1.0 calls for documenting test sets and measuring performance under conditions similar to deployment. It is a voluntary risk framework. For this pilot, turn that principle into a comparison between a manual handoff and an assisted handoff using the same task notes.

Why the New Jersey context matters

The New Jersey Division on Civil Rights publishes guidance on algorithmic discrimination covering automated decision tools in employment, housing, public accommodations, credit, and contracting. A business owner evaluating automation should identify whether a proposed workflow touches those areas before treating it as routine administration.

Keep this first experiment limited to summarizing existing internal work. Do not expand it into ranking job applicants, deciding who receives a service, or determining eligibility. If the intended use affects people in those ways, take the proposed data flow and decision process to qualified advisers for a separate review. A handoff pilot does not establish suitability for those decisions. The state guidance is a reason to define the boundary, not a certification for this example.

Build a comparison record before connecting systems

In the example, a supervisor selects a bounded set of synthetic task notes representing ordinary work and exceptions. Each note has a task identifier, a recorded status, a responsible role, and a next step if one exists. The supervisor writes the expected handoff directly from those notes before seeing the assistant output. Keep customer details and confidential records outside this initial test.

Record the time spent creating the manual summary, checking the draft, and correcting it. Count missing tasks, incorrect statuses, unsupported next steps, and cases needing clarification. Keep preparation time and review time visible rather than counting only how quickly a draft appears. These are proposed pilot measures, not reported savings.

The NIST AI RMF Playbook recommends selecting measurements for the system’s purpose and defining acceptable performance limits. Apply that guidance by deciding what counts as an acceptable summary before the trial. For example, every unresolved task must appear with its recorded status, and an absent next step must stay absent. Agree who records errors and who can stop the experiment.

Use an unresolved task to test the draft

Suppose a synthetic note says a replacement part has been requested but has no confirmed arrival date. The expected summary preserves that uncertainty. A draft that adds an arrival date or says the work is ready fails the check, even if the sentence reads well. Ask the reviewer to identify the source note for each assertion.

Repeat the comparison with conflicting statuses, a missing owner, a duplicate task identifier, an empty note, and an unavailable source. Define the expected result for each case: preserve the conflict, request clarification, or stop with an explanation. The assistant must not silently resolve an ambiguity that the supervisor cannot resolve from the same evidence.

NIST’s Generative AI Profile describes confabulation as confidently presented false or erroneous content. Its suggested actions include checking sources and citations during testing and ongoing monitoring, and avoiding broad performance conclusions from narrow assessments. This example tests those risks at the handoff level. Passing it supports only the tested task and conditions.

Make the next decision from the complete workload

At the end of the trial, review the comparison record with the person who normally receives the handoff. Did the draft preserve unresolved work? Did corrections consume the time it appeared to save? Were exceptions easy to recognize? If review takes longer or uncertainty remains hidden, revise the task or stop. A polished demonstration is not an operating result.

If the evidence supports a further pilot, define its duration, approved inputs, access, retention, monitoring owner, and stop conditions before introducing live data. Keep external messages and record changes outside the initial scope. Recheck the comparison after changing instructions, source formats, or connections. NIST AI RMF 1.0 treats production monitoring as part of measurement; the pilot decision should therefore include who reviews failures after release.

AI Gaur builds AI systems and workflow automation. Bring one process and a sample with confidential details removed to discuss a bounded workflow assessment. A next step could be paid discovery or a limited pilot with agreed deliverables, acceptance criteria, usage limits, and support coverage. The purpose is to find evidence for a useful system before committing to broader implementation.