A pilot can produce promising results and still struggle to become part of daily work. The demonstration may not include the team's difficult cases. The output may need more review than expected. Or nobody may be responsible for keeping it working after the initial project.
Here are three areas I would examine before deciding whether to continue.
Is the task specific enough to test?
A broad goal such as “help the sales team use AI” leaves too much room to disagree about success. Choose a task with identifiable inputs, users, and a result you can evaluate.
Document classification, invoice coding, and ticket routing can be useful candidates when the volume and requirements justify the work. A customer-facing assistant may also be appropriate, but its errors and support needs will require a different test.
The choice should follow the problem you need to solve. Write down what the proposed change is expected to improve and what evidence would support that expectation.
Who owns the work around the model?
Map where the input comes from, who reviews the result, and where it goes next. Include what happens when a document changes, an integration fails, or the system is uncertain.
Agree on who is responsible for those situations. The process owner and the technical owner may be different people, but both need to know what they are taking on.
The vendor's responsibilities also need to be clear. A product can operate as designed while leaving work your team did not expect to inherit.
Are you measuring a useful result?
Accuracy and latency tell you something important about the system. They do not, on their own, tell you whether the process has improved.
Measure the time needed to review and correct outputs. Check whether people use the result and whether it changes the task they were trying to complete. An answer that is quick to generate but slow to verify may offer little benefit.
Choose an operational measure alongside the technical tests. It could be average handling time, cost per invoice, or time from request to fulfillment. Include quality and consequences of errors in the decision.
Decide what to do next
A stalled pilot does not automatically need a larger rollout or a different model. Review the evidence and identify what the next test would resolve.
You may need to narrow the task, improve the data, change the integration, or give the team more time to learn the process. You may also find that the benefit does not justify further work.
If you would like help assessing a pilot, talk it through with Ron.
Ron Davis
Founder
Three decades building enterprise platforms. Started Joust to close the gap between strategy decks and the work they're supposed to change.