Most large organisations now have AI pilots. Far fewer have AI in production doing something that shows up in a business metric. This paper sets out why that gap exists and what distinguishes the initiatives that cross it.
The constraint is rarely model capability. It is the absence of a defined decision to improve, trustworthy data to reason over, and an operating model that permits a probabilistic system to run in production. This paper covers all three, with the questions to settle before committing budget.
1. Why pilots stall
Pilots stall for reasons that are visible at selection time, if anyone is looking. Four recur.
No defined decision. The use case is described as a capability (“summarise documents”) rather than as a change to how work happens. Nobody can therefore say what success is, and the pilot ends with a demonstration rather than a decision.
No baseline. Nobody measured how well the task was performed before. Any claim about improvement afterwards is contested, and the initiative is judged on impressions.
Data discovered late. The prototype used a curated extract. Production requires the real sources, which turn out to have ownership, quality and access problems that were never in the plan.
No route to production. There is no agreed position on what the system may do unsupervised, what is logged, who reviews outputs, or who answers when a regulator or customer asks why a decision was made. The pilot succeeds and then has nowhere to go.
2. Selecting use cases
We recommend scoring candidates on two independent axes and treating them separately, because conflating them is how organisations end up with a portfolio of things that are either valuable and impossible or feasible and pointless.
Value
- Volume. How often does this task occur? High frequency makes modest improvements material and supplies evaluation data.
- Cost of the current approach. Time, rework, delay, or decisions made with incomplete information.
- Consequence of improvement. Does a better outcome here move something the organisation actually reports on?
- Beneficiary. Is there a named person or team who will be measurably better off, and do they know about it?
Feasibility
- Data proximity. Is the required information accessible, governed and of known quality?
- Error tolerance. What happens when the system is wrong, and is a human already positioned to catch it?
- Workflow insertion point. Is there a clear place in an existing process where this fits, or would adoption require new behaviour?
- Evaluability. Can you tell a good output from a bad one at scale, or does every judgement require an expert?
The last point is underrated. If quality can only be assessed by a senior specialist reviewing each case, you cannot evaluate the system economically, cannot detect regression, and cannot safely change anything once live.
3. The data foundations that actually matter
A full data programme is not a prerequisite. A specific set of properties, for the data the use case touches, is.
Access with governance intact. The system must reach the data through a path that respects existing permissions. Where outputs are personalised or user-specific, entitlement must be enforced at retrieval, not filtered afterwards.
Known authority. Where the same fact exists in several systems, one must be identified as authoritative — otherwise the system will confidently reconcile a contradiction in a way nobody chose.
Currency. Staleness tolerance should be explicit. A policy document from two versions ago produces an answer that is wrong in a way that is hard to detect.
Defined terms. Where business language is ambiguous, the ambiguity propagates. This is the point at which AI initiatives routinely surface data governance issues the organisation has tolerated for years.
4. Evaluation
Evaluation is the discipline that separates initiatives that can improve from those that can only be replaced.
Build a representative test set early — real cases, including the awkward ones, with agreed correct outcomes. Keep it under version control and treat it as an asset. Establish what proportion of outputs must be acceptable, and what an unacceptable one costs.
Run it on every change. Prompt adjustments, model updates and retrieval changes all shift behaviour, sometimes in ways that improve the average and degrade an important subset. Without a regression check this is discovered by users.
Instrument production as well. Test sets age; real usage reveals cases nobody anticipated. Log inputs, outputs and corrections in a form that supports review, subject to your privacy obligations.
5. Governance and oversight
Decide these before production, not during an incident:
- What the system may do without human review, and what it may not.
- Who reviews, on what sample, and what happens when they disagree with an output.
- What is logged, for how long, and who may access it.
- How a customer-facing or regulator-facing explanation is produced.
- What triggers suspension, and who has the authority to invoke it.
- How personal data is handled at every stage, including in logs and evaluation sets.
In regulated environments these connect to existing quality and validation frameworks rather than sitting beside them. Organisations subject to GxP, financial services rules or comparable regimes should establish that mapping early; it strongly influences architecture.
6. A phased approach
Phase one — frame (2–4 weeks). Define the decision. Measure the baseline. Confirm data access. Agree what would justify proceeding. Deliverable: a one-page statement of the decision, the baseline, and the success threshold.
Phase two — prototype (4–8 weeks). Build against real data with real users. Construct the evaluation set. Surface the data problems. Deliverable: a working prototype, measured against the baseline, and an honest list of what production would require.
Phase three — production (8–16 weeks). Integrate into the workflow. Implement oversight, logging and entitlement. Establish monitoring and the feedback loop. Deliverable: a live system with a named owner and a measured effect.
Phase four — extend. Only now consider adjacent use cases, reusing the foundations. Most of the durable value comes from this phase, which is why the earlier ones should be built to be reused.
7. Questions for the next conversation
- Which specific decision or task will change, and who performs it today?
- How well is it performed now, measured how?
- What data does it depend on, who owns that data, and how good is it?
- What happens when the system is wrong, and who notices?
- Who will own this once it is live?
- What would tell us to stop?
Organisations that can answer these are usually ready to proceed. Those that cannot have learned something more valuable than another pilot would have told them.
How VYSoft can help
VYSoft combines technology consulting with specialised delivery capability across AI, data, cloud and regulated environments. We work with teams on use case selection, data readiness, evaluation design and the operating foundations that let an initiative move from prototype to production.