Why AI document pilots stall between the demo and production

Most AI pilots do not fail loudly. The demo works. It reads twenty documents, pulls the right fields, writes a tidy summary, and everyone in the room agrees it is impressive. Then the project goes quiet. Three months later it is still a pilot, the team that built it has moved on, and nobody can quite say what is blocking it.

What blocks it is almost always the same thing: the demo answered an easy question, and production asks five hard ones. A pilot that has not answered them is not a smaller version of the finished system. It is a different thing, and buying the build on the strength of it is where the money goes.

The demo tests the easy middle

A demo is assembled from documents someone chose. Usually they are clean, recent, and typical, because the person choosing them wanted to see whether the idea works at all. That is a reasonable first question. It is just not the question that decides whether the system can run.

Real document flows have a long tail. Scanned faxes, pages photographed at an angle, a supplier who changed their template last March, a contract with a handwritten amendment in the margin, a claim file where the one fact that matters is on page forty. In most operations this tail is a small share of the volume and a large share of the effort. It is also exactly where the team's manual time is going today, which means it is the part the automation was bought to fix.

A pilot that never met the tail has measured the easy middle and priced the whole.

Five questions production asks

How accurate is it, measured how? Not "it looked right," but a hand-checked sample drawn from the real document mix, tail included, with a number attached. The sample needs to be fixed before the build starts and never used for tuning, or the score measures memory rather than accuracy. If the pilot report has no such number, it has an impression.

What does it do when it does not know? Every system meets documents it cannot read well. A production system has a defined behaviour for that moment: it flags the document, routes it to a person, and says why. A demo system guesses, confidently, because nothing ever asked it to do otherwise. The difference between those two behaviours is most of the difference between a tool your team trusts and a tool your team checks.

Where does a person look, and when? Someone has to review something. The useful design question is which outputs, at what threshold, in which queue, and how long that queue will be on a normal Tuesday. A human review point that was never sized becomes a backlog within a month, and a backlog is how a working system turns into a stalled one.

Can every output be traced to its source? When an extracted figure is challenged, whether by a client, an auditor, or a regulator, someone needs to open the exact page it came from in seconds. Systems that record where each answer came from survive their first dispute. Systems that do not get switched off after it, however accurate they were on average.

Who owns it after launch? Document formats drift, suppliers change templates, the model underneath gets updated, and a system that scored well in March quietly degrades by September. Production needs someone watching a small set of numbers and re-running the evaluation sample on a schedule. If the plan ends at launch, the system has an expiry date nobody wrote down.

Why these questions get skipped

None of this is exotic. The questions get skipped because answering them is slower and less exciting than building the demo, and because the demo creates momentum that nobody wants to interrupt. There is also a quieter reason: a vendor paid by the hour, or paid to reach a launch date, has no particular reason to surface the tail before the contract for the next phase is signed.

The fix is to make the five answers part of what the pilot delivers, in writing, rather than something the build is expected to discover.

How to tell whether a pilot answered them

Before funding the production build, ask for four things.

The evaluation sample, and the number. How many documents, how they were chosen, and the score on each document type separately, because a blended number hides whichever type is quietly failing.

The failure list. The documents the system got wrong or refused, with the reason for each. A pilot with an empty failure list was not tested hard enough.

The review design. Which outputs a person checks, the threshold that sends them there, and the expected weekly volume in that queue.

A recommendation, including the option to stop. An honest pilot can conclude that the workflow is not ready, or that the tail is too large for the value on offer. If stopping was never a possible outcome, the pilot was a sales step, not a test.

A pilot that hands you those four things has done its job whatever it concluded. A pilot that hands you a demo and a proposal has done half of it, and the other half will be billed to the build.

How we run it

This is how our Proof pilots are built: one workflow, one data source, a hand-checked evaluation sample fixed before any tuning, a written findings document with the failures in it, and a clear recommendation, which is sometimes to stop. It costs a fraction of what the build would, and it is designed to answer the five questions while they are still cheap.

The first step is free. We send a written diagnostic of one workflow, two pages, no call required, including the single biggest risk we would expect a pilot to surface. If that risk is one you already know about, you have lost a few minutes of reading. If it is one you did not, it was worth finding out before the build.