From AI PoC to MVP: What Actually Makes It to Production

In a PoC you judge output by looking at it. That works for twenty examples and stops working immediately after.

Before you write production code, you need a held-out evaluation set — realistic inputs with agreed expected outcomes — and a score you can re-run on every change. It does not need to be sophisticated. For most products a spreadsheet of 100 to 300 cases and a pass/fail rubric is transformative, because it converts "the new prompt feels better" into a number.

Without this you cannot safely change anything. Every prompt tweak, model upgrade, or retrieval change becomes a gamble, and teams freeze. We have seen products stuck on an old model for a year purely because nobody could prove an upgrade was safe.

What to measure

• Task success on your evaluation set, split by input category so you can see which segments are weak.
• Failure mode distribution — not just how often it is wrong, but how. Refusals, hallucinated specifics, and formatting breaks need different fixes.
• Latency and cost per request, measured at the percentiles your users actually feel.

In the PoC, the context you feed the model was assembled by hand. In production it has to be assembled by code, from live systems, in a few hundred milliseconds, for inputs nobody anticipated.

This is where the majority of engineering time goes, and it is ordinary software work: connectors, schemas, permissions, caching, incremental sync, and handling the document that is 400 pages long or empty or in the wrong language.

On the FakeCheck engagement, the interesting part was never the model call. It was defining how a submission moves through the system, what gets stored, what gets retried, and what the user sees while it is in flight. Once that was settled, the AI step was a component rather than the architecture.

Our advice: design the data path before you tune the prompt. If the retrieval is returning the wrong context, no amount of prompt work will save you, and you will spend weeks confusing a data bug for a model limitation.

PoC economics are irrelevant — a few dollars of usage over a week. MVP economics decide whether the product is viable.

Three numbers matter, and you should be able to state all three before you build:

The usual levers, in the order we reach for them: cache aggressively, route easy cases to a smaller model, shorten context before you shorten quality, and only then consider fine-tuning.

A PoC is judged on its best output. A product is judged on its worst.

Every AI feature that survives contact with users has an explicit answer to four questions:

• How does the user know how confident the system is? Not a fake percentage — a meaningful signal, such as showing sources, or degrading to a narrower claim.
• What happens when it cannot answer? A clean, honest fallback beats a confident guess every time, and users forgive it.
• How does the user correct it? Editing, overriding, or flagging. This is also your best source of evaluation data.
• What is the blast radius of a wrong answer? If it is high, the feature needs a human checkpoint before it acts, not after.

This is a product design problem more than an engineering one, which is why we bring it into the UX phase rather than treating it as error handling at the end.

The order we use on AI engagements, and why:

1. Define the task and the failure cost. What exactly is the system deciding, and what does a wrong answer cost the user? This determines how much verification you need.
2. Build the evaluation set by hand. Before any code. It forces the specification to be concrete.
3. Build the data path with a stub model. Prove the plumbing, permissions, and UX with a hard-coded response.
4. Introduce the model and measure. Now every change is comparable against a baseline.
5. Harden: cost, latency, fallbacks, monitoring.
6. Ship to a narrow user group and grow the evaluation set from real traffic.

The step people skip is three. Building the plumbing against a stub feels like wasted effort until the first time you change models and nothing else breaks.

A short checklist we run before committing to a build:

• You cannot state the success rate as a number.
• The context is assembled manually, or from a fixed sample of documents.
• Nobody has priced a heavy user.
• There is no defined behaviour for "the system does not know".
• The only people who have used it built it.

Each of these is fixable in days, not months — but only if you address them before the build rather than after the first user complaint.