Services
CompanyReproducibility comes first
The failure that costs small teams the most is not drift or scaling. It is being unable to rebuild a model you shipped four months ago.
Someone asks why a prediction looks wrong, and you find that the training script has changed, the data has been overwritten, and the person who ran it has moved on. You cannot reproduce it, so you cannot debug it, so you retrain from scratch and hope.
The fix is not a platform. For every model that reaches production, record four things in one place:
- The exact training data — a snapshot or an immutable query with a timestamp, not "the users table".
- The code commit that produced it.
- The configuration — hyperparameters, feature list, preprocessing versions.
- The resulting metrics on a fixed evaluation set.
A directory per model version with a JSON manifest is entirely sufficient. The discipline matters; the tooling does not.
A fixed evaluation set you never touch
The second highest-value practice is a held-out evaluation set that does not change when the model does.
This sounds obvious and is violated constantly, usually gradually: someone adds recent data to the test set, someone removes cases the model handles badly because "they are not representative", and six months later your metrics are not comparable to anything.
Keep the evaluation set immutable and version it. When you genuinely need to expand it, create v2 and re-score the old models so you have a comparable baseline. Report both until v1 is retired.
Add a second, smaller set of cases that must never regress — the handful of inputs that map to real user harm or an embarrassing failure. Run it on every candidate model as a gate, not a metric.
Monitor inputs before outputs
Most small teams that monitor anything monitor prediction accuracy, which requires labels, which usually arrive late or never.
Input monitoring is cheaper and catches more problems sooner. Track the distribution of the features going in: missing-value rates, category frequencies, and simple summary statistics per feature per day.
Nearly every production incident we have seen on a small ML system announced itself in the inputs first — an upstream field renamed, a client sending a new enum value, a data pipeline silently returning empty results. You do not need drift detection theory for this. An alert on "missing rate for this field jumped from 2% to 40%" catches the overwhelming majority.
Add output monitoring where it is free: the distribution of predicted classes or score ranges over time. A model that suddenly predicts one class 95% of the time is broken, and you will know within a day rather than a quarter.
Make rollback trivial
You will ship a bad model. The question is how long it takes to undo.
The requirement is simple: serving must be able to point at a previous model version without a redeploy. Model version as configuration, not as a build artifact baked into the image.
With that in place, a bad release is a two-minute config change. Without it, it is a full build-and-deploy cycle under pressure, which is when second mistakes happen.
Keep at least the last three versions loadable, and make sure the preprocessing code is versioned alongside the model — a surprising share of rollback failures are a new model rolled back while its feature transformation stayed in place.
What you can safely skip for now
Deliberately not on the list, with the trigger that should change your mind:
| Skip | Until |
|---|---|
| Feature store | Multiple models share features, or training/serving skew has actually bitten you. |
| Workflow orchestrator | Retraining has real dependencies and a schedule. A cron job and a script is fine before that. |
| Automated retraining | You have proven you can retrain manually and safely, twice. |
| A/B testing infrastructure | You have enough traffic for a result to be significant within a sprint. |
| Dedicated model registry | The manifest directory becomes genuinely hard to navigate. |
Adopting these early is not neutral. Each one is infrastructure to run, upgrade, and debug, and it competes with the model work that actually moves your product.
The order to build it in
If you are starting from nothing, this sequence gets you to a defensible position in about two weeks of part-time work:
- Write the manifest for the model currently in production, even retroactively. If you cannot, that is your first finding.
- Freeze an evaluation set and score the current model. That number is your baseline.
- Add input distribution logging and one alert on missing rates.
- Move the model version into configuration and test a rollback in staging.
That is the whole minimum. It fits in a small repository, needs no new services, and covers the failures that actually take small teams down.