Data Engineering Comes Before AI

Before committing to an AI build, we work through a short list. It is deliberately unglamorous:

• Can you get the data at all? Not "does it exist" — can a system read it programmatically, on a schedule, without someone exporting a spreadsheet?
• Do the definitions agree? If sales, finance, and operations each have a customer count, do they match? If not, which is correct, and who decides?
• Is history intact? Many systems overwrite rather than append. If a record's current state is all you have, anything involving trends or training on past outcomes is impossible.
• How fresh does it need to be? Daily is straightforward. Minutes is a different architecture. Deciding this late is expensive.
• Who is allowed to see what? If your source data has permission rules, the AI system inherits them, and retrofitting access control to a retrieval layer is painful.

If more than two of these are unresolved, the honest recommendation is a data phase first. It is a less exciting proposal and it is the difference between shipping and not.

The technical work of moving data between systems is well understood. The genuinely difficult part is agreeing what things mean, because it is an organisational question wearing a technical costume.

A typical example: "active user". Marketing means anyone who opened an email in 90 days. Product means someone who performed a core action in 30 days. Finance means a paying account. All three are correct in context, all three produce different numbers, and a model trained on the wrong one produces confidently wrong output.

The fix is a written definition layer — one place where each business concept has an owner, a definition, and the query that implements it. It does not need special tooling. A documented set of views in the warehouse, with an owner named for each, resolves most of it.

Expect this to surface disagreements that predate the project. That is the point. Better to have the argument during a data phase than to discover it in a model's output.

The pipeline itself should be unremarkable, but a few properties are worth insisting on because retrofitting them is disproportionately expensive:

Idempotency

Re-running yesterday's load must not double anything. This single property turns pipeline failures from incidents into a re-run, and it is very hard to add later.

Append-only history

Keep raw ingested data immutable and derive everything else from it. When a transformation turns out to be wrong — and one will — you can rebuild. If you transformed on ingest and discarded the original, you cannot.

Explicit schema handling

Decide what happens when a source adds a column, removes one, or changes a type. Silent failure here is the most common cause of quietly corrupted downstream data.

Freshness and volume monitoring

Alert on "this table has not updated in 26 hours" and "row count changed by more than 40% day over day". These two checks catch the large majority of data incidents and take an afternoon to implement.

Data readiness is not a maturity model you have to climb to the top of. For most AI use cases, "good enough" is genuinely modest:

• The relevant data lands in one queryable place on a known schedule.
• The concepts the use case depends on have agreed definitions.
• Raw history is retained.
• Freshness and volume are monitored with alerts someone receives.
• Access rules are represented in the data, not only in the source application.

That is achievable in weeks for most mid-sized companies, and it unblocks not just the AI project but every analytics question that has been answered by hand until now — which is usually how the work pays for itself before the model ships.

Data readiness does not have to be a gate that blocks everything. The sequence that works well in practice runs the two in parallel with a deliberate join:

1. Define the AI use case precisely, including which data it needs. This scopes the data work — you are not building a warehouse, you are making one use case possible.
2. Build the evaluation set by hand from whatever data you can extract manually. This proves the use case is real and gives you a target.
3. Build the pipeline for exactly those inputs, properly, with the properties above.
4. Build the AI system against the pipeline, not against extracts.
5. Extend the pipeline as further use cases justify it.

The mistake in both directions is common: teams either build a two-year data platform before attempting any AI, or build an AI prototype on manual extracts and discover at deployment that no pipeline exists. Scoping the data work to a specific use case avoids both.

A quick diagnostic when an AI initiative is stuck. Ask the team to run the current prototype on last week's real data, end to end, with no manual steps.

If they can, and the output is poor, you have a model problem — evaluation, prompting, retrieval, or the use case itself.

If they cannot, you have a data problem, and no amount of model work will move it. In our experience it is the second far more often than teams expect, which is why we ask the question before recommending a build rather than three months into one.