The slide deck and the reality
Every AI project I’ve been called in to assess in the last two years has had roughly the same shape on the slide deck. There’s an agent at the centre. Around it are some boxes labelled “RAG,” “tools,” “evaluation,” and “knowledge base.” Arrows go in and out. A vendor logo is in the corner. The deck is confident, modern, and entirely about the model layer.
The reality of where the project is stuck is almost always somewhere else. The agent works fine in the notebook. It falls over in production because the data going into it is inconsistent, duplicated, ungoverned, and impossible to audit. The eval set was assembled by one engineer two months ago, isn’t representative, and nobody knows what it tests. The pipeline that’s supposed to keep the retrieval index fresh runs nightly when it should run hourly and breaks silently when it does run.
The diagnosis is the same almost every time: this is not an AI problem. It’s a data problem with an AI layer on top. And it’s failing where data projects always fail, just with a fashionable label on the failure.
This is the unglamorous read most AI consultancies don’t lead with, because “we’ll fix your data engineering” is a worse pitch than “we’ll build you agents.” But the unglamorous work is what decides the outcome.
Where AI projects actually die
Five places, in rough order of frequency.
1. Entity resolution. Your “knowledge base” thinks Acme Corp, Acme Corporation, and Acme Inc are three different companies. Your customer table has the same person three times under slightly different spellings. Your retrieval system pulls the wrong record half the time. The model is confidently wrong, which is worse than refusing to answer, because users believe it. I’ve written about the practical mechanics of entity resolution in Postgres, most teams under-invest here by an order of magnitude.
2. No lineage, no governance. You can’t answer “where did this answer come from?” or “what did the model see when it was trained?” In a regulated environment that ends careers. In an unregulated environment it ends trust the first time a customer disputes an output.
3. Pipeline architecture. The retrieval index is rebuilt nightly from a flat dump of your CRM. Anything changing in the day is invisible to the agent until tomorrow. Your customers notice. Your agents look stupid. The fix isn’t a smarter model; it’s an event-driven pipeline you didn’t budget for.
4. Eval set quality. “Looks good in vibes” is the actual release criterion on most AI projects. A real eval set, representative, versioned, expanded as the system fails, is something almost nobody starts with and almost everyone needs. Without it, you cannot tell when a prompt change made things better or worse. You’re flying blind, and you’ll discover the regression when a user does.
5. The data layer can’t actually scale. Your beta worked because the retrieval index had ten thousand records. At ten million, query latency went from 50ms to 5 seconds, you started timing out, you bolted on caching, the cache went stale, and now your agent is fast and wrong instead of slow and right. None of this is a model problem.
In every one of these failures, the model layer is fine. The model layer is doing its job. The data layer is sabotaging the model layer’s output, and the team is trying to fix the symptom instead of the cause.
The honest order of operations
If I’m running an AI engagement from scratch, the order of operations is roughly the inverse of the slide deck. The slide deck starts with the agent and works outward. The honest order starts with the data and works toward the agent.
Data audit. What sources do you have? What’s the quality? Where are the duplicates? What’s the freshness? What’s auditable, what isn’t? This isn’t a week of work; this is genuinely several weeks of careful inventory and quality measurement, and it ends with a list of datasets that are AI-ready, datasets that need work, and datasets to leave behind.
Entity resolution and normalisation. Aggressive cleanup. Standardisation. The boring work that turns “Acme Corp / Acme Corporation / Acme Inc” into one canonical entity with a stable ID. This is where the bulk of the unglamorous engineering lives. Most teams skip it; most teams pay for the skip later.
Pipeline architecture. Event-driven where freshness matters, batch where it doesn’t, idempotent everywhere, observable, replayable. Pipelines that fail loudly, not silently. Pipelines that survive the team changing.
Governance and lineage. Catalog, lineage, access control. Every record can be traced back to its source. Every model output can be traced back to the records it used. This is unsexy and load-bearing.
Retrieval design. Now we’re starting to look like an AI project. Vector embeddings where the problem is genuinely semantic, lexical search where it’s not, hybrid where both matter. Reranking. Caching. Eval sets to tell whether the changes are helping.
Agent and prompt engineering. The thing the slide deck started with. By the time you get here, the system has a chance of working, because everything underneath it is sound. Without the foundations, this is rearranging deck chairs.
Evaluation harness. A real eval set, representative of production, versioned, expanded with every failure mode you encounter. Regression suites that run on every prompt or model change. A pass-rate dashboard that anyone on the team can look at.
Monitoring and drift detection. Production observability for the AI system: latency, cost per request, success rate, eval pass rate on production traffic, hallucination signals, agent step-completion metrics.
Most “AI projects” jump from step 1 (or worse, skip it entirely) to step 6, then spend the next year discovering that steps 2-5 needed to happen first. Every shortcut at the top of this stack compounds at the bottom.
What “AI ready” actually means
When a vendor tells you that you need to “get your data AI-ready,” they usually mean “buy our vector database” or “buy our data platform.” That’s not what it means. AI-ready data has five specific properties:
- Canonical. Each real-world entity exists once, with a stable identifier, and every reference to it points at that identifier.
- Governed. Every record has a source, a timestamp, an owner, and an access policy. You can answer “where did this come from” and “who’s allowed to see it” without thinking.
- Fresh. The data the model sees reflects reality within the time window your use case needs. If decisions are made minute-to-minute, the data is updated minute-to-minute.
- Observable. Pipelines have monitoring. Failures are loud. Stale partitions don’t sit unnoticed for a week.
- Versioned. You can reconstruct what the model saw at any point in the past. Training data, retrieval index, prompts, all under version control.
A team with all five is genuinely AI-ready. A team without them is going to ship something that demos well and fails in production. There is no model on the market that compensates for a data layer that lacks these properties, you can swap GPT-5 for Claude Opus for a fine-tuned Llama and the failure mode doesn’t change.
The honest test
If you’re sitting on an AI project and unsure whether you’re in this trap, here’s the diagnostic. Ask three questions:
“How many records in our retrieval system are duplicates or near-duplicates?” If nobody knows, the answer is: a lot. Entity resolution work hasn’t happened.
“What does our eval set look like, and when was it last expanded?” If the answer is “we have one somewhere” or “we eyeball it,” the AI project is shipping on vibes.
“If a customer complains about an output we generated last week, can we reconstruct what data the model saw at that moment?” If the answer is no, you have a lineage problem, and the first compliance or product-trust incident will be expensive.
Three “I don’t know”s out of three means the project is in the data-layer trap. The fix is to slow down, pause the model-layer work, do the data engineering that should have happened first, and resume the AI work when the foundation is sound. It feels like backwards motion. It isn’t. The backwards motion is shipping an agent that’s confidently wrong.
Why this keeps happening
Two reasons, and they reinforce each other.
First, AI is the loudest market in software right now. Founders, boards, and procurement all want to be seen doing AI. Saying “we’re building agents” is fundable; saying “we’re doing entity resolution on our customer database” is not. So the projects get scoped, sold, and staffed at the AI layer, even when the work is at the data layer.
Second, the consultancies and vendors selling AI services know this. Most of them will happily sell you AI when the right thing to sell you is data engineering, because data engineering is harder to package and less exciting on a website. The customer wants AI; the vendor wants revenue; the actual gap goes unaddressed.
The way out is to be specific about what your project actually is. If the data layer is sound, you can build an AI system on top of it that holds up. If the data layer is unsound, no amount of model-layer work will save the project, and the longer you spend pretending otherwise, the more expensive the eventual remediation will be.
The unsexy answer is almost always the right one: do the data work first. Most of “doing AI” is “doing the data engineering you should have done before AI existed.” The companies that ship AI systems that actually work in production are the ones that internalised this early. The ones that didn’t are still rebuilding their agents every six months and wondering why.
If you’re trying to figure out where your project actually sits on this spectrum, that’s exactly the conversation our AI data preparation engagements start with, and we’ll tell you honestly whether the gap is at the model layer or the data layer, even if the answer means a different scope of work than the one you were planning to fund.
Ex-NASA engineer and cloud architect with over a decade of experience building scalable systems for startups and enterprises.
Work with Tom →