Why "Do We Have Enough Data?" Is the Wrong AI Readiness Question

The Question Everyone Asks Is Measuring the Wrong Thing

"Do we have enough data?" sounds like due diligence. It isn't. It's a volume question, and volume was never the thing that sinks a pilot. We've watched teams clear a data-volume check with ten thousand clean-looking rows and still fail in production, because nobody asked the question that actually matters: if three different people handled the same case today, would they make the same call?

If the answer is no, you don't have a data problem you can fix by collecting more rows. You have a consistency problem, and no model, no matter how good, converges on a reliable answer from inconsistent ground truth. It will learn the inconsistency. Garbage in, garbage modeled, just with more confidence than the humans ever had.

The Real Diagnostic: Walk the Workflow and Find the "Just Know" Moments

Here's the test we actually run, and you can run it yourself before you spend a dollar on a build. Take the workflow you want to automate and walk it end to end, step by step, the way a new hire would have to learn it. At every step, ask: where would someone need to "just know" something nobody wrote down?

In practice, that shows up as:

• Tribal knowledge. "We always flag anything from that vendor for manual review, everyone on the team just knows that." Everyone except the model.
• Inconsistent labeling. The field says "resolved," but three different reps mean three different things by it depending on the day.
• Shift-dependent exception handling. The exception gets handled one way on the morning shift and another way by whoever's covering on a Friday afternoon.

None of that shows up in a data-volume check. All of it shows up the first time you try to turn that workflow into something a model has to replicate without a human quietly smoothing over the gaps.

Why This Matters More in Budget-Planning Season

Right now, a lot of teams are scoping next year's AI roadmap and deciding which pilot deserves real investment. The instinct is to greenlight the one with the most data behind it. That's backwards. The pilot that deserves funding is the one where the workflow is already consistent, because that's the one that will actually ship. A data-rich, consistency-poor workflow will eat a budget line and produce a case study nobody wants to present to the board.

The Mid-Pilot Failure Pattern We See Constantly

The pattern is almost always the same shape. A team runs a fast data check: enough volume, fields populated, looks fine. They scope a build. Weeks one through three go smoothly, because that's when you're building the scaffolding, not testing it against real edge cases. Then week four or five hits, the model starts processing a wider slice of real cases, and the team discovers the inputs were never standardized in the first place. What looked like one workflow was actually three or four slightly different workflows wearing the same label.

At that point, you're not scoping anymore. You're doing expensive rework on a timeline and budget that assumed the data was already production-ready. That's the costliest possible place to discover a consistency gap: after the build has started, not before it.

This is the same failure mode we talk about in why most AI pilots die from an accountability gap, not a technology gap: the technology almost never breaks first. The undocumented human judgment behind the workflow breaks first, and the technology just makes the break visible.

How to Catch This Before You Scope a Build, Not After

The fix isn't a longer data audit. It's a readiness check scoped to surface exactly this kind of gap, early and cheaply, before a production timeline and budget are locked in. That's the entire premise of our AI Opportunity & Readiness Sprint: a two-week, fixed-fee engagement built to find the consistency gaps, the shift-dependent exceptions, the tribal knowledge, before you commit to a 6 to 10 week Pilot-to-Production Build.

Two weeks to find out your workflow isn't standardized is a scoping conversation. Week five is a change order. The sprint exists specifically so you have the first conversation, not the second one. If you want to see what that diagnostic actually covers, our take on what an AI readiness assessment should actually test goes deeper on the full checklist.

Consistency Isn't Just a Pilot Problem, It's an Eval Problem Too

Here's the part teams miss until it's too late: you can't build a meaningful evaluation harness on top of a workflow that never had a stable ground truth. An eval set assumes there's a correct answer to check the model against. If three humans would have given three different answers on the same case last quarter, you don't have ground truth, you have a vote, and your quality gate is going to be grading the model against noise.

Fix the consistency problem first, and the evaluation harness becomes straightforward: you're checking the model against a standard that actually exists. Skip it, and you'll spend the Enablement phase arguing about whether the eval is wrong or the model is wrong, when the real issue was decided months earlier, back when nobody walked the workflow end to end.

If this sounds like the stage you're at right now, that's exactly what our services are built around: finding the highest-leverage workflow and proving it's actually ready before anyone commits a quarter of engineering time to it.

What to Do With This Before You Finalize Next Year's Roadmap

Before you put a pilot on next year's roadmap, don't ask how much data backs it. Ask whether the humans doing the work today would agree with each other on how to handle the same case twice. That single question will tell you more about whether a pilot will reach production than any volume metric on a dashboard.

Run the two-week AI Opportunity & Readiness Sprint before you scope the build. Visit horizontwolabs.com to see how we pressure-test workflow consistency before anyone commits to a production timeline.