Put the Quality Bar in the Business's Hands

The quality bar for your AI lives in a spreadsheet on an engineer's laptop. That's the problem.

I keep seeing the same setup. The technical team builds the AI, defines what "good enough" means, and sets the threshold. The business signs off once and never thinks about it again. Then the complaints start. The AI passes its internal checks but fails in the real world, because the people who defined quality never talked to the people who feel the quality.

The fix is uncomfortable for both sides: put the quality bar in the hands of the business. Let the people who do the work set the standard, and let the engineers build to it.

Why the engineers shouldn't own the bar

Engineers own the plumbing. They don't own the judgment.

"Good enough" is a business decision, not a technical one. What counts as an acceptable error rate depends on what the output is for. A wrong answer in an internal draft costs an hour. A wrong answer in a customer-facing price quote costs an account. A wrong answer in a compliance document costs a regulator's visit. The same model, the same accuracy number, and three completely different quality bars.

An engineer looking at a model's accuracy can tell you the number. They can't tell you which errors are tolerable, because that depends on the customer, the contract, the stakes — the things the business knows and the engineer was never told. When the bar is set by people who don't feel the consequences, the bar is set wrong.

Quality isn't a technical measurement. It's a business decision about what errors you can live with. The people who feel the errors get to make the call.

The three-question rubric

Here's the part that makes this practical instead of philosophical. The business doesn't need an ML degree to set a quality bar. They need three questions, applied to every output the AI produces.

Question one: would you ship this to a customer as-is? This is the floor. If the answer is no, the output doesn't go out. No exceptions, no workarounds.

Question two: if this is wrong, what does it cost? This is the calibration. A summary for internal use has a low cost of error. A price quote has a high one. The answer to this question decides how carefully the output gets checked — and how much checking the AI has to pass before it reaches a human.

Question three: what would make you trust this without checking? This is the goal state. It's the question that forces people to name their actual bar instead of gesturing at it. When the answer is specific, like "when the part numbers match the system, I trust it," the engineers have something to build toward. When the answer is vague, like "when it's right," you've found the gap between the model and the business.

Three questions. No rubric to memorize, no threshold to define in the abstract. The business answers them per task, and the answers become the spec.

How the review actually works

The business sets the bar, and the bar gets enforced in the work, not in a meeting.

Pick the highest-stakes output first. The one where being wrong hurts most. Sit the operators down with real examples — good outputs, bad outputs, borderline outputs — and have them sort. This is calibration, and it's worth doing in person, because the sorting conversation surfaces the judgment the engineers never see. "This one's fine because the customer already knows the context." "This one's not fine because it reads like a robot wrote it." That's the actual quality bar, in the words of the people who feel it.

Then the operators review live output against the bar for a set period, and the log of their decisions becomes the training data and the evaluation set. The AI gets tuned until the operators stop flagging. Not until the engineer's metric says it's good. Until the operators say it's good.

That's the loop: business sets the bar, operators enforce it, the log teaches the model, and the model improves until the operators go quiet. Every pass through the loop moves the quality decision further into the business's hands.

The resistance and why it's worth pushing through

The pushback comes from both sides. Engineers say the business doesn't know what it's asking for. The business says it doesn't have time to review AI outputs. Both are right, and both are missing the point.

The business is the one that feels the errors. Their review time is the price of getting the bar right, and it's cheaper than the cost of a wrong bar. The engineers get something better than a vague spec — they get a concrete, example-driven definition of good, which is exactly what makes their job easier.

The reward shows up in the metrics. Fewer escalations, because the outputs are calibrated to what the business actually tolerates. Faster approvals, because the operators trust the system they helped define. And a model that improves because it's being trained on the judgment of the people who use it, not the guess of the people who built it.

The quality bar belongs to the people who feel the errors. Three questions, real examples, and a review loop that teaches the model. That's how you build AI the business actually trusts — because the business built the standard.

Think this argument fits your event? Tell me about the room — the calendar is selective.