Scale — Week 38
You've put an AI system into some part of the business — customer support replies, lead follow-up, document review, scheduling. It's been running for a while. Nobody has complained. The dashboard, if there is one, shows green.
Is it working?
That question is harder to answer than it sounds. The easiest version to check — has anything gone wrong that I've heard about — is not the same question as "is this system doing what we need it to do", and the gap between them is where quiet underperformance lives.
Why "no complaints" isn't evidence
Silence from customers or staff tells you the errors haven't been bad enough to surface yet. It doesn't tell you the error rate, and it doesn't tell you whether the system is drifting — getting slightly worse, slightly more often, as inputs change and edge cases accumulate that nobody modelled at launch.
There is no obvious framework for asking better questions than "how's it going." Vendors, understandably, tend to answer with reassurance rather than evidence — a general sense that things are fine, rather than a specific account of what's being measured and how. That's not necessarily dishonest. It can simply be that nobody set up the measurement in the first place, so there's nothing more specific to report.
The result is that a deployment can run for months without anyone — owner, vendor, or user — actually knowing the failure rate, what the failures look like, or whether they're being caught before they reach a customer.
What to ask for
You don't need to understand the model to ask good evaluation questions. You need to ask for the same kind of evidence you'd expect from any other operational system.
What does "good" mean here, specifically? Before a system goes live, someone should have defined what a correct output looks like for the task it's doing. Ask to see that definition. If the answer is vague — "helpful and accurate" — push for specifics: accurate compared to what, helpful according to whom. A concrete answer names the checks: does the output match the data in the source system, does it follow the tone and policy guidelines, does it avoid the categories of claim the business isn't willing to make.
How is it actually being checked, on an ongoing basis? Not at launch — now, this month. Ask whether outputs are sampled and reviewed, how often, and by whom. A good answer names a cadence and a person. A weak answer describes a process that existed once, at launch, and hasn't been repeated since.
What happens when it gets something wrong? Every AI system has a failure mode. The question isn't whether failures happen — they will — it's whether the system is built to catch them before they reach a customer, and whether someone is looking at those catches to see if they cluster around a particular type of request. Ask for a description of the escalation path: what triggers a handoff to a person, and what happens to that case afterwards.
Is performance measured against a real baseline, or against perfection? The right comparison for an AI system usually isn't "did it get everything right" — it's "how does this compare to how the task was done before." Ask what the system is being measured against. If nobody can answer that, the "it's working" claim doesn't have anything underneath it.
Can I see a sample of real outputs, not curated ones? This is the simplest and most useful ask on the list. Not a demo, not a highlight reel — a genuine, recent sample of what the system has actually produced, including the ones that needed correcting. If a vendor or internal team resists this, that resistance is itself information.
What a good answer sounds like
A vendor or internal team that has actually built the evaluation layer will answer these questions with specifics: what's measured, how often, who reviews it, what the escalation criteria are, and what's changed as a result of past reviews. They'll be able to tell you about a correction that got fed back into the system, or a category of request that got rerouted to a person after it turned out to be handled poorly.
A team that hasn't built this layer will answer in generalities — "it's performing well," "we haven't had issues," "customers seem happy." That's not necessarily a red flag on its own, particularly for a system that's new. But it is a gap, and it's worth naming as a gap rather than accepting as an answer.
The friction worth acknowledging
Building proper evaluation into an AI deployment takes time and, usually, a person whose job includes reviewing outputs on a schedule. For a small or mid-sized business, that's a real cost, and it's tempting to treat it as optional — something to add later, once the system has proven itself.
The trade-off runs the other way. Evaluation is most valuable early, while the system is still learning the edges of what it's good at and what it isn't. Skipping it at launch doesn't remove the cost — it defers it, usually to the point where a customer-facing error has already happened and you're building the monitoring layer under pressure instead of on your own schedule.
There's also no single benchmark that applies across every system and every business. What counts as an acceptable error rate for an internal scheduling assistant is different from what's acceptable for something replying to customers directly. Part of the evaluation work is deciding, deliberately, what the acceptable threshold is for this specific task — not importing a number from somewhere else.
What this means for you
You don't need to be technical to evaluate an AI system properly. You need to ask for the same rigour you'd expect from a new hire doing the same job: a clear definition of what good performance looks like, a way of checking it on an ongoing basis, a plan for what happens when something goes wrong, and evidence rather than reassurance when you ask how it's going.
If a vendor or internal team can answer those questions specifically, that's a good sign regardless of what the answers are. If they can't, that's worth addressing before you scale the deployment further — not after.
If you're trying to work out what proper evaluation should look like for a system you've already deployed, or one you're about to, get in touch. It's a reasonable conversation to have before the stakes get higher.
