Your AI Feature Passed. It Was Still Wrong.

A support bot answered a billing question with total confidence. It got the refund policy wrong. Nobody noticed for three days, because the response looked fine. Grammatically correct, on-topic, politely worded. It just wasn't true.

That gap, between "the answer looked good" and "the answer was good," is the thing your dashboards keep missing. If you ship AI, measuring that gap is now part of your job, not something you hand off and hope for. Let me catch you up on what changed and what to do about it.

The test you write before you have users

The usual advice for evals starts with your data. Read the traces, find the failures, sort them. But on day one of a new feature you have no traces. So you build the test out of what you already know.

Daniel McKinnon, who was a PM on the Llama models at Meta, has a blunt version of this. As Aakash Gupta wrote it up, when a partner team wants Llama to do something, McKinnon tells them not to send a PRD. "Cut the middleman and send me an eval directly." The examples become the spec.

The build is fast. Write one sentence for what the feature does. Find the easiest real case and check the model can even do it. Find a case you think nothing can solve. Then fill the middle with 20 cases at a time until you have around 100. You use AI to build the test for the AI, which is why this takes 90 minutes instead of two weeks.

A green dashboard is not a passing grade

Standard logging was never built for this. A REST API returns the right JSON or it doesn't. An LLM call can be fast, cheap, syntactically perfect, and still wrong in a way no HTTP status code will ever surface. That is the point the observability write-up makes about your 2 a.m. page: you pull the logs and find a request ID, a 200 status, a latency number, and the actual conversation is gone.

Here is how bad the blind spot gets. Mutagent AI Labs built an agent that drafts emails, runs them through a required review step, then sends. They found a send with no review in front of it. The message went out. Nothing checked it. Every outcome check passed, because a send is a completed action and this one completed.

The cause was in the path, not the result. A healthy send runs eight to eleven steps. This one ran three. No draft, so the review had nothing to run on, so it never ran. Grade an agent only by whether its final action succeeded, and a message sent without review counts as a win.

Right answer is one signal. Right for the right reason is another.

Mutagent splits the work into two jobs your team probably treats as one. Evaluation answers a closed question: is this right, yes or no, against a reference. Diagnostics answers an open one: what happened, why, and where did it start. A score that drops six points is a fire alarm. It tells you the building is burning and nothing about which room.

They also learned one rule the expensive way. Two of their own quality checks agreed on about 6,300 traces out of 6,300. For a day that looked like validation. One check had been derived from the other, so the agreement was just arithmetic repeating itself. It would have held even if both were wrong about every trace. As they put it, "You cannot grade an exam with an answer sheet the student wrote."

On hallucinations specifically, there is a cleaner move than a single score. GraphEval, from Amazon researchers and walked through by KDnuggets, breaks a response into small facts and checks each one against known-good source text. It does not just tell you the answer was off. It points at the exact claim that was made up.

What the user feels is its own number

None of the above tells you whether people trust the thing. That is a separate measurement problem, and Jeff Sauro's work at MeasuringU gives you the shape of it. Attitude metrics like productivity, trust, dependency, anxiety, and personification, each measured with real items, not a vibe check.

The numbers here should worry you a little. A Melbourne-KPMG survey of 48,000 people across 47 countries found fewer than half of regular AI users were willing to trust it. In that same survey, 66% said they relied on AI output without checking accuracy, and 56% said they made mistakes at work because they accepted an output uncritically. Your users trust your feature less than you think and lean on it more than they should.

So track both sides. Action metrics like completion rates and errors tell you what people do. Attitude metrics tell you what they think. Ship on one and you are guessing about the other.

The deep cut

Do not put a detector in the pipeline and call it a safeguard. Hamza Boulahia's piece walks through the classifiers, watermarks, and the theoretical ceiling, and lands on a hard result: as AI text gets more fluent, detectors slide toward random guessing. His own tests on a commercial detector turned up false positives on plain human writing. Treat detector scores as weak signals, never proof.

Which points back at the one move that actually pays off. You cannot buy your way out of this with a tool that stamps outputs "true" or "AI." You get reliability by writing the eval set yourself, from cases you know the answers to, and by grading the path and not just the ending. That is a PM and design call, not an engineering handoff. When your eval comes back at 50%, you decide which slices are good enough to ship and put guardrails on the rest. You hand your team a target instead of a feeling.

Three questions for your team

1. For the AI feature we are shipping next, do we have 100 cases with known correct answers, or are we relying on "it looked fine in the demo"?
2. Can we tell the difference between a right answer and a right answer for the wrong reason, or do our checks only read the final output?
3. Are we tracking user trust and over-reliance as real numbers, or assuming people use the feature the way we hoped?