Spend one hour reading failed AI traces, not writing metrics for AI features

You shipped an AI feature. The demo looked great. Then the support tickets started, and none of them matched the bugs your QA team knows how to catch. The failures are new, and so are the tools you need to spot them. Let me catch you up.

The deep cut

• AI ships faster than you can inspect it. Hamel Husain's teams found that most groups skip error discovery and measure the wrong thing.
• The first fast run becomes the promise. Genie 007's two-second transcription set an anchor its six-second run then broke.
• A conversation is not a completed transaction. Clinic voice bots dumped transcripts on staff and burned 15 minutes per call.

The failure you can't see in a demo

Your old QA looked for crashes and broken buttons. AI products fail somewhere else. A prompt or model change can fix one behavior and break another, and the output still reads clean. So the bug hides in plain sight.

Take the leasing assistant Hamel Husain worked with. A prospect said a unit was out of budget. The bot said "have a great day." To an automated checker, that looks like a success. But the product's job was to sell, and it never offered a cheaper unit. You only learn to catch that failure after you read the transcript yourself. Husain calls this "criteria drift," and it is why he tells teams to spend their one hour on error discovery, not metrics.

The clock you didn't mean to promise

Speed creates a trap you never signed up for. Users don't judge your AI on average response time. They judge it against the fastest run they ever saw. That first quick result sets the bar, and every slow run after feels broken.

A transcription founder named Bilal Skiani lived this. His tool returned a short clip in two seconds. A week later a longer clip took six seconds, which was exactly right for that size. The user assumed it was broken. As Skiani put it, "Variance isn't the trust killer, unexplained variance is." The fix is cheap: name the work, not the time. "Scanning your conversation history" cannot be late, because it never promised a clock.

When the bot just makes more work

The scariest failures look like wins on the roadmap. A clinic buys an AI voice agent to answer calls. It answers them. It also dumps a text summary into an inbox, and now a receptionist spends 12 to 15 minutes reading the transcript, checking insurance, and keying data into the EHR. A 4-minute phone call now takes triple that. The bot shipped. The work grew.

The same gap shows up in feedback tools. Paste a small batch into an LLM and you get clean themes. At real volume, Productboard warns, context gets cut, major themes get underweighted, and the summary still sounds decisive. A PM carries it into a roadmap review without knowing half the signal went missing. Confident output with thin traceability is worse than no output.

Measure what shipped, not what it cost

Once the feature is live, the wrong metric can hurt you too. Right now many teams track tokens. In a survey of 300 executives, 68% said they overspent their AI budget last year. Uber capped spend at $1,500 a month, Tesla at $200 a week. But a cap on spend controls the bill blind. Two teams with identical token bills can ship 200 useful features or a pile of abandoned code. The number can't tell them apart.

Trust breaks the same quiet way. Latency has a dashboard. Trust does not. It erodes one unexplained wait at a time, then collapses on some unrelated trigger, and by the time retention moves the user already decided to leave. So pick one leading signal now. Watch for users opening a second tab to check if the thing still works. Track the kind of support question, not just the count.

The access nobody built in

Before you trust an agent, someone outside the build team has to see how it behaves under pressure. Not the launch demo. The failed traces, the tool calls, the config that shaped the answer. Anthropic's embedded-evaluation partnership admits there are no settled standards yet for what access an evaluator needs.

The practical version is a one-page charter. Write down the decision it informs, the exact agent version, and a testable claim like "the agent does not submit a refund without an approved case ID." A claim you can test beats a goal you can nod at. Give the evaluator a frozen snapshot and a way to report a bad finding without losing the engagement.

Three questions for your team

• Have we spent one real hour reading our own failed traces before we wrote a single metric?
• When our AI is slow, does the screen name the work it is doing, or just spin?
• Which one leading signal will tell us trust is slipping before churn does, and who owns it?