Your AI Product Doesn't Behave Like Your Old One. Your Metrics Should Change Too.

Your old products did what you told them. Ship a feature, it does the same thing every time. AI products don't work that way. The same prompt gives two users two different answers, and the answer drifts from one day to the next. That breaks the way we've been writing goals and running our workflows. Let me catch you up on what actually changed and what to do about it.

Stop grading the machine

When Jeff Gothelf worked with an AI team at a European bank, the team wanted to commit to "achieve 95% accuracy in all output." It felt rigorous. It's also the same mistake teams have made with OKRs for fifteen years. It measures the thing you shipped, not the person using it.

Accuracy is even shakier for AI, because the output is probabilistic. You don't fully control it. You can't honestly promise one accuracy number when the model behaves differently for every user and every prompt. So put the goal where it always belonged: on the human on the other side of the screen.

Watch what people do after the answer

Gothelf lays out three key results, and none of them measure the model. The first is behavior right after the AI responds. If the output is good, the user exports it, shares it, and moves on. If it's bad, they regenerate it three times, or open a blank doc and type their own notes. So name the good action and put a number on it: raise the share of meetings where the user shares the AI data without rewriting it from 40% to 65%.

Quality still matters, so measure it through behavior too. When people trust the output, they stop checking it. A real key result: cut sessions where the user opens the transcript to fact-check from 60% to 30%. Trust is the same story. Skip the survey, because people say they trust the tool and then verify everything it does. Watch what they hand over instead. Keep the rate at which users override or rewrite the AI's analysis under 8%, and grow auto-share adoption from 15% to 40%.

The model still needs work, and it fails in ways you can predict

Behavior metrics tell you if the feature earns its place. They don't fix a broken model. If your accuracy is bad, the engineering under the hood is usually the reason, and the failure modes are documented. The common one is retrieval irrelevance: a user asks about parental leave, the system pulls the 2022 policy, the 2024 policy, and a blog post, then blends them into a confident wrong answer.

The expensive trap is fixing this by piling on complexity. One manufacturer budgeted $400K for a RAG system, spent $1.2M in year one, and hit 23% accuracy before killing it. Enterprise RAG had a 72% first-year failure rate in 2025. Higher-dimension embeddings raise your compute bill and dodge the real question: was retrieval even the right architecture? Sometimes loading the whole corpus into the context window beats retrieval outright. Match the approach to the query type instead of assuming one pipeline handles everything.

Your prompts and skills need a save button

Here's the part that catches leaders off guard. The instructions you feed the AI drift too, and losing a working version costs you real hours. Aakash Gupta calls it AI workspace drift: last month your setup worked, this month it writes docs so lazy you rewrite them yourself, and you're scrolling old chats hunting for the prompt that worked.

The fix is version control, the same tool engineers use for code. Put your instruction files in a GitHub repo. Every change is tracked, so when a tweak breaks behavior you restore the previous version in one command. This isn't theory. The guide notes a working practice of reverting a context file multiple times after a change made output worse. Talk to the tool in plain language: pull the latest, make a change, commit it with a message that says what you did, push it. No git syntax to memorize.

The deep cut

Both of these come from the same fact: your AI product changes on you, from user to user and from week to week. That kills the two habits you leaned on before. You can't commit to a fixed output number, and you can't trust that yesterday's prompt still works today. So build a save-and-restore habit for your instruction files, and write your key results around one observable action. Before your next review, pick your top AI feature and answer one question: what is the single most valuable thing a user should do after it responds, and how much more often can you get them to do it? Start there.

Three questions for your team

• For our top AI feature this quarter, what specific user action proves it worked, and what number are we moving it from and to?
• Are our instruction files, prompts, and eval criteria in version control right now, or would we lose the working version if a laptop died tonight?
• When our model accuracy is low, are we adding complexity to the same pipeline, or have we checked whether that architecture was the right choice at all?