Your Feedback Loop Was Built for a Product That No Longer Exists

You built a feedback loop for software that behaved the same way every time. Then you shipped an AI feature, and that stopped being true. The same user gets a different answer on Tuesday than on Monday. Two users ask the same thing and get different results on the same day. Your surveys, your tickets, your NPS all assume the thing being described stays put. It doesn't anymore. Let me catch you up on what changed and what to do about it.

When one bug report no longer speaks for everyone

Here is the old world, in one line from Jeff Gothelf: "When we built software whose behavior we specified, one person's bug report described everyone's product." You could reproduce the steps. You could take a screenshot that stood in for every screen.

AI breaks that in two directions at once. The same user gets different output on different days, and different users get different output on the same day. So a single piece of feedback no longer describes your product. It describes one draw from a distribution. Gothelf's own house is the example: his wife asks Claude something, it fails, he asks it the same thing his way, and he gets a completely different set of answers.

The question you ask changes. Not "what is the product doing?" but "what is the product doing across the range of what it does, and what are people doing about it?"

Watch what they do next, not what they say

The most honest feedback a user gives an AI feature is the action right after they see the output. Did they take it as-is? Edit it heavily? Re-query three times? Copy it out and then abandon the flow? Those are correction and override events, and you can measure them in a way opinions can't touch.

Start with one behavior. Gothelf points to override rate, the share of AI outputs a user changes or throws away before using. That number tells you more than a thumbs-up ever will. What the user does next is the signal.

Then widen the lens. Once a week, pull 30 to 50 real outputs at random and score them against three questions, not thirty: accurate, useful for what the user was doing, appropriate in tone. Don't chase the average. Look at the spread. The worst ten percent is your feedback backlog. Use real outputs with real context, because the test prompts you invent will always be kinder than the ones your users live in.

The pattern still decides whether the tap works

While the AI conversation eats the oxygen, the plain interaction rules did not get repealed. Usman Writes walks through the moment we all know: you tap the hamburger, wait for the drawer to lurch open, hunt for one link buried in a list of eleven, tap the wrong one, back out. Thirty seconds later you're on a competitor's site and you can't say why.

That failure wasn't taste. The icons were fine, the colors matched the brand. Someone picked the wrong pattern for the job. There are twelve ways to build mobile navigation, and the choice comes down to two questions: how often does someone need this, and how deep does it sit in the app.

A hamburger drawer for something people tap ten times a session adds a wasted animation to a one-glance task. A bottom tab bar stuffed with forty settings categories turns into a scroll mess. Match the pattern to the behavior, or bleed conversion in a spot no survey will flag.

Faster tools don't mean better tools

The build side is moving fast too, and it comes with the same trap. Pablo Stanley traded Figma and a sketchbook for a terminal, where he spends his days talking to agents and spinning up more of them. His verdict: he's never been more productive, building in a day what used to take weeks. His subtitle also says it's "great and horrible at the same time," which is the part worth sitting with.

Speed hides a quality problem. Nick Babich installed 20 popular community skills for Claude Code, the saved shortcuts that promise better output with no work. The hard truth: the vast majority did nothing at all. An independent study he cites, "Do Agent Skills Actually Help in Real-World Software Engineering?", backs him up.

So the grab-it-off-the-shelf move fails the same way copied feedback methods fail. It looks like the obvious win and delivers noise. Test whether a tool actually changes the output before you trust it.

The deep cut

The thread through all of this: stop trusting the artifact and start watching the behavior. A ticket, a survey score, a downloaded skill, a shipped menu, none of them tell you if value landed. The action does. Override rate tells you if the AI output was any good. The next tap tells you if the menu pattern fit. A measured before-and-after tells you if the tool earned its place.

So this quarter, pick one AI feature and instrument the next action after its output. Pull 30 real outputs Friday and score the spread, not the average. You'll walk into your next review with something you can put on a roadmap instead of "the AI is sometimes wrong."

Three questions for your team

1. For our top AI feature, can anyone say what users do right after they see the output? If not, what event do we instrument first, override rate or abandon rate?
2. Which mobile navigation choices did we make on taste, and which on how often people actually use them and how deep they sit? Where might that be costing us taps?
3. What tools and skills did we adopt because they looked like a free win, and have we measured whether any of them changed the output?