Your AI Dashboard Is Lying to You

Here's where we are. Leaders are getting asked, "How's the AI rollout going?" and they're answering with the number that's easiest to grab. Tokens used. Tool adoption. Weekly active users. Those numbers go up and to the right, so they make a clean slide. But they tell you almost nothing about whether the work got better. Let me catch you up on why that gap matters, and what to put on your next deck instead.

The number people game the second you watch it

Amazon started measuring employees on how many AI tokens they burned. Within days, people gamed it. According to Jeff Gothelf's writeup, employees wrote scripts to run dummy prompts overnight, padded prompts with filler to cost more, and asked the AI questions they already knew the answers to. The token meter kept running. The work did not get better.

The name for this is old. Goodhart's Law: when a measure becomes a target, it stops being a good measure. Token use is an output, not a result. When the metric your execs watch counts activity, the work people do is the work that makes the activity number bigger. You get tokenmaxxing, not value.

Why everyone reached for the easy number

This isn't really an Amazon story. For two decades the slow part of building was execution, so we measured the things that came with building: story points, velocity, tickets closed. None of those told you if you built the right thing, but they tracked shipping, and shipping was the bottleneck. The proxy was good enough.

AI broke that bottleneck. The cost of producing output dropped close to zero. So the old proxies stopped meaning anything, and leadership reached for the next easy thing to count: are people using AI? That's how you end up with token charts in the all-hands deck and 1:1s that open with "how are you using AI?" instead of "how did your work change a customer's behavior?"

The big public numbers are just as fuzzy

This trap isn't limited to internal dashboards. Benedict Evans points out that Google and Microsoft now show charts of "tokens generated," and it looks a lot like reporting bandwidth growth in 1996. The line goes up, but there are too many multipliers to know what it means. Models got more efficient. Agents and media use far more tokens per request. Google shows AI Overviews to everyone. Same chart, very different stories underneath.

The headline user counts have the same problem. OpenAI reports weekly active users, even though Sam Altman knows from his social media days that WAU is a weak metric. If someone uses a thing once a week, it isn't changing their life. OpenAI's own 2025 data showed 80% of users sent fewer than 1,000 messages all year, which is under three prompts a day. Usage a mile wide, an inch deep.

The harder question that actually works

The fix is not to stop measuring AI use. It's to measure the right thing. Gothelf gives a clean test: a good AI metric points at customer behavior or a business outcome, it can move in either direction with a thoughtful person able to defend the call, and it survives the question "would we still care about this number if AI didn't exist?"

Token consumption fails all three. "Did the customer come back next week?" passes all three. So does "did the feature ship faster and make the customer's job easier?" and "did the experiment teach us something we didn't know?" Notice these are the questions you were asking before AI showed up. That's the point. The job is to supervise direction, not count activity.

The deep cut

The real failure isn't the dashboard, it's that nobody agreed on the outcome first. Without an outcome, the only thing left to measure is activity, and the only thing people can do with an activity metric is perform it. That's tokenmaxxing in one sentence. And don't fall for the idea that a better model fixes engagement. Even OpenAI admits a "capability gap" between what models can do and what people do with them, which Evans reads as a polite way to say there's no clear product-market fit yet. A better model doesn't help someone who can't think of anything to do with the last one.

So before you add another "AI usage" chart to the Q3 deck, write down one sentence: what is the customer supposed to do differently on the other side of all this AI work? If you can't answer that, the chart is hiding the problem, not measuring it.

Three questions for your team

1. For every AI usage number on our dashboard, is there a paired outcome number? If "tokens consumed" has no "customer behavior that changed" next to it, which one are we actually managing to?
2. What is the one customer behavior we expect this AI investment to move, and would we still care about that number if AI didn't exist?
3. In our 1:1s and reviews, are we asking people to show their prompts or show their results? If it's prompts, what do we change next week so the answer is results?