AI Observatory: half of real chatbot conversations get filtered out of the usage reports

Your dashboard says AI usage is up and to the right. Your vendor's deck says the same. Both are built on numbers that fall apart the second someone independent tries to check them. Let me catch you up on what a few researchers found when they looked hard at the data everyone keeps citing.

The deep cut

• Vendor data shows you the flattering half. The AI Observatory found 48% of real conversations would be filtered out of Anthropic's own index.
• A giveaway buys downloads, not paying users. Perplexity's India revenue rose, but Sensor Tower can't tell buyers from people who forgot to cancel.
• A blended score can hide two opposite facts. Citations and mentions correlate at negative 0.229, so averaging them is a category error.

The half of usage nobody reports

AI companies publish reports on how people use their tools. They release the data that looks good. When researchers behind the AI Observatory ran Anthropic's own filtering method on their dataset, nearly half of all conversations, 48%, got tossed out. Those tossed conversations skewed toward health, relationships, and adult topics. The reports focus on work. Real people use these tools for a lot more.

The usage also changes by model and over time. People turned to Claude for coding, Gemini for roleplay, and ChatGPT for homework. Conversations got longer and chattier, while the assistant disclosed less often that it was a bot. None of that shows up cleanly in a single company report. As one co-lead put it, "No single company report tells the whole story."

When two honest numbers can't agree

Here is how far this goes. Two credible providers measured the same thing, US searches that end without a click, over the same window. One published 22.4%. The other published 68.01%. Neither is lying. The gap comes from different definitions and different device panels, and one big chunk of it lives in a footnote about panel composition, not in any number you could correct for.

The lesson carries straight into AI-visibility tools. Citation overlap between AI platforms runs below 50% for every pair. Google's own AI Mode and AI Overviews share only 13.7% of cited URLs while reaching the same answer 86% of the time. There is no "AI search" in the aggregate. There are engines, each with its own behavior. A finding proven on one is a finding about that one.

The score that averages two opposite things

The sharpest warning here is about blended metrics. Being the most-cited domain in a category and being the most-mentioned brand correlate at negative 0.229. They move in opposite directions. Citations get decided at retrieval, so a page can win on shape alone. Mentions get decided at generation, and that runs on standing. Wikipedia gets cited 4.3x more than it gets mentioned. Reference sites get quoted. Brands get recommended.

So any dashboard that averages citation share and mention share into one "AI visibility score" is blending two weakly anti-correlated numbers made by different machinery. On top of that, Ahrefs found that 28% of brand mentions carry a link, and on Google AI Overviews roughly nine in ten carry none. A brand can be named a million times and show up nowhere in a referral log.

The growth chart that hides the churn

Perplexity ran one of the biggest growth experiments in AI. It gave 360 million Airtel customers in India a free year of Pro, normally worth $200. Downloads jumped 625% in one month, per Sensor Tower. Monthly active users peaked at 22 million. Impressive on a slide.

The catch sits in the fine print. The free plans auto-renewed. Users who did not want to pay had to cancel first. Revenue rose about 60%, but nobody can tell buyers from people who missed the cancel date. As Appfigures' CEO cautioned, the publicity may have pulled in paying users who were never part of the giveaway at all. The number went up. What it means is still unknown.

What to actually trust

The pattern across all of this is simple. Numbers that are easy to measure get measured, and the theory of value behind them gets skipped. John Cutler makes the point about "return on tokens" in his piece on curious proxies: tokens are very easy to measure, so people spend all their time on the "I" side of ROI and almost none on the "R." Same story as hours and story points. The measurement problem is old. AI just made it loud.

If you want a number worth trusting, MeasuringU built a validated questionnaire across 420 users of four chatbots. It found Claude leading on productivity and ChatGPT lagging on trust, and it can back those claims with factor loadings and reliability scores above 0.80. The point is not the ranking. It is that they showed their instrument. Ask your team to do the same before you act on any figure.

Three questions for your team

• When we cite an AI usage or visibility number in a review, can someone rebuild the denominator? If the definition and panel live in a footnote, we cannot compare it to anything.
• Are we averaging citation share and mention share into one score? If so, we are blending two anti-correlated numbers. Split them and report each engine on its own.
• On our own growth chart, how much is real intent versus friction? If a lift depends on auto-renewal or a giveaway, name that before we call it demand.