You Finally Have a Yardstick for AI Chat. Now Watch What It Can't Measure.

For a year you've been shipping chat features on gut feel. No baseline, no way to tell your team "good" from "fine." That just changed. There's now a set of 2026 numbers you can hold people to, and two research papers that tell you where the numbers stop being enough. Let me catch you up.

The scores are in, and the pack is tight

MeasuringU ran a study of 420 U.S. users across ChatGPT, Claude, Gemini, and Grok in May 2026. The headline: these products are close. On perceived ease and usefulness, the spread was just 5.5 points, from 77.8 for Grok to 83.3 for ChatGPT. On the System Usability Scale, the gap between first and last was 3.1 points. None of that was statistically significant.

So when your team says "our chat scored a 79 on SUS," you now know that lands right in the same band as the big four. The average SUS across 500-plus products is 68. All four AI tools beat it. Use these as your floor, not your finish line.

Word of mouth is where the real gap opened

Usability scores clustered. Recommendation scores did not. In 2025 Claude's Net Promoter Score sat 4 points above ChatGPT. In 2026 that gap blew out to 21 points, 28 percent for Claude against 7 percent for ChatGPT. Every product measured in both years saw its NPS fall, but Claude pulled ahead on the strength of its growth.

Here's what should catch your eye: Claude wins on people wanting to recommend it while still lagging on perceived ease. Usefulness climbed, ease stayed behind. People will tell friends about a tool that helps them even when it's a little clunky. If you're picking what to fix next, that reorders your list.

What people actually complain about

The verbatim comments are the part to read out loud in your next review. Across ChatGPT, Gemini, and Grok, the top gripe was accuracy. One ChatGPT user put it plainly: "It's very confident in its errors, so I need to pay careful attention." Grok users hit hallucination loops. Gemini users ran fact-checks on what it found.

Claude stood apart. Its complaints were about usage limits and misread prompts, not wrong answers. Slow responses dogged Gemini and Grok. So "trust" and "speed" aren't one problem, they're two, and which one bites you depends on where your product sits. Pull your own support tickets and sort them the same way before you assume you have an accuracy problem.

The behavior fooled everyone, including the researchers

Here's where the benchmarks run out of road. A chatbot that talks smoothly feels like it understands you. It doesn't. A paper in Trends in Cognitive Sciences argues that no current AI, ChatGPT included, is conscious, because the appearance is built in a way that isn't similar enough to how we process information. Their line: something can behave as if conscious without being conscious.

Watch what that means in practice. In the grief-bot study from CU Boulder, researchers built two versions, one that spoke as the dead person and one that only described them. Participants kept treating the describer bot as the person anyway, ignoring the line the researchers drew. Users will read intention and feeling into your product no matter how you label it. Your framing is weaker than the illusion.

The deep cut

The grief study found people cared more about whether the tone "felt right" than whether it was factually accurate. One user got the closure she needed from a bot speaking as her late loved one. Another worried out loud: "I am worried that over time I will come to be reliant on the voice."

Sit with both. Your users will accept warmth over accuracy, and they'll form attachments you didn't design for and can't see in an NPS number. The paper calls for weighing emotional benefit against dependency, plus consent and family governance before shipping to grieving people. You don't need a grief bot for this to matter. Any chat product that sounds caring is making a promise. Decide now, on purpose, how far you want that promise to go, because your users will take it further than your spec sheet says.

Three questions for your team

1. Our chat scores near the 68-point SUS average and inside the big-four band. Are we treating that as passing, or as the floor we build up from?
2. Sort our last 100 complaints into accuracy, speed, and limits. Which one actually costs us recommendations, and are we fixing that one?
3. Where does our product sound more caring or more certain than it should? What's our written line on attachment, consent, and what we promise a user in a hard moment?