NN/g tested synthetic users against 604 real people and the AI dropped 40 answers

The pitch for synthetic users is simple. Skip recruiting, skip scheduling, get answers in an afternoon. The pitch for AI interfaces is the same promise with a different coat: let the system decide what each person sees. Both save you time. Both ask you to trust a machine's version of what your users think. The research just caught up with how much that trust costs. Let me catch you up.

The deep cut

• AI gives you the middle, not the edges. NN/g's synthetic respondents matched the headline but dropped the uncommon answers real people gave.
• An eval is a product spec in disguise. Your360 built personas from failed calls because "accurate" meant nothing until they defined it.
• A link is a promise about the next page. Hudson's scent model says vague labels and AI-drafted "Learn More" make users backtrack.

Same answer, fewer voices

NN/g ran the cleanest test yet of synthetic users. They took one open-ended question from a survey of 604 real people and posed it to 120 AI-generated respondents. The big conclusion held: UX is still the dominant name for the work, in both groups. If you only needed the top-line answer, the robots got it.

But the humans used more than 50 distinct labels. The synthetic ones collapsed onto about a dozen. NN/g calls this mode collapse, where the model keeps spitting out the common answer and drops the long tail. The odd, specific terms real people gave, like "human-factors engineering" or "business analysis," never showed up.

It got worse inside subgroups. In the human data, 44% of product managers used UX terms. In the synthetic data, 93% did. The AI didn't just lose variety overall, it pulled every role toward the same framing. If you study groups to find how they differ, synthetic data will hide the differences you came to find.

Define good before you build it

AI products don't pass or fail. They land somewhere on a spectrum, and someone has to decide where "good enough" sits. The Product Journey makes the case that this is product work, not an engineering afterthought. A support agent told to be "accurate" can close a ticket while promising something the company can't deliver. "Accurate" was never a spec.

Writing an eval forces the questions out. What counts as resolved, the ticket closing or the customer confirming the fix? Anthropic's guidance, quoted in the same piece, puts it plainly: defining eval tasks stress-tests whether your requirements are concrete enough to build against. If you can't write the test, your spec has a hole.

At Your360 AI, Teresa Torres walks through how the team built personas from calls that failed, then used them to regression-test every model upgrade. Two kinds of checks: structural ones that catch misattributed feedback, and substantive ones that re-run the analysis to find patterns the first pass missed. The warning underneath: a system can get better at passing your tests while getting worse for your customers. Your definition is the product.

Adaptive is a promise you have to keep

The newest pitch is interfaces that reshape themselves per person. A banking app that guides the new customer and gives the pro a compact dashboard. The case for adaptive UX is real: remove friction, meet people where they are. The catch is that changing navigation unexpectedly makes familiar features vanish, and a confident wrong guess feels intrusive.

This connects to an old idea worth reviving. William Hudson's scent of information says every link is a promise about the page behind it. Front-load the specific words, since Nielsen's eye-tracking found the first two carry the weight. AI writing tools drift toward "Learn More" and "Read About Our Services," which break that promise and send users backtracking. Dynamic content inherits the same risk at scale.

The deeper problem is coordination. One UX Design piece describes an AI assistant telling a finance manager an invoice is ready to pay when it's actually blocked. Approval and payment eligibility are different concepts, owned by different teams, stitched into one broken answer. The user sees one experience. Your org chart sees five handoffs.

The method that still beats the model

When you want to know how your users think, there's still no shortcut that works. Card sorting asks people to group items and name the groups, surfacing their mental model instead of yours. Hudson is blunt about where AI fits: it drafts a plausible structure by copying common conventions and your own prompt, so it reproduces your assumptions rather than revealing your users'. Use it to draft, then run the sort to see if real people agree.

The craft still has rules. Online sorts cap around 60 to 70 items before fatigue sets in. Superficial matching, where people group items that share a word like "manage," is the top source of bad data, so strip repeated words first. Fifteen to twenty participants minimum for a quantitative sort.

The same discipline runs through all of it. NN/g's maps glossary exists because teams confuse a journey map with an experience map with a service blueprint, and the wrong artifact aligns people around the wrong thing. Naming is research. Synthetic users, evals, and adaptive UI are all bets on a machine's version of your user. The named method is how you check the bet.

Three questions for your team

• Where are we treating synthetic respondents as evidence instead of as a hypothesis, and which decisions depend on the long tail the AI drops?
• For every shipped AI feature, can we write down one real-world success and one failure a customer wouldn't notice? If not, what's our spec actually saying?
• Before we ship adaptive behavior, which navigation and labels stay fixed no matter what, so a wrong guess doesn't make familiar features disappear?