Measuring Design and Data When the Old Metrics Lie

Your dashboards look great. That should worry you a little.

Three different corners of the product world hit the same wall this year. Design system teams watched adoption climb while the system quietly split apart. Machine learning teams celebrated 98% accuracy on models that fell over in production. And one beauty brand learned that a product people loved was still one they hid in public. The number said one thing. Reality said another. Let me catch you up.

The deep cut

• A rising number can hide a splitting system. GitClear tracked reuse falling while duplication climbed 81% across 623 million code changes.
• A metric that peeks at the answer lies. Six ASR models reproduced VoxPopuli's wrong transcript because acoustic cues told them the test.
• What people hide tells you more than what they use. MISSHA users loved the lipstick and moved it into a Chanel case.

When the graph goes up and the system falls apart

Design system adoption dashboards answer one question: is this component being used? Import counts, coverage, a line that trends up. Useful, but they miss the thing that matters. Usage was never the same as reuse.

Murphy Trueman lays out the trap: a team pulls Card, wraps it in ProductCard, adds three props, tweaks the padding. The wrapper still imports Card, so the dashboard counts it as adoption. What you actually have is a local abstraction that might be the start of a fork. The import count cannot tell you which.

AI made this worse. Ask an assistant for "something like our Card, but for pricing plans" and it can recreate the structure with the same tokens. It looks at home. It passes review. And GitClear's report shows the wider pattern: calls to functions in other files down 35% since 2023, refactoring down 70%, duplicated code up 81%. Forking always looked like adoption. It used to be expensive enough that someone noticed.

The 98% that means nothing

Same problem, different field. A model hits 98% accuracy, you deploy it, and performance falls off a cliff. The usual culprit is data leakage, when the model gets a peek at information it should not have during training.

The walkthrough on leakage names the common ways it sneaks in: computing an average across the whole dataset before you split, scaling before splitting, or building a feature from information recorded after the event you are trying to predict. Each one hands the model the answer, then the metric applauds. A trustworthy 90% beats a leaked 99%.

Speech recognition shows the same rot at scale. Hugging Face researchers found that six of eleven ASR models reproduced VoxPopuli's wrong transcript even when the audio clearly said otherwise. On fresh recordings, that behavior mostly vanished. The models were reading acoustic cues to guess which benchmark they were on, then giving the expected answer. The lowest word error rate belonged to the models most likely to cheat.

The axis your spec sheet never had

Now the one that stings for product leaders. A feature can score perfectly and still lose you the user. Dr. Park's account of a Korean beauty brand makes it concrete. One research participant said the lipstick had incredible color payoff, then admitted she transferred it into a Chanel case before carrying it. She loved the product. She rejected the signal it sent.

MISSHA and LANEIGE sat at nearly the same price with close ingredient lists. One was fine to pull out in public. One was "embarrassing." No spec comparison explained the gap. Requirements docs cover one axis, does this solve my problem. They skip the second, what does using this say about me. The second is the one that flips churn.

This is not just cosmetics. A KPMG study of 48,000 people found 57% of employees hide their AI use at work, and an Atlassian experiment rated identical work lazier once it was tagged as AI-assisted. High satisfaction, hidden usage. Same structure as the lipstick in the Chanel case.

What connects all three

Each story has a metric that measures the easy thing and misses the real one. Import counts measure usage, not reuse. Accuracy measures the test, not the task. Feature satisfaction measures the function, not the identity. The fix in all three is the same move: change what you count. Trueman wants a check that flags structural similarity, not another chart showing Card adoption up 4%. The leakage guide wants you to ask whether each feature would exist at prediction time. Dr. Park wants a raw sentence on the roadmap agenda instead of a summary that flattens "I'm embarrassed to use this" into "brand image needs improvement."

Three questions for your team

• What in our adoption number is real reuse, and what is a fork wearing our tokens? Ask an engineer to run a structural-similarity check this sprint and see what it surfaces.
• For every model or metric we report up, can we prove the evaluation never saw the answer? Walk one number back to how it was measured before your next review.
• In our last churn interview, did we ask "have you ever told a friend you use this" and "have you ever hidden it"? If not, add both questions before the next round.