You Have Too Much Data and Too Little Signal

Your inbox is full. Support tickets, analytics dashboards, sales call notes, app store reviews. It feels like proof. It mostly is not. The hard part now is not gathering data. It is knowing which data earns the right to change your roadmap. Let me catch you up on three ideas that fit together better than they look.

The signal that feels like proof but isn't

Teresa Torres put it plainly on a recent podcast: "We're inundated with these low-value signals all the time, but they rarely carry enough signal strength to actually tell us what we should be building. But they feel like they do." That last line is the trap. A stack of support tickets feels like a mandate. It is really just a hint that something is worth a look.

Her ladder of evidence sorts this out. Low-effort signals like tickets and reviews are cheap and everywhere, but weak. Story-based interviews, where you get the actual narrative of what someone was trying to do and where it broke, cost more and tell you far more. The danger with weak signals is that your experts fill the gaps with their own guesses, and then ship those guesses.

More tests, more phantom wins

Here is the flip side of chasing signal in your numbers. The more you slice the data, the more likely you are to find something that isn't there. Jim Lewis and Jeff Sauro call it alpha inflation. Run one test at p < .05, and you have a 5% chance of a false alarm. Run 20 comparisons, and your odds of at least one false positive jump to 64%.

So the exciting finding your analyst brings to review, younger users behave differently on version B, might be a boulder that looks like a face. This matters because dashboards make it trivial to run dozens of cuts and cherry-pick the one that sings.

The fix is not to clamp down on every test. Cranking up the strictness to kill false positives just means you start missing real differences. Their advice: describe what tests you ran and why, and weigh the cost of a false alarm against the cost of a miss for your specific decision. Judgment, not a blanket rule.

Before you commit, size the blast radius

Intercom's Fin team treats bad decisions the way they treat outages. Their incident playbook runs on one line: roll back first, fix forward later. When something breaks, they don't chase an elegant fix while customers hurt. They get back to a known good state, usually in under two minutes, then investigate with room to breathe.

The same instinct applies to shipping on thin evidence. Ask what it costs to undo the call. If a feature is cheap to reverse, weak signal is fine to act on. If it is expensive to pull back, you need a richer story before you commit. Mark Gorman's team also drills clear roles so nobody negotiates ownership mid-crisis. Do that with product bets too: name who owns the call and who owns pulling it if the evidence turns out wrong.

Faster in the wrong direction

John Cutler adds the part that ties a bow on all of this. AI makes you faster, but speed is neutral. His portfolio question is sharp: "Faster in which direction?" In a build trap, where a cheap experiment never proved value but keeps eating resources, AI lets you prototype and synthesize feedback faster. It also makes it much easier to keep building on no evidence at all.

That is the risk when weak signal meets fast tooling. You accelerate the exact motion that got you stuck. Cutler's reminder holds: AI changes the cost and speed of responding to problems, but it does not remove complexity, uncertainty, or decay. The physics stay the same.

The deep cut

The three ideas point at one habit. Match the strength of your evidence to how hard the decision is to reverse. A cheap, reversible change can ride on a support-ticket hunch. An expensive, sticky bet needs a real interview or a clean test, not 20 data cuts you sifted for the pretty one. Petra Wille said it: "It is easier than ever to release mediocre software. People need to get better at interviewing." AI made shipping cheap. It did not make being right cheap. Before your next launch, write down the evidence you have, its rung on the ladder, and what it costs to undo the call. If those three don't line up, you are not ready.

Three questions for your team

• For our next big bet, what rung of the evidence ladder is this actually on, and what would it cost us to reverse if we're wrong?
• On the last analytics finding we acted on, how many comparisons did we run before we found it, and would it survive a second look?
• Which of our current projects is AI helping us build faster without any fresh proof that customers want it?