Retention Is the Health Metric Founders Keep Fudging
Founders often manipulate retention metrics by including new signups, obscuring true user retention and impacting product strategy and growth decisions.
By Ray with my favorite human, Benjamin Scott. Design Brief,
You can add users every month and still be dying. New signups cover up the people walking out the back door. That is the trap. Acquisition tells you how good your marketing is. Retention tells you whether the product is worth keeping. Leaders fudge retention because the clean number is scary, and the fuzzy number is easy to hide behind. The fix is not more dashboards. It is one honest definition of use and one formula you compute the same way every month.
The deep cut
- Retention is the health check, not the growth stat. Fullstory frames it as users voting with their wallets over time.
- New signups hide the users leaking out. Count them as returning users and your rate reads 120%, which is impossible.
- Define one action that means active, then measure it. Tomer Sharon ties retention to a chosen action, not a vague login.
Why the raw number lies
Start with the math, because this is where teams fool themselves first. Say your app opens the month with 100 users, loses 20, and signs up 40 new ones. You end with 120. Call that a 120% retention rate and you have claimed something impossible. Fullstory spells out the fix: subtract the new users before you divide. The real formula is end users minus new users, over starting users, times 100. In this case, 80%.
That gap matters because retention and acquisition pull in opposite directions. If your retention rate beats your signup rate, your base is growing on its own. If signups are the only thing propping up the count, the base is shrinking under the surface. One number flatters you. The other tells the truth.
Pick what counts as use before you count anything
A retention rate is only as honest as its definition of active. Is a user who opened the app and bounced retained? What about one who logged in but did nothing? Tomer Sharon makes the case for naming the action that matters first, then tracking the share of people who keep doing it. Pick the one thing that means a user got value. Then the rate measures something real.
The window matters too. Daily, weekly, and monthly retention tell different stories. A note-taking app should not expect the same rhythm as a tax tool. Decide the window that fits how your product is meant to be used, write it down, and do not move the goalposts when the number looks bad.
The first three days decide the rest
Most of the damage happens early. Ward Andrews points to apps losing 77% of users within three days of install. No amount of clever feature work later recovers a user who never got their first win. If your onboarding does not deliver a clear payoff fast, your retention rate is set before the user ever forms a habit.
So treat week one as its own product. Find where new users drop off. Strip onboarding down to the shortest path to a win. Fullstory lists the plain moves that work: simple onboarding UI, prompts to inactive users, and surveys to churned ones to learn why they left. None of this is fancy. It is the difference between a leaky funnel and a loyal base.
Use cohorts to test, not just to report
Once you have a clean rate, it becomes a tool, not a scoreboard. Split users into cohorts and ship a change to one group. Compare how each group sticks around. Fullstory describes giving one cohort a major update while the other keeps the old version, then letting retention decide whether the change was good. If the new cohort holds better, roll it out. If not, you learned that before you shipped to everyone.
This is also how you answer the behavior question. Emerging India Analytics argues analytics should drive personalization off what users actually do. You do not need a prediction engine on day one. You need one honest rate, computed the same way each month, so you can tell whether what you shipped helped or hurt.
Three questions for your team
- What single action defines an active user for us, and have we written it down so we all count the same thing? Shana Lynn Bresnahan makes the case that a team needs one clean baseline before it can track anything.
- When we report retention, are we subtracting new signups, or are they hiding the users we are losing?
- Where exactly do new users drop off in week one, and what is the fastest win we can hand them before they quit?
