Your Agent's First Reply Is a Retention Decision

Here's where we are. Teams ship an AI agent, then chase first-message numbers like that's the win. The data says the real decision happens one beat later, in the first reply and the first session. Whether a user sends a second message, saves an output, or walks away is getting set by things you probably haven't touched: your starter prompts, your agent's tone, and whether your eval signal even talks to your product analytics.

Let me catch you up on what changed and what to bring to your next review.

The second message is the real activation

Forget the first message. People type something just to see what the thing does. The harder test is whether your reply earns a follow-up. Amplitude's own retention data found users who sent two or more messages in their first week retained at 2.5x the rate of users who sent only one.

That reframes activation for any agent you own. The North Star isn't "got them to ask once." It's "got them to come back for a second turn." A low second-message rate is a flashing sign that your first responses aren't landing. Start measuring it like you measure signups.

The starter prompt you forgot about

Most first impressions aren't typed by the user. They're chosen from your starter prompts and placeholder text. That UI you set once and forgot is doing more work than you think.

Amplitude found one default prompt, "Explain what I can do in Amplitude," drove 53% of all first messages. It also retained at just 14%. People treated a strong analytics agent like a support bot, because that's what the prompt told them to ask. Swap in prompts tied to the user's own work, like "What's changed in my key metrics this week?", and retention jumped to 34%, about 2.4x better.

Here's the move. Pull your prompt distribution. If one or two defaults carry the load, segment retention by first prompt. That 14% versus 34% gap was invisible at the top level. It only showed up after they sliced by which prompt users picked.

Tone is a product surface, not vibes

Agent personality usually shows up by accident, whatever the model and system prompt happen to produce. Amplitude treated two traits as deliberate choices: inquisitiveness, meaning does it ask a clarifying question, and helpfulness, meaning does it do the task or just hand you instructions.

Their first instinct was wrong. They told the agent not to ask follow-ups, figuring questions would make people bail. The agent got overconfident and gave wrong answers. When they A/B tested an inquisitive version, those users had longer conversations and saved more analyses. On helpfulness, an evaluator that read every conversation found the agent gave instructions when it could have done the work about 3% of the time. That number, not the loudest Slack complaint, is what gave them confidence to tune it up.

The first session is the whole decision

Here's the catch that changes your roadmap. For brand-new users, the first session is the entire adoption call. Calibre and Amplitude's analysis of 20K+ users found a strictly positive first week was worth a 3x retention multiplier over users whose first session tripped even one failure flag. Users who saved an output in that first clean session retained at 3x the rate of users who hit any failure, holding steady through week four.

Now the twist. For existing power users, eval scores and retention were basically unrelated, all four eval cohorts within 5% of each other. Power users push the agent hard, hit dead ends, and come back anyway. So if your aggregate retention looks fine, don't relax. Your veterans are hiding the damage being done to new users.

The deep cut

The reason most teams can't run this analysis is plumbing. Eval scores sit in one warehouse, product analytics in another, and any real question needs a JOIN and a stack of meetings. The 3x finding was only possible because session-level eval flags and product actions like saving a chart fired into the same event stream, scoped to the same user, with consistent naming.

So the action isn't "write better prompts." It's get your eval data and your product analytics sharing one schema, then watch one cohort: new users who had a clean first session AND saved something AND came back in week two. That number is your agent's future. Most of your save moments happen within an hour of first contact, so that hour is where you spend your attention.

Three questions for your team

1. What's our second-message rate, and have we ever segmented retention by which starter prompt a new user picked first?
2. Did we choose our agent's tone on purpose, or did we inherit it from the system prompt, and can we size a tone problem with data instead of Slack threads?
3. Can we run one query joining first-session eval outcome to whether a new user saved an output and returned, or does that still take a warehouse JOIN and three meetings?