Amplitude turned ten years of analyst judgment into files an agent can run

Amplitude just shipped a stack of AI releases in a short window. Agents that watch your dashboards. Skills that run your weekly review. Connectors that pull Jira and GitHub into your analytics. A governed mode for banks. A whole self-improving product agent called Wave. It is a lot, and it is all branded. Let me catch you up on what actually changed and what you can take from it without buying the pitch.

The deep cut

• Agents move analytics from asking to answering. Smartsheet PMs now get instant funnel reads that once sat in a weeks-long SQL backlog.
• A high score means nothing without a trace behind it. Amplitude's own eval guide shows an agent can return clean JSON and still be useless.
• Encoded judgment is the asset, not the agent. Amplitude packaged ten years of how its analysts run reviews into skill files.

From ticket queue to self-serve answers

The pattern under all these releases is the same. The data team stops being the gate. Business users ask questions in plain language and get answers back without waiting on a ticket. At Smartsheet, a funnel analysis that took weeks of backlog now runs instantly. At NTT DOCOMO, Japan's largest carrier with over 100 million customers, product and marketing teams run cross-dashboard investigations without the analyst bottleneck.

This is the upside worth stealing even if you never touch Amplitude. The analyst role shifts from filling orders to checking work. Someone with context still has to verify the output before it ships. That job does not go away. It gets more important.

So the move for your team is not "replace the analyst." It is "free the analyst to validate and dig." Palo Alto Software's Shawn Hymer put the limit plainly: "Amplitude AI does not remove complexity, it helps teams navigate it without cutting corners."

The number that lies to you

Here is the catch nobody puts in the demo. AI agents do not behave like the buttons and forms your analytics was built for. Ask the same question twice, you get two different answers. A dashboard that tracks clicks never sees inside that mess.

Amplitude's own guide for product managers is the most useful thing in this whole batch, and it is not really selling you anything. It explains evals: repeatable tests that score whether an agent's output is good. The warning that matters most is about trusting the wrong signal. "A judge that gives every answer a passing score is worse than no judge at all because it creates false confidence." An agent can return valid JSON and still be useless.

Bring this to your next review. If your team is shipping any AI feature, ask what they measure beyond a pass rate. If the answer is "the model scores high on our test set," that test set only covers what someone thought to include. Real users type things no one anticipated.

Packaging the know-how, not the tool

The sharpest idea here is the skill file. Amplitude turned ten years of how its analysts run weekly reviews, audit replays, and score opportunities into files an agent like Claude or Cursor can execute. Agent-generated insights jumped from under 1 percent of all insights in October to 22.8 percent by May. Sandhya Hegde of Calibre Labs called it: "the skill file is where institutional knowledge lives now."

That is the part you can act on without a vendor. The weekly brief skill hardcodes one judgment call most agents miss: it compares the current week against a four-week baseline instead of reading a single snapshot. The opportunity skill only promotes a finding with multi-source evidence and a score over 100. A single anomaly is a story. Corroboration is what earns a spot on the backlog.

Write down the rules your best analyst follows and you have an asset that outlasts any one hire. That is the real lesson, with or without the marketplace.

When the real blocker is legal, not the model

The governed-analytics release is aimed at a different problem, and it is honest about it. "AI adoption doesn't fail because of a lack of interest; it stalls because of risk." Amplitude's financial services pitch scopes the AI down hard: no autonomous credit or pricing decisions, no touching regulated systems, no access beyond what a user already has.

The useful framing for any regulated leader is the three questions a security team actually asks. Can we trust your platform with our data? Can we trust the outputs? Will this deliver value without new risk? Most legal roadblocks are a lack of clarity, not a real objection. Answer those three up front with a low-risk pilot that avoids sensitive data, and reviews move faster.

Three questions for your team

• Which AI features are we shipping right now with no eval behind them, just a demo that looked good? Name them this week.
• What institutional knowledge lives only in one analyst's head, and what would it take to write it into a skill file or a checklist?
• Before our next AI rollout, can we answer the three security questions with real documentation, or are we going to find out in legal review?