When an agent cuts coding 80% and leaves you 6% worse off

The agent demos all look the same. An agent picks up a ticket, edits some files, runs the tests, opens a pull request. Ten minutes instead of an hour. Your inbox is full of these. The pitch is simple: ship more, faster, with fewer people.

The demo is real. The math around it is where teams win or lose. A cluster of sharp, practical pieces landed this month, and together they tell you how to separate what you ship from what you pilot. Let me catch you up.

The deep cut

• The agent is not the unit of trust. Murphy Trueman says you score the action, not the bot: a lint fix and an API change are different decisions.
• Execution time is one term in a seven-term cost. The break-even test shows an 80% cut to coding can still leave you 6% worse off.
• A model with an API key is not a system. Databricks Genie wrote a wrong MRR formula into a weekly report because a definition lived in the wrong place.

The ten minutes nobody budgets for

The demo clocks ten minutes of agent time and compares it to sixty minutes of a human doing the task by hand. Those are different quantities. The honest comparison, laid out in the break-even test for engineering leaders, splits human labor into seven stages: prep, execution, validation, correction, integration, supervision, maintenance. Agents shrink execution. The other six are where the cost quietly moves.

Run the numbers. A task takes 100 human minutes. Execution is 30% of that. An agent cuts execution by 80%, saving 24 minutes. If review and supervision add 20 minutes, you are 4% better off. If they add 30, you are 6% worse. An 80% win on the one stage everyone demos lands somewhere between a small gain and a small loss, decided by a number no demo shows you.

One ratio is worth tracking: the cost of verifying an agent's output divided by the cost of just doing the work by hand. When checking costs as much as doing, delegation cannot pay. That is why test coverage, typed interfaces, and reconciliation checks are agent investments, even though none of them is an AI project.

Score the action, not the agent

The common question is "can we trust this agent?" Wrong question, says Murphy Trueman, because agents don't come with trust ratings. What you can score is the action, by one test: how bad is it if the agent is wrong?

So you assign actions to tiers. Lint fixes, accessibility labels, doc typos are deterministic and low-cost, so they auto-merge. Token values and component descriptions go to a draft PR, because the agent can spot an inconsistency but can't reason about what breaks across every product that consumes it. Component API changes, deprecations, governance calls are suggest-only, with a human signing off every time. Not because the agent can't find the problem, but because deciding what to do needs context it doesn't have.

The boundary has to live in the wiring, not in a policy wiki. GitHub Primer ships an MCP server that only exposes read access to components and tokens, nothing that touches the repo. Trueman's warning is the part to write on the wall: when nobody draws the line, the default is that the agent decides, and by the time you notice, unwinding it costs more than the review would have.

Context the model doesn't have, and where to put it

A smart model is a bad coworker if it doesn't know your business. One Databricks Genie test makes this concrete. Asked to build a weekly revenue brief, the agent searched the workspace for a definition of MRR, found one from a different demo, and wrote that formula straight into the skill. It summed annual plans as stored instead of dividing by 12, used the calendar quarter when the fiscal year starts February 1, and counted upgrades as churn. Because the wrong formula was baked into the skill, it would have repeated those mistakes every Monday.

The fix is placement. Company-wide definitions go in a shared layer every agent reads. Team rules go in instructions. Repeatable tasks go in skills. Personal preferences go in memory, and never a definition, since private memory means your teammate gets a different answer than you. A good skill names the term and points to the shared definition, so you fix it in one place.

This is the real line between an agent and what the towardsai team calls an AI coworker: not more intelligence, but connectivity to governed data, long-term memory, and an audit trail. As the governed data infrastructure piece puts it, a model with an API key is not an AI system. The access boundary lives in the index, not in a prompt a clever question can talk its way around.

Cheap generation moves the bottleneck

More output is not more value. Anthropic's dynamic workflows let Claude orchestrate its own fleet of sub-agents, and the presenters were candid: the stock-scanner flow cost "much more" than a single agent while finding more detail. One engineer watching a pile of PRs becomes the breaking point. Their answer to runaway sub-agents was blunt: evals and lower defaults.

The break-even study names the trap. If agents produce 100 changes a day and reviewers can clear 60, the backlog grows by 40 a day and accepted output stays at 60. Upgrading the generator to 200 makes the queue grow faster and ships nothing more. Cheap generation moves the constraint to whatever stage didn't get cheaper, usually review.

It can be moved. One AI-forward company doubled reviewer load, handed much of it to automated review, and held merge and revert rates steady while merged PRs hit 2.09 times baseline. The point from Replit's Amjad Masad holds: keep human judgment in the middle of the loop. The constraint does not move by itself.

Three questions for your team

• For our top pilot, what are the real numbers in all seven labor stages, not just execution? If we can't measure supervision and review, we can't claim a saving.
• Have we written down which agent actions auto-merge, which go to draft, and which always need a human, with the reasoning for each edge? If not, the agent is deciding for us by default.
• Where do our business definitions live, and would an agent find the right one or a stale one from another project? Pick the shared layer before we wire anything up.