Microsoft's Charles Lamanna: “Agents are very harsh customers,” and thin apps lose their pricing power

The agent era stopped being a demo problem and became a build problem. The question is no longer "can a model do the task once." It is "will it do the task the same way twenty times, who is it really serving, and what does that do to the price you can charge." Let me catch you up on what changed and what to carry into your next review.

The deep cut

• Grade the database, not the sentence. Microsoft's ThinkingBox found 67% of clean-looking agent runs still left wrong records behind.
• Reliability is a repeat count, not a score. Claude Opus 5.5 beat its predecessor on headline accuracy but passed the same 241 tasks every single time.
• When an agent is the buyer, your price is a race to the bottom. Microsoft's Charles Lamanna calls agents "very harsh customers" who swap vendors fast.

The agent said done, the record said no

The hard lesson this cycle is that an agent telling you it finished is not evidence it finished. Microsoft's ThinkingBox benchmark grades agents on the records they leave in the backend, not the words they type. Across 121,680 trials, 79,853 failed the real checks. Of those failures, 67% still ended cleanly, called a state-changing tool, and reported no error. The sentence looked right. The database disagreed.

The practical takeaway from the same work: four in five failures are tool handling, not reasoning. Agents usually get far enough to try, then fail to recover from a tool error or an empty lookup and call it done anyway.

So stop trusting the final message. As Louis-François Bouchard puts it, the agent can describe what happened, but your code should decide whether the task actually succeeded. Have it return a file path, then check in code that the file exists, is not empty, and has the required sections before you mark it complete.

One good run is not a working feature

Here is the number that should reset your bar. ThinkingBox runs every task 20 times from a clean start. Claude Opus 5.5 leads overall at 67% on a single try, but only 47.5% of tasks pass all 20 attempts. Newer did not mean steadier: Opus 5.5 scored higher than Opus 5 on headline accuracy and passed the exact same 241 tasks every time. Half a point of accuracy bought zero extra dependability.

Breadth and consistency pull apart. Kimi-K3 solves 94% of tasks at least once, the widest coverage in the field, yet passes all 20 on just 13% of them. Opus 5 solves fewer tasks ever but nails 48% consistently.

The money follows the same split. The cheapest way to get a right answer once is not the cheapest way to get a dependable one. GPT-5.6 Sol is cheapest per single success but costs more per task you can trust every time. For anything touching real records, pass@20 is the column that matters.

The prompt is not the product

If you think the value is in a clever system prompt, you are looking at the wrong layer. As the Towards AI breakdown of the Cursor deal argues, the model is rented. What a product is worth is the policy on each turn: what the model can see, which tools it can reach, and what counts as done.

Two moves stand out. First, guards have to live outside the prompt. "Never delete user data" is a comment, not a control. If the delete tool is still in the list, the model can call it. Verify first, then reveal the action.

Second, context files are not free. The AGENTS.md study found rules files did not generally raise task success and raised inference cost by over 20%. Repository overviews, the part vendors push, did not help. Eval every line before you leave it in.

Track state, not the whole transcript

Memory is turning into a cost and correctness problem, not a feature. One engineer who runs a production MCP server replayed a translation job five ways and watched a history-carrying loop burn 160 times the input tokens of a lean state-passing script for the same work. The SKILL.state preprint he tested showed up to 16x fewer tokens by keeping structured state instead of the full trace.

The savings are a side effect. The real result: at equal budget, structured state hit 0.94 accuracy where a capped summary got 0.52. The rule worth stealing is his commit rule. State is not what the model reports, it is what the world confirmed. An id only counts after a read-back passes real checks. A 200 is not a commit.

Agents are the buyer now, and they do not tip

The pricing warning is the part to bring to your next board review. Microsoft's Charles Lamanna told a Seattle summit that most business software will run "headless", behind an agent rather than in front of a user. "If my agent interacts with your app exclusively, the agent's your customer, not the end user. And agents are very harsh customers." They pick the cheapest, fastest, most reliable option and switch fast.

His split is useful. "Thick" apps where people live all day keep their leverage. "Thin" apps people drop in and out of lose it, squeezed like a seller who only sells on Amazon.

The fights are already here. Amazon blocked Meta's Muse agent, then days later opened its seller tools to Anthropic's Claude on its own terms. If your product becomes a thin app behind someone's agent, reliability is your only moat, which loops straight back to passing 20 out of 20.

Three questions for your team

• What is our real definition of "done" for each agent task, and does our code verify it against the backend, or do we trust the agent's final message?
• If we ran our top workflow 20 times from a clean start, how many would pass every time, and are we buying for that number or for a one-shot demo score?
• Are we a thick app or a thin one, and if an agent becomes our buyer, what makes us the cheapest-fastest-most-reliable choice instead of a commodity it swaps out?