ChatGPT wrote "$3.68M in sales" while its own code said 3675.49

Your AI features look great in the demo. That is exactly the problem. The demo is one input, run once, by someone who wants it to work. Your customers run thousands of inputs, and the model gets a different answer each time. Let me catch you up on how teams are learning to measure quality and defend spend before they ship, not after a customer files a ticket.

The deep cut

• Confident prose is not measured data. ChatGPT wrote "$3.68M total sales" and "the data is clear" while its own code computed 3675.49.
• Define good before you measure it. Teresa Torres's variant prompt cut fabricated quotes from 12.5% to 1.7% by naming the rules.
• Bill the outcome, not the token. Madrona found 77% of enterprises re-evaluate AI vendors every six months.

What a demo hides

An LLM is probabilistic. Send it the same input twice and you can get two different answers, so testing it once tells you nothing. Teresa Torres puts it plainly: a unit test expects the code to always be right, but an eval measures how often the model gets it right. Those are different jobs.

The catch is that "right" is your call to make. Torres warns against letting a vendor define correctness for you. A verbose answer teaches a student and annoys a professor. Before you can score anything, you have to write down what good looks like for your product and your users.

The mistakes that pass every test

The scary errors are the ones that run clean. When KDnuggets ran three datasets through ChatGPT, the model measured ship-to-door when asked order-to-door. Valid Pandas, wrong columns, answer 2.4 times too small. No exception fires on a correct subtraction between the wrong two dates.

Worse, the numbers in the prose do not have to match the numbers in the code. One run reported "$3.68M total sales" and signed off with "the data is clear" while the code that ran, if it ran at all, produced 3675.49. Another dropped 226 blank rows without a word, shrinking the sample by 64%. Then they asked the model to check its own work. It fixed one error and approved two it had the evidence to catch sitting on screen.

Building a judge you can trust

You cannot hand-grade thousands of outputs, so teams use an LLM to score another LLM. Rohan Mistry is blunt about the risk: your judge might be lying to you. His fix is to treat the judge as a small ML system. Start from real failure modes, force binary decisions, make it reason before it votes, and calibrate it against human labels.

Torres's test harness shows the mix in practice. A code assertion checks that every quote appears word for word in the transcript. An LLM judge fact-checks claims, and it ships with 29 hand-labeled examples so you can check the judge. When she added grounding rules to the prompt, fabricated quotes dropped from 12.5% to 1.7% and status errors dropped to zero.

The number your CFO actually asks about

Cost per token is the wrong metric. An agent is a loop, so one completed outcome can take a dozen model requests. A wrong turn calls the wrong tool and loops to recover, burning tokens on turns that should never have happened. The number to defend is cost per successful outcome. An optimization that lowers cost per request while adding turns has made things worse, and only the outcome number shows it.

This connects straight to your revenue. Madrona's research found 77% of enterprises re-evaluate AI vendors every six months, and buyers surveyed by a16z want to pay for work produced, tickets closed or reports processed, not tokens burned. If you charge for outcomes, you had better be able to measure them.

The work AI hands back to you

Not every use of AI needs a heavy eval rig. John Cutler points at the grunt work of reshaping the same information for ten stakeholders who each want it "a certain way." That translation used to eat hours or get skipped. Now you write a skill per audience and let the team work how it wants. The win is doing what you always knew you should do but never had time for.

That still needs a quality bar. MIT Technology Review notes that 80% of the Fortune 500 have adopted agentic AI, but scale stays stuck in isolated pilots. Arun Chandra of NiCE argues agents should be held to the same standards as human workers. You cannot do that without evals.

Three questions for your team

• What is our cost per successful outcome for each shipped AI feature, and can we prove it, not estimate it?
• Have we written down what "good" looks like for our top workflow, or are we still trusting the demo?
• When our LLM judge scores an output, who calibrated it against human labels, and how often do we re-check its work?