The Agent Got It Right. It Also Got It Silently Wrong.

Your team is under pressure to point an AI agent at your data and let people ask questions in plain English. Sounds great. Someone types a question, the agent writes the query, the numbers come back. No SQL, no bottleneck. Before you greenlight that, let me catch you up on what a careful head-to-head test just showed.

The tie nobody expected

Someone ran the same three interview questions against SQL, Pandas, and a Claude agent, then timed each one over 500 runs. On the easy question, all three returned the same answer. On the medium question, same. On the hard question, three tables, window functions, running totals, the agent got it right too. It even used a different pattern than the reference solution and still matched exactly.

So the headline is boring in a good way. A modern agent, given a clear prompt, can write correct analytics code across easy, medium, and hard problems. That is real. If your team is still hand-writing every ad hoc query, an agent can take that load off.

The catch is what made the agent right, and what happens when that thing is missing.

The sentence doing all the work

On the medium question, the agent nailed it because the prompt spelled out one rule: users who never started a feature count as zero percent complete. That sentence was load-bearing. Drop it, and the agent writes an inner join that quietly drops the non-starters. Every average goes up. The numbers come back clean, and they are wrong.

That is the part worth stopping on. A wrong SQL join is visible in the text. You can read the query and spot the bad condition. When the agent fails this way, there is no error. You would only catch it if you already knew the right answer. That is the opposite of what you want from a tool people trust without checking.

Speed cuts the same way. SQL ran in fractions of a millisecond. The agent added two to four seconds of thinking time before any query ran, and each call produced different code. Fine for a low-stakes question a human reviews. Risky for a number that lands in a board deck.

The mess that comes before the query

Here is what the agent demos skip. Real data is not a clean table. A walk through cleaning a customer CSV shows the usual damage: column names with stray spaces, ages stored as text, three different date formats, dollar signs glued to numbers, and "N/A", "unknown", and blank cells all meaning missing but written five ways. Ten rows in, one was already a duplicate.

None of that throws an error either. Numbers stored as text sort wrong. A duplicate row inflates a count. An outlier can wreck an average all on its own, which is why there are several ways to catch them, from a simple IQR fence to isolation forests for messier data. Point an agent at dirty data and it will happily write a correct query over garbage. Correct query, wrong answer.

Cheap gut checks before you trust a number

Before anyone acts on a metric, run a fast read of the data behind it. Pandas' plain describe() gives you a five-number summary in one line, and chaining groupby in front of it splits that by segment, so you can spot when one group looks off. A step further, a tool called skimpy prints a full summary with a tiny histogram per column, so you see the shape of each distribution and the missing-value percentage without opening a chart.

That missing-value number is the one to watch. If a column is 20 percent empty and your query silently dropped those rows, your average is off and nothing warned you. A ten-second scan catches it. That habit costs your team almost nothing and saves the review where a leader asks why two dashboards disagree.

The deep cut

The agent is not the risk. The blank prompt is. Every silent failure here traces back to something a human forgot to say: how to treat non-starters, that a column was 20 percent missing, that spend was stored as text. The agent fills that gap with a guess and writes clean code around it.

So the useful move is not "agent yes" or "agent no." It is deciding where a wrong answer is cheap and where it is expensive. Let agents write the exploratory, low-stakes queries a person eyeballs before acting. For any number that drives a decision or ships to leadership, keep a reviewed, version-controlled query and a data check in front of it. Same rule you would use for a fast junior analyst: great for a first pass, checked before it counts.

Three questions for your team

• Which numbers in our current dashboards come from queries nobody has re-read in six months, and would we catch a silent wrong answer in them?
• For the metrics leadership acts on, do we know the missing-value rate and duplicate count of the tables underneath, or are we trusting a clean-looking output?
• If we let people ask our data questions in plain English, where exactly is the line between "answer they can act on now" and "answer a human signs off first"?