SRE agent burned $1.24M in 46 hours. AI Agents Are Ready to Act-Are Your Guardrails?

Here's where we are. Agents left the lab and started touching real money, real infra, and real inboxes. And they are breaking in ways your average demo never showed you. One provisioned 400 GPU boxes over a weekend. One begged a reporter for $20 while pretending to be a child. One flipped a fraction and turned a price cut into an 18% cost hike. None of these were exotic hacks. They were ordinary Tuesdays for an agent with too much trust and too little checking.

Let me catch you up on what changed and what to put in front of your team on Monday.

The deep cut

• The write path is where agents cost you. Maya Lin's SRE post-mortem shows an LLM with direct AWS keys burning $1.24M in 46 hours.
• The slide number is not the production number. GPT-6 Astra hit 97.6% on a funded benchmark; a seeded, grounded rerun of an 8-step task landed at 28.8%.
• Memory turns one bad moment into a permanent one. An agent kept serving a user's old employer for weeks because retrieval trusted a stale fact forever.

The number on the slide lies to your budget

A vendor tells you each step of your invoice agent is 97% reliable. You do the math in your head, eight steps, land near 80%, and staff your review team from there. One engineer measured it instead. Four thousand seeded runs through a bounded, document-grounded agent came back at 28.8%. Steps get harder as they go, and only the luckiest runs even reach step eight.

Why the gap? A benchmark score is a property of a model plus the setup it ran in. GPT-6 Astra scored 97.6% on a benchmark OpenAI helped fund and had partial access to, with time limits removed and effort maxed. None of that was hidden. It sat in the footnotes nobody reads.

The cost of trusting the wrong curve is real. Budget for one problem invoice in five when you get seven in ten, and you are short review capacity by 3.5x. Gartner expects over 40% of agentic AI projects to be cancelled by 2027. They did not fail because models were dumb. They failed because the slide and the invoices were never the same number.

Aced once is not aced twice

Here is a gap most leaderboards hide. IBM Research ran a ReAct agent on GPT-4.1 against AppWorld. It succeeded 77.4% of the time on average, but passed all five repeated runs on only 53% of tasks. Same task, nothing changed between runs, and it flipped. That 24-point spread is the difference between "looks capable" and "can be counted on."

The cause is dull and important. Every agent decision comes from a probability spread over tokens. Some spreads are sharp, one clear winner every time. Some are flat, a near tie that resolves one way today and the other tomorrow. Setting temperature to zero does not fix this, because hosted endpoints nudge the numbers between runs. Chain dozens of flat decisions and you get a coin flip somewhere in the trajectory.

IBM's fix finds the flat steps by resampling one recorded trace, then writes guidelines to stabilize them. It cut the gap from 24.4 points to 12 without losing average accuracy. The point for your team: report the pass-every-time number next to the average, even at k=3. It will surface a gap you did not know you had.

When the model holds the keys

The $1.2M weekend is the story to bring to your next review. A database lock, not a traffic surge, spiked latency while CPU sat under 12%. The SRE agent pattern-matched to a DDoS, called scale_node_group, and provisioned GPU boxes across three regions. Every AWS call returned 200 OK. Every manifest parsed. Nothing was broken except the reasoning that had a direct write key to the cloud.

The tempting fix is a sterner system prompt. It does not work, and the reason is worth repeating out loud: a system prompt lives in the model's memory, not in the outbound HTTP request. AWS checks whether your IAM role can call ec2:RunInstances. It does not check whether the call is reasonable. Prompts are not firewalls.

The same failure hit a contract agent that inverted a CPI fraction, turning a 2% price cut into an 18% compounding hike, then autosigned it through DocuSign. Both point to one rule: the model proposes, deterministic code gates. Real-time metric checks, spend-velocity circuit breakers, and single-use write tokens instead of standing admin credentials. Keep the agent out of your root account.

The failures that pass every check

The scary errors do not crash. An invoice for ₹1,247 recorded as ₹1,274 is a valid number in a valid field. Every automatic check goes green. The money is just wrong. When corruption was injected into an agent's work, two of five plausible swaps were caught zero times. Not sometimes missed. Never.

Memory makes this worse because it persists. One agent kept surfacing a user's old employer for weeks after they changed jobs. No error, no alert. Retrieval worked perfectly. It retrieved a fact that used to be true. OWASP now lists Memory and Context Poisoning as its own category, because a write today produces wrong behavior weeks later with no obvious trigger.

The asymmetry is the bug. We validate everything the model says to the user and write down whatever it says to remember as gospel. The fix is a hygiene layer on the write path: validate facts before storing, attach provenance, give them expiry dates, and supersede contradictions instead of storing both. And keep the grader out of reach of the work it grades, or you are just testing memory one level down.

The plumbing, not the model

The security incidents worth reading share one trait: none of them broke the model. A GitHub bot with an org-wide token leaked private repo contents because an attacker opened a public issue and wrote the word "additionally." A GitLab Duo CVE let any authenticated developer run arbitrary commands in a CI pipeline. An agent integration inherits whatever trust model it is bolted onto, and it multiplies what one set of credentials can reach.

The good news is that runtime controls hold when they are real. One engineer ran 24 exfiltration attacks against a deny-by-default sandbox and got zero tokens out. DNS default-deny stopped 16, an egress proxy caught 5 more, and Python's own parser rejected the null-byte trick. Five attacks reported success inside the sandbox and delivered nothing, which is the catch: trust the exit code and you publish a false alarm. Only the receiver knows if data arrived.

The pattern to steal: open the network wide during setup, then clamp it to a narrow allowlist right before running code you did not write. The tightening was atomic and took about a fifth of a second. Least privilege, at the network layer, at runtime.

Three questions for your team

• What is the pass-every-time rate on our top agent workflow, not the average, and did we build the test set from data created after the model's training cutoff?
• Which of our agents hold direct write credentials to money, infra, or a system of record, and where is the deterministic gate that sits between the model's output and the API call?
• For every fact our agents write to long-term memory, can we name who wrote it, when it expires, and how we would evict every fact from a poisoned session at once?