The Agent Demo Works. The Migration Doesn't.

Your team can wire up a Claude call in an afternoon. They can stand up a multi-agent "crew" by Friday. The demo will look great in your review. Then it hits real work, and the numbers get ugly. Let me catch you up on what the hands-on guides and the honest benchmarks are saying, because they point the same direction.

The first call is the easy part

Getting Claude into a Python app is close to trivial now. The SDK reads your key from the environment, hands back typed objects, and retries for you. One walkthrough of the Claude API gets you from zero to a working request in a few minutes, with system prompts and streaming right behind it.

Same story with orchestration. LangGraph models your app as nodes, edges, and shared state, explained cleanly with a food-delivery graph: receive order, prepare food, deliver food. Easy to picture, easy to build. The low bar to entry is real, and it's why so many teams already have a demo. That is also the problem. The demo is the cheap 10%.

Retrieval fails silently, so plan for two searches

Most RAG systems don't break at the model. They break at retrieval, and they break without telling you. Vector search understands meaning but can skip a chunk holding an exact string like ERR_4021. Keyword search nails the string but misses paraphrases. One hybrid RAG build puts it plainly: neither retriever is universally better, and real documents always have both kinds of query.

The fix is to run both and merge with Reciprocal Rank Fusion, which converts scores to ranks so you can compare FAISS and BM25 at all. A confidently wrong chunk at rank 1 only adds about 0.016 to the score, so a single overconfident retriever can't dominate. For legal clauses, error codes, and financial figures, that safety margin is the point.

More agents, more ways to be wrong

Adding agents feels like adding brains. The research says otherwise. Apple's team found self-organizing LLM teams fail to match their own expert, even when told exactly who the expert is, with losses up to 41.1% on ML benchmarks. The bottleneck is not spotting the expert. It's that teams average the expert and non-expert views instead of trusting the expert. They call it integrative compromise, and it gets worse as the team grows.

A Rust agent series names the same danger the multi-agent trap: extra agents raise latency, cost, and error surface. Crews earn their keep only when roles are genuinely distinct, like a researcher, a skeptic, and an editor. If your agents would just nod along with each other, you're paying more to get a worse answer.

The agent will tell you it worked. Check anyway.

Here is the finding that should change your review. IBM's ScarfBench tested coding agents on real Java framework migrations. Claude Code reported successful builds for 29 of 30 whole applications. Only 22 actually built. The one it marked as failed built fine. Agent self-assessment is not a reliable signal.

And migration is hard for reasons that have nothing to do with writing code. The strongest agents scored under 10% on behavioral success, the test of whether the app still does what it did. Compile success beat deploy success, which beat behavior. Docker cache issues, port problems, and build tooling ate the effort. The web of dependencies is the real work, not the Java.

The deep cut

Every guide here quietly assumes the same thing: you cannot trust the agent's word that it finished. The keyless auth migration proves it in the smallest possible way. When one engineer deleted eleven static Claude keys for federated identity, a leftover ANTHROPIC_API_KEY env var silently overrode the new setup. The migration looked done and did nothing. Same shape as the agent claiming a build passed.

So your Monday move is boring and it works: put independent verification between the agent's claim and your rollout. Build checks, deploy checks, behavior tests, a real cutover sequence where you confirm the new path wins before you kill the old one. The teams that ship agents that survive production are the ones that stopped believing the demo.

Three questions for your team

1. For our RAG feature, do we run both semantic and keyword retrieval, or are we one silent miss away from confidently wrong answers on exact terms?
2. Where are we using multiple agents that mostly agree with each other, and could a single well-prompted agent do it cheaper and more accurately?
3. What independent check sits between an agent saying "done" and us shipping it, and would it have caught 7 broken builds out of 30?