Your AI Feature Ships When It Learns to Say No

Your team has a demo that works. The problem is the word "demo." It works on the happy path, in the meeting, on the example you picked. Then real users show up and it invents a deductible, loops forever, or reports success on a task it never finished.

The writing this month is less about smarter models and more about plumbing. Three moves keep coming up: force the shape of the output, teach the system to refuse, and turn every failure into a test that never disappears. Let me catch you up.

Stop hoping for clean JSON

Prompting a model to "please return valid JSON" is a wish, not a guarantee. The newer approach controls the output at the moment of generation. When you watch XGrammar work, the first step of a JSON object leaves exactly one legal token out of fifteen: the opening brace. The other fourteen get their odds set to negative infinity before the model even votes.

The outlines library does the same thing with a friendlier front door. You hand it a Pydantic schema or a short list of choices, and it masks illegal tokens during generation instead of trying to repair bad text after. No trailing commas that crash your parser. No "kelvin" sneaking into a field that only allows three values.

One caution worth putting in front of your team: this guarantees structure, not truth. The JSON will be shaped right. Whether the values inside are correct is a separate fight.

Refusing is a feature, not a bug

A chatbot that always answers is easy. The one worth shipping knows when the evidence isn't there. Pariv Shah built a PDF question app that answers "what is my wind/hail deductible?" with the real clause and $2,500, then declines to answer "what is the capital of France?" because the insurance policy says nothing about it.

The refusal runs on three cheap layers. A retrieval gate checks whether the best chunks are actually related, and if the scores are weak, it refuses before the model is ever called. A strict prompt tells the model to answer only from the excerpts. A post-check catches the model when it still gives up. Tighten the distance thresholds for fewer hallucinations, loosen them for fewer false declines on reworded questions.

The retrieval itself uses two searches at once. Vectors find sections that mean something like the question, so "storm damage" still finds "wind/hail." BM25 keyword ranking catches the exact terms, like a section number or the word "deductible." Meaning finds the neighborhood, exact terms pick the right house.

The failure that comes back next week

Traditional software has a tight loop. Bug reported, reproduce, write a failing test, fix it, test passes, merge. That test stays in the suite forever, so the bug can't sneak back. Rashmi's point in the eval flywheel piece is that AI systems usually skip the failing-test part, so the same production failure resurfaces months later and nobody notices until a user does.

The fix is a discipline, not a tool. Every production incident gets triaged and labeled, reduced to a minimal case, graded with the right method, then added to an eval dataset that runs in CI and blocks the release if it regresses. Grading matters here: exact match for some cases, field diffs for others, rule-based checks, and an LLM-as-judge only when you actually need one, since judge grading is one of the listed failure modes.

This matters more for agentic features because they are non-deterministic. The same input can produce a different path twice. Without a growing regression suite, you are shipping on vibes.

Reliability has a bill attached

Every guardrail adds work, and work costs latency and money. The twelve latency and cost tactics start with a reminder your team probably skips: measure the right things first. Track time to first token, inter-token latency, cache hit rate, and P95 and P99, not just the average. Tail latency is usually what users feel.

The biggest savings come from cutting work you never needed. Route easy tasks like sentiment or extraction to a smaller model. Combine three sequential calls into one structured prompt. Use plain code for date formatting and permission checks instead of paying a model to do them. And put stable content, like system instructions and tool definitions, at the front of the prompt so prefix caching turns a long repeated context into a cache hit.

Anthropic's Building Effective Agents says the same thing from a design angle: agents bring higher costs and compounding errors, so reach for the simplest thing that works and add autonomy only when the problem demands it.

The deep cut

These three moves are not separate projects. They chain. Constrained output gives you a stable, gradeable shape. The refusal path gives you a clear, testable behavior, either a grounded answer or an honest decline. The eval flywheel captures both when they break and stops them from breaking twice. Structure is what makes grading possible, and grading is what makes the flywheel run.

So the practical order is backwards from how teams usually build. Before you add more model calls or a bigger context window, force your output into a schema you can check, define what "I don't know" looks like, and capture your next production failure as a test. That is the difference between a feature you demo and a feature you can leave running while you sleep. The KDnuggets roundup puts it plainly: making something work is the demo, knowing whether it works is the job.

Three questions for your team

• For our top agentic feature, what does a correct refusal look like, and do we have a single test that fails when the model answers a question it shouldn't?
• When we fixed our last production AI failure, did that fix become a regression test in CI, or did we just patch it and move on?
• Which of our model calls could be a smaller model, deterministic code, or a cache hit, and what would that save on P95 latency and cost?