Your Job Just Moved From Making the Thing to Judging the Thing

AI is shrinking the gap between an idea and a working thing. A prototype now stands in for problem framing. A fast release stands in for evidence. That's real, and it's not going away.

But here's the part that should change how you run your team. When the machine makes the artifact, the human job moves to judging it. Defining what "good" means. Catching weak reasoning before it ships. Let me catch you up on what that means for how you staff, train, and run reviews.

When you can't write the spec anymore

For years the deal was simple. You wrote a spec, engineering built exactly that, QA checked it matched. That worked because old software was deterministic. Same input, same output, every time.

Generative AI breaks that deal. As Adam Elman at NN/g puts it, AI is nondeterministic. The same input can give you different outputs. Ask it about the weather and it might dump wind chill on you, or just say "It's nice out." The model is making design decisions, and you can't pin down every one in advance.

So the task flips. You stop specifying exact behavior. You start defining what good looks like and what it doesn't. The judgment still comes from research and design sense. You're just writing it down in a new form.

Critique becomes the work, not the meeting

Elman runs what he calls a judge-evaluate-iterate loop. You set criteria for "good," you score the actual output against them, then you fix the implementation and the criteria both. Repeat. The criteria are where your team does its real authorship now.

The hard part is making criteria objective without making them dumb. "Response must be under 10 seconds" is useless. So is "does it feel too long." His fix: classify the response type first, then judge each type by its own rule. An open-ended answer passes if it "fully answered the question and included at most one or two highly relevant extra facts." Still a little fuzzy, but tight enough that two reviewers agree.

One warning worth pinning to the wall. You can hand judging to another AI, the "LLM as a judge" move, but only if you calibrate it against real human ratings first. He treats an F1 score of 0.8 as the bar for trusting it. Skip the calibration and your eval data can quietly make the product worse.

Speed is not the same as getting smarter

Here's the trap. AI makes the work faster, and fast looks like progress. Synthesis that took days takes minutes. But the Discovery Judgment Framework makes a sharp point: running faster doesn't make your reasoning better. You still have to ask whether you framed the problem right, whether the evidence meant what you thought, whether to stop or push on.

Those judgment moments are the layer AI can't touch. And they only sharpen through deliberate practice, not more reps. Gale Robins names two habits: write down the reasoning behind a decision as you make it, then schedule a look back after the outcome lands. Not "what went well," but "what did we predict, where were we wrong, and what does the gap say about how we think."

The State of UX 2025 caught the cost of skipping this: operations speeding up, outcomes not keeping pace. Accumulation gives you experience. Only examined reflection turns it into judgment.

The people building with AI mostly can't explain it

NN/g studied nondevelopers building real agentic systems, people they call vibe architects. A head of product put his whole team on a Claude-based system that replaced meetings. An ops specialist deploys web apps. None of them can really explain how it works.

The behavior should worry you. One participant clicked "Accept" on every permission request without reading it. Another got carpal tunnel from it. One found a governance file Claude had written and maintained on its own, and was surprised by what was in it. Their systems decay every few weeks and need rebuilding. They learn from Twitter and Reddit, not the products themselves.

This is your near future. Capability is outpacing knowledge. People on your team will ship things they can't account for. The check on that is exactly the thing we keep circling: someone who can judge the output and define the boundary, instead of accepting it blind.

The deep cut

Don't buy the line that engineering was the bottleneck and now it's gone. John Cutler nails why in TBM 427: product work was never one clogged pipe. And here's the kicker. If coding really was the only thing holding you back, you were competing on speed in a world where everyone builds the same thing. Once AI removes that constraint for everyone, you lose.

So the practical move is to make critique an explicit, staffed role, not a vibe. Build a shared definition of "good" for your AI features and write it down as testable criteria. Calibrate any automated judge against human ratings before you trust it. And give people a persistent way to carry context, the way one designer built identity files for Claude so it knows his voice, his principles, and his failure modes without re-explaining every session. That same act of writing down how you operate is what sharpens the judgment in the first place.

Three questions for your team

1. For our top AI feature, can we write down right now what "good output" means in criteria two reviewers would agree on? If not, that's this week's work.

2. Who owns critique here? Name a person who defines the eval criteria and judges output, the way you'd name a tech lead, and make it part of their job, not a side task.

3. After our last big release, did anyone examine whether our reasoning going in was sound, and write it down? If the answer is no, we shipped but learned nothing about how we decide.