On this page

Teams get evaluation wrong in two opposite directions.

Some skip it entirely. The feature demos well, everyone nods, it ships. Quality becomes whatever the loudest user says it is.

Others decide they need the full research setup before they can ship anything. Benchmark suites, eval platforms, a dashboard nobody opens. The feature sits in a branch for two months while the harness gets built.

Both versions lose. The first one ships blind. The second one never ships.

There's a smaller thing in the middle that works, and you can build it in an afternoon.

🔗The demo lied to me first

I've been building voice-first field entry for beekeeping operators. The pipeline is simple on paper: short voice note, transcription, then extraction into structured inspection fields.

In the demo it looked solved. I recorded a clean note at my desk, the fields filled in, everyone's happy.

Then I listened to real field notes. Wind. Gloves tapping the phone. Half-sentences, because the person is holding a frame of bees and does not care about my parser. Jargon the model had never seen used that way.

The extraction wasn't bad everywhere. It was bad in specific, repeatable ways. But I only knew that because I sat down and read about thirty real transcripts next to what the pipeline produced. Not a dashboard. A table with two columns and my own eyes.

That table was the first eval. Everything since has been making it cheaper to redo.

🔗What an eval actually is at this scale

Strip the vocabulary away and an eval is three things:

  1. A folder of real inputs — not synthetic ones you wrote to make yourself feel good
  2. What the output should have been, written down
  3. A script that runs your pipeline over the folder and shows you the diff

That's it. Mine is a directory of audio clips and expected field values, and a script that prints a pass/fail line per case with the failures expanded.

No platform. No vendor. A junior engineer can build this in a day, and the first version is allowed to be ugly.

The point is not statistical rigor. The point is that when you change a prompt, swap a model, or a provider silently updates something under you, you can answer "did this get better or worse?" in five minutes instead of finding out from a user.

🔗Keep the set small enough to read

My rule: the eval set should be small enough that I can read every failure end to end. For most features that's somewhere between thirty and a hundred cases.

That sounds unserious to people who came up on ML benchmarks. It's the opposite. A thousand-case suite with an aggregate score hides the story. Sixty cases you actually read tell you which failures you have:

  • wrong but confident — the dangerous one
  • right but mangled formatting — annoying, cheap to fix
  • refused when it shouldn't have — usually a prompt problem
  • correct but slow or expensive — an architecture problem wearing a quality costume

Those four failure modes want four different fixes. An aggregate score would have averaged them into one misleading number.

🔗Failures are free labels. Take them.

Every case in my set came from a real miss. A user corrected an extracted field, and that correction is the ground truth I wish I'd had on day one.

If your feature lets users edit AI output — and your first features should — you are sitting on a labeling pipeline you didn't have to build. Capture the before and after. When the edit rate on a field spikes, that field's cases go in the set.

The eval set grows the way a good test suite grows: every bug becomes a regression test. You don't schedule a labeling sprint. You just stop throwing the labels away.

🔗The habit is the hard part

The mechanics are an afternoon. The discipline is the actual work.

The rule I hold myself to: the eval runs on every prompt change, the same way tests run on every commit. Prompt tweaks feel harmless. They are code changes with worse version control and bigger blast radius, and they deserve the same gate.

The second rule: write down what "correct" means before arguing about whether the model achieved it. This is the step that embarrasses teams. More than once I've watched a group discover, mid-argument, that they never agreed on what the right output was. The model was taking the blame for a spec that didn't exist.

🔗What the loop buys you

A few things I didn't fully appreciate until I had it:

  • You can say no to upgrades. New model drops, everyone wants to switch. Run the set. If it's not better on your cases, the launch-day benchmarks are someone else's story.
  • You catch drift you didn't cause. Providers update models behind stable API names. Without a fixed set, that just feels like the feature "getting weird lately."
  • Cost conversations get honest. Cost per successful case is a much better number than cost per call. Sometimes the cheaper model is more expensive once you count the retries and the human cleanup.
  • You ship faster, not slower. This is the counterintuitive one. Confidence is speed. When the loop is green, I ship prompt changes the same day I write them.

🔗Final thought

If your team has an AI feature in production and no eval loop, you don't have a quality process. You have a complaint process, and your users are the harness.

Start with thirty real examples and a script that diffs. Read every failure. Add every miss. Run it on every change.

It's not research. It's just the discipline of looking, made cheap enough that you'll actually keep doing it.

🔗Next step

If you have an AI feature in production and no way to tell whether it's getting better or worse, that's a solvable problem.