LLM evals that actually catch regressions

Here is the pattern I have seen at four companies now. There is an eval suite. It has 200 cases. It has passed every run for five months. And support tickets about wrong answers have tripled in that time.

The suite was not lying. It was measuring the wrong thing, on cases that were never hard, with a scorer generous enough to accept anything plausible. This post is about building the other kind.

Start from failures, not from the happy path

Most eval suites are written the day the feature ships, from the developer's imagination. They encode what the system was designed to do, which is precisely the part that already works.

Useful suites are grown from real failures. Every time a human corrects the system, every time a ticket says the answer was wrong, every time you see a weird trace, that case goes into the suite with the correct outcome attached. Six weeks of doing this produces a suite with actual teeth, because every case is something that genuinely broke once.

Score the parts that are objectively checkable

Teams jump to fuzzy quality scoring far too early. Most of what matters is deterministic and cheap to check.

• Schema validity. Does the output parse against the expected shape, every time.
• Field-level exactness. Did it extract the right invoice number, date, total, party names. Exact match, no mercy.
• Tool trajectory. Did the agent call the right tools, and did it avoid calling the ones it should not have.
• Grounding. Does every factual claim map to a retrieved source. This catches hallucination without needing a judge.
• Refusal correctness. Did it decline the cases that should be declined, and not decline the ones that should not.
• Budget. Tokens, tool calls, and turns per case, with a ceiling. Quality that costs three times more is a regression too.

Only after those are locked down does a judged rubric for tone, clarity, or helpfulness earn its place.

Making a judge behave

LLM-as-judge works when you treat it as a narrow classifier and fails when you treat it as a critic. Four constraints have made it dependable for me:

1. One criterion per call. Never ask for an overall score.
2. A three-point scale with written anchors for each point. Ten-point scales produce noise.
3. Give it the source material and require it to cite the span supporting its verdict.
4. Spot check 20 judgments a week by hand. When the judge drifts, you find out from the sample, not from a customer.

Wire it into CI or it will rot

An eval suite that runs when someone remembers is a document, not a gate. The setup that has held up for me:

• On every relevant pull request. A fast subset, 20 to 40 cases, under two minutes. Blocks merge on regression against the baseline.
• Nightly. Full suite, results posted to the same channel that gets production alerts.
• On model version change. Non-negotiable. Provider updates are the single most common cause of silent quality shifts.
• Compare to a pinned baseline, not to an absolute threshold. What you care about is direction.

Set the failure condition on a delta, for example any drop over two percentage points on the exactness metrics, or any new hard failure. And cap total eval spend per run so the suite does not become the expensive part of your build, which pairs well with the tactics in prompt caching.

If your coding agents also open pull requests, evals become the guardrail that makes that safe. That whole setup is in putting AI coding agents in CI.

We had monitoring for errors and no monitoring for being wrong.

Offline evals do not replace production signals

A suite tells you whether a change is safe. It cannot tell you that your input distribution shifted last Tuesday when a client started uploading scanned faxes. For that you need production signals: human correction rate, escalation rate, grounding failures, retry rate, and cost per successful outcome. The logging spec for those lives in AI agent observability.

The loop between them is the point. Production surfaces a new failure mode, that becomes an eval case, the case guards against recurrence forever. That loop is the entire quality system. Everything else is detail.

A two-week plan

1. Week one: pull 40 real cases from traces and tickets, half of them known failures. Write expected outcomes by hand.
2. Week one: implement the deterministic scorers. Schema, fields, trajectory, grounding, budget.
3. Week two: wire the fast subset into CI with a pinned baseline and a delta gate.
4. Week two: add the rule that every bug fix ships with a case, and put nightly results where people already look.

That is achievable alongside other work, and it is the difference between knowing your system is fine and hoping it is.

If you want an outside read on where your current system is likely to drift, the AI production readiness audit includes an eval gap review, and agent development engagements ship with a suite by default.

Frequently asked