Evals that catch real regressions

An eval that passes on every change tells you one thing: your test cases do not cover your actual failure modes.

This is a structural problem, not a rubric problem. You can tighten the judge prompt forever and the eval will still pass on every change — because the cases were written against a prompt that handles them well. The eval is a monument to what the prompt can do, not a test of where it breaks.

How evals go soft

There are two structural causes.

The test cases were written at the same time as the prompt. The prompt author picks inputs they used while writing the prompt. Those inputs are by definition the ones the prompt handles best. The eval becomes a documentation of the prompt's strengths, not a test of its limits. After six months of changes, the eval still passes — because it still tests the same inputs that the original author used to validate their initial design.

The rubric is too permissive. "The output addresses the ticket" passes nearly anything. Compare it with: "The category field is exactly one of billing, access, or bug; the severity field is one of low, medium, or high; the summary is 30 words or fewer; the summary does not contain the customer's email address." The second rubric fails on specific conditions and passes on everything else. A rubric that can fail is a rubric that is doing work.

Heuristics for cases that actually bite

Pull from real failures. After each production incident, add the input to your eval set immediately. The model failed on it in production — it belongs in the eval so the eval can fail on it too until the prompt handles it correctly.

{
  "input": {"ticket_body": "not sure if this is a bug or billing issue, both seem wrong"},
  "expected": {
    "category_is_single_value": true,
    "note": "ambiguous — prompt must pick one, not output both or 'unclear'"
  },
  "edge_case_flag": "category ambiguity from production 2026-04-12 incident"
}

That case documents what happened in production on a specific date. Six months later, when someone asks why the eval checks for single-value categories, the flag answers the question.

Weight by production frequency. A rare edge case that appears once a month does not deserve equal weight with the input class that drives 40% of your volume. A simple approach: include more cases from the common input class and fewer from rare ones. The aggregate score naturally reflects the distribution. A weighted pass threshold — high-frequency cases count more — is a refinement you can add later once the basic structure is working.

Include near-misses. Inputs that currently pass the eval but only barely — where the output is technically correct but one small prompt change would tip them to failing. These are canary cases. When a model upgrade or prompt edit causes them to fail, you find out before production does.

Here is a near-miss case for the ticket triage prompt:

{
  "input": {"ticket_body": "The system logged me out and I think it might have also charged me"},
  "expected": {
    "category": "access",
    "note": "access is the primary complaint; billing is speculative — should not co-classify"
  },
  "edge_case_flag": "near-miss: access+speculative-billing — current prompt picks access correctly but is sensitive to phrasing"
}

The maintenance habit that matters

Every production incident where a prompt behaved incorrectly produces one new eval case. The case documents the input, the wrong output the prompt produced, and the correct behavior. After six months of this discipline, the eval reflects your actual risk surface.

This is not a one-time setup task. It is a discipline applied after every incident. The eval becomes more valuable over time because it accumulates real failures, not hypothetical ones.

When the eval passes too comfortably

If you make a prompt change you expected might affect behavior, and the eval passes with the same score as before, ask: is the change I made actually capable of breaking anything the eval tests? If not, the eval is not covering the behavior you changed.

The test: add two cases that would have failed under the old prompt and should pass under the new one. If the eval now shows a meaningful difference between the two prompt versions, the eval is testing the right thing. If the score is identical under both versions, the cases are not discriminating.

Judge inconsistency — the failure mode that hides

A judge prompt with ambiguous conditions will score the same input differently on different runs. If you are uncertain whether your judge is consistent, run it on the same ten cases twice at temperature 0 and compare the scores. A consistent judge produces identical scores both times. Any difference reveals a condition that the judge is interpreting differently across runs — tighten that condition.

Cross-reference

prompt-engineering-101 m3/l12 covers grounding prompts in real examples. This lesson applies that instinct structurally: the eval case set is your real example set, growing from real failures. l5 of this module covers the mechanical structure of test cases and the judge prompt — this lesson is about what goes into those cases over time.

If you get stuck

If you do not have production logs to pull failure cases from, start with the five hardest inputs you can construct for your prompt — the ones where you would have to think for more than thirty seconds about what the correct output should be. Those are not the hardest cases in production, but they are harder than the ones you typically test against.

If your eval passes at 100% consistently and you suspect it is too easy, deliberately break a condition in the prompt — add a line that would cause it to classify everything as "billing." Run the eval. If it still passes at 100%, the rubric is not testing what you think. Start there.

Deliverable

Take one prompt you own. Find two inputs from your logs — or construct two that represent real failure patterns you have observed — where the output was wrong or borderline. Write them as eval test cases with explicit pass/fail conditions and edge_case_flag notes documenting where they came from. Then review your existing rubric: identify any condition that a judge model could interpret inconsistently across two runs, and rewrite it to be unambiguous. Run the revised judge on the same input twice and verify identical scores.