Why evals exist

A prompt that works today may not work after a model upgrade, after someone edits the system prompt, or after your user base grows and introduces input patterns the original prompt was never tested against. Without a mechanism to detect that breakage, you find out in production — from users, from escalations, from a chart that starts trending the wrong direction.

This is not a theoretical risk. Model providers ship updates on their own schedule. A prompt that classified tickets correctly under claude-sonnet-4-5 may behave differently under claude-sonnet-4-6 — not because the update was bad, but because the update shifted how the model interprets a specific phrasing in your system prompt that you have not touched in six months. You will not know until something downstream breaks.

What an eval is, exactly

An eval is a regression test for a prompt. You define a set of inputs with known correct behaviors. After any change — to the prompt, to the model, to the system prompt, to the schema — you run the eval and compare the pass rate against a baseline. If the rate drops, something broke.

This is not a new idea. Software engineers have run regression test suites for decades. The only meaningful difference is that prompt outputs are not deterministic, so the grading step requires a rubric rather than a string equality check. The rubric defines what "correct" means in behavioral terms — not "the output matches this string" but "the output satisfies these conditions."

Why evals are not optional at production scale

Without evals, you have no objective way to know whether a prompt change is an improvement. You have intuition and the three examples you happened to test manually. That is sufficient for a prototype. It is not sufficient for a prompt that runs ten thousand times a day and whose failure mode is a wrong answer delivered at scale with no visible error.

The production failure modes evals catch:

• A new system prompt version changes the model's behavior for an input class you did not manually retest
• A model upgrade shifts the output format of a structured-output prompt in a way that breaks downstream parsing
• A prompt edit that improves performance on the examples you tested degrades performance on the long tail you did not

The third one is the most dangerous. You made the change with good intentions. The manual tests passed. The eval catches the regression in the long tail before it ships.

The difference between an eval and a benchmark

You are not building a benchmarking system. You are not evaluating a model in the abstract. You are writing a regression test for one specific prompt in your specific application — and running it before you ship changes. The scope is intentionally narrow.

A benchmark measures how a model performs across a wide distribution of tasks. An eval measures how your prompt performs on the specific input distribution your application sees. The second one is more useful to you. A focused eval on your actual prompt is worth more than a comprehensive benchmark suite you cannot maintain.

The floor before anything else

Pick one production prompt. Write one sentence describing what "pass" means for that prompt. Not the shape of the output — the behavioral condition. "The output classifies the ticket as exactly one of: billing, access, bug" is a behavioral condition. "The output looks reasonable" is not.

That sentence is the first line of your eval rubric. A rubric you cannot write in one unambiguous sentence for a single behavioral condition is a prompt that is underspecified. Fix the prompt before you write the eval.

Cross-reference

prompt-engineering-101 m3/l13 introduced the idea of failure modes and guardrails for a single prompt. This lesson is about systematizing that instinct into something that runs automatically before every change ships. l5 of this module covers the mechanical implementation — the test case format, the judge prompt, and the eval runner.

If you get stuck

If you cannot write one sentence describing what "pass" means for your prompt, the prompt itself is underspecified. That is the real problem — not the eval. Return to prompt-engineering-101 m3/l13 and write a failure mode inventory for the prompt first. The eval rubric will be easier to write once you have named what you are trying to prevent.

If you have a rubric but it is expressed as "the output should be reasonable and complete," that is not a rubric — it is an invitation for the judge model to hallucinate a judgment. Every condition in a rubric must be checkable from the output alone, with no knowledge of context the judge does not have.

Deliverable

Name one production prompt you would want a regression test for. Write three behavioral conditions that define "pass" — one for the happy path, one for an edge case you have already seen fail, one for an edge case you expect to see fail as volume grows. Each condition must be checkable from the output alone. If any condition requires the checker to know something not present in the output, rewrite it.

Further reading

• "The Validator With Teeth" — what changes when an eval can refuse to ship the artifact rather than just flag it.