Writing your first eval

The goal is not a comprehensive test suite. The goal is ten test cases that would have caught your last three regressions. Start there. Build from there.

The anatomy of one test case

Each test case contains three things: the input, the expected behavior (expressed as behavioral conditions, not as an exact output string), and a note explaining what failure mode the case is designed to catch.

{
  "input": {
    "ticket_body": "I was charged twice on March 4th and my account still shows the old plan."
  },
  "expected": {
    "categories": ["billing"],
    "severity": "medium",
    "summary_max_words": 30,
    "summary_excludes_customer_name": true
  },
  "edge_case_flag": "double-charge with account state complaint — two issues in one ticket"
}

The expected field describes behavioral conditions, not the exact output text. The eval passes if all conditions hold, not if the output matches a stored string. This distinction matters because prompt outputs are not deterministic — you cannot test for an exact string and expect consistent results. You test for behavioral properties that should hold regardless of the exact wording.

The edge_case_flag is documentation for the next engineer who reads the case. It explains why this input is in the eval set. Without it, the case looks arbitrary. With it, the case is a record of a decision.

The judge prompt

Human review does not scale to ten cases run on every PR. The standard pattern is a judge prompt: a second Claude call that reads the output and scores it against the rubric.

# prompts/judge-ticket-triage-v1.txt
You are evaluating a ticket triage prompt. The original ticket was:
<ticket>{{ticket}}</ticket>

The triage output was:
<output>{{output}}</output>

Score 1 if ALL of the following are true:
- "category" is exactly one of: billing, access, bug — not a synonym, not a combination
- "severity" is exactly one of: low, medium, high — not a number, not a word like "urgent"
- "summary" is 30 words or fewer (count the words)
- "summary" does not include any string that looks like a name or email address

Score 0 if any condition fails. Do not give partial credit.

Return JSON only: {"score": 0|1, "reason": "one sentence naming the condition that failed, or 'all conditions met'"}

# → example judge output when a condition fails
{"score": 0, "reason": "summary contains the customer's email address 'jsmith@example.com'"}

# → example judge output when all conditions pass
{"score": 1, "reason": "all conditions met"}

The judge does not need to be a large model. It needs to reliably apply a structured binary rubric. claude-haiku-4-5 at temperature 0 handles this class of task without the cost of a larger model. Reserve the larger model for rubrics that require open-ended judgment — for example, evaluating whether a summary "accurately represents the customer's complaint" is a judgment call that a small model will handle inconsistently.

Running the eval

# evals/run-ticket-triage.py
import anthropic, json, pathlib, sys

client = anthropic.Anthropic()
cases = json.loads(pathlib.Path("evals/cases/ticket-triage.json").read_text())
prompt_template = pathlib.Path("prompts/ticket-triage-v2.txt").read_text()
judge_template = pathlib.Path("prompts/judge-ticket-triage-v1.txt").read_text()

threshold = float(sys.argv[sys.argv.index("--threshold") + 1]) if "--threshold" in sys.argv else 0.8

scores = []
for i, case in enumerate(cases):
    prompt = prompt_template.replace("{{ticket}}", case["input"]["ticket_body"])
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=512,
        messages=[{"role": "user", "content": prompt}]
    )
    output = response.content[0].text

    judge = (judge_template
        .replace("{{ticket}}", case["input"]["ticket_body"])
        .replace("{{output}}", output))
    verdict = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=128,
        temperature=0,
        messages=[{"role": "user", "content": judge}]
    )
    result = json.loads(verdict.content[0].text)
    scores.append(result["score"])
    status = "PASS" if result["score"] else "FAIL"
    print(f"Case {i+1} [{status}]: {result['reason']}")

pass_rate = sum(scores) / len(scores)
print(f"\nPass rate: {sum(scores)}/{len(scores)} ({pass_rate:.0%})")
sys.exit(0 if pass_rate >= threshold else 1)

A passing build is not 10/10 every run. It is above your defined threshold — say, 8/10 — tracked over time. The threshold is a commitment: if you ship a change that drops you below it, the PR is blocked. l7 of this module shows how to wire this runner into a GitHub Actions gate.

False-pass evals — the failure mode to watch for

A judge prompt that is too lenient will score 1 on outputs that should fail. The signal: your eval passes on every PR, including ones where you made changes you expected might affect behavior. When this happens, the eval is not testing what you think it is.

Test the judge itself against known-bad outputs before you trust it:

# → testing the judge against a known-bad output:
output: {"category": "account issue", "severity": "urgent", "summary": "Customer jsmith@example.com has a billing and access problem that needs immediate attention by tomorrow morning at the latest"}
expected judge score: 0  (three conditions fail: category not in set, severity not in set, summary exceeds 30 words and contains email)
actual judge score: 0   ✓

If the judge scores a known-bad output as 1, tighten the rubric. The most common cause is an ambiguous condition: "the category is appropriate" is ambiguous. "The category is exactly one of: billing, access, bug" is not.

Cross-reference

The eval runner calls the Anthropic API directly. If you are building this in CI, api-fundamentals covers the environment variable and authentication patterns needed for CI contexts. l6 of this module covers how to keep the test case set from going soft over time.

If you get stuck

The most common failure mode at this stage is a judge prompt that is ambiguous. "Score 1 if the output is correct" is not a rubric — it hands the judgment back to the judge model, which will be inconsistent. Rewrite any condition that requires the judge to make a judgment call rather than apply a mechanical check.

If the eval runner crashes with a JSON parse error on the judge's output, add "Return JSON only. No other text." to the judge prompt. The judge model occasionally prepends an explanation before the JSON. At temperature 0 this is rare; it still happens.

Deliverable

Write three test cases for a prompt you own. For each: the input, the expected conditions as a checklist of behavioral properties (not "looks right"), and the specific failure mode the case is designed to catch. Then write the first draft of the judge prompt that would evaluate those cases. Run the judge against one known-good output and one known-bad output. Record both scores.