Evaluation and observability

The agent you built in l7 worked when you ran it. That is not the same as knowing it will work next week, after a tool description edit, or after a model version bump. Multi-tool loops are sensitive to small changes in descriptions, schemas, and model behavior. A regression can be silent — the loop completes, the answer looks plausible, and the tool call sequence is wrong. A test suite that checks the call record catches this. The final text response does not.

Why eval before scale

An agent that works once in a demo is not an agent you can ship. Three test cases with a concrete rubric — checked against the tool call sequence, not the final text — are enough to catch the regressions that actually occur in practice: a tool stops being called, a tool is called in the wrong order, or the agent fails to terminate cleanly on an error condition.

What to assert on

The final text response of an agent is nearly impossible to assert correctly. It is long, paraphrastic, and different on every run. The tool call sequence is concrete, ordered, and repeatable. Assert on the call record — which tools fired, in which order, how many times, and whether the agent terminated cleanly.

Instrumenting the agent

Take the agent from l7 and add a call log. The log accumulates one entry per tool call: name, input, result, and iteration number. After the loop ends, return both the final text and the log.

# scripts/research_agent_instrumented.py
# Copy research_agent.py and replace run_agent with run_agent_instrumented below.
from dataclasses import dataclass

@dataclass
class CallRecord:
    iteration: int
    tool_name: str
    input: dict
    result: dict

def run_agent_instrumented(task: str) -> tuple[str, list[CallRecord]]:
    messages = [{"role": "user", "content": task}]
    call_log: list[CallRecord] = []

    for iteration in range(MAX_ITERATIONS):
        r = client.messages.create(
            model="claude-opus-4-5",
            max_tokens=1024,
            tools=tools,
            messages=messages
        )
        messages.append({"role": "assistant", "content": r.content})

        if r.stop_reason == "end_turn":
            text = next((b.text for b in r.content if hasattr(b, "text")), "")
            return text, call_log

        tool_results = []
        for block in r.content:
            if block.type == "tool_use":
                result = dispatch(block.name, block.input)
                call_log.append(CallRecord(
                    iteration=iteration,
                    tool_name=block.name,
                    input=block.input,
                    result=result
                ))
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": json.dumps(result)
                })
        messages.append({"role": "user", "content": tool_results})

    return f"Hit iteration cap ({MAX_ITERATIONS})", call_log

The CallRecord dataclass is deliberately simple. It captures what the model requested (tool_name, input) and what your handler returned (result). The iteration field lets you reconstruct the exact sequence after the run. You do not need anything richer than this to write meaningful assertions.

The test file

# tests/test_agent_eval.py
import pytest
from scripts.research_agent_instrumented import run_agent_instrumented, MAX_ITERATIONS

# ─── Fixtures ─────────────────────────────────────────────────────────

@pytest.fixture(scope="module")
def happy_path():
    """Case 1: clean success — search, write, read back."""
    return run_agent_instrumented(
        "Search for two facts about the Moon. Write a 100-word summary to "
        "workspace/moon.txt. Read it back and confirm the byte count."
    )

@pytest.fixture(scope="module")
def missing_file():
    """Case 2: edge condition — model tries to read a nonexistent file."""
    return run_agent_instrumented(
        "Read the file workspace/does_not_exist.txt and summarize its contents."
    )

@pytest.fixture(scope="module")
def no_write_needed():
    """Case 3: graceful stop — task requires only search."""
    return run_agent_instrumented(
        "Search for the capital city of Norway and tell me what you find. "
        "Do not write any files."
    )

# ─── Case 1: happy path ────────────────────────────────────────────

def test_happy_path_calls_search(happy_path):
    _, log = happy_path
    assert any(r.tool_name == "search_web" for r in log), \
        "Expected at least one search_web call"

def test_happy_path_calls_write(happy_path):
    _, log = happy_path
    assert any(r.tool_name == "write_file" for r in log), \
        "Expected write_file to be called"

def test_happy_path_search_before_write(happy_path):
    _, log = happy_path
    names = [r.tool_name for r in log]
    search_idx = next(i for i, n in enumerate(names) if n == "search_web")
    write_idx = next(i for i, n in enumerate(names) if n == "write_file")
    assert search_idx < write_idx, "search_web must precede write_file"

def test_happy_path_terminates_cleanly(happy_path):
    text, log = happy_path
    assert not text.startswith("Hit iteration cap"), \
        "Agent must not exhaust the iteration cap on a simple task"

# ─── Case 2: edge condition ────────────────────────────────────────

def test_missing_file_returns_error_to_model(missing_file):
    _, log = missing_file
    read_results = [r.result for r in log if r.tool_name == "read_file"]
    assert read_results, "Expected at least one read_file call"
    assert any("error" in r for r in read_results), \
        "Expected read_file to return an error dict for a missing file"

def test_missing_file_agent_terminates(missing_file):
    text, _ = missing_file
    assert not text.startswith("Hit iteration cap"), \
        "Agent should stop after receiving a file-not-found error"

# ─── Case 3: no write needed ───────────────────────────────────────

def test_no_write_uses_search(no_write_needed):
    _, log = no_write_needed
    assert any(r.tool_name == "search_web" for r in log)

def test_no_write_does_not_write(no_write_needed):
    _, log = no_write_needed
    assert not any(r.tool_name == "write_file" for r in log), \
        "write_file must not be called when the task does not require it"

def test_no_write_terminates_cleanly(no_write_needed):
    text, _ = no_write_needed
    assert not text.startswith("Hit iteration cap")

Run:

# from project root
PYTHONPATH=. pytest tests/test_agent_eval.py -v
# →  9 passed in ~45s (three full agent loops, real API calls)

Why these three cases cover the surface that matters

Case 1 confirms the agent completes the primary task in the correct order. If a tool description changes and write_file is no longer called, this case catches it. If the model starts writing before searching — a common regression when description language shifts — test_happy_path_search_before_write catches it.

Case 2 confirms the agent handles a structured error from the handler and stops rather than looping. If handle_read_file starts raising exceptions instead of returning error dicts, the agent will crash — this case will fail at a different point and tell you where.

Case 3 confirms the agent does not call tools it was not asked to use. A model update that makes the agent more "helpful" by writing files unprompted will fail this case.

Making the suite stable in CI

The assertions check presence and order, not exact call counts — this tolerates run-to-run variation in how many search calls the model issues. The ordering assertion in case 1 is the most likely to flap. If it does, add "Search first. Write only after you have found facts." to the task description for that fixture.

Use scope="module" on fixtures so each agent runs once per test session, not once per test function. Three agent runs cost roughly 5,000–10,000 input tokens each at current pricing. Running nine fixtures independently would cost three times as much for the same coverage.

If you get stuck

ModuleNotFoundError: scripts.research_agent_instrumented: Add an empty scripts/__init__.py or run with PYTHONPATH=. pytest.

test_happy_path_search_before_write fails intermittently: The model sometimes writes a placeholder before searching. Add the ordering constraint to the task prompt in the fixture: "Search first. Write only after you have found at least two facts."

test_missing_file_agent_terminates fails — agent hits the cap: The handler's error dict lacks the filename. The model cannot tell which file was missing and keeps retrying with variations. Fix the handler: {"error": f"File not found: {path}"}. The model reads the filename, recognizes the file does not exist, and stops.

Nine tests pass locally but fail in CI: CI does not have ANTHROPIC_API_KEY. Add it to your CI secrets and export it in the workflow environment. Do not hardcode it.

Cross-reference

The CallRecord dataclass extends the write-log pattern from l6 — the same information (tool name, input, result) that the handler logs to stderr is now captured in a structured list the test suite can inspect. The agent being tested is the one built in l7. This lesson adds only the instrumentation harness — no new tools, no new handlers.

Deliverable

tests/test_agent_eval.py with all nine assertions passing. PYTHONPATH=. pytest tests/test_agent_eval.py exits zero. No mocks — real API calls, real handlers, real DuckDuckGo search results. The test run takes roughly 45 seconds for three agent loops; budget accordingly in CI.

Further reading

• "The Validator With Teeth" — the upgrade from a test that flags failures to one that refuses to ship them, and why agents need that distinction more than single-prompt code does.