MemberPress AI Academy memberpress.ai

Courses

Tool Use — From Calculator to Agent

From calculator-as-a-tool to a small agent — define tool schemas, handle tool-use loops, instrument for production.

advanced2h 55m9 lessonsTaught by Seth Shoultes

PrimitiveThe Tool-Use Primitive

1. A calculator as a tool 15 min

Lesson notes

You sent Claude a math question and it answered from memory. Now you want it to answer from code — code you control, code that is always correct. The gap between "Claude answers" and "Claude runs your function" is exactly one JSON schema and one extra API call.

What a tool actually is

A tool is a JSON schema you attach to a Messages API request. The schema tells Claude what functions exist, what arguments they take, and what they return. Claude does not execute the function. Claude decides whether to call it and, when it does, emits a tool_use content block containing the function name and a validated input dict. Your code reads that block, runs the function, and hands the result back. The model never touches your runtime.

Here is the smallest possible tool definition:

# scripts/calculator.py
import anthropic, json

client = anthropic.Anthropic()

tools = [{
    "name": "add",
    "description": "Add two integers and return their sum.",
    "input_schema": {
        "type": "object",
        "properties": {
            "a": {"type": "integer", "description": "First addend"},
            "b": {"type": "integer", "description": "Second addend"}
        },
        "required": ["a", "b"]
    }
}]

def add(a: int, b: int) -> int:
    return a + b

response = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=256,
    tools=tools,
    messages=[{"role": "user", "content": "What is 41 plus 1?"}]
)

print(response.stop_reason)          # → tool_use
print(response.content)              # → [ToolUseBlock(id='...', name='add', input={'a': 41, 'b': 1})]

The response has stop_reason: "tool_use" and its content array contains a ToolUseBlock. That block has three fields you will use every time: id (a string like "toolu_01..."), name (the function name), and input (a dict matching your schema). Nothing about this block changes across any tool you will ever build.

Completing the round-trip

Once you have the tool call, you run the function and send the result back as a second request. That second request must include both the original assistant turn and a new user turn carrying the result:

# scripts/calculator.py (continued)
tool_block = next(b for b in response.content if b.type == "tool_use")
result = add(**tool_block.input)          # → 42

final = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=256,
    tools=tools,
    messages=[
        {"role": "user", "content": "What is 41 plus 1?"},
        {"role": "assistant", "content": response.content},   # full assistant turn
        {
            "role": "user",
            "content": [{
                "type": "tool_result",
                "tool_use_id": tool_block.id,
                "content": str(result)
            }]
        }
    ]
)

print(final.content[0].text)   # → "41 plus 1 is 42."

The assistant turn in the second request is response.content verbatim — not just the text, the full list of content blocks. Discarding it and sending only the tool_result is the single most common beginner error. The API will accept the malformed request but Claude will not know what it asked, and the answer will be incoherent.

Why start with addition

Addition has no side effects, no authentication, no error states, and no ambiguity. The result is always correct or obviously wrong. This means every bug you encounter is in the envelope — the tool definition, the round-trip structure, the result format — not in the tool logic. Learn the envelope here, with no noise, before adding any real complexity.

If you get stuck

Schema validation error on the first request: The input_schema must have "type": "object" at the top level. A bare {"type": "integer"} for a single-argument tool will fail. Always wrap in an object even if you have one parameter.

stop_reason is "end_turn" instead of "tool_use": The model decided not to call the tool. Usually the prompt does not require computation — try phrasing the question so the answer is a specific number, not something the model can guess from training data. "What is 41 plus 1?" works because the model recognizes the tool is the right channel. "How much is forty-one and one?" sometimes does not.

tool_result gets a validation error: The content field of a tool_result block must be a string (or a list of content blocks for richer returns). Passing an integer directly raises a validation error. Cast to str().

Cross-reference

If the two-turn structure feels unfamiliar, the api-fundamentals course, lesson l2, covers multi-turn message shape in detail. This lesson extends that directly — the only new element is tool_use blocks in the assistant turn and tool_result blocks in the user turn.

Deliverable

scripts/calculator.py — a self-contained script that defines the add tool, sends one prompt, executes the function, completes the round-trip, and prints the final text response. Thirty lines. No imports beyond anthropic. When you run it, the last line of output is "41 plus 1 is 42." or equivalent.

Further reading

Try these prompts

  • Define a JSON schema for a multiply tool that takes two integers and returns their product. Include the description field, required array, and one field description example.
  • Write the Python code to extract the tool_use block from a Messages API response and call the corresponding function.
  • Explain why the content field of a tool_result block must be a string, not an integer, and what error results from passing the integer directly.

Back to contents

2. How tool calls round-trip 18 min

Lesson notes

The calculator lesson showed one tool, one call, one result. Most real tasks require more than one tool and sometimes more than one call per turn. When Claude emits two tool_use blocks in a single response, both results must come back in a single user turn. Getting this wrong produces a validation error that tells you nothing useful about which rule you broke.

The full message structure, turn by turn

A tool-use exchange requires at minimum two API calls. Knowing the exact shape of each turn — every field, not just the conceptual summary — prevents the class of bugs that manifest as mysterious validation errors or incoherent model responses.

Turn one request:

# scripts/round_trip.py
import anthropic, json

client = anthropic.Anthropic()

tools = [
    {
        "name": "get_weather",
        "description": "Return the current temperature in Celsius for a given city.",
        "input_schema": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
        }
    },
    {
        "name": "get_humidity",
        "description": "Return the current relative humidity percentage for a given city.",
        "input_schema": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
        }
    }
]

messages = [{"role": "user", "content": "What is the weather and humidity in Oslo right now?"}]

r1 = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=512,
    tools=tools,
    messages=messages
)

print(r1.stop_reason)   # → tool_use
print(json.dumps([b.model_dump() for b in r1.content], indent=2))

Turn one response content (two tool calls in one turn):

[
  {
    "type": "tool_use",
    "id": "toolu_01ABCxyz",
    "name": "get_weather",
    "input": {"city": "Oslo"}
  },
  {
    "type": "tool_use",
    "id": "toolu_02DEFabc",
    "name": "get_humidity",
    "input": {"city": "Oslo"}
  }
]

Turn two — collect all results, return in one user message:

# scripts/round_trip.py (continued)
def get_weather(city: str) -> str:
    return "4"          # Celsius — real handler would call a weather API

def get_humidity(city: str) -> str:
    return "78"         # percent

HANDLERS = {"get_weather": get_weather, "get_humidity": get_humidity}

tool_results = []
for block in r1.content:
    if block.type == "tool_use":
        result = HANDLERS[block.name](**block.input)
        tool_results.append({
            "type": "tool_result",
            "tool_use_id": block.id,
            "content": result
        })

messages.append({"role": "assistant", "content": r1.content})
messages.append({"role": "user", "content": tool_results})

r2 = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=512,
    tools=tools,
    messages=messages
)

print(r2.stop_reason)          # → end_turn
print(r2.content[0].text)      # → "Oslo is currently 4°C with 78% humidity."

The three mistakes this structure prevents

First: discarding the assistant turn. The second request's messages list must include the full assistant response as it arrived — {"role": "assistant", "content": r1.content}. Sending only the tool_result without that preceding assistant turn causes a validation error: messages: roles must alternate between "user" and "assistant".

Second: splitting multi-tool results across multiple user turns. When the model emits two tool_use blocks in one response, both results go into a single user turn with two tool_result blocks. Returning them one at a time violates the alternation rule and will fail.

Third: mismatching tool_use_id. Each tool_result must carry the exact id from the corresponding tool_use block. A typo here produces a validation error on turn two: tool_use_id 'toolu_...' does not match any tool_use block.

What the message list looks like after turn two

After a complete two-call exchange, your messages list is:

[
  {"role": "user",      "content": "What is the weather and humidity in Oslo?"},
  {"role": "assistant", "content": [ToolUseBlock(...get_weather...), ToolUseBlock(...get_humidity...)]},
  {"role": "user",      "content": [tool_result(get_weather), tool_result(get_humidity)]},
]

This is the canonical state after a two-call exchange. Every subsequent turn appends to this list in the same alternating pattern. The model reads the entire history on each call — there is no session or context object that persists server-side. You are responsible for keeping the list intact.

If you get stuck

400 messages: roles must alternate: You forgot the assistant turn, or you returned multi-tool results in separate user messages. Print messages before the failing call and check the role sequence.

400 tool_use_id does not match: Check that the id you pass in tool_use_id is the exact string from the tool_use block, character for character. These are case-sensitive. Copy the id directly from block.id — do not reconstruct it.

Model makes a second tool call instead of answering: The model decided it needs more information. Handle it the same way — look for stop_reason: "tool_use", run the handlers, append results, call again. This is the beginning of an agent loop, covered in l7.

Cross-reference

The alternating-roles shape is introduced in the api-fundamentals course, lesson l2. Tool-use turns are an extension of that same pattern, not a separate protocol. The agent loop in l7 of this course wraps this same structure in a while loop with a termination condition.

Deliverable

scripts/round_trip.py — a script with two stub handlers and the full two-turn exchange. The final print outputs the model's text answer. Add a third tool (e.g. get_wind_speed), confirm that three results in one user turn still produces a coherent response, and note the output.

Try these prompts

  • Write the Python code to collect tool results from a response that contains two tool_use blocks — get_weather and get_humidity — and return both results in a single user turn.
  • Explain the three mistakes the two-turn message structure prevents: discarding the assistant turn, splitting multi-tool results, and mismatching tool_use_id.
  • Write what the messages list looks like after a complete two-call exchange with one user question and two tool results. Show the role and content shape for each entry.

Back to contents

3. When the model picks the wrong tool 18 min

Lesson notes

You have three tools defined. The user asks a question that should trigger one of them. The model answers from memory instead, or calls the wrong one, or calls none. The error is not in your handler. The error is in the signals you gave the model to make the decision — the tool name, the description, and the schema constraints. These are the only inputs the model has. Fix them and the behavior changes.

Tool selection is a model decision, not a random one

Claude chooses which tool to call — and whether to call any at all — based on three signals in order of importance: the tool's name, its description, and the constraints in its input schema. These signals are read every time. Change any of them and the selection behavior changes. This lesson is about reading the signals the model actually uses, not guessing at them.

Version 1 — the bad definition

# scripts/selection_demo.py
import anthropic

client = anthropic.Anthropic()

tools_v1 = [{
    "name": "db_op",
    "description": "Performs a database operation.",
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {"type": "string"}
        },
        "required": ["query"]
    }
}]

r = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=256,
    tools=tools_v1,
    messages=[{"role": "user", "content": "What orders does customer 42 have?"}]
)
print(r.stop_reason)    # often "end_turn" — model answers from memory instead of calling
print(r.content)

The description "performs a database operation" gives the model nothing. It cannot tell whether this tool reads, writes, which table it touches, or what inputs make sense. The model defaults to answering from its weights — which for a question about a specific customer's orders will produce a fabricated or generic answer.

Version 2 — improved description

A good description names what the tool does, names what it explicitly does not do, and provides at least one example of when to use it. These three elements are the minimum for reliable selection.

tools_v2 = [{
    "name": "query_orders",
    "description": (
        "Query the orders table. Returns the ten most recent orders for a given "
        "customer ID. Use this when the user asks about order history, recent "
        "purchases, or what a specific customer has bought. Do NOT use for "
        "inventory lookups or product searches."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "customer_id": {"type": "integer", "description": "Numeric customer ID"}
        },
        "required": ["customer_id"]
    }
}]

r = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=256,
    tools=tools_v2,
    messages=[{"role": "user", "content": "What orders does customer 42 have?"}]
)
print(r.stop_reason)                           # → tool_use
print(r.content[0].name)                       # → query_orders
print(r.content[0].input)                      # → {"customer_id": 42}

Version 3 — schema constraints

Schema constraints are the second dial. Add them after you have a good description. They do not replace the description — they complement it by eliminating impossible inputs before the call reaches your handler.

tools_v3 = [{
    "name": "query_orders",
    "description": (
        "Query the orders table. Returns up to 10 recent orders for a given "
        "customer ID and optional status filter. Use for order history queries."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "customer_id": {
                "type": "integer",
                "minimum": 1,
                "description": "Numeric customer ID, must be positive"
            },
            "status": {
                "type": "string",
                "enum": ["pending", "shipped", "delivered", "cancelled"],
                "description": "Filter by order status. Omit to return all statuses."
            },
            "limit": {
                "type": "integer",
                "minimum": 1,
                "maximum": 10,
                "default": 10
            }
        },
        "required": ["customer_id"]
    }
}]

The enum on status means the model will only pass one of those four values. It will not invent "processing" or "in_transit". The minimum/maximum on limit means it will not request 1000 rows. Schema constraints shrink the surface of malformed calls without requiring you to validate inside the handler.

tool_choice — use sparingly

# Force the model to call a specific tool, ignoring its selection logic
r = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=256,
    tools=tools_v3,
    tool_choice={"type": "tool", "name": "query_orders"},
    messages=[{"role": "user", "content": "Tell me about customer 42."}]
)

tool_choice: {"type": "tool", "name": "..."} bypasses selection entirely. Use it for structured-extraction steps where you are certain the tool should fire — not as a workaround for a bad description. If you need tool_choice to get reliable behavior, fix the description first. tool_choice is a shortcut; a bad description is still a broken contract.

tool_choice: {"type": "any"} forces the model to call some tool without specifying which one. Rarely useful outside evaluation harnesses.

When to merge tools instead of fixing descriptions

If two tools overlap enough that the model consistently picks the wrong one despite good descriptions, the tools may be conceptually too similar. Merge them into one tool with a type or action parameter:

# Instead of query_orders and query_returns as separate tools:
{
    "name": "query_customer_records",
    "description": "Query order or return records for a customer.",
    "input_schema": {
        "type": "object",
        "properties": {
            "record_type": {"type": "string", "enum": ["orders", "returns"]},
            "customer_id": {"type": "integer"}
        },
        "required": ["record_type", "customer_id"]
    }
}

Merging removes the ambiguity at the selection step by making it impossible to call the wrong tool — there is only one tool.

If you get stuck

Model consistently returns end_turn instead of using the tool: The description does not match the user's phrasing. The model cannot tell the tool applies. Add concrete use-case language to the description — the exact phrases the model is likely to see from users.

Model calls the tool but passes bad inputs: Add schema constraints. Check that required lists all fields the handler actually needs. If the model passes null for a required field, it read the schema and could not determine the value — your description is not specific enough about what to pass.

Model calls the wrong tool when you have many: Two descriptions overlap in vocabulary. Either split them more precisely (distinct verbs, distinct nouns, explicit "do NOT use for X") or merge the overlapping tools. Running both prompts side by side against the same question surfaces the overlap quickly.

Cross-reference

The tool_choice parameter interacts with the multi-turn loop in l7. When you force a tool call in an agent, the model cannot choose to stop — you must ensure the iteration cap is in place or the loop will never terminate on a forced-tool turn.

Deliverable

scripts/selection_demo.py — run the same prompt against all three tool versions, log stop_reason and content[0] for each. Add a comment above each version explaining what changed and why the behavior shifted. The v1 run should show end_turn; v2 and v3 should show tool_use with a valid input dict.

Further reading

  • "Switch Channels, Don't Dig" — when the model keeps picking the wrong tool, the operator's instinct is to add constraints. The essay names why that fails and what to do instead.
  • "Skipping Is a Feature" — sometimes the right behavior is no tool call at all, and the description has to license that explicitly.

Try these prompts

  • Rewrite this tool description to be a reliable decision rule: "Performs a database operation." The tool queries the orders table for a given customer ID.
  • Write a tool description that includes a positive use-case trigger and a negative exclusion, for a tool that queries order history but should not be used for inventory lookups.
  • Explain when to use tool_choice: {"type": "tool", "name": "..."} and when to fix the description instead. Give a concrete example of each.

Back to contents

Real ToolsTools That Do Real Work

4. Database read tools 18 min

Lesson notes

The model can answer questions about your data — but only if you give it a channel into the database. The channel has to be narrow enough that the model cannot accidentally mutate anything, and transparent enough that when the model generates a bad query you can see it. Building that channel is what this lesson is about.

Why read-only first

Write access is earned. A read-only SQL tool has a blast radius of zero — the worst outcome is a slow query, not a corrupted table. Build the read boundary cleanly, then expand permissions deliberately when you can audit the full call record. Many production use cases never need write access at all.

The tool definition

The model needs to know what it can query, not how it connects. Give it a query parameter and document the schema in the description. Never include the connection string in any field the model can read.

# scripts/db_tool.py
import anthropic, sqlite3, json

DB_PATH = "data/orders.db"     # create this with sqlite3 for the exercise
MAX_ROWS = 20

client = anthropic.Anthropic()

tools = [{
    "name": "query_db",
    "description": (
        "Execute a read-only SQL SELECT query against the orders database. "
        "Tables: orders(id, customer_id, status, total, created_at), "
        "customers(id, name, email). "
        "Returns up to 20 rows. For larger result sets, add a WHERE clause "
        "to narrow the query. Do not use INSERT, UPDATE, DELETE, or DROP."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {
                "type": "string",
                "description": "A valid SQLite SELECT statement."
            }
        },
        "required": ["query"]
    }
}]

The handler

def handle_query_db(query: str) -> dict:
    # Block write statements before they reach the database
    normalized = query.strip().upper()
    for banned in ("INSERT", "UPDATE", "DELETE", "DROP", "CREATE", "ALTER"):
        if normalized.startswith(banned) or f" {banned} " in normalized:
            return {"error": f"Write statements are not permitted. Received: {query[:80]}"}

    try:
        conn = sqlite3.connect(DB_PATH)
        conn.row_factory = sqlite3.Row
        cursor = conn.execute(query)
        rows = cursor.fetchmany(MAX_ROWS)
        data = [dict(row) for row in rows]
        conn.close()
        return {
            "rows": data,
            "row_count": len(data),
            "truncated": len(data) == MAX_ROWS
        }
    except sqlite3.OperationalError as e:
        return {"error": str(e), "query": query}

Three things this handler does that matter: it blocks write statements before they reach SQLite, it caps the result at MAX_ROWS, and it returns errors as dicts rather than raising exceptions. Claude can read a structured error and adapt — if it receives {"error": "no such table: order"}, it will correct the table name on the next call. An uncaught exception crashes the loop.

Handling the round-trip

def run_db_agent(user_question: str):
    messages = [{"role": "user", "content": user_question}]

    while True:
        r = client.messages.create(
            model="claude-opus-4-5",
            max_tokens=512,
            tools=tools,
            messages=messages
        )
        messages.append({"role": "assistant", "content": r.content})

        if r.stop_reason == "end_turn":
            return r.content[0].text

        tool_results = []
        for block in r.content:
            if block.type == "tool_use":
                result = handle_query_db(**block.input)
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": json.dumps(result)
                })
        messages.append({"role": "user", "content": tool_results})

print(run_db_agent("Which customers placed orders over $500 last month?"))

Handling large result sets

When truncated is True in the response, Claude will often follow up with a narrower query. This is the correct behavior. You do not need to handle it specially — just let the loop continue. The pattern of "broad query, see truncation, narrow query" is reliable because the model reads the result dict.

If the model keeps issuing broad queries that return 20 rows without narrowing, add a sentence to the tool description: "If results are truncated, issue a more specific WHERE clause before summarizing." One sentence in the description is enough — the model follows it.

If you get stuck

sqlite3.OperationalError: no such column: The model hallucinated a column name. Fix: add the exact column names to the tool description. The model reads them and will use them. A description that says "Tables: orders(id, customer_id, status, total, created_at)" gives the model the exact column list it needs.

Model issues a SELECT but gets back 0 rows, then asserts "no orders exist": The query has a logic error — likely a date format mismatch or a wrong join condition. Return the query in the error dict so you can inspect it. {"error": str(e), "query": query} gives you the exact SQL in the tool result, which you can then run manually to diagnose.

Model tries to INSERT despite the ban: The keyword-match logic above uses startswith which catches INSERT INTO ... but not SELECT * FROM (INSERT ...). For production, use a proper SQL parser or open the SQLite connection with conn.execute("PRAGMA query_only = ON") before running the query. The pragma raises an OperationalError on any write attempt at the database level, which is harder to circumvent than string matching.

Cross-reference

The agent loop above is simplified — it has no iteration cap and no dispatch() abstraction. l7 covers the full dispatch table, loop termination conditions, and iteration caps. Build the read-only tool here and plug it into the full agent architecture in l7.

Deliverable

scripts/db_tool.py — a SQLite database with at least two tables, the query_db tool, the handler with write-blocking, and the agent loop. Run it with two different natural-language questions and confirm the model issues syntactically correct SQL both times. Check that a question phrased to require a JOIN actually produces a JOIN in the emitted query.

Try these prompts

  • Write a tool description for a read-only SQL query tool that documents its table schema inline and instructs the model not to use INSERT, UPDATE, or DELETE.
  • Write the Python handler function that blocks write statements before they reach SQLite and returns a structured error dict instead of raising an exception.
  • Explain why the handler returns {"error": str(e), "query": query} instead of just {"error": str(e)} and how the model uses the query field.

Back to contents

5. External API tools 18 min

Lesson notes

Your handler calls an API the model has never seen. The model passes a city name; your code geocodes it, fetches weather, and returns three fields. The model does not see the geocoding step, the HTTP headers, or the full upstream response. It sees what you choose to return. That boundary — what you expose versus what you hide — is the entire design problem.

The adapter contract

An external API tool is a thin adapter between the model and an upstream service. The model sees: parameter names, descriptions, and the return structure you choose to expose. The model does not see: your API key, your retry logic, HTTP headers, or the full upstream response envelope. Your job is to decide exactly where that boundary sits and keep it narrow.

A complete example: Open-Meteo weather

Open-Meteo is a free, no-auth weather API — useful here because it removes the credential step and lets you focus on the adapter structure.

# scripts/weather_tool.py
import anthropic, httpx, json, os

client = anthropic.Anthropic()

tools = [{
    "name": "get_current_weather",
    "description": (
        "Return current weather conditions for a city. "
        "Use when the user asks about temperature, wind, precipitation, or "
        "general weather. Returns temperature_c, wind_kph, and condition. "
        "Do not use for historical data or forecasts beyond today."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "City name in English, e.g. 'Oslo', 'Tokyo', 'Chicago'"
            }
        },
        "required": ["city"]
    }
}]

The handler with real error handling

GEOCODE_URL = "https://geocoding-api.open-meteo.com/v1/search"
WEATHER_URL = "https://api.open-meteo.com/v1/forecast"

def get_current_weather(city: str) -> dict:
    # Step 1: geocode the city name to coordinates
    try:
        geo = httpx.get(GEOCODE_URL, params={"name": city, "count": 1}, timeout=8)
        geo.raise_for_status()
        results = geo.json().get("results", [])
        if not results:
            return {"error": f"City not found: {city}"}
        lat = results[0]["latitude"]
        lon = results[0]["longitude"]
    except httpx.TimeoutException:
        return {"error": "Geocoding timed out. Try again."}
    except httpx.HTTPStatusError as e:
        return {"error": f"Geocoding failed: {e.response.status_code}"}

    # Step 2: fetch weather
    try:
        params = {
            "latitude": lat, "longitude": lon,
            "current": "temperature_2m,wind_speed_10m,weathercode",
            "timezone": "auto"
        }
        weather = httpx.get(WEATHER_URL, params=params, timeout=8)
        weather.raise_for_status()
        current = weather.json()["current"]

        # Map WMO weather code to human-readable condition
        code = current["weathercode"]
        condition = {0: "clear", 1: "mainly clear", 2: "partly cloudy",
                     3: "overcast", 45: "foggy", 61: "light rain",
                     63: "moderate rain", 80: "rain showers"}.get(code, f"code {code}")

        # Return only the three fields the model needs
        return {
            "temperature_c": current["temperature_2m"],
            "wind_kph": current["wind_speed_10m"],
            "condition": condition
        }
    except httpx.TimeoutException:
        return {"error": "Weather fetch timed out. Try again."}
    except httpx.HTTPStatusError as e:
        return {"error": f"Weather API returned {e.response.status_code}"}

Three things worth noting. First: every error path returns a dict, not a raised exception. The model receives the dict and can decide whether to retry or report. Second: the return value is three fields stripped from a deeply nested response — the model does not need to parse nested JSON; you flatten it. Third: timeout handling is explicit. Eight seconds is generous for a geocoding call; do not let httpx default to infinity, or a slow upstream will stall your loop indefinitely.

Authentication — when you need it

For APIs that require a key:

API_KEY = os.environ["WEATHER_API_KEY"]   # read from environment, never from the tool call

headers = {"Authorization": f"Bearer {API_KEY}"}
response = httpx.get(url, headers=headers, timeout=8)

Never include the key in the tool definition, the tool description, or the tool_result. The model has no need to see it. If you log tool_result content for debugging, scrub credentials before logging. If the key appears in a tool description, it will appear in the conversation history returned by the API and may be stored in logs you do not control.

Rate limits

If the upstream API rate-limits (HTTP 429), return a structured error that includes a retry_after field. This gives the model information it can act on — it may decide to wait and retry, or to tell the user the data is temporarily unavailable.

if response.status_code == 429:
    retry_after = response.headers.get("Retry-After", "60")
    return {"error": "Rate limited", "retry_after_seconds": int(retry_after)}

In an agent loop (l7), the model may issue multiple weather calls in rapid succession. If the upstream API limits at ten requests per minute, build a simple call counter in the handler and sleep before the request when the counter approaches the ceiling.

If you get stuck

Model calls the tool with a city name in a non-English script: Add "City name in English" to the parameter description. The geocoding API may handle it anyway, but making the expectation explicit reduces malformed calls and gives you a clear reason when it fails.

Handler returns the full upstream JSON and the model hallucinates field names: You returned too much. The model treats every key in the result dict as a signal. Flatten to the three or four fields the model's downstream reasoning needs and nothing else.

httpx.ConnectError propagates as an uncaught exception: Catch it explicitly. ConnectError is not a subclass of HTTPStatusError. Add a bare except Exception as e: return {"error": str(e)} as a final catch after all specific exception handlers.

Cross-reference

Rate-limit handling within a multi-turn loop is covered in l7. The pattern of returning a structured error that the model reads and acts on is the same whether the error comes from an upstream API or from your own database handler in l4. The handler interface is identical — return a dict, never raise.

Deliverable

scripts/weather_tool.py — a complete script with the tool definition, the Open-Meteo handler, and an agent loop. Run it with a city that exists, a city that does not exist, and a misspelled city name. Confirm the model handles all three cases and produces a coherent response without crashing the loop.

Further reading

Try these prompts

  • Write the adapter contract for a weather tool that calls Open-Meteo — what three fields it returns to the model, what error dict it returns on timeout, and what it never exposes.
  • Write the Python handler block for an HTTP 429 rate-limit response that returns a structured error dict with a retry_after_seconds field.
  • Explain in two sentences why returning the full upstream JSON response to the model degrades reasoning quality, and what to do instead.

Back to contents

6. Filesystem tools and safety 18 min

Lesson notes

A bad SQL query returns wrong data. A bad filesystem write can destroy data, expose it, or execute it. The model does not know this. It will pass "../../etc/passwd" as a path if nothing stops it — not maliciously, but because it constructs paths from the inputs it received without understanding what lies outside the sandbox. Path validation belongs in the handler, not in the tool description, and it must run before any file operation.

Why filesystem tools are different

The blast radius of a filesystem tool scales with what the model can reach. Read-only tools with a confined root are low risk. Write tools with no path validation are not. A single os.path.join(SANDBOX, user_path) without realpath can be escaped with "../../../home/user/.ssh/authorized_keys". That call succeeds silently. You will not see it in the tool result.

The sandbox function

# scripts/fs_tools.py
import os, json, logging, anthropic
from datetime import datetime, timezone

SANDBOX = os.path.realpath(os.path.join(os.path.dirname(__file__), "../workspace"))
os.makedirs(SANDBOX, exist_ok=True)

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
logger = logging.getLogger("fs_tools")

def safe_path(user_path: str) -> str:
    """Resolve user_path relative to SANDBOX. Raise ValueError if it escapes."""
    resolved = os.path.realpath(os.path.join(SANDBOX, user_path))
    # The trailing sep prevents /var/app/workspace-evil from matching /var/app/workspace
    if not resolved.startswith(SANDBOX + os.sep) and resolved != SANDBOX:
        raise ValueError(f"Path escapes sandbox: {user_path!r} → {resolved!r}")
    return resolved

The os.path.realpath call resolves symlinks, .. components, and . — all of the ways a path can be constructed to escape a directory while looking innocent. Call it before any read or write. Never call os.path.join(SANDBOX, user_path) without realpath. The trailing os.sep in the startswith check is also non-optional — without it, a sandbox at /var/app/workspace would pass paths rooted at /var/app/workspace-evil.

The tool definitions

Expose read and write as separate tools. This gives you independent permission control, and it makes the model's intent explicit in the call record. If the model calls write_file, you know it intended to write — you do not have to infer it from a generic fs_op tool with an action parameter.

tools = [
    {
        "name": "read_file",
        "description": (
            "Read the contents of a file in the workspace. "
            "Paths are relative to the workspace root. "
            "Returns the file contents as a string, or an error if not found."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "path": {"type": "string", "description": "Relative path, e.g. 'notes/draft.txt'"}
            },
            "required": ["path"]
        }
    },
    {
        "name": "write_file",
        "description": (
            "Write text content to a file in the workspace. "
            "Creates the file if it does not exist, overwrites if it does. "
            "Paths are relative to the workspace root. Do not use absolute paths."
        ),
        "input_schema": {
            "type": "object",
            "properties": {
                "path": {"type": "string", "description": "Relative path, e.g. 'output/result.txt'"},
                "content": {"type": "string", "description": "Text to write"}
            },
            "required": ["path", "content"]
        }
    }
]

The handlers

def handle_read_file(path: str, tool_use_id: str = "") -> dict:
    try:
        resolved = safe_path(path)
        with open(resolved, "r", encoding="utf-8") as f:
            content = f.read()
        logger.info("READ  id=%s path=%s bytes=%d", tool_use_id, resolved, len(content))
        return {"content": content, "bytes": len(content)}
    except ValueError as e:
        return {"error": str(e)}
    except FileNotFoundError:
        return {"error": f"File not found: {path}"}
    except Exception as e:
        return {"error": f"Read failed: {e}"}

def handle_write_file(path: str, content: str, tool_use_id: str = "") -> dict:
    try:
        resolved = safe_path(path)
        os.makedirs(os.path.dirname(resolved), exist_ok=True)
        with open(resolved, "w", encoding="utf-8") as f:
            f.write(content)
        byte_count = len(content.encode("utf-8"))
        logger.info(
            "WRITE id=%s path=%s bytes=%d ts=%s",
            tool_use_id, resolved, byte_count,
            datetime.now(timezone.utc).isoformat()
        )
        return {"written": byte_count, "path": path}
    except ValueError as e:
        return {"error": str(e)}
    except Exception as e:
        return {"error": f"Write failed: {e}"}

Every write logs four things: the tool_use_id (so you can correlate to the conversation turn), the resolved path (the actual file that changed on disk), the byte count, and the UTC timestamp. If the model writes something unexpected, this log tells you exactly when and where. The tool_use_id links back to the specific assistant turn in the message history.

Connecting handlers to the dispatch table

client = anthropic.Anthropic()

HANDLERS = {
    "read_file": handle_read_file,
    "write_file": handle_write_file
}

def run(question: str):
    messages = [{"role": "user", "content": question}]
    while True:
        r = client.messages.create(
            model="claude-opus-4-5", max_tokens=1024,
            tools=tools, messages=messages
        )
        messages.append({"role": "assistant", "content": r.content})
        if r.stop_reason == "end_turn":
            return r.content[0].text
        results = []
        for block in r.content:
            if block.type == "tool_use":
                result = HANDLERS[block.name](**block.input, tool_use_id=block.id)
                results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": json.dumps(result)
                })
        messages.append({"role": "user", "content": results})

print(run("Write a short poem about Oslo to output/oslo.txt, then read it back and confirm."))

If you get stuck

safe_path raises for a path that should be valid: Check whether your SANDBOX string ends with a path separator before the check. The expression SANDBOX + os.sep is correct. If SANDBOX already ends with /, you get // which can produce false positives on some systems. Use os.path.realpath when setting SANDBOX and you will not see this.

Model constructs an absolute path instead of a relative one: The description says "Do not use absolute paths" — if the model still passes /workspace/output/file.txt, strip leading slashes in the handler before calling safe_path. user_path.lstrip("/") is sufficient for this case.

Model writes a file and the byte count is unexpectedly small: The model may have passed only a fragment of the content it intended to write. Check the call log — the tool_use_id links the write to the conversation turn, and you can inspect what content the model actually passed.

Cross-reference

The dispatch table pattern (HANDLERS dict) is formalized in l7 with error handling, an iteration cap, and a dispatch() wrapper function. The audit log format here — tool_use_id, resolved path, timestamp — is the basis for the CallRecord instrumentation in l8.

Deliverable

scripts/fs_tools.py — sandbox function, two handlers with logging, dispatch loop. Test it by asking the model to write a file, read it back, and append a second sentence. Confirm the log shows two writes with different byte counts and that the second write's byte count is larger than the first.

Try these prompts

  • Write the safe_path function in Python that resolves a user-supplied path relative to a SANDBOX directory and raises ValueError if it escapes. Explain why os.path.realpath is required.
  • Write the tool definitions for read_file and write_file as separate tools, including the description and input_schema for each. Explain why they should be separate rather than one fs_op tool.
  • Write the four fields that every file write should log — tool_use_id, resolved path, byte count, and UTC timestamp — and explain which field links the log entry back to the conversation turn.

Back to contents

AgentsFrom Tools to Agents

7. A small agent that uses three tools 25 min

Lesson notes

Everything in the previous lessons was one prompt, one tool call, one result. An agent is what happens when you remove the one-call constraint. The model calls tools repeatedly, in whatever order it decides, until it has enough information to answer. Your code does not decide when to stop. The model does. Your code's only job is to keep running until the model says it is done — or until you hit a safety limit you set yourself.

What an agent actually is

An agent is a while loop. It runs until the model returns stop_reason: "end_turn" with no tool_use blocks in the response. Each iteration: send the message history, receive the response, execute any tool calls, append results to the history, call the API again. The tools are the same handlers you built in earlier lessons. What changes is the driver that orchestrates them.

Two failure modes end an agent loop badly: the loop never terminates because the model keeps calling tools (unbounded loop), or the loop crashes because the dispatch table raises on an unknown tool name (uncaught KeyError). Both are preventable with two additions: a maximum iteration cap and a safe dispatch function.

The full agent

# scripts/research_agent.py
import anthropic, json, os, httpx

client = anthropic.Anthropic()
SANDBOX = os.path.realpath(os.path.join(os.path.dirname(__file__), "../workspace"))
os.makedirs(SANDBOX, exist_ok=True)

# == Tool definitions ==
tools = [
    {
        "name": "search_web",
        "description": (
            "Search the web and return a list of relevant results with titles and snippets. "
            "Use to find factual information, recent events, or source material. "
            "Returns up to 5 results."
        ),
        "input_schema": {
            "type": "object",
            "properties": {"query": {"type": "string"}},
            "required": ["query"]
        }
    },
    {
        "name": "read_file",
        "description": "Read a file from the workspace. Path is relative to workspace root.",
        "input_schema": {
            "type": "object",
            "properties": {"path": {"type": "string"}},
            "required": ["path"]
        }
    },
    {
        "name": "write_file",
        "description": "Write text to a file in the workspace. Creates or overwrites.",
        "input_schema": {
            "type": "object",
            "properties": {
                "path": {"type": "string"},
                "content": {"type": "string"}
            },
            "required": ["path", "content"]
        }
    }
]

# == Handlers ==
def handle_search_web(query: str) -> dict:
    # DuckDuckGo Instant Answer API — free, no key required, returns limited results
    try:
        r = httpx.get(
            "https://api.duckduckgo.com/",
            params={"q": query, "format": "json", "no_redirect": 1},
            timeout=8
        )
        r.raise_for_status()
        data = r.json()
        results = []
        for topic in (data.get("RelatedTopics") or [])[:5]:
            if "Text" in topic:
                results.append({"title": topic.get("Text", "")[:100],
                                "url": topic.get("FirstURL", "")})
        if not results and data.get("AbstractText"):
            results.append({"title": data["AbstractText"][:200],
                            "url": data.get("AbstractURL", "")})
        return {"results": results, "count": len(results)}
    except Exception as e:
        return {"error": str(e)}

def _safe_path(p: str) -> str:
    resolved = os.path.realpath(os.path.join(SANDBOX, p))
    if not resolved.startswith(SANDBOX + os.sep) and resolved != SANDBOX:
        raise ValueError(f"Path escapes sandbox: {p!r}")
    return resolved

def handle_read_file(path: str) -> dict:
    try:
        with open(_safe_path(path), "r", encoding="utf-8") as f:
            return {"content": f.read()}
    except FileNotFoundError:
        return {"error": f"Not found: {path}"}
    except ValueError as e:
        return {"error": str(e)}

def handle_write_file(path: str, content: str) -> dict:
    try:
        resolved = _safe_path(path)
        os.makedirs(os.path.dirname(resolved), exist_ok=True)
        with open(resolved, "w", encoding="utf-8") as f:
            f.write(content)
        return {"written": len(content.encode()), "path": path}
    except ValueError as e:
        return {"error": str(e)}

# == Dispatch table ==
HANDLERS = {
    "search_web": handle_search_web,
    "read_file": handle_read_file,
    "write_file": handle_write_file
}

def dispatch(name: str, input_dict: dict) -> dict:
    handler = HANDLERS.get(name)
    if handler is None:
        return {"error": f"Unknown tool: {name!r}"}
    try:
        return handler(**input_dict)
    except TypeError as e:
        return {"error": f"Bad arguments for {name}: {e}"}

# == Agent loop ==
MAX_ITERATIONS = 12

def run_agent(task: str) -> str:
    messages = [{"role": "user", "content": task}]

    for iteration in range(MAX_ITERATIONS):
        r = client.messages.create(
            model="claude-opus-4-5",
            max_tokens=1024,
            tools=tools,
            messages=messages
        )
        messages.append({"role": "assistant", "content": r.content})

        if r.stop_reason == "end_turn":
            text_blocks = [b for b in r.content if hasattr(b, "text")]
            return text_blocks[0].text if text_blocks else "(no text response)"

        if r.stop_reason != "tool_use":
            return f"Unexpected stop_reason: {r.stop_reason}"

        tool_results = []
        for block in r.content:
            if block.type == "tool_use":
                print(f"[iter {iteration+1}] {block.name}({json.dumps(block.input)[:120]})")
                result = dispatch(block.name, block.input)
                print(f"           → {json.dumps(result)[:120]}")
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": json.dumps(result)
                })
        messages.append({"role": "user", "content": tool_results})

    return f"Agent hit iteration cap ({MAX_ITERATIONS}). Last response was tool_use."

if __name__ == "__main__":
    task = (
        "Search for three facts about the Aurora Borealis. "
        "Write a 150-word summary of what you find to workspace/aurora_summary.txt. "
        "Then read the file back and confirm the byte count."
    )
    print(run_agent(task))

What the loop does not do

The loop does not decide which tool to call, does not retry on its own, and does not stop early because the task looks finished. Those are the model's decisions. The loop's job is to execute what the model requests and keep the message history intact. The only thing the loop decides unilaterally is when to give up — that is the iteration cap.

The dispatch table pattern

HANDLERS maps string names to callables. dispatch() wraps the lookup with two error paths: unknown tool name (.get() returns None), and bad arguments (TypeError when **input_dict does not match the function signature). Both return structured errors, not raised exceptions. The model reads the error and can adapt — passing a corrected argument on the next call.

When you add a new tool, you add one entry to HANDLERS. The loop does not change. The dispatch function does not change. This is the correct separation: tools are data, the loop is infrastructure.

If you get stuck

Agent hits the iteration cap every run: The model is stuck in a pattern — often it keeps searching because the search results are thin. Add a constraint to the system prompt or task description: "If search results are insufficient after two attempts, write a summary based on what you have found and stop." One sentence breaks the loop reliably.

TypeError in dispatch because the model passed an unexpected keyword: The model extrapolated a parameter that is not in the schema. Add "additionalProperties": false to the input_schema object. This causes a schema validation error before the call reaches your handler, and the model will correct the argument on the next attempt.

Tool results are too long and the context window fills up: Truncate long results in the handler. search_web already caps at 5 results with 100-character snippets. For read_file, add a max byte count (content[:8000]) and append a truncation notice to the returned string.

Model calls write_file before searching: The task description did not make the dependency explicit. Rewrite the task: "Search first. Write only after you have found at least two facts." The model follows explicit sequencing instructions reliably.

Cross-reference

The safe_path and write handler here are simplified from l6 — the audit logging is stripped. The full instrumented version of this agent, with CallRecord capture and a pytest eval suite, is in l8.

Deliverable

scripts/research_agent.py — the complete script above, runnable with python scripts/research_agent.py. The agent must complete the Aurora Borealis task: search, write, read back, confirm byte count. The printed output must show at least two [iter N] lines (one search call and one write call at minimum) and end with a non-empty summary paragraph.

Try these prompts

  • Write the dispatch function that looks up a tool name in a HANDLERS dict and returns a structured error dict for both an unknown tool name and a TypeError from bad arguments.
  • Explain what the agent loop does not decide — which tool to call, when to retry, when to stop early — and why those are the model's decisions, not the loop's.
  • Write the one-sentence task constraint to add when an agent hits the iteration cap because it keeps searching with thin results instead of writing its summary.

Back to contents

8. Evaluation and observability 30 min

Lesson notes

The agent you built in l7 worked when you ran it. That is not the same as knowing it will work next week, after a tool description edit, or after a model version bump. Multi-tool loops are sensitive to small changes in descriptions, schemas, and model behavior. A regression can be silent — the loop completes, the answer looks plausible, and the tool call sequence is wrong. A test suite that checks the call record catches this. The final text response does not.

Why eval before scale

An agent that works once in a demo is not an agent you can ship. Three test cases with a concrete rubric — checked against the tool call sequence, not the final text — are enough to catch the regressions that actually occur in practice: a tool stops being called, a tool is called in the wrong order, or the agent fails to terminate cleanly on an error condition.

What to assert on

The final text response of an agent is nearly impossible to assert correctly. It is long, paraphrastic, and different on every run. The tool call sequence is concrete, ordered, and repeatable. Assert on the call record — which tools fired, in which order, how many times, and whether the agent terminated cleanly.

Instrumenting the agent

Take the agent from l7 and add a call log. The log accumulates one entry per tool call: name, input, result, and iteration number. After the loop ends, return both the final text and the log.

# scripts/research_agent_instrumented.py
# Copy research_agent.py and replace run_agent with run_agent_instrumented below.
from dataclasses import dataclass

@dataclass
class CallRecord:
    iteration: int
    tool_name: str
    input: dict
    result: dict

def run_agent_instrumented(task: str) -> tuple[str, list[CallRecord]]:
    messages = [{"role": "user", "content": task}]
    call_log: list[CallRecord] = []

    for iteration in range(MAX_ITERATIONS):
        r = client.messages.create(
            model="claude-opus-4-5",
            max_tokens=1024,
            tools=tools,
            messages=messages
        )
        messages.append({"role": "assistant", "content": r.content})

        if r.stop_reason == "end_turn":
            text = next((b.text for b in r.content if hasattr(b, "text")), "")
            return text, call_log

        tool_results = []
        for block in r.content:
            if block.type == "tool_use":
                result = dispatch(block.name, block.input)
                call_log.append(CallRecord(
                    iteration=iteration,
                    tool_name=block.name,
                    input=block.input,
                    result=result
                ))
                tool_results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": json.dumps(result)
                })
        messages.append({"role": "user", "content": tool_results})

    return f"Hit iteration cap ({MAX_ITERATIONS})", call_log

The CallRecord dataclass is deliberately simple. It captures what the model requested (tool_name, input) and what your handler returned (result). The iteration field lets you reconstruct the exact sequence after the run. You do not need anything richer than this to write meaningful assertions.

The test file

# tests/test_agent_eval.py
import pytest
from scripts.research_agent_instrumented import run_agent_instrumented, MAX_ITERATIONS

# ─── Fixtures ─────────────────────────────────────────────────────────

@pytest.fixture(scope="module")
def happy_path():
    """Case 1: clean success — search, write, read back."""
    return run_agent_instrumented(
        "Search for two facts about the Moon. Write a 100-word summary to "
        "workspace/moon.txt. Read it back and confirm the byte count."
    )

@pytest.fixture(scope="module")
def missing_file():
    """Case 2: edge condition — model tries to read a nonexistent file."""
    return run_agent_instrumented(
        "Read the file workspace/does_not_exist.txt and summarize its contents."
    )

@pytest.fixture(scope="module")
def no_write_needed():
    """Case 3: graceful stop — task requires only search."""
    return run_agent_instrumented(
        "Search for the capital city of Norway and tell me what you find. "
        "Do not write any files."
    )

# ─── Case 1: happy path ────────────────────────────────────────────

def test_happy_path_calls_search(happy_path):
    _, log = happy_path
    assert any(r.tool_name == "search_web" for r in log), \
        "Expected at least one search_web call"

def test_happy_path_calls_write(happy_path):
    _, log = happy_path
    assert any(r.tool_name == "write_file" for r in log), \
        "Expected write_file to be called"

def test_happy_path_search_before_write(happy_path):
    _, log = happy_path
    names = [r.tool_name for r in log]
    search_idx = next(i for i, n in enumerate(names) if n == "search_web")
    write_idx = next(i for i, n in enumerate(names) if n == "write_file")
    assert search_idx < write_idx, "search_web must precede write_file"

def test_happy_path_terminates_cleanly(happy_path):
    text, log = happy_path
    assert not text.startswith("Hit iteration cap"), \
        "Agent must not exhaust the iteration cap on a simple task"

# ─── Case 2: edge condition ────────────────────────────────────────

def test_missing_file_returns_error_to_model(missing_file):
    _, log = missing_file
    read_results = [r.result for r in log if r.tool_name == "read_file"]
    assert read_results, "Expected at least one read_file call"
    assert any("error" in r for r in read_results), \
        "Expected read_file to return an error dict for a missing file"

def test_missing_file_agent_terminates(missing_file):
    text, _ = missing_file
    assert not text.startswith("Hit iteration cap"), \
        "Agent should stop after receiving a file-not-found error"

# ─── Case 3: no write needed ───────────────────────────────────────

def test_no_write_uses_search(no_write_needed):
    _, log = no_write_needed
    assert any(r.tool_name == "search_web" for r in log)

def test_no_write_does_not_write(no_write_needed):
    _, log = no_write_needed
    assert not any(r.tool_name == "write_file" for r in log), \
        "write_file must not be called when the task does not require it"

def test_no_write_terminates_cleanly(no_write_needed):
    text, _ = no_write_needed
    assert not text.startswith("Hit iteration cap")

Run:

# from project root
PYTHONPATH=. pytest tests/test_agent_eval.py -v
# →  9 passed in ~45s (three full agent loops, real API calls)

Why these three cases cover the surface that matters

Case 1 confirms the agent completes the primary task in the correct order. If a tool description changes and write_file is no longer called, this case catches it. If the model starts writing before searching — a common regression when description language shifts — test_happy_path_search_before_write catches it.

Case 2 confirms the agent handles a structured error from the handler and stops rather than looping. If handle_read_file starts raising exceptions instead of returning error dicts, the agent will crash — this case will fail at a different point and tell you where.

Case 3 confirms the agent does not call tools it was not asked to use. A model update that makes the agent more "helpful" by writing files unprompted will fail this case.

Making the suite stable in CI

The assertions check presence and order, not exact call counts — this tolerates run-to-run variation in how many search calls the model issues. The ordering assertion in case 1 is the most likely to flap. If it does, add "Search first. Write only after you have found facts." to the task description for that fixture.

Use scope="module" on fixtures so each agent runs once per test session, not once per test function. Three agent runs cost roughly 5,000–10,000 input tokens each at current pricing. Running nine fixtures independently would cost three times as much for the same coverage.

If you get stuck

ModuleNotFoundError: scripts.research_agent_instrumented: Add an empty scripts/__init__.py or run with PYTHONPATH=. pytest.

test_happy_path_search_before_write fails intermittently: The model sometimes writes a placeholder before searching. Add the ordering constraint to the task prompt in the fixture: "Search first. Write only after you have found at least two facts."

test_missing_file_agent_terminates fails — agent hits the cap: The handler's error dict lacks the filename. The model cannot tell which file was missing and keeps retrying with variations. Fix the handler: {"error": f"File not found: {path}"}. The model reads the filename, recognizes the file does not exist, and stops.

Nine tests pass locally but fail in CI: CI does not have ANTHROPIC_API_KEY. Add it to your CI secrets and export it in the workflow environment. Do not hardcode it.

Cross-reference

The CallRecord dataclass extends the write-log pattern from l6 — the same information (tool name, input, result) that the handler logs to stderr is now captured in a structured list the test suite can inspect. The agent being tested is the one built in l7. This lesson adds only the instrumentation harness — no new tools, no new handlers.

Deliverable

tests/test_agent_eval.py with all nine assertions passing. PYTHONPATH=. pytest tests/test_agent_eval.py exits zero. No mocks — real API calls, real handlers, real DuckDuckGo search results. The test run takes roughly 45 seconds for three agent loops; budget accordingly in CI.

Further reading

  • "The Validator With Teeth" — the upgrade from a test that flags failures to one that refuses to ship them, and why agents need that distinction more than single-prompt code does.

Try these prompts

  • Write the CallRecord dataclass for an instrumented agent loop. Include iteration, tool_name, input, and result fields. Explain why iteration is load-bearing for assertions.
  • Write a pytest assertion that checks search_web was called before write_file in a recorded call log. Show how to get the index of each tool name from the log.
  • Explain why pytest fixtures should use scope="module" for agent eval tests, and what the cost difference is between module scope and the default function scope.

Back to contents

9. Tools built for agents — the Hyperframes pattern 14 min

Lesson notes

Eight lessons asked: how do you build a tool a model can use? This one flips it. What does a tool look like when an agent authors with it, not just calls it?

Run init and read what shows up

npx hyperframes init
# → Created hyperframes.json
# → Created package.json
# → Created index.html
# → Created meta.json
# → Created AGENTS.md
# → Created CLAUDE.md

Six files. Two of them — AGENTS.md and CLAUDE.md — exist for one reason: so the agent never has to leave the repo to learn how to use the tool.

The cognitive surface

Hand someone a power saw without the safety label and you've handed them a different tool. The blade is the same. What changed: whether the warning sits next to the cut. The agent has the same constraint — it sees what's in the working directory. A docs site lives somewhere it doesn't.

CLAUDE.md lists thirteen slash-skills the project owns: /hyperframes, /gsap, /lottie, /three, /website-to-hyperframes. The docs aren't a site you read. They're a tool you invoke — same packaging as Anthropic's superpowers.

The edit surface

One file. index.html. Each .clip div carries data-start, data-duration, data-track-index. Animations live in GSAP timelines registered on window.__timelines["main"], created paused.

<!-- index.html -->
<div class="clip" data-start="0" data-duration="2" data-track-index="0">
  <h1 class="title">Hello</h1>
</div>

<script>
  const tl = gsap.timeline({ paused: true });
  tl.from(".title", { y: 100, opacity: 0, duration: 0.8 });
  window.__timelines["main"] = tl;     // load-bearing line
</script>

That last line is the whole lesson on this mechanic. The renderer is a headless browser that must seek to timestamps to capture frames. A running timeline doesn't expose seek; only a paused timeline registered on window.__timelines does. Forget the registration and the browser preview still plays — but the render freezes on frame 0. Same shape as the os.path.realpath boundary in l6: the failure is invisible until something outside the local environment exercises it.

The lint pass

npm run check
# → composition validated
# → 0 console errors
# → WCAG-AA: pass

Same discipline as the pytest suite in l8: fail loudly before shipping, with concrete checks, not vibes. The browser hid the console error; the headless run surfaces it.

The catalog

npx hyperframes add data-chart
# → compositions/data-chart.html (458 lines)
# → snippet copied to clipboard

shadcn-style. The file is in your repo. You own it, you edit it, no runtime import to a black-box component.

The four-signal test

When evaluating any tool that claims agent support:

  1. Does it ship an AGENTS.md or CLAUDE.md at init?
  2. Does it provide slash-skills or equivalent invocable docs?
  3. Is the edit surface one file, not a component tree?
  4. Does a lint pass catch headless-render failures before they ship?

Three of four missing means the tool was built for humans and retrofitted with an MCP server. The retrofitted version still works — but the agent spends tokens reading a website to learn what AGENTS.md could have told it for free.

Hyperframes vs. Remotion

Code-first agents → Hyperframes. The agent writes HTML it can read back; the renderer is local.

React-first humans → Remotion. Type-safe composition trees, prop-based reuse, JSX a person edits.

Same shape, different authors. Pick by who is writing.

If you get stuck

Composition animates in the browser preview but freezes on frame 0 in the render: the GSAP timeline is not registered on window.__timelines["<id>"]. Add the assignment line and confirm the timeline was created with paused: true.

Render passes locally but npm run check reports a console error you didn't see: the browser swallowed it. Open the page in devtools, reproduce, and fix the underlying error — the lint pass is reading the same buffer.

Agent imports a block from npm instead of running add: it pulled in a registry your project does not own. Delete the import, run npx hyperframes add <name>, edit the file that lands in compositions/.

Cross-reference

l7 and l8 built the consumer posture — your code calls tools the model authors. l9 is the author posture — you evaluate tools built for the model to author with. Same vocabulary, opposite end of the contract.

Deliverable

A 6-second composition rendered locally with npx hyperframes to a 200-300KB h264 file in out/. Plus notes/tool-evaluation.md applying the four-signal test to one other tool you use that claims agent support — name which signals are present, which are missing, and what the next agent author must read a website to learn.

Further reading

  • "What Agents Actually Need From Your API" — same boundary problem one layer down. The response shape that worked for human developers degrades reasoning when the caller is a model. Tooling has the same gradient.

Try these prompts

  • Run npx hyperframes init in a scratch directory, then describe what each of the six scaffold files is for and which two exist solely so an agent can read them before writing any code.
  • Apply the four-signal agent-first test (AGENTS.md ships at init, slash-skills as docs, single-file edit surface, lint pass catches headless-render failures) to one tool you currently use that claims agent support; name which signals are present and which are missing.
  • Explain in two sentences why a GSAP timeline that animates correctly in the browser preview can freeze on frame 0 in the headless render, and what one line of code prevents it.

Back to contents