Long-running agents and recovery

A long-running agent fails at the worst moment. Mid-migration. Halfway through a 200-file refactor. At step 47 of 50. The question is not whether this will happen — it will — but whether the failure leaves the system in a resumable state or forces a restart from zero.

The answer is checkpointing. Not as an afterthought, and not as a feature you add after the first crash. As a design constraint you enforce before the first run.

The checkpoint schema

{
  "task_id": "refactor-imports-2026-05-05",
  "schema_version": 1,
  "current_step": 23,
  "total_steps": 200,
  "next_input": {
    "file": "/src/components/Button.tsx",
    "target_import": "@/utils/cn",
    "replacement": "~/lib/cn"
  },
  "completed": [
    "/src/components/Alert.tsx",
    "/src/components/Badge.tsx"
  ],
  "errors": []
}

The checkpoint is written before the next step begins, not after. If the agent completes a step and crashes before writing the checkpoint, the step is lost but the file is already changed — you have a partial state with no record of where you are. If the agent writes the checkpoint and then crashes before the next step, you know exactly where to resume, and the incomplete step was not started.

This is the only rule about checkpointing that matters: write before the destructive operation, not after.

The recovery loop

# scripts/resume-refactor.py
import json, pathlib

CHECKPOINT = pathlib.Path("checkpoint.json")
SCHEMA_VERSION = 1

def load_checkpoint():
    if CHECKPOINT.exists():
        state = json.loads(CHECKPOINT.read_text())
        if state.get("schema_version") != SCHEMA_VERSION:
            print("Checkpoint schema mismatch — starting from zero")
            return {"current_step": 0, "completed": [], "errors": []}
        return state
    return {"current_step": 0, "completed": [], "errors": []}

def save_checkpoint(state):
    state["schema_version"] = SCHEMA_VERSION
    CHECKPOINT.write_text(json.dumps(state, indent=2))

state = load_checkpoint()
files = get_file_list()  # full list, same order, every run

for i, file in enumerate(files):
    if file in state["completed"]:
        continue  # already done, skip
    state["current_step"] = i
    state["next_input"] = {"file": file}
    save_checkpoint(state)        # write BEFORE the operation
    try:
        process_file(file)
        state["completed"].append(file)
        save_checkpoint(state)    # write after success
    except Exception as e:
        state["errors"].append({"file": file, "error": str(e)})
        save_checkpoint(state)
        break                     # stop on first error, inspect, resume

The loop is intentionally simple. The complexity is in the discipline: checkpoint before the destructive operation, every time, without exception.

Idempotent operations

Every operation in a long-running agent should be safe to repeat. If an API call creates a record, add a lookup-or-create pattern. If a file write applies a patch, check whether the patch is already applied before writing. The test: if you ran the same step twice in sequence, would the second run produce a different result or an error?

# Not idempotent — creates a duplicate on second run
resp = api.create_record(data)

# Idempotent — returns existing record if already created
existing = api.find_record(data["external_id"])
resp = existing if existing else api.create_record(data)

Audit every operation for idempotency before the first run. The time to discover a non-idempotent operation is in code review, not at 2 AM after a crash at step 140 that left 40 duplicate records in the database.

If you get stuck

The agent resumed but re-processed files already in completed. The write order is wrong — the completed entry was added to the in-memory state after a crash that prevented the checkpoint write. Fix the write order: add to completed, write the checkpoint, then advance to the next file. The entry must be on disk before you move on.

The checkpoint file is valid JSON but the schema_version does not match. You changed the checkpoint format — added a field, renamed one — and there is a stale checkpoint from a previous run. The version check in load_checkpoint above handles this: mismatched version restarts from zero. This is intentional. A checkpoint from a different schema cannot be safely resumed — resuming it silently is worse than restarting cleanly.

The recovery loop runs but operations are not idempotent, so resuming creates duplicates or overwrites changes. There is no shortcut here. You must audit every operation before the first run. If you discover a non-idempotent operation after a crash, fix it, clean up the partial state manually, and restart from the last clean checkpoint. Document what you cleaned up — the next crash will happen, and you will want the record.

Cross-reference

l7 covers the "almost done" loop in PR-shipping agents — the hard stop condition in the brief is the brief-level fix, and this lesson's checkpointing is the infrastructure-level fix. l3's worktree pattern becomes essential for long-running code agents: a crash leaves the worktree in a partial state, so the checkpoint must record the worktree path and branch so recovery can verify the state before resuming.

Deliverable

Write a checkpoint schema for one long-running task you run or plan to run. The schema must include: a schema_version field, a step index, a completed list, an errors list, and the next input. Then write the idempotency check for the most dangerous operation in the pipeline — the one that creates data or modifies an external resource. These two artifacts together are the pre-flight for any autonomous agent you ship.

Further reading

• "The Office That Worked While He Was Out" — what an autonomous Claude Code daemon actually produced over weeks of unattended runs, and the three architectural layers that kept it resumable.
• "The Office Held a Vote" — a worked example of a long-running pipeline crossing dream → debate → plan → build → QA → ship without a human in the loop, and what failed mid-run.