Rolling back a bad prompt

The on-call engineer at 2 a.m. does not want to read your prompt engineering notes. They want a procedure they can execute without understanding the prompt's intent, without making judgment calls, and without spending more than five minutes before service is restored.

If your rollback procedure requires understanding the prompt, it will take longer than five minutes. If it requires a judgment call — "is this the right version to revert to?" — it will take longer than five minutes. Write the procedure for someone who has never read the prompt and is operating under pressure. Test it before you need it.

What the playbook must contain

A rollback playbook is not documentation. It is a sequence of mechanical steps with no branches that require judgment. Every element below is required:

1. How to identify which prompt version is currently live
2. How to find the last known-good version
3. The exact commands to change the version and redeploy
4. How to verify the rollback took effect
5. Who to notify and what to log

Each element must be specific enough that the person executing it does not have to make a decision.

A complete example

# Rollback Playbook — ticket-triage prompt
Last updated: 2026-05-01 / Owner: @sethshoultes

## 1. Identify the current live version
Check the config store:
  curl https://api.yourapp.com/debug/config | jq .prompt_versions
→ Look for: TICKET_TRIAGE_PROMPT_VERSION
→ Record the current value here before changing anything: ____________

## 2. Find the last known-good version
In the repo:
  git log --oneline prompts/ticket-triage-*.txt
→ The commit immediately before the one that introduced the current version
  is the previous version. Record it: ____________
→ If unclear, check the deploy log for the timestamp of the incident start
  and find the commit active at that time.

## 3. Revert the version

Option A — environment variable only (fastest, no code change):
  1. Open the config store (Vercel / Railway / your platform)
  2. Set TICKET_TRIAGE_PROMPT_VERSION to the known-good version
  3. Trigger a redeploy if the app reads config at startup (not at request time)
  → Expected time to effect: 60–90 seconds

Option B — git revert (if prompt file and config changed in the same commit):
  1. git revert <commit-sha> --no-edit
  2. git push origin main
  3. CI deploys automatically
  → Expected time to effect: 3–5 minutes (CI run)

Use Option A first. Use Option B only if the config store is not accessible or
the prompt file itself needs to be reverted (not just the config value).

## 4. Verify the rollback took effect

Run the smoke eval:
  ANTHROPIC_API_KEY=<key from 1Password: "eval-key-prod"> \
  python evals/run-ticket-triage.py \
    --cases evals/cases/ticket-triage-smoke.json \
    --threshold 0.9

→ Expected: 9/10 or better
→ If below threshold: do not stop here. Escalate to @sethshoultes.
  Do not attempt further changes without the prompt owner present.

## 5. Notify and log
Post in #incidents:
  "ticket-triage prompt rolled back from <bad-version> to <known-good-version>
   at <UTC time>. Smoke eval: <score>."

Create a follow-up issue:
  Title: "Post-mortem: why did <bad-version> pass the eval gate?"
  Assign to: prompt owner
  Due: within 48 hours of incident close

Before closing the incident:
  Add the input that triggered the failure to evals/cases/ticket-triage.json.
  If you do not have the exact input, record the best approximation you have.
  The case can be refined later — it must exist before the incident closes.

The smoke eval: a smaller, faster subset

The full eval runs ten or more cases and may take a few minutes. A smoke eval runs three to five — the highest-risk cases, the ones representing your highest-volume input class. It exists specifically for the rollback verification step, where you need a fast signal under pressure.

// evals/cases/ticket-triage-smoke.json
[
  {
    "input": {"ticket_body": "I was charged twice and need a refund"},
    "expected": {"category": "billing"},
    "edge_case_flag": "highest-volume class: billing complaint"
  },
  {
    "input": {"ticket_body": "I can't log in after resetting my password"},
    "expected": {"category": "access"},
    "edge_case_flag": "second-highest-volume class: access failure"
  },
  {
    "input": {"ticket_body": "the export button crashes the app every time"},
    "expected": {"category": "bug"},
    "edge_case_flag": "bug classification: deterministic action + crash"
  }
]

The smoke eval is not a replacement for the full eval. It is the verification tool you reach for during an incident. Run the full eval during the post-mortem to understand the scope of the regression.

The five-minute constraint is not a figure of speech

Every element of the playbook that requires a judgment call will cost you two to five minutes under pressure. Name the judgment call and eliminate it before it shows up at 2 a.m.

• "Find the appropriate version" is a judgment call. Replace it with: look at git log --oneline prompts/ticket-triage-*.txt and take the commit immediately before the current HEAD on that file.
• "Verify the fix worked" is a judgment call. Replace it with: run the smoke eval command above and verify the score is 9/10 or better.
• "Notify the appropriate people" is a judgment call. Replace it with: post in #incidents and assign the follow-up to the prompt owner.

Run the playbook as a dry run before an incident. Have someone who did not write it execute it against a staging environment. Time them. Anywhere they pause is a gap in the procedure. Any question they ask is a line you need to add.

Why rollback procedures take 20 minutes instead of 5

The incidents that run long share a pattern: the version identifier is not in a single canonical place, or the person executing the rollback is not sure whether the version they reverted to is actually earlier than the broken version, or the verification step requires judgment about whether the output "looks right." Each ambiguity costs minutes. The playbook eliminates all of them in advance — not during the incident.

Cross-reference

The smoke eval in step 4 uses the eval runner from l5 and the prompt versioning structure from l7. The rollback procedure is the downstream dependency of every practice in this module: without versioned prompts in git, there is nothing stable to revert to. Without the config-layer version reference, rollback requires a code deploy. Without an eval, there is no mechanical way to verify the revert took effect. This lesson is where the module's practices converge into a single operational procedure.

If you get stuck

If you have never deployed a prompt change, invent a plausible scenario: a triage prompt with three versions in git, a config store that references the active version, and an incident where the current version started misclassifying a high-volume input class. Write the playbook for that scenario. The exercise of writing it will surface every ambiguity a real on-call engineer would encounter — every place where you wrote "find the right version" instead of "run this command."

Deliverable

Write the rollback playbook for one production prompt you own or would own. It must contain all five elements. It must be executable by someone who has never read the prompt. Time yourself reading it aloud and executing each step — if any step takes more than 90 seconds because of ambiguity, that step contains a judgment call that must be eliminated. The playbook is done when someone else can execute it in under five minutes on the first try.

Further reading

• "The Deploy Told You Nothing" — when a rollback completes "successfully" but the prompt is still broken, and what the playbook needs to verify beyond the version pointer.