Versioning prompts in git

A prompt that lives in a configuration UI will be edited without review, without history, and without any mechanism to connect a change to an outcome. Someone will edit it on a Tuesday afternoon and you will find out on Wednesday when a chart moves. A prompt that lives in git gets a diff, a reviewer, and a timestamp.

The argument for storing prompts in git is the same argument for storing any other piece of logic that governs production behavior in git. A prompt is code. It has inputs, outputs, and failure modes. It should be reviewed before it ships and auditable after it breaks.

The prompt-as-code directory structure

Store each production prompt as a plain text file in your repository. One file per prompt. Not an overwrite — a new file with a version suffix. The old version stays in the repository so you can read it, compare it, and revert to it.

prompts/
  ticket-triage-v1.txt
  ticket-triage-v2.txt       ← current version in production
  ticket-summary-v1.txt
  judge-ticket-triage-v1.txt

evals/
  cases/
    ticket-triage.json
    ticket-triage-smoke.json
  run-ticket-triage.py

schemas/
  claims-v1.json
  claims-v2.json             ← current version, bumped when field names changed

The active version is referenced by an environment variable or a config key, not by a hardcoded filename:

# .env / config store
TICKET_TRIAGE_PROMPT_VERSION=v2
TICKET_SUMMARY_PROMPT_VERSION=v1

Your application loads the prompt at startup by reading the file named ticket-triage-${TICKET_TRIAGE_PROMPT_VERSION}.txt. Rolling forward or back is a one-character change to the config value, not a file edit. No deployment required if your application reads the config at startup.

Why code review matters for prompts

A diff on a prompt file is readable. A reviewer can look at it and ask: what failure mode does this addition address? What edge case might it introduce? What changed about the output schema that might affect downstream consumers?

That conversation happens before the change ships — not after the incident. The pull request is the record of the intent behind the change. Six months later, when the prompt behaves unexpectedly, you can read the PR and understand why the line was added. Without that history, you are guessing.

Here is a prompt diff that a reviewer should flag:

# prompts/ticket-triage-v2.txt
- Classify the ticket as exactly one of: billing, access, bug.
+ Classify the ticket as one of: billing, access, bug, or other.

The addition of "other" is not self-evidently wrong. But a reviewer who knows that step 2 in the chain checks if category == "billing" will immediately ask: what happens when this returns "other"? That question catches a downstream bug before it ships. Without a PR, no one asks the question.

The eval-on-PR gate in CI

# .github/workflows/eval.yml
name: Eval gate
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install anthropic
      - name: Run ticket-triage eval
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: python evals/run-ticket-triage.py --threshold 0.8

The PR cannot merge if the eval score drops below threshold. This is not a heavy lift — the eval runner from l5 is the entire test suite. The GitHub Actions step calls it, passes the threshold, and the runner exits with code 1 if the score falls below it, which GitHub Actions interprets as a failed step.

The cost is a few API calls per PR — roughly ten calls to the prompt under test, ten calls to the judge. At current API pricing that is negligible for the assurance that the change does not regress the cases you have already identified as important.

Schema versioning as a dependency declaration

When a prompt change produces a different output shape, bump the schema version file and update the prompt version config key in the same commit. The commit diff shows both: the prompt changed, and the schema changed with it.

A downstream step that pins to claims-v1.json will produce a visible mismatch in code review — the reviewer sees that the downstream step was not updated to consume claims-v2.json. The schema version is how you make dependencies between steps visible at review time rather than discoverable at incident time.

Cross-reference

api-fundamentals covers the environment variable and authentication patterns needed for CI contexts, including how to store ANTHROPIC_API_KEY as a GitHub Actions secret. l8 of this module is the direct downstream dependency of this lesson — the rollback procedure only works if versioned prompts exist in git and the config layer knows which version is live.

If you get stuck

The most common obstacle is the organization's existing tooling — prompts are already stored in a database or a CMS that the team does not want to abandon. The minimum viable version of prompt-as-code does not require moving everything. Store only the prompts that have CI coverage in git. Reference them by version from the existing tooling. Migrate the rest as evals are written for them. A partial migration is not failure — it is progress.

Deliverable

Design the directory structure for prompts, evals, and schemas for one of your projects. Write the environment variable naming convention you would use for version references. Draft the two-line CI step that would run the eval and block the merge if the pass rate falls below your threshold. Describe in one sentence what a reviewer should check in a prompt diff before approving it.