Rate limits and the cost of context

Rate limits and token costs are not independent problems. Understanding one without the other will leave you with surprises in production.

Your tier's limits

Anthropic assigns accounts to tiers based on usage history and prepayment amount. Each tier has two limits: tokens per minute (TPM) and requests per minute (RPM). You can see your current tier at console.anthropic.com under Settings > Limits.

The limits reset every 60 seconds. A single large request can exhaust most of your TPM quota for that window. If your application sends bursts — ten requests simultaneously at startup — you can hit the RPM limit even on a generous tier.

Input tokens are the cost that compounds

Every token in the messages array costs money and counts against your TPM limit — not just the new user message, but every prior turn you include. This is where most production cost overruns originate.

Run this and watch:

# scripts/measure_context.py
import anthropic

client = anthropic.Anthropic()
history = []

topics = [
    "What is an HTTP header?",
    "What is JSON?",
    "What is REST?",
    "What is idempotency?",
    "What is a webhook?",
]

for i, topic in enumerate(topics, 1):
    history.append({"role": "user", "content": topic})
    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=256,
        messages=history,
    )
    reply = response.content[0].text
    history.append({"role": "assistant", "content": reply})
    print(f"Turn {i}: input_tokens={response.usage.input_tokens}, output_tokens={response.usage.output_tokens}")

Each turn's input_tokens will be meaningfully larger than the last. The fifth call sends the first four exchanges — question and answer both — plus the fifth question. If the assistant replies verbosely, the compounding accelerates.

Three patterns that keep cost predictable

First: trim history. For most applications, the model does not need every prior turn — it needs the last few. A sliding window is the simplest approach:

MAX_TURNS = 4  # keep last 4 message objects (2 user + 2 assistant)

def trimmed_history(history):
    return history[-MAX_TURNS * 2:]

Add this and re-run measure_context.py. Compare input_tokens at turn 5 before and after. The trimmed version stops compounding after the window fills.

Second: move stable context to prompt caching (covered in l6). If you send the same system prompt and document on every request, caching it costs roughly 10% of the normal input price on cache hits.

Third: set max_tokens to the minimum the use case requires. Output tokens cost more than input tokens on most models — do not request 4096 tokens for a route that produces 100-word answers.

Reading the usage block

The usage object appears in every response:

"usage": {
  "input_tokens": 847,
  "output_tokens": 134
}

Log both fields for every production request. After a week of real traffic, query your logs to find which routes are sending the largest payloads. That is where optimization has the highest return. A route that sends a 2,000-token system prompt on every call and handles 500 requests per day is spending a million input tokens per day on that prefix alone — the same prefix that prompt caching would serve at 10% of the cost.

If you get stuck

• input_tokens is not growing between turns: confirm you are appending the assistant reply to history before the next call. If only user messages accumulate, the count grows but not as fast as when both sides are included.
• TPM limit hit on a single large request: reduce the document size, enable caching for the stable prefix, or request a tier upgrade from the console.
• Cost higher than expected: run the sliding window and compare input_tokens before and after. The difference is almost always the conversation history. Check whether verbose assistant replies are being included in full on every subsequent call.
• usage field missing from streaming response: call get_final_message() after the stream completes to get the finalized usage block.

Cross-reference

l6 (Prompt caching) shows how to attach a cache_control marker to make the stable prefix of every request dramatically cheaper. l4 (Errors, retries, and timeouts) covers 429 rate_limit_error, which is what you will see when TPM or RPM is exhausted.

Deliverable

Run measure_context.py as written, then add the sliding window and run it again. Compare the input_tokens values at turn 5 between the two runs. The trimmed version should show meaningfully fewer input tokens on the later turns.

Further reading

• "The Pipeline Rate-Limits Itself, Or It Eats Itself" — what happens when a multi-step Claude pipeline ignores TPM and stampedes its own quota, and the operator's fix.