# Rate limits and the cost of context

Web page: https://academy.memberpress.ai/courses/api-fundamentals/l5/

Lesson 5 of 9 in [Anthropic API Fundamentals](https://academy.memberpress.ai/courses/api-fundamentals/), 16 min.

Video: https://www.youtube.com/watch?v=k0nfqHU2f0Y

Rate limits and token costs are not independent problems. Understanding one without the other will leave you with surprises in production.

**Your tier's limits**

Anthropic assigns accounts to tiers based on usage history and prepayment amount. Each tier has two limits: tokens per minute (TPM) and requests per minute (RPM). You can see your current tier at `console.anthropic.com` under Settings > Limits.

The limits reset every 60 seconds. A single large request can exhaust most of your TPM quota for that window. If your application sends bursts — ten requests simultaneously at startup — you can hit the RPM limit even on a generous tier.

**Input tokens are the cost that compounds**

Every token in the `messages` array costs money and counts against your TPM limit — not just the new user message, but every prior turn you include. This is where most production cost overruns originate.

Run this and watch:

```python
# scripts/measure_context.py
import anthropic

client = anthropic.Anthropic()
history = []

topics = [
    "What is an HTTP header?",
    "What is JSON?",
    "What is REST?",
    "What is idempotency?",
    "What is a webhook?",
]

for i, topic in enumerate(topics, 1):
    history.append({"role": "user", "content": topic})
    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=256,
        messages=history,
    )
    reply = response.content[0].text
    history.append({"role": "assistant", "content": reply})
    print(f"Turn {i}: input_tokens={response.usage.input_tokens}, output_tokens={response.usage.output_tokens}")
```

Each turn's `input_tokens` will be meaningfully larger than the last. The fifth call sends the first four exchanges — question and answer both — plus the fifth question. If the assistant replies verbosely, the compounding accelerates.

**Three patterns that keep cost predictable**

First: trim history. For most applications, the model does not need every prior turn — it needs the last few. A sliding window is the simplest approach:

```python
MAX_TURNS = 4  # keep last 4 message objects (2 user + 2 assistant)

def trimmed_history(history):
    return history[-MAX_TURNS * 2:]
```

Add this and re-run `measure_context.py`. Compare `input_tokens` at turn 5 before and after. The trimmed version stops compounding after the window fills.

Second: move stable context to prompt caching (covered in l6). If you send the same system prompt and document on every request, caching it costs roughly 10% of the normal input price on cache hits.

Third: set `max_tokens` to the minimum the use case requires. Output tokens cost more than input tokens on most models — do not request 4096 tokens for a route that produces 100-word answers.

**Reading the usage block**

The `usage` object appears in every response:

```json
"usage": {
  "input_tokens": 847,
  "output_tokens": 134
}
```

Log both fields for every production request. After a week of real traffic, query your logs to find which routes are sending the largest payloads. That is where optimization has the highest return. A route that sends a 2,000-token system prompt on every call and handles 500 requests per day is spending a million input tokens per day on that prefix alone — the same prefix that prompt caching would serve at 10% of the cost.

**If you get stuck**

- `input_tokens` is not growing between turns: confirm you are appending the assistant reply to `history` before the next call. If only user messages accumulate, the count grows but not as fast as when both sides are included.
- TPM limit hit on a single large request: reduce the document size, enable caching for the stable prefix, or request a tier upgrade from the console.
- Cost higher than expected: run the sliding window and compare `input_tokens` before and after. The difference is almost always the conversation history. Check whether verbose assistant replies are being included in full on every subsequent call.
- `usage` field missing from streaming response: call `get_final_message()` after the stream completes to get the finalized usage block.

**Cross-reference**

l6 (Prompt caching) shows how to attach a `cache_control` marker to make the stable prefix of every request dramatically cheaper. l4 (Errors, retries, and timeouts) covers `429 rate_limit_error`, which is what you will see when TPM or RPM is exhausted.

**Deliverable**

Run `measure_context.py` as written, then add the sliding window and run it again. Compare the `input_tokens` values at turn 5 between the two runs. The trimmed version should show meaningfully fewer input tokens on the later turns.

**Further reading**

- ["The Pipeline Rate-Limits Itself, Or It Eats Itself"](https://sethshoultes.com/blog/the-pipeline-rate-limits-itself.html) — what happens when a multi-step Claude pipeline ignores TPM and stampedes its own quota, and the operator's fix.

## Try these prompts

- Write a Python sliding-window function that trims a messages history array to the last 4 message objects before passing it to the API.
- Explain why input_tokens grows on every turn of a multi-turn conversation, and name two techniques to stop the compounding.
- Write the log line you would add after every API call to record input_tokens, output_tokens, and the route name for cost tracking.

Next: [Prompt caching](https://academy.memberpress.ai/courses/api-fundamentals/l6/)