Prompt caching: you're paying to re-read the same prompt

There’s a mental model of the API that almost everyone starts with, because the chat interface trains it into you: you’re in a conversation, the model remembers what’s been said, and each message just adds to it. The API doesn’t work like that. It’s stateless. Every request stands completely alone, and the “conversation” is you re-sending the entire transcript every single time: the tool definitions, the system prompt, every user turn, every tool result, all of it, followed by the one new thing that actually changed.

For a chat app that’s a curiosity. For an agent it’s a cost model. An agent loop doesn’t make one request per user question, it makes one request per step: call a tool, get the result, send everything back, call another tool, send everything back again. Ten tool calls means the model reads your tool definitions ten times, your system prompt ten times, the first tool’s result nine times. I didn’t fully register this until I started watching input token counts in the logs while building my own agent: output tokens trickled, input tokens climbed like a staircase, and nearly all of each request was a byte-for-byte copy of the request before it. Most of what an agent sends the model, it has already sent before, and you can stop paying full price for it with one line.

What gets cached isn’t your prompt

The intuitive picture of prompt caching is a text cache: the provider stores your prompt string, sees the same string again, and skips something. That picture is wrong in a useful way. Storing text is free, anyone can keep a string in a database, and if that were the trick it wouldn’t be worth writing about. What’s expensive is what the model computes from your text. When the model reads a prompt, it converts every token into a stack of internal numerical representations, the processed form it actually works with when generating. Building those numbers is where the GPU time goes, and it’s what your input token bill is really paying for.

Prompt caching stores that: the computed representation, not the words. The technical name is the KV cache, for the key and value vectors the model produces per token, and that’s as deep as this article needs to go. What matters is the reframe. You’re not caching text. You’re caching computation. The provider keeps the model’s finished working-state for your prompt’s opening warm for a few minutes, and the next request that starts the same way picks up that state instead of rebuilding it.

Two boundaries to draw before going further, because both bite in practice. First, this is not response caching: the model still generates fresh every time, and the output is byte-identical to what you’d get without caching. Nothing about quality changes, only what you pay to get there. Second, the match is exact and prefix-only. The cache is keyed on the precise sequence of tokens from the very first one; change a single character at position 100 and everything from there onward gets rebuilt, while everything before it survives. Why it works that way falls out of how the model reads, which is the next section.

The expensive half of inference is the reading

To see why caching the model’s working-state is such a big deal, you need one fact about how inference actually runs: it has two halves, and they do very different work.

The first half is prefill. Before the model can generate anything, it ingests your entire prompt in one big parallel pass, and this is when all those per-token representations from the last section get computed. Prefill is the compute-heavy part, its cost grows with the length of your prompt, and it’s the work your input tokens are billed for. Send a 20,000-token prompt and the model does the full reading job on all 20,000 tokens before producing a single character of output.

The second half is decode: generating the response, one token at a time. Here’s the part that surprised me when I first dug into it. The model doesn’t re-read the prompt as it generates. Each new token consults the stored representations that prefill built, and it does this on every token it produces. When the model is 900 tokens into a response and still follows an instruction from your system prompt, that isn’t some vague memory at work; it’s a physical read of the stored state for those system prompt tokens, happening right then. The cache isn’t cold storage sitting next to the model. It’s the most-read data structure in the entire process.

Once you see the two halves, prompt caching stops being magic: it moves the prefill work across requests. The expensive reading happened once, on some earlier request. The next request that opens with the same tokens picks up the saved state and skips straight past the part you were paying for. And the reason a provider can safely do this is one structural fact about how these models read: strictly left to right. The stored state for token 500 depends only on tokens 1 through 500, and nothing that comes later can change it. Identical opening, identical state, guaranteed, every time. That’s the whole legality argument, and it’s also the exact reason caching is prefix-only: edit the token at position 100 and the stored state for every token after it was computed while looking at something that no longer exists, so it all has to be rebuilt. Everything before position 100 never saw the change and survives untouched.

Prefill builds the memory once. Decode reads it forever. Caching just refuses to build it twice.

An agent loop is the best-case scenario for caching

Everything about an agent loop that makes it expensive is exactly what makes it cache well. Each iteration re-sends the whole request, and between one iteration and the next, almost nothing changed: the tools are identical, the system prompt is identical, the history is identical right up to the newest tool result. The repeated prefix that caching wants is the same repeated prefix the loop was already paying for. You couldn’t design a better customer for this feature if you tried.

On Anthropic’s API, turning it on is one field. Requests are assembled in a fixed order, tools, then system, then messages, and the cache follows that same hierarchy: a breakpoint caches everything from the start of the request up to and including the block it sits on. So you place cache_control on the last tool in your array, and the entire tool-definitions prefix is covered:

tools = [
    {
        "name": "get_weather",
        "description": "Get the current weather in a given location",
        "input_schema": {...},
    },
    {
        "name": "search_docs",
        "description": "Search the internal documentation",
        "input_schema": {...},
        "cache_control": {"type": "ephemeral"},  # caches the whole tools prefix
    },
]

That covers the static front of the request. For the growing conversation behind it, Anthropic now offers automatic caching: a single cache_control at the top level of the request, and the API moves the breakpoint forward to the end of the history as the conversation grows, so each iteration reads everything the previous iteration wrote. The prompt caching docs cover both patterns; for an agent loop you want both at once.

A breakpoint is a write, not a bookmark

The mental model that makes the rest of the behaviour predictable: marking a block with cache_control doesn’t highlight some content as cacheable, it tells the API to hash the entire prefix ending at that block and store one cache entry under that hash. One entry, at that exact position, nothing else. The hash is cumulative, covering everything from the first token of the first tool up to the breakpoint, which is why a change anywhere before the marker produces a different hash and a clean miss. On the next request, the API computes the hash at your breakpoint and looks for a match; if there isn’t one, it walks backwards, block by block, checking whether the prefix at each earlier position matches an entry some previous request already wrote. That backwards walk is capped at 20 blocks, and it’s looking for prior writes, not for content that happens to be stable. If nothing was ever written at a position, there’s nothing there to find, no matter how unchanged the content is.

You get up to four breakpoints per request, and they’re free: you pay for what’s written and read, not for the markers. Most agents need two, the explicit one on the last tool and the automatic one riding the end of the conversation. Reach for more when sections change at different frequencies (tools never, a context document daily, history every turn), or when single turns add more than 20 blocks at once and the lookback would slide past the last write. The docs work a full growing-conversation example if you want to trace it turn by turn.

The economics settle any hesitation. A cache write costs 1.25x the normal input price, a cache read costs 0.1x, and the default entry lives for 5 minutes, with the clock reset for free every time it’s hit. Break-even is a single reuse: one write plus one read is 1.35x, against 2.0x for paying full price twice, and every read after that is 90% off. An agent mid-task hits the cache every few seconds, so its prefix effectively never expires while it’s working.

One habit to adopt from day one: don’t trust that it’s working, check. The usage block in every response splits your input three ways: cache_read_input_tokens (served from cache at 0.1x), cache_creation_input_tokens (written this request at 1.25x), and input_tokens (everything after your last breakpoint, full price). The number you want to see is reads dominating from the second request onward. If reads stay at zero, you don’t have caching, you have a 25% surcharge: you’re paying for a write on every request and never collecting.

The checklist

Everything above compresses into a short list. If your cache hit rate is disappointing, the answer is almost always in here:

• Order the request by volatility. Stable content first (tools, system prompt, examples), changing content last (per-request context, the newest message). The cache is prefix-only, so anything volatile poisons everything behind it.
• Put the breakpoint on the last block that is byte-identical across requests. A breakpoint after a timestamp caches nothing, ever, and charges you the write price for the privilege - the lookback can’t save you, because it only finds entries that earlier requests actually wrote.
• Treat tool definitions as frozen. They sit at the front of the hierarchy, so any change to a name, description, or schema invalidates the entire cache. That includes invisible changes: a JSON serializer that doesn’t guarantee key order will break your cache without changing a single word.
• Keep request settings stable. Toggling tool_choice, adding or removing images, or changing extended thinking settings invalidates the messages cache even when the text is identical.
• Know the minimum. Prompts below the model’s minimum cacheable length (1,024 tokens for Sonnet-class models) are silently processed without caching. No error, no warning, just full price.
• Read the usage fields on every deploy. cache_read_input_tokens should dominate from the second request onward. Reads at zero with writes climbing means you’re paying the surcharge and collecting nothing.

What this changes about how you build

The real shift isn’t the discount, it’s that prompt structure becomes a cost decision instead of a style decision. Once you know the cache is prefix-only, the ordering habit writes itself: everything stable goes at the front (tools, system prompt, examples, reference documents), everything volatile goes at the end (per-request context, timestamps, the newest message). That habit costs nothing to adopt on day one, and it pays 90% on the most repeated part of every request for the life of the product. Retrofitting it later, once your prompt assembly has timestamps and per-user fragments scattered through it, is a genuinely annoying refactor. I’d put it in the same category as structured logging: trivial if you start with it, painful if you don’t.

It also kills an instinct most of us carried over from the early days: keeping the system prompt short to save money. With caching, a rich 5,000-token system prompt full of worked examples costs almost nothing amortized across an agent’s requests, because after the first pass it’s a 0.1x read forever. The trade-off that used to exist between a thorough prompt and a cheap one has mostly collapsed. Spend the tokens where they teach the model something; the cache means you only pay full price for them once.

This coverage of prompt caching deliberately stayed at the surface of what the model is actually storing. Those key and value vectors, why the model builds them, and what they do during generation deserve a proper treatment rather than a paragraph, and at some point I’ll write that one up: how attention works, what actually lives in the cache, and why generation reads it on every token. For now: order your prompt by volatility, mark the last stable block, and read the usage fields. That’s 90% of the value for one line of code.

Worth reading

The primary sources are good here, and a couple of adjacent pieces round out the picture if you’re building agents rather than just calling the API:

• Prompt caching - Anthropic. The reference this article leans on: breakpoints, the lookback window, invalidation rules, and the usage fields, all in one place. Read the “common mistake” example even if you skip the rest.
• Tool use with prompt caching - Anthropic. The agent-specific half: exactly what to mark in a tools array and what toggling tool_choice costs you.
• Prompt caching with Claude - the original announcement, worth it for the headline benchmark numbers on long-document chat and multi-turn agents.
• Prompt caching cookbook notebook - Anthropic. Runnable end-to-end examples if you’d rather read code than docs.