Prompt caching
Prompt caching is when an LLM provider or inference server stores the processed form of a prompt's opening tokens and reuses it for later requests that start with the same tokens. What it reuses is the prefix's KV cache. Cached tokens skip prefill, so the answer starts sooner and those tokens are billed at a lower rate.
How it works
A cache hit needs an exact match from the first token up to the cache boundary. Put stable content first, such as the system prompt, tool definitions and reference documents, and the user's question last. Entries expire after a few minutes without use, and the first request that writes the cache can cost more. Take a 10,000-token system prompt and a 200-token question. If cached reads cost 10% of the normal input price, as on some APIs, a hit is billed like 10,000 × 0.1 + 200 = 1,200 tokens instead of 10,200, about 88% less.
Why it matters in production
In chat and agent workloads, most input tokens are often repeated context, not the new question. One timestamp or user ID near the start of the prompt changes the prefix and turns every request into a miss. Track cache hit rate next to time to first token, so a prompt change that breaks caching shows up as a cost and latency jump.
See also
On the One2N blog
Further reading
