Prompt caching, and the rest of the bill

Nearly every agent bill I have audited had the same composition. Output tokens were a rounding error. Input tokens were the bill, and most of those input tokens were the same content being resent turn after turn.

That is good news, because repeated identical content is the cheapest thing in the world to optimize. Here is the order I work in, from highest return to lowest.

Step zero: know your composition

Before changing anything, break your token spend into four buckets: system and rules, tool definitions, retrieved material, and conversation history. Add output as a fifth for completeness.

Teams are consistently surprised. I have seen tool definitions alone account for a quarter of spend on an agent with too many tools. You cannot prioritize what you have not measured, and the measurement is one field in your span logging.

Step one: cache the stable prefix

Providers price cached input tokens far below fresh ones. The mechanic that matters is ordering: the cache works on a prefix, so everything stable must come first and everything volatile must come after.

1. System prompt and rules. Identical every run.
2. Tool definitions. Identical until you ship a change.
3. Stable reference material. Policies, schemas, style guides.
4. Few-shot examples, if you use them.
5. Then, and only then, task state, retrieval, and history.

Verify with your provider's cache-hit metrics rather than by assumption. Log cached and uncached input tokens as separate fields so a regression is visible on a chart instead of on an invoice.

Step two: stop sending what you do not need

Caching makes repeated context cheap. Removing it makes it free, and usually improves quality at the same time.

• Trim tools per task. If a job type only uses four tools, do not ship twelve. Fewer options also means fewer wrong choices.
• Compress history past a threshold. Keep recent turns verbatim, replace older ones with a structured state summary of decisions and constraints.
• Rerank retrieval down to three to five passages. Retrieve widely, pass narrowly.
• Cap tool output. Return the top N plus an omitted count. Unbounded results are a cost and a quality problem at once.

The full reasoning behind these cuts is in context engineering is the whole job now.

Step three: route by difficulty

Using one frontier model for every call is the most common source of avoidable spend. Most agent turns are not hard.

• Small model. Classification, routing, formatting, yes-no decisions, short summarization of clean text.
• Best model. Structured extraction from messy input, drafting anything a customer will read, multi-step reasoning, code.
• Escalation path. Try small, validate the output against a schema, escalate on failure. This gives you cheap throughput with a quality floor.

Every route gets its own eval slice before it ships, per LLM evals that actually catch regressions. Routing without measurement is how quality quietly slides.

Step four: stop paying for repeated work

• Deduplicate identical requests. Agents call the same tool with the same arguments inside a single run more often than you would believe. A short-lived response cache removes it.
• Cache derived artifacts. If a document's extraction is deterministic, store it keyed on a content hash and never pay twice.
• Batch offline work. Anything not user-facing goes through batch endpoints where available, at a substantially lower rate.
• Kill loops early. A run that spins for thirty turns is pure waste. Turn caps and repeat detection are cost controls as much as reliability controls.

Eighty-one percent of their input tokens were a system prompt being resent, uncached, because a greeting with the user's first name sat above it.

The metric to actually watch

Total spend is a vanity metric. Track cost per successful outcome, split by job type. It captures the thing you care about, catches silent retry storms, and makes optimization arguments concrete. A change that halves cost per success without moving eval scores is unambiguously good. A change that halves cost and drops accuracy four points is a decision, not a win.

Pair it with a daily spend alert and a hard kill switch above it. AI systems fail expensively, and the failure is usually a loop rather than an outage.

A one-day plan

1. Log cached and uncached input tokens separately by span.
2. Reorder your prompt so every stable block sits before every volatile one. Verify cache hits.
3. Trim the tool list per job type.
4. Add history compression and cap tool output size.
5. Route one obvious task class to a smaller model with eval coverage.
6. Ship a cost-per-success chart and a daily spend alert.

That is a day of work and it typically removes most of the bill. If you want it done and verified against your own traces, cost review is part of the AI production readiness audit, and agent builds ship with caching and routing in place from the start.

Frequently asked