Context engineering for AI agents

When an agent gives a wrong answer, inspect the material it received on that turn. Missing task state, irrelevant sources or truncated results can explain a failure that a prompt rewrite would leave unresolved.

That is the job now. Models got good enough that wording matters less than selection. Deciding what an agent sees, in what order, and what it forgets, is where the remaining quality lives.

Bigger windows made this worse, not better

The intuitive response to a struggling agent is to give it more. More history, more documents, more instructions. Large context windows make that easy, and it is usually a mistake.

Two things degrade as context grows. Relevant material competes with irrelevant material for attention, and instructions buried in the middle of a long payload lose force against instructions at the edges. Test a shorter context against the same cases and compare the results before adopting the change.

The four layers of an agent's context

It helps to stop thinking about "the prompt" and start thinking about four layers with different lifetimes.

1. Identity and rules. Who the agent is, what it may never do, output format. Stable across every run, so it is also the layer you cache.
2. Task state. The current objective, what has been completed, what is blocked, what the constraints are. This is the layer teams most often fail to make explicit, and it is the one that keeps long runs coherent.
3. Retrieved knowledge. Documents, records, and prior examples fetched for this specific turn. Tight, ranked, and cited.
4. Working history. Recent turns and tool results. Grows every turn, so it needs an eviction policy from day one.

Once the layers are separate you can tune them independently, cache the stable ones, and compress the growing one without touching the rest.

Retrieval is where most context rot starts

Inspect the retrieved text that reaches the prompt. Look for page headers, navigation, duplicate passages and relevant material pushed behind unrelated results.

• Retrieve more candidates than you need, then rerank and pass the top three to five. Recall at retrieval, precision at the prompt.
• Deduplicate aggressively. Three chunks saying the same thing crowd out the chunk that says something different.
• Carry metadata with every passage: source, section, date. It improves the model's judgment and gives you citations for free.
• Strip furniture before embedding. Headers, footers, and legal boilerplate poison similarity.

The extraction and chunking side of this is a whole discipline of its own, which I cover in a RAG pipeline that survives real documents.

Compression that preserves decisions

Long-running agents have to forget things. The question is what survives. A prose summary of a conversation loses exactly the parts that matter. A structured state object keeps them.

A state summary can keep decisions made and why, constraints discovered, identifiers touched, open questions, and failed approaches already ruled out. That last one prevents the most annoying agent behavior there is, which is confidently retrying something that already failed twice.

Everything else can go. The full transcript lives in your logs, where it belongs, and feeds your eval suite later.

Tool results are context too

A tool that returns 300 rows of JSON has just made a context decision on your behalf, and a bad one. Tool output should be shaped for a reader with limited attention: capped, summarized, and explicit about truncation.

Return the top N with a count of what was omitted, plus a way to ask for more. Check whether capped results preserve the information needed for the next decision.

When testing a context cut, use the same inputs and scoring rules before and after. Record both quality and token use.

Ordering, and where instructions go

Position matters more than people expect. One ordering to test is rules and identity first, then task state, then retrieved material, then recent history, then the immediate instruction restated last. The restatement at the end costs a few dozen tokens and measurably improves adherence on long payloads.

Mark boundaries clearly. Untrusted retrieved content should be visibly fenced and labeled as data, never blended into instruction text, for both quality and security reasons.

How to actually work on this

1. Log the full rendered context for a sample of production runs. Read ten of them yourself. This is uncomfortable and always instructive.
2. Measure token composition by layer. Teams are routinely shocked by how much of the bill is boilerplate.
3. Cut one layer at a time and run evals. Keep the cuts that hold.
4. Add a state object if you do not have one. Then cache the stable prefix and watch the bill drop, per prompt caching.

Use traces and evaluation cases to check what the agent receives before changing the model. If you have a stalled product build that has to work for a real customer, pilot, or launch, send the project brief. Scope and price are agreed before work starts.

Frequently asked