Context engineering is the whole job now

Almost every agent I have been asked to fix had the same diagnosis. Not a weak model. Not a bad prompt, exactly. The agent was being handed the wrong material at the wrong moment, and then blamed for the result.

That is the job now. Models got good enough that wording matters less than selection. Deciding what an agent sees, in what order, and what it forgets, is where the remaining quality lives.

Bigger windows made this worse, not better

The intuitive response to a struggling agent is to give it more. More history, more documents, more instructions. Large context windows make that easy, and it is usually a mistake.

Two things degrade as context grows. Relevant material competes with irrelevant material for attention, and instructions buried in the middle of a long payload lose force against instructions at the edges. I have watched accuracy improve by cutting a prompt from 40,000 tokens to 5,000 with no other change. Same model, same task, fewer distractions.

The four layers of an agent's context

It helps to stop thinking about "the prompt" and start thinking about four layers with different lifetimes.

1. Identity and rules. Who the agent is, what it may never do, output format. Stable across every run, so it is also the layer you cache.
2. Task state. The current objective, what has been completed, what is blocked, what the constraints are. This is the layer teams most often fail to make explicit, and it is the one that keeps long runs coherent.
3. Retrieved knowledge. Documents, records, and prior examples fetched for this specific turn. Tight, ranked, and cited.
4. Working history. Recent turns and tool results. Grows every turn, so it needs an eviction policy from day one.

Once the layers are separate you can tune them independently, cache the stable ones, and compress the growing one without touching the rest.

Retrieval is where most context rot starts

Teams tune chunk sizes for a week and never look at what actually enters the prompt. When I do look, I usually find page headers, navigation boilerplate, duplicated near-identical passages, and one genuinely useful paragraph ranked fourth.

• Retrieve more candidates than you need, then rerank and pass the top three to five. Recall at retrieval, precision at the prompt.
• Deduplicate aggressively. Three chunks saying the same thing crowd out the chunk that says something different.
• Carry metadata with every passage: source, section, date. It improves the model's judgment and gives you citations for free.
• Strip furniture before embedding. Headers, footers, and legal boilerplate poison similarity.

The extraction and chunking side of this is a whole discipline of its own, which I cover in a RAG pipeline that survives real documents.

Compression that preserves decisions

Long-running agents have to forget things. The question is what survives. A prose summary of a conversation loses exactly the parts that matter. A structured state object keeps them.

What I keep, in order of importance: decisions made and why, constraints discovered, identifiers touched, open questions, and failed approaches already ruled out. That last one prevents the most annoying agent behavior there is, which is confidently retrying something that already failed twice.

Everything else can go. The full transcript lives in your logs, where it belongs, and feeds your eval suite later.

Tool results are context too

A tool that returns 300 rows of JSON has just made a context decision on your behalf, and a bad one. Tool output should be shaped for a reader with limited attention: capped, summarized, and explicit about truncation.

My default is to return the top N with a count of what was omitted, plus a way to ask for more. That single change fixed more "the agent lost the thread" complaints than any prompt rewrite I have done.

We removed 34,000 tokens of "helpful background" and the accuracy went from 71 percent to 89 percent. Nothing else changed.

Ordering, and where instructions go

Position matters more than people expect. What has worked reliably for me: rules and identity first, then task state, then retrieved material, then recent history, then the immediate instruction restated last. The restatement at the end costs a few dozen tokens and measurably improves adherence on long payloads.

Mark boundaries clearly. Untrusted retrieved content should be visibly fenced and labeled as data, never blended into instruction text, for both quality and security reasons.

How to actually work on this

1. Log the full rendered context for a sample of production runs. Read ten of them yourself. This is uncomfortable and always instructive.
2. Measure token composition by layer. Teams are routinely shocked by how much of the bill is boilerplate.
3. Cut one layer at a time and run evals. Keep the cuts that hold.
4. Add a state object if you do not have one. Then cache the stable prefix and watch the bill drop, per prompt caching.

If you would rather have someone else read your traces and tell you what to cut, that is a large part of what the AI production readiness audit does in 48 hours.

Frequently asked