AI agent observability, what to log

Traditional monitoring answers whether the service is up. Agent monitoring has to answer something harder: whether the system is doing the right thing. Those are different questions, and a green uptime dashboard tells you nothing about the second one.

Below is the instrumentation spec I now install on every agent I ship. It is deliberately small. Six fields per span and four dashboards catch almost everything.

One trace per run, spans for everything else

The unit of work is a run: one webhook, one job, one user request. Give it a trace ID and carry that ID into every downstream call. Then every model call and every tool call becomes a span inside it.

This structure is what makes an incident tractable. Somebody says "the thing was wrong for customer 4471 this morning" and you pull one trace and read the decisions in order. Without it, you are grepping.

What each span carries

• Name and kind. Model call or tool call, and which one.
• Input. Arguments for tools, and for model calls a hash of the rendered prompt plus the full text on a sampled subset.
• Output. Result or error, truncated with a length marker.
• Tokens and cost. Input, output, and cached input separately. Cached versus uncached is where cost optimization lives.
• Latency. Including time waiting on the provider, separately from your own processing.
• Outcome. A small enum: ok, invalid output, tool error, refused, budget exceeded, loop broken.

What the run carries

• Objective or job type, so you can slice by workload.
• Final status, using the same enum vocabulary as spans.
• Total cost and total turns.
• Whether a human intervened, and whether the human changed the outcome. This is your most honest quality metric.
• Model and prompt version. Without these, comparison is guesswork.

The four dashboards

1. Outcomes over time. Stacked run statuses by day. A rising slice of invalid output or tool error is the earliest visible signal of drift.
2. Cost per successful outcome. Not total spend. Spend divided by runs that actually worked. This number going up while volume stays flat means something is retrying quietly.
3. Human correction rate. How often a person edits or overrides the result, sliced by job type. The best proxy for quality that does not need a judge model.
4. Turn distribution. Histogram of turns per run. Loop problems show up here as a long tail before they show up on the invoice.

Alerts worth waking up for

Alert on rates and deltas, never on individual failures. Agents fail individually all the time and that is fine.

• Failure rate above a threshold over a rolling window.
• Daily spend above a ceiling, plus a hard kill switch above that.
• Any run hitting the loop breaker or budget breaker. These are always bugs.
• Grounding failures, meaning claims without a retrieved source, above a small percentage.
• Human correction rate rising more than a few points week over week.

Every request returned 200. The tool was returning an empty list and the agent was politely reporting that there was nothing to do.

That one taught me to log the shape of tool results, not just their status. An empty result where results are expected deserves its own outcome code.

Privacy, retention, and the parts people skip

You need real payloads to debug and you do not need to hoard them.

• Redact at the logging boundary with a deny list of field names plus a pattern pass for emails, cards, and national identifiers.
• Sample full prompts at a low percentage, hash the rest. Sampled traces are enough to reconstruct a class of bug.
• Short retention for full payloads, long retention for aggregates. Thirty days and thirteen months is a reasonable default pairing.
• Keep an immutable audit log of write and irreversible tool calls, separate from debug logs and with a longer horizon.

Your traces are also your eval dataset

This is the part teams miss. Every trace where a human corrected the agent is a labeled example with a known correct answer. Pipe those directly into the suite described in LLM evals that actually catch regressions, and your observability spend pays for itself twice.

Traces are also how you find context bloat. Read ten rendered prompts and you will usually find several thousand tokens nobody can justify, which is the exercise in context engineering is the whole job now.

Start here, this week

Add a trace ID per run. Add spans with the six fields. Ship the outcomes chart and the cost-per-success chart. Alert on failure rate and daily spend. That is two days of work and it converts your agent from a black box into a system you can reason about.

If you want the gaps found for you, the AI production readiness audit covers observability, evals, cost, and permissions in one pass, and every agent build I deliver ships with this instrumentation already in place.

Frequently asked