AI agent security, what actually breaks

Conference talks about AI agent security are mostly about attackers. The incidents I actually get called into are about operations. An agent had a key it should never have had. A retry loop sent the same email nine hundred times. A tool that was meant for a staging tenant was pointed at production because both used the same token.

Nobody was attacked. The system did exactly what it was permitted to do. That is the category most teams are unprepared for, so that is what this post is about. Adversarial input has its own post, prompt injection is not a prompt problem, and the protocol layer is covered in MCP server security and auth.

The incidents are boring, and that is the point

Five real shapes, anonymized, from the last eighteen months of client work.

• A support agent with a full database role, added during a demo, never narrowed. It updated a column it was never meant to see.
• One shared API key across three agents. When it leaked, nobody could say which system to shut off, so all three went down for a day.
• A retry with no idempotency key. The provider timed out, the workflow retried, and the customer got charged twice.
• A logging call that wrote full tool payloads, including the contents of identity documents, into a log tool half the company could read.
• A cron-triggered agent that kept running for six weeks after the project was cancelled, because nothing owned its lifecycle.

None of these need an adversary. They need an afternoon of attention before launch.

Blast radius is the only number that matters

Before an agent goes live, answer one question in writing. If this thing does the worst permitted action a thousand times in a row, what happens?

That is blast radius. It is a property of the credential and the tool list, not of the prompt, so it is measurable. Write it down, then shrink it until you can live with the answer.

1. One service identity per agent. Never a shared key, never a developer key, never the deploy key.
2. Scope to the smallest set of operations the job needs, then remove one more and see if anything fails.
3. Rate limit at the tool layer, not just at the provider. Per run, per hour, per tenant.
4. Cap spend and volume with hard stops that page a human instead of degrading quietly.

The agent should never hold the key

The instinct is to put the secret in the agent's environment and let it call the API. That collapses two very different jobs into one. Instead, keep every credential behind a tool the agent calls by name.

The tool holds the secret. It verifies who is calling, validates the arguments against a narrow schema, enforces its own limits, writes an audit record, and returns a small structured result. The agent never sees the credential, so a leaked transcript is embarrassing rather than expensive.

• Verbs, not endpoints. Give it refund_order, not http_request. A generic fetch tool is a credential with extra steps.
• Bounded arguments. Amount less than or equal to the original charge. Recipient inside the current task context. Table allowlisted, not templated.
• Short-lived tokens. Mint per run where the provider supports it. Long-lived secrets are a permanent liability.
• Idempotency by default. Every write tool takes a key derived from the run and the payload. Retries then stop being incidents.

Gate the irreversible, let the rest run

Approval on everything is the same as approval on nothing, because people start clicking without reading. Sort actions by whether you can take them back, then spend the human attention where it counts.

• Reversible. Drafts, internal notes, status changes, enrichment. Unattended.
• Customer visible. Email, SMS, public posts, tickets. Template allowlist plus recipient checks, or a human on anything outside the templates.
• Money or deletion. Payments, refunds, cancellations, permission changes, destructive writes. A human, or a one-time signed grant tied to that exact payload.

The approval should read like a receipt, not a puzzle. Two lines of summary, the exact payload, one button. If a reviewer has to open three tabs to understand what they are approving, the gate is decoration.

The agent did nothing unusual. It just did the usual thing to nine hundred customers before anyone looked at the queue.

Logging that survives an audit

Security work you cannot reconstruct is a story, not a control. Every write and every irreversible call needs a record you can hand to a customer, a partner, or a lawyer without editing.

1. Append-only. If the agent's role can delete audit rows, the audit log is a suggestion.
2. Correlated by run ID, so one incident reads as one story across tools, retries, and handoffs.
3. Record the authorization decision, not just the action. Which policy allowed this, and which identity asked.
4. Redact contents, keep the shape. Field names, sizes, and hashes are enough to debug. Document bodies in a log tool are a second breach waiting to happen.
5. Alert on denied-by-scope attempts. Those are either bugs or probes, and both deserve a look the same day.

The wider version of this, including which metrics are worth a dashboard, is in AI agent observability, what to log. Failures caught there belong in your eval suite the same week, so the same break cannot ship twice.

Ownership, the control everyone forgets

An agent with no owner becomes a permanent unmonitored process with production credentials. Give every deployed agent a named human, a review date, and a documented off switch. Then actually use the review date.

• Who gets paged when it misbehaves at 2am.
• What the kill command is, and who has run it once on purpose.
• When the credential rotates, and whether rotation is tested.
• What happens to its access when the project ends.

The pre-launch checklist

1. One scoped service identity per agent. No shared or human keys.
2. Blast radius written down and accepted by a named owner.
3. Credentials behind tools. Verbs with bounded arguments, no generic fetch or exec.
4. Idempotency keys on every write. Retries proven safe.
5. Rate limits and hard spend caps that page instead of degrading.
6. Irreversible actions behind a human gate or a signed one-time grant.
7. Append-only audit log, run-correlated, redacted payloads.
8. Alerts on anomalous and denied tool calls, plus a weekly manual read.
9. Tested kill switch, rotation plan, and a review date on the calendar.

Nine items. A focused afternoon for a single agent, less for the second one, because most of it becomes shared infrastructure. If you want an adversarial read on an agent you already run in production, that is a core part of the AI production readiness audit.

Frequently asked