Putting AI coding agents in CI

Coding agents got good enough that leaving them in a terminal on your laptop started to feel wasteful. The obvious next move is CI, where they can pick up work while nobody is watching.

The obvious next move after that is an incident. Here is the setup that gets the upside without it.

The three non-negotiables

1. Ephemeral sandbox. Fresh container per job, no reused state, no access to production networks or secrets stores. Egress allowlisted to your package registry and the model provider, nothing else.
2. A token that cannot merge. Contents write for branches, pull request create, and nothing more. No merge, no release, no environment secret access, no repository administration.
3. A gate before merge. Human review, or a deterministic check set strong enough to be trusted for the narrow class of task in question. Branch protection enforces it so the rule does not depend on discipline.

Pick tasks where success is machine-checkable

The pattern that separates useful from dangerous is verifiability. If a test suite, a type checker, or a linter can tell you the work is correct, an agent can do it unattended. If correctness requires taste or product judgment, it cannot.

• Excellent. Dependency bumps where tests must pass. Type and lint error cleanup. Adding tests for uncovered branches. Mechanical renames and API migrations. Generating a diagnosis comment on a failing build.
• Workable with review. Small bug fixes with a reproduction test written first. Documentation from code. Adding instrumentation.
• Do not. Schema migrations. Auth and permission logic. Anything touching payments. Broad architectural refactors. Anything without a test that would fail if the change were wrong.

Trigger deliberately, not ambiently

Agents on every push is how you get a hundred pull requests nobody asked for and a bill nobody predicted. Every agent job I run is triggered on purpose:

• A label on an issue, meaning a human decided this one is suitable.
• A schedule, for maintenance classes like dependency hygiene.
• A failed build, for triage and a diagnosis comment only.
• An explicit comment command on a pull request, with an allowlist of who can issue it.

Budgets and breakers, in the job definition

Agents in CI fail by spinning, exactly as they do everywhere else, except now nobody is watching the terminal.

1. Max turns per job, plus a wall-clock timeout on the job itself.
2. Token budget per job, enforced in the loop rather than hoped for.
3. Per-repository daily spend cap with an alert before the ceiling.
4. Repeat detection, so identical tool calls abort the run with a partial result and a clear reason.

These are the same guards from the Claude Agent SDK in production, with the difference that CI has no human in the loop to notice, so the breakers have to be real.

Deterministic checks are your actual safety net

The agent's own claim that it verified something is not evidence. Every agent pull request runs the same pipeline as a human one, plus a few additions worth having:

• Full test suite, type check, lint, build.
• A diff size ceiling. A job scoped to fix a lint error that produces 900 changed lines gets rejected automatically.
• A path allowlist per job type. A dependency job has no business editing authentication code.
• Dependency and secret scanning, because agents copy patterns from wherever they learned them.
• Your eval suite if the change touches prompts, tools, or retrieval. That is the only way to catch a behavior regression that compiles fine.

It fixed the failing test by deleting the assertion. Every check passed. The diff ceiling would have caught it. We had not added one yet.

Make the output reviewable

Half the value of an agent pull request is lost if reviewing it takes as long as writing the code. Require a fixed template in the description: what was asked, what changed and why, what was verified and how, what was deliberately not done, and what the reviewer should look at first.

Then keep the diff small on purpose. One task per pull request, hard ceiling on scope. The review checklist for what to look at is in how to review agent-written pull requests.

Instrument it like any other agent

Log per job: trigger, task type, turns, tokens, cost, duration, outcome, and whether the pull request was merged, edited, or closed. That last field is your quality metric. A task class with a low merge rate should be turned off, not tuned forever. Same discipline as agent observability generally.

A working starting configuration

1. One job type only: dependency upgrades on a weekly schedule, tests must pass, diff ceiling of 200 lines, lockfile and manifest paths only.
2. Ephemeral container, allowlisted egress, branch-and-PR-only token, branch protection requiring one human approval.
3. Turn cap, token cap, job timeout, daily repository budget with an alert.
4. Template-enforced pull request description and merge-rate tracking.
5. Add a second task type only after four weeks of clean results on the first.

Boring on purpose. Boring is what makes it still running in six months.

If you want this wired into an existing pipeline, or an audit of an agent setup that already has more access than it should, that is the AI production readiness audit and consulting engagements.

Frequently asked