agent-self-evaluation

Agent Self-Evaluation

After finishing a non-trivial task, score your own output before the user has to. This is a reflection step, not a pass/fail gate: it catches omissions and overconfidence while they are still cheap to fix. The rule that makes it work: every score below 5 must cite specific evidence. Show the gap, don't just name it.

When to run

• Code changes spanning 3+ files or 50+ lines
• Multi-step workflows (implement, test, review)
• Debugging sessions that took 3+ attempts
• Design docs, architecture decisions, written analysis
• User asks "how good was that?"

The five axes

Scale: 5 exceptional, 4 minor nits, 3 meets request with one notable weakness, 2 clear gap affecting usability, 1 misses the request.

Workflow

1. Collect raw material. The original request, your final output, tool outputs that prove correctness (test results, exit codes, lint), and any mid-task user corrections.
2. Score each axis independently. Read the axis question, find evidence in the output, assign 1-5. Do not pick an overall feel first and work backwards.
3. Write the report. One-line summary, five scores each with evidence, overall average to one decimal, then 1-3 improvements ranked by impact. One improvement per weak axis, the highest-impact gap only.
4. Act on it. Any axis at 3 or below: fix it now if the fix takes under a minute (missing link, unclear sentence). Otherwise flag it explicitly: "X scored 2 because [evidence]. Rerunning with [specific fix] would raise it to [score]." Get confirmation before redoing external side effects (sends, publishes).

Good vs bad

Good: "Completeness: 4. Covered happy path plus 3 error cases. Missing: timeout handling for hung connections." The gap is named and locatable.

Bad: "Completeness: 5. Everything covered." No evidence. That is self-congratulation, not evaluation. A real 5 requires proving there is nothing left to improve.

Rules that keep scores honest

• Evaluate the output, not the process. Iteration count is irrelevant to the user.
• Evaluate against what was asked, not what you could have built. "Didn't handle WebSockets" is not a gap if WebSockets were never requested.
• No re-litigating. If you disagreed with the approach, that argument happened before delivery. Score what shipped.
• Preference is not evidence. "I don't like decorators" is not a 3. Cite a concrete readability, testability, or correctness concern, or leave it at 4+.
• Use tool outputs as proof. Tests passed: cite them. Lint clean: cite it. Don't guess.
• If every axis looks like a 5, try harder. Ask: "If I were the user, what would annoy me about this output?"

Verification

Do this check on your finished scorecard: for every score below 5, confirm the evidence sentence names a specific, locatable gap (a file, a case, a missing step). Expect zero scores that read "could be better" without a noun. If any exist, rewrite that line with the concrete gap or raise the score you cannot justify.

Completion checklist

• [ ] All five axes scored independently
• [ ] Every score below 5 cites specific evidence
• [ ] Overall average computed, one decimal
• [ ] 1-3 improvements ranked by impact
• [ ] Axes at 3 or below: fixed now or explicitly flagged with the fix
• [ ] Self-check answered: "Would the user agree with this assessment?"

Any box unchecked: not done. Fix or say so.

Footguns

• Halo scoring. Deciding "this went well" then assigning 4s and 5s to match. Fix: score axes in a fixed order, evidence first, number second.
• Score inflation on accuracy. "Code looks right" is not accuracy evidence. Fix: only tool output (tests, execution, lint) justifies accuracy of 4+, otherwise cap at 3 and say the code is unverified.
• Listing five fixes for one axis. Dilutes attention, nothing gets fixed. Fix: one highest-impact improvement per weak axis, tied to user impact ("missing error handling means the API call crashes silently").