SKILL.md

Skill Eval Runner

Skill libraries drift: imports bring foreign conventions, bulk edits break links, bodies bloat past what an agent will actually read. Run this audit after every import or cleanup pass, before declaring the work done.

Structural Checks

Run these against every skill folder under the skills root ($SKILLS_DIR):

SKILL.md exists in every skill directory.
Frontmatter has `name` and `description`, and description stays under the Agent Skills limit (about 500 chars, and long enough to route on).
Folder slug equals frontmatter `name`.
Body size: SKILL.md under 500 lines. Bigger means content belongs in references/.
Local links resolve: every relative markdown link points at a file that exists.
Risky patterns surfaced for manual review: curl | sh, eval, base64-decode-then-execute, install hooks, network calls in bundled scripts.

Inline commands that cover most of it:

# skills missing SKILL.md
for d in "$SKILLS_DIR"/*/; do [ -f "$d/SKILL.md" ] || echo "MISSING: $d"; done

# slug vs frontmatter name
for f in "$SKILLS_DIR"/*/SKILL.md; do
  slug=$(basename "$(dirname "$f")")
  name=$(awk -F': *' '/^name:/{print $2; exit}' "$f")
  [ "$slug" = "$name" ] || echo "MISMATCH: $slug vs $name"
done

# oversized bodies
find "$SKILLS_DIR" -name SKILL.md | xargs wc -l | awk '$1>500 && $2!="total"'

# risky executable patterns
grep -rnE "curl .*\| *(ba)?sh|eval\(|base64 (-d|--decode)" "$SKILLS_DIR" || echo "clean"

Manual Review After the Script Pass

Structure passing does not mean the skill is good. Still inspect:

Is the description a useful trigger surface (what, "use when", "not for"), not a workflow summary?
Is the body actually concise, or padded to just under the limit?
Are bundled scripts necessary, documented, and free of hidden secret requirements?
Should this be a reference doc instead of a skill?
Do sibling skills point at each other in negative scope?

Verification

Run the four command blocks above against $SKILLS_DIR. Expect: no MISSING lines, no MISMATCH lines, no files over 500 lines, and "clean" (or only known, reviewed hits) from the risky-pattern grep. Any other output is a failure list: fix each item and re-run until clean or every remaining warning is documented with a reason.

Good vs Bad

Bad: "I imported six skills and skimmed them, they look fine." Two have folder names that do not match frontmatter, so routing silently fails, and one links to a references file that was never copied.

Good: Import, run the audit, fix the two mismatches and the dangling link, re-run to clean, then note the one accepted warning (a documented network call in a publisher skill) in the run log.

Footguns

Auditing only the skills you touched. Bulk edits (rename, sed passes) break neighbors too. Fix: always run against the whole skills root.
Passing structure, dead content. A skill can be structurally perfect and still reference tools or paths that no longer exist. Fix: for changed executable workflows, validate behavior with a real invocation, not just the audit.
Silencing warnings instead of documenting them. A grep exclusion added to make the run "clean" hides the next real hit. Fix: document accepted warnings in the run log; never tune the check to skip them.

Completion Checklist

[ ] Audit run against the full skills root, not a subset
[ ] Structural failures fixed and re-run confirms clean
[ ] Risky-pattern hits reviewed, each accepted one documented
[ ] Descriptions and body size manually sanity-checked on changed skills
[ ] Run log or decision note written for major imports

Any box unchecked: not done. Fix or say so.

skill-eval-runner

SKILL.md