Skill Quality Auditor

Navigation hub for evaluating, maintaining, and improving skill quality with 9-dimension framework scoring.

Quick Start

Build once, then audit:

Build once:

bun run build:skill-auditor

Run audits:

# Single skill
skill-auditor evaluate <domain>/<skill-name> --json --store

# Batch with grade gate
skill-auditor batch <skill1> <skill2> --fail-below B --store

When to Use

• Evaluate skills before merge or publication using 9-dimension scoring
• Generate remediation plans, detect duplication (>20% threshold), or enforce CI quality gates
• Validate eval scenario coverage and artifact conventions

When Not to Use

• Write the skill first — do not audit an unfinished draft
• Avoid using this as a substitute for peer review of logic or domain accuracy

Workflow

1. Run skill-auditor evaluate <skill> --json --store
2. Check artifacts and eval coverage using deterministic criteria
3. Generate a remediation plan with T-shirt sizing and score delta estimates
4. Run the auditor again to verify improvement; if score is below target, check remediation-plan.md and focus on the lowest-scoring dimension

Mindset

• Use scores as directional signals, not absolute truth.
• Apply deterministic, reproducible checks over manual review.
• Use threshold-based evaluation rather than relative comparisons.
• Keep audit rules strict for safety and consistency; stay flexible elsewhere.

Anti-Patterns (Summary)

• NEVER skip baseline comparison in recurring audits — WHY: score regressions go undetected without a prior audit.json
• NEVER ignore Knowledge Delta below 15/20 — WHY: low D1 means the skill adds no value over LLM baseline
• NEVER apply subjective scoring — WHY: scores drift between evaluators and cannot be automated in CI
• NEVER create kitchen-sink skills covering unrelated tasks — WHY: broad scope kills D7 and prevents correct triggering
• NEVER use harness-specific paths in skill content — WHY: absolute paths break when installed in a different repo
• NEVER list references without "When to Use" conditions — WHY: unconditional loading bloats context and penalises D5

Ensure you review Detailed Anti-Patterns for all WHY/BAD/GOOD failure modes including agent name references and D4 heading rules.

Examples

Remediation workflow:

skill-auditor evaluate documentation/markdown-authoring --json --store
# Score: 98/140 (C+) -> review remediation-plan.md -> fix -> re-audit -> 128/140 (A)

PR-scoped triage:

skills=$(git diff --name-only origin/main | grep "skills/.*/SKILL.md" | sed 's|skills/||;s|/SKILL.md||' | tr '\n' ' ')
skill-auditor batch $skills --fail-below B --store

Audit all skills:

skill-auditor batch $(find skills -name "SKILL.md" | sed 's|skills/||;s|/SKILL.md||' | tr '\n' ' ')

See Audit Workflow Examples for input/output pairs and CI quality gate examples.

Self-Audit

skill-auditor evaluate agentic-harness/skill-quality-auditor --json
# Expected: A grade, total >= 126/140

References

Framework

Operations