eval-audit

Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-analysis, write-judge-prompt, or validate-evaluator instead).

Quality

81%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, actionable audit playbook with concrete checks, skill-linked fixes, and a report template. It is slightly held back by minor over-explanation of familiar concepts and the absence of explicit feedback loops, though the latter is less critical for a read-only audit.

Suggestions

Trim explanatory rationale that restates concepts Claude already knows (e.g., the class-imbalance accuracy walkthrough and Likert calibration explanation) to tighten conciseness toward a 5.

Add a short 'verification' note for findings (e.g., re-confirming a problem against a second trace sample before reporting) to give the workflow an explicit checkpoint without changing its read-only nature.

Consider extracting the six diagnostic-area checklists into a reference file so the SKILL.md overview stays leaner, which would push progressive_disclosure toward 5.

Dimension	Reasoning	Score
Conciseness	The body is mostly lean and purposeful, but includes a few explanatory sentences of concepts Claude already knows (e.g., the class-imbalance accuracy example and Likert-scale calibration rationale) that could be trimmed, fitting the 'efficient with minor over-explanation' anchor rather than the lean 5.	4 / 5
Actionability	Checks are concrete and directive ('Flag any that use Likert scales', 'Look for: labeled trace datasets...'), findings link to specific sibling skills and articles, and a copy-paste report template is provided; minor gaps remain because artifact inspection steps depend on the user's MCP/files rather than being exact commands, keeping it just below fully executable 5.	4 / 5
Workflow Clarity	A clear three-stage sequence (gather artifacts → run six diagnostic checks → produce prioritized report) with per-area 'determine whether the problem exists, and record a finding if it does' checkpoints; it lacks explicit error-recovery feedback loops, but since this is a read-only audit rather than a destructive/batch operation, the destructive-cap rule does not apply and it sits at 4 rather than 5.	4 / 5
Progressive Disclosure	The skill is well-organized into clear sections (Overview, Prerequisites, Diagnostic Checks, Report Format, Anti-Patterns) with external article links and sibling-skill references clearly signaled; no bundle files exist, and the inline diagnostic content is core to the audit so it is appropriately placed, though the body is long enough that some detail could arguably be split out, keeping it just below 5.	4 / 5
	Total	16 / 20 Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that clearly states what the skill does and when to use it, with concrete problem examples and an explicit negation distinguishing it from related skills. The only gap is that it describes a single audit action rather than a broad menu of capabilities.

Dimension	Reasoning	Score
Specificity	Names the domain ('LLM eval pipeline') and a concrete action ('surface problems') with specific examples ('missing error analysis, unvalidated judges, vanity metrics'), but the skill performs essentially one action rather than a list of multiple distinct operations, so it falls below the comprehensive-coverage anchor of 5.	4 / 5
Completeness	It explicitly answers both what ('Audit an LLM eval pipeline and surface problems: ...') and when ('Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point...') with concrete trigger phrases, matching the top anchor.	5 / 5
Trigger Term Quality	'Use when inheriting an eval system', 'when unsure whether evals are trustworthy', and 'as a starting point when no eval infrastructure exists' are natural phrases a user would say, giving good keyword coverage, but a few common synonyms are absent so it does not reach comprehensive 5.	4 / 5
Distinctiveness Conflict Risk	The explicit negative trigger ('Do NOT use when the goal is to build a new evaluator from scratch (use error-analysis, write-judge-prompt, or validate-evaluator instead)') carves out a clear niche and disambiguates from sibling skills, giving minimal conflict risk.	5 / 5
	Total	18 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: hamelsmu/evals-skills
Path: skills/eval-audit/SKILL.md
Commit: 22418da

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.