Name: testland/llm-eval-anti-patterns
Rating: 75.2 (1 reviews)
Author: testland

testland/llm-eval-anti-patterns

Audits an existing LLM evaluation suite for eight methodology errors that make its numbers untrustworthy: too few cases per capability, single-provider lock-in, exact-match assertions on open-ended output, no semantic-similarity check on paraphrase-tolerant output, no baseline comparison in CI, no cost or latency ceiling, unpinned model identifiers, and no adversarial coverage. Supplies harness-neutral detection cues, the reason each error invalidates the result, a concrete fix, a Critical/Warning/Info severity scheme, and a findings-table output shape. Covers validating an LLM judge against human labels before its verdicts count as evidence. Use when an eval suite already exists and its pass rate is about to gate a release, a model swap, or a prompt change, and nobody has audited how the suite itself was built.

Quality

94%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

85%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable and well-structured with a clear review workflow, real one-level-deep references, and a concrete worked example. Its only weakness is conciseness — it is long and re-explains some statistical concepts Claude already knows.

Suggestions

Trim the 'Why it invalidates' sections that re-teach general statistics (sampling error, confidence intervals) down to the eval-specific claim, since Claude already knows the underlying statistics.

Move the memorization-vs-exploitation contamination digression in anti-pattern 1 into a one-line pointer or into references/citations.md, keeping only the practical import inline.

Consider tightening the severity-scheme and 'Reading an eval definition' prose to bullet-level brevity to reduce the ~600-line footprint while preserving the actionable structure.

Dimension	Reasoning	Score
Conciseness	The body is dense and actionable, but several 'Why it invalidates' sections re-explain concepts Claude already knows (sampling error, confidence intervals, the memorization/exploitation split) and the catalog runs ~600 lines, so it is mostly efficient but could be tightened rather than fully lean.	2 / 3
Actionability	Every anti-pattern gives concrete detection cues (named Promptfoo assertion types like 'equals, contains, icontains, regex, llm-rubric, g-eval') and executable fixes (e.g. 'promptfoo eval --grader <provider>', copy-paste YAML, a full worked config), matching the 'fully executable, copy-paste ready' anchor.	3 / 3
Workflow Clarity	The review process is clearly sequenced — locate the five harness slots, apply the eight checks, assign severity, emit a findings table, then a verdict block — and a complete worked example demonstrates the flow end-to-end; this is a read-only audit so the destructive-operation validation cap does not apply.	3 / 3
Progressive Disclosure	The core checks stay inline in SKILL.md while genuinely offloadable material is split into two clearly signaled, verified one-level-deep references (references/citations.md and references/harness-mapping.md), giving easy navigation without nested indirection.	3 / 3
	Total	11 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific, complete, and distinctive, with concrete actions and an explicit 'Use when...' trigger clause in third-person voice. It is a strong, low-conflict description that clearly signals when to invoke the skill.

Dimension	Reasoning	Score
Specificity	The description enumerates eight distinct methodology errors by name ('too few cases per capability, single-provider lock-in, exact-match assertions on open-ended output...') plus concrete deliverables ('Critical/Warning/Info severity scheme', 'findings-table output shape'), matching the 'lists multiple specific concrete actions' anchor.	3 / 3
Completeness	It answers both 'what' ('Audits an existing LLM evaluation suite for eight methodology errors...') and 'when' with an explicit 'Use when...' clause, matching the highest anchor that requires clear what-and-when with explicit triggers.	3 / 3
Trigger Term Quality	The trigger clause uses natural practitioner phrasing — 'an eval suite already exists and its pass rate is about to gate a release, a model swap, or a prompt change' — terms a user would actually say when reaching for this skill, giving good coverage rather than jargon-only keywords.	3 / 3
Distinctiveness Conflict Risk	The niche is narrow and specific — auditing the methodology of an already-existing eval suite before its pass rate gates a release — with triggers unlikely to fire for unrelated skills, matching the 'clear niche with distinct triggers' anchor.	3 / 3
	Total	12 / 12 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

3 days ago

Table of Contents

Discovery Implementation Validation