CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/llm-eval-anti-patterns

Audits an existing LLM evaluation suite for eight methodology errors that make its numbers untrustworthy: too few cases per capability, single-provider lock-in, exact-match assertions on open-ended output, no semantic-similarity check on paraphrase-tolerant output, no baseline comparison in CI, no cost or latency ceiling, unpinned model identifiers, and no adversarial coverage. Supplies harness-neutral detection cues, the reason each error invalidates the result, a concrete fix, a Critical/Warning/Info severity scheme, and a findings-table output shape. Covers validating an LLM judge against human labels before its verdicts count as evidence. Use when an eval suite already exists and its pass rate is about to gate a release, a model swap, or a prompt change, and nobody has audited how the suite itself was built.

72

Quality

90%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

82%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A strong, expert-level catalog: every anti-pattern comes with named harness keys, an executable detection procedure, a severity, and a specific fix, and the reporting format is fully specified with a worked example. The two real critiques are that the ~435-line body could offload the worked example and more per-harness detail to the references directory, and that a few rhetorical passages and the scope table could be trimmed for token efficiency.

Suggestions

Move the worked example (promptfooconfig.yaml + eval.yml + findings table) into a references file (e.g., references/worked-example.md) and keep a two-line pointer plus the output-format spec in SKILL.md, cutting roughly 50 lines from the always-loaded body.

Consolidate the repeated per-harness detail in anti-patterns 3, 4, 5, and 8 into the existing references/harness-mapping.md, keeping only the harness-neutral detection cue inline so the checks stay one page each.

Trim the framing prose (the opening three paragraphs and the 'What this owns' table overlap with the frontmatter description) and relocate 'The judge is a model under test' to follow anti-pattern 8 so the numbered check sequence reads uninterrupted.

DimensionReasoningScore

Conciseness

The body assumes Claude's competence throughout — it never explains what an eval, CI, or an assertion is, and it leads with curated specifics (exact assertion families, provider flags, thresholds) rather than tutorial prose. It is not a 5 because rhetorical framing ("can be green and still be worthless", "A threshold copied from a tutorial is a threshold nobody has validated") and the ownership/scope table partially duplicate what the description and the per-check Detect sections already establish, so a modest trim is available. It is well above the 3 anchor, where unnecessary explanation of known concepts would appear.

4 / 5

Actionability

Every check gives an executable detection procedure plus a concrete fix: named Promptfoo assertion types (`equals`, `contains`, `icontains`, `regex`, `is-json` vs `llm-rubric`, `g-eval`, `factuality`, `similar` with a cosine `threshold`), copy-paste YAML for `providers`, `latency`/`cost` thresholds, the three-level grader override chain (`--grader`, `defaultTest.options.provider`, per-assertion `provider` with documented precedence), a threshold-setting procedure ("run the current known-good outputs through it... place the threshold below the worst acceptable output"), and a complete worked example with an eight-row findings table in the specified output format. This matches the fully-executable, common-cases-covered anchor; the 4 anchor's 'minor gaps' are not present.

5 / 5

Workflow Clarity

The sequence is clear and checkpointed: map the five slots first (numbered 1–5), apply the eight Detect/Why/Fix checks, then emit the ordered findings table and verdict with explicit validation rules ("Every finding must point at a line", "A finding with no location is not reviewable", "A suite with an outstanding Critical finding is not passing"). It is not a 5 because there is no re-check or error-recovery loop (e.g., what to do when a slot cannot be located or when findings conflict), and "The judge is a model under test" interrupts the numbered anti-pattern sequence between 4 and 5, slightly blurring the main flow. It clears the 3 anchor easily since checkpoints are explicit rather than implicit.

4 / 5

Progressive Disclosure

Structure is good: citations and the per-harness slot mapping are correctly split out to two real, one-level-deep, clearly-signaled files (references/citations.md, references/harness-mapping.md) linked both inline and in a closing References section, and headers make the 435-line body navigable. It is not a 5 because the body itself is long for a SKILL.md — the worked example and some per-harness detail in anti-patterns 3–8 are candidates for reference files — so the split is good but not maximally considerate of the context window. It is well above the 3 anchor, where bulk reference material would sit inline with weak signaling.

4 / 5

Total

17

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A dense, explicitly bounded description: it names all eight checks, the deliverables, and a precise use-when clause in third person with no padding beyond the enumeration itself. The only soft spot is that a few natural synonym triggers ("benchmark", "test suite") are not covered, and the enumeration makes it long for a description field even though every clause is informative.

DimensionReasoningScore

Specificity

The description enumerates all eight concrete checks ("too few cases per capability, single-provider lock-in, exact-match assertions on open-ended output, no semantic-similarity check... no baseline comparison in CI, no cost or latency ceiling, unpinned model identifiers, and no adversarial coverage") plus the concrete outputs it supplies ("detection cues... a concrete fix, a Critical/Warning/Info severity scheme, and a findings-table output shape"), matching the comprehensive multiple-specific-actions anchor. It is not the level below (4) because coverage is exhaustive rather than having minor gaps, and there is no vagueness that would place it lower.

5 / 5

Completeness

Both halves are explicit: the "what" ("Audits an existing LLM evaluation suite for eight methodology errors that make its numbers untrustworthy... Supplies harness-neutral detection cues, the reason each error invalidates the result, a concrete fix...") and a concrete "Use when" clause ("Use when an eval suite already exists and its pass rate is about to gate a release, a model swap, or a prompt change, and nobody has audited how the suite itself was built"). This matches the anchor for a clear, explicit what-and-when with concrete trigger phrases; the 4 anchor would require the 'when' to be less specific than it is.

5 / 5

Trigger Term Quality

Natural domain terms are present and phrased the way practitioners speak: "LLM evaluation suite", "pass rate", "gate a release", "model swap", "prompt change", "nobody has audited how the suite itself was built". It is not a 5 because common synonyms a user might say — "benchmark", "test suite", "regression", "CI" — are absent, leaving a few natural entry phrases uncovered; it clears the 3 anchor comfortably because the terms present are natural, not jargon-only.

4 / 5

Distinctiveness Conflict Risk

The niche is sharp — auditing an already-existing eval suite's methodology — and the trigger is fenced on three sides ("an eval suite already exists", "pass rate is about to gate a release, a model swap, or a prompt change", "nobody has audited how the suite itself was built"), which rules out the adjacent skills (building a suite, benchmarking, code review) it could otherwise collide with. It is not a 4 because the residual overlap with generic review or prompt-engineering skills is closed by those explicit boundary conditions.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

Table of Contents