testland/llm-eval-anti-patterns
Audits an existing LLM evaluation suite for eight methodology errors that make its numbers untrustworthy: too few cases per capability, single-provider lock-in, exact-match assertions on open-ended output, no semantic-similarity check on paraphrase-tolerant output, no baseline comparison in CI, no cost or latency ceiling, unpinned model identifiers, and no adversarial coverage. Supplies harness-neutral detection cues, the reason each error invalidates the result, a concrete fix, a Critical/Warning/Info severity scheme, and a findings-table output shape. Covers validating an LLM judge against human labels before its verdicts count as evidence. Use when an eval suite already exists and its pass rate is about to gate a release, a model swap, or a prompt change, and nobody has audited how the suite itself was built.
75
94%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
View guide
Passed
No findings from the security scan
harness-mapping.mdreferences/
Mapping the five slots across harnesses
The eight checks are structural, not syntactic. Every harness has the same five slots - case set, system under test, assertions/scorers, judge configuration, and the CI gate - under different vocabulary.
- Promptfoo:
tests,providers,assert, the grading provider, and the CI job (per promptfoo.dev/docs/configuration/guide and promptfoo.dev/docs/configuration/expected-outputs). - LangSmith: the dataset, the target application, the evaluators (human review, code-based rules, LLM-as-judge, or pairwise comparison), and the experiment (per docs.langchain.com/langsmith/evaluation).
- Braintrust: an eval is data plus a task plus scorers, recorded as an experiment that CI compares over time to catch regressions (per braintrust.dev/docs/guides/evals).
- DeepEval: the parameterized test data, the metric objects, and their thresholds (per deepeval.com/docs/metrics-llm-evals).
- OpenAI Evals: the JSON data file and the YAML eval parameters (per github.com/openai/evals).
Where a check in SKILL.md names an exact key, that key is verified in that one harness's linked documentation. Everywhere else the description is deliberately structural so it transfers to a bespoke in-house harness once its five slots are mapped by hand.