CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

promptfoo-evals

Design and run promptfoo evaluations for LLM behavior regressions or comparisons across prompts, agent rules, and models. Use for repeatable behavioral cases with explicit assertions, not ordinary unit tests or static configuration checks.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is an efficient, well-organized set of evaluation discipline guidance with genuine non-obvious insights and a clear phase structure. Its main gap is actionability: it tells Claude how to think about cases and graders but never shows a concrete promptfoo case or assertion example, leaving the authoring step to be improvised.

Suggestions

Add a minimal worked example in 'Build a useful case and grader' — a short promptfooconfig.yaml snippet with one test case and a deterministic assertion (e.g., an assert-type with a threshold) — so the core authoring task is concrete rather than principle-only.

Turn 'Define what is being evaluated' through 'Run and interpret' into an explicitly numbered sequence, and add a short error-recovery loop (what to do on provider failure, flaky cases, or failing grader checks) to strengthen the workflow's feedback checkpoints.

Name the common promptfoo assertion types or link where to find them (e.g., 'assertions: contains, javascript, llm-rubric') so grader guidance maps directly to real configuration syntax.

DimensionReasoningScore

Conciseness

The body is lean with no padding and no explanation of concepts Claude already knows — every line is non-obvious judgment guidance ('A client passing a case does not prove the harness caused the behavior', 'Unavailable providers are not passes'). It assumes Claude's competence and each directive earns its tokens, matching the top anchor rather than the 4 anchor's 'minor instances of over-explanation'.

5 / 5

Actionability

There is some concrete guidance — 'inspect evals/promptfoo/promptfooconfig.yaml', 'make install-promptfoo', 'make eval-promptfoo', 'pin the evaluator and commit the dependency lockfile' — but it is incomplete for the core task: no example promptfoo case, assertion syntax, or config snippet showing how to actually author an evaluation. It is not a 4 because the central 'build a case and grader' section stays at the level of principles ('Prefer deterministic assertions for structure, exact facts, and prohibited actions') without a concrete worked example, and not a 2 because commands and file paths do give real, executable hooks.

3 / 5

Workflow Clarity

A clear define → build → run-and-interpret sequence with most checkpoints present: 'Exercise the grader with a known correct response and plausible incorrect responses before trusting it', 'check its judgments against examples reviewed by a person', and 'Report behavioral failures separately from provider errors, timeouts, missing credentials'. It falls short of the 5 anchor because error-recovery feedback loops (what to do when a run fails or flakes) are implied rather than explicit, and the steps are prose sections rather than an explicit ordered sequence.

4 / 5

Progressive Disclosure

No bundle files exist (references/, scripts/, assets/ are absent), and the skill needs none: the ~68-line body is self-contained, cleanly sectioned, and everything it contains is core guidance that belongs in SKILL.md. The one file pointer it gives (the Arnesto config path and make targets) is clearly signaled and one level deep, matching the well-organized-sections standard for simple self-contained skills.

5 / 5

Total

17

/

20

Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it states concrete capabilities, includes an explicit 'Use for' trigger clause with both positive and negative boundaries, and occupies a distinct niche. The only minor weaknesses are slightly limited action coverage and a few missing natural synonyms such as 'evals' or 'A/B testing prompts'.

DimensionReasoningScore

Specificity

Names concrete actions — 'Design and run promptfoo evaluations', 'comparisons across prompts, agent rules, and models', 'explicit assertions' — with only minor gaps in coverage (e.g., no mention of report generation or CI integration). It is not a 5 because it stops short of the comprehensive multi-action coverage of the top anchor, and not a 3 because it lists several specific actions rather than just one or two.

4 / 5

Completeness

Explicitly answers both questions: what ('Design and run promptfoo evaluations for LLM behavior regressions or comparisons across prompts, agent rules, and models') and when ('Use for repeatable behavioral cases with explicit assertions, not ordinary unit tests or static configuration checks'). The 'Use for' clause includes concrete trigger phrasing plus negative boundaries, matching the top anchor; a 4 would require the 'when' to be less explicit, which it is not.

5 / 5

Trigger Term Quality

Good natural keyword coverage: 'promptfoo evaluations', 'LLM behavior regressions', 'prompts', 'agent rules', 'models', 'assertions', 'unit tests'. A few natural phrasings users would say are missing, such as 'evals', 'evaluate prompts', or 'A/B test prompts', keeping it below the 5 anchor's synonym-and-variation coverage.

4 / 5

Distinctiveness Conflict Risk

'promptfoo evaluations' carves out a clear niche with distinct triggers, and the explicit exclusion ('not ordinary unit tests or static configuration checks') prevents conflicts with general testing and linting skills. Minimal overlap risk with closely related skills like generic test-writing.

5 / 5

Total

18

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

saski/arnesto

Path

.agents/skills/promptfoo-evals/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.