promptfoo-evaluation

Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like "promptfoo", "eval", "LLM evaluation", "prompt testing", or "model comparison".

1.59x

Quality

87%

Does it follow best practices?

Impact

97%

1.59x

Average score across 3 eval scenarios

Securityby

Low

Low-risk findings worth noting

Quality

Content

82%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable with executable examples and a clear sequence, and it correctly offloads detail to a real reference file. It loses points for repeated relay/maxConcurrency guidance and for inlining API-reference detail that duplicates the reference file.

Suggestions

De-duplicate the relay/maxConcurrency and llm-rubric-provider rules: state them once in the Relay section and Troubleshooting, and remove the repeated restatements in Configuration Structure and Best practices.

Move the Common Assertion Types table and provider/CLI reference detail into references/promptfoo_api.md, keeping only a short pointer and the most-used types inline in SKILL.md.

Add an explicit validation checkpoint before the first eval run, e.g. preview with the echo provider or run `npx promptfoo@latest eval -j 1` on one test case to confirm the config loads.

Dimension	Reasoning	Score
Conciseness	Mostly efficient with tight code blocks and high-value gotchas, but the relay/maxConcurrency and llm-rubric-provider rules are repeated across Configuration Structure, Relay section, Troubleshooting, and Best practices, so it sits below the lean level-5 anchor yet above the padded level-3 anchor.	4 / 5
Actionability	Provides copy-paste-ready YAML and Python plus specific commands ('npx promptfoo@latest eval --filter-metadata category=math') covering common cases, matching the fully-executable anchor; not below because examples are concrete and complete rather than pseudocode.	5 / 5
Workflow Clarity	The init → configure → run → view sequence is clear with a Troubleshooting error→fix section and an echo-provider preview loop, but the main eval flow lacks an explicit 'validate config before first run' checkpoint, placing it below the level-5 anchor with explicit validation.	4 / 5
Progressive Disclosure	Clear section structure with one-level-deep references to the real references/promptfoo_api.md and scripts/metrics.py, but a substantial amount of API/assertion-table detail is inlined in SKILL.md and duplicated in the reference file, so it is above the level-3 'should be separate is inline' anchor but below the cleanly-split level-5 anchor.	4 / 5
	Total	17 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is strong: it states five concrete capabilities, gives explicit 'Use when' trigger guidance with natural keywords, and occupies a clear niche. The only gap is trigger-term coverage could include file extensions and a few more synonyms.

Dimension	Reasoning	Score
Specificity	Lists five concrete actions—'Configures and runs LLM evaluation', 'creating evaluation configs (promptfooconfig.yaml)', 'writing Python custom assertions', 'implementing llm-rubric', 'managing few-shot examples'—matching the comprehensive-coverage anchor; not below because it is not limited to 1-2 actions.	5 / 5
Completeness	Explicitly answers 'what' (configure/run evaluations and sub-tasks) and 'when' ('Use when setting up prompt testing, creating evaluation configs...') with concrete trigger phrases, matching the top anchor; not below because both halves are explicit, not weakly implied.	5 / 5
Trigger Term Quality	Quotes natural trigger phrases ('promptfoo', 'eval', 'LLM evaluation', 'prompt testing', 'model comparison') that users would say, but misses common synonyms/file extensions like '.yaml' or 'model grading', so it sits below the comprehensive level-5 anchor.	4 / 5
Distinctiveness Conflict Risk	Tied to a specific framework (Promptfoo) with distinct triggers and minimal overlap with other skills, matching the clear-niche anchor; not below because the triggers are specific rather than broad.	5 / 5
	Total	19 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: daymade/claude-code-skills
Path: promptfoo-evaluation/SKILL.md
Commit: b8556f2

Reviewed: 3 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.