CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

eval-plan

Builds an eval plan for an AI feature - error analysis on real traces, failure-mode ranking, code checks and binary LLM judges validated against human labels, CI tests, production monitoring, and a launch bar. Use when you ask how do we eval this, how do I test an AI feature, our LLM outputs are inconsistent, are the evals good enough, LLM-as-judge, hallucinations, or what accuracy is good enough to launch. Draws on 41 Lenny's Podcast guests incl. Hamel Husain, Shreya Shankar and Sander Schulhoff.

75

Quality

94%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

88%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A high-quality instruction-only skill: dense, specific, and sequenced with explicit 'Done when' checkpoints, decision routing, and a paste-ready deliverable template. The two sub-top dimensions are minor: some quote content is duplicated between the body and references/quotes.md, and the body is long enough that a few sections could move to reference files.

DimensionReasoningScore

Conciseness

The body is consistently dense with non-obvious specifics (sample sizes, budgets, named tactics) and assumes Claude's competence without explaining basic concepts. It falls short of the top anchor because the 'Receipts' quotes partially restate points already made in the body and the closing 'Draws on...' provenance adds length; trimming those would reach 'every token earns its place'.

4 / 5

Actionability

For an instruction-only skill the guidance is fully actionable: a copy-paste-ready Deliverable markdown template, concrete numbers ('Sample ~100 real traces', '4-7 judges', '30-40 failing examples', 'about 1,000 a day'), exact judge-writing rules, and specific vendor questions to ask. This matches the 'specific examples cover the common cases' top anchor; per the scoring notes, absence of code is not penalized when guidance is this concrete.

5 / 5

Workflow Clarity

The process is clearly sequenced (Diagnose with at most 4 questions, a routing table to plays, then Plays A-E) with explicit completion checkpoints ('Done when: a ranked failure-mode table exists...') and feedback loops ('iterate until both approach zero', 'review before locking rubrics', 're-calibrate after model swaps'). This matches the top anchor's explicit validation steps and error-recovery loops; no destructive/batch cap applies.

5 / 5

Progressive Disclosure

References are real, one level deep (no nested .md links inside them), and clearly signaled in 'Go deeper' with what each contains; the operational playbook stays in SKILL.md while bulk material (all frameworks, 22 quotes) lives in references/. It is not the top anchor because the body's 'Receipts' section duplicates a subset of references/quotes.md and the 173-line body could shed some content to the reference files, fitting 'minor organization gaps'.

4 / 5

Total

18

/

20

Passed

Description

96%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: concrete action list, explicit 'Use when' clause with natural trigger phrases, and a clearly defined niche. The only weakness is a couple of broad trigger terms that could fire for adjacent skills, and the provenance sentence about podcast guests adds tokens without aiding triggering.

DimensionReasoningScore

Specificity

Lists six concrete actions spanning the full workflow: 'error analysis on real traces, failure-mode ranking, code checks and binary LLM judges validated against human labels, CI tests, production monitoring, and a launch bar'. This matches the comprehensive-coverage anchor; score 4 would require minor gaps, but the action list covers diagnosis through monitoring and launch criteria.

5 / 5

Completeness

Explicitly answers what ('Builds an eval plan for an AI feature' plus its components) and when ('Use when you ask how do we eval this...') with concrete trigger phrases. Both halves are present and explicit, matching the top anchor; a score of 4 would require the 'when' to be less specific, which it is not.

5 / 5

Trigger Term Quality

Includes many natural user phrasings with variations: 'how do we eval this', 'how do I test an AI feature', 'our LLM outputs are inconsistent', 'are the evals good enough', 'LLM-as-judge', 'hallucinations', 'what accuracy is good enough to launch'. Coverage of how a user would actually phrase the need is comprehensive, matching the top anchor rather than the 'a few natural terms missing' level below.

5 / 5

Distinctiveness Conflict Risk

The eval-planning niche is clear and most triggers ('LLM-as-judge', 'are the evals good enough') are distinct, but broad single-word triggers like 'hallucinations' and mentions of 'production monitoring' and 'CI tests' create minor overlap risk with adjacent testing, metrics, or monitoring skills. This fits 'mostly distinct; minor overlap risk' better than the minimal-conflict anchor above.

4 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

menkesu/awesome-pm-skills

Path

skills/eval-plan/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.