judgment-list-author • testland • Registry • Tessl

testland/judgment-list-author

Bootstraps human-relevance judgment lists (query sets, grading scales, rater guidelines, inter-rater agreement, Quepid tooling, TREC-style pooling, and refresh cadence) that serve as ground truth for search-relevance test suites. Use when a team needs to create or refresh the judgment corpus before running NDCG / MRR / Recall@k evaluations.

Quality

87%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

81%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body delivers a clear, validated workflow with highly actionable formats and code, well-sectioned for navigation. Its main weaknesses are re-explaining standard statistics (Cohen's kappa formula and thresholds) Claude already knows, duplicated link citations, and a monolithic length that would benefit from splitting detailed material into reference files.

Suggestions

Trim Step 5's kappa formula, Landis-Koch table, and sklearn snippet to just the decision thresholds and the action rule (kappa < 0.60 -> revise and re-judge), keeping only a pointer to the source.

Cite each external source once and drop the redundant per-claim inline link repetitions; a single References section suffices.

Move the rater-guidelines template details and Quepid setup/export specifics into reference files (e.g., references/rater-guidelines.md, references/quepid-setup.md) so SKILL.md remains a lean overview.

Dimension	Reasoning	Score
Conciseness	Mostly efficient domain-specific content (tier sampling, Quepid workflow, pooling depth, refresh triggers), but Step 5 restates the kappa formula, Landis-Koch thresholds, and a trivial sklearn snippet Claude already knows, and the same external links are cited inline multiple times and again in the References section. Not 4: the known-knowledge repetition and link duplication are more than minor trimmable instances.	3 / 5
Actionability	Fully executable throughout: exact 4-column qrels format, copy-paste output JSON schema, runnable pool_results() and cohen_kappa_score code, concrete Quepid setup steps, spreadsheet column layout, and a refresh-trigger table. Specific examples cover the common cases.	5 / 5
Workflow Clarity	Seven clearly sequenced steps with explicit validation: the kappa >= 0.60 gate with a revise-guidelines/calibrate/re-judge feedback loop, guideline-change invalidation of old rounds, and a monthly staleness check with a 20% threshold. Checkpoints and error recovery are explicit, matching the top anchor.	5 / 5
Progressive Disclosure	Well-organized single-file skill with clear section headers and one-level-deep external references only. Not 5: at roughly 300 lines, content such as the Quepid setup flow, rater-guideline templates, and grading-scale tables could be split into bundle reference files so SKILL.md stays an overview.	4 / 5
	Total	17 / 20 Passed

Description

88%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific, complete, and in proper third-person voice with an explicit 'Use when' clause covering concrete triggers. Trigger-term coverage is strong though it lacks a few common synonyms (qrels, relevance judgments), and the evaluation-oriented trigger creates minor overlap risk with sibling relevance-testing skills.

Suggestions

Add common synonyms such as 'qrels' or 'relevance judgments' to the description so users who use those terms find the skill.

Sharpen the 'when' clause to emphasize authoring/creating ground truth (e.g., 'Use when a team needs to author or refresh qrels/judgment lists') to reduce overlap with the evaluation-runner skills.

Dimension	Reasoning	Score
Specificity	"Bootstraps human-relevance judgment lists (query sets, grading scales, rater guidelines, inter-rater agreement, Quepid tooling, TREC-style pooling, and refresh cadence)" lists seven concrete, distinct capabilities with comprehensive coverage of the judgment-authoring task. Not 4: there are no meaningful gaps in the capability list.	5 / 5
Completeness	Explicitly answers what ("Bootstraps human-relevance judgment lists ... that serve as ground truth for search-relevance test suites") and when ("Use when a team needs to create or refresh the judgment corpus before running NDCG / MRR / Recall@k evaluations") with concrete trigger phrases, in third-person voice. Both anchors are fully satisfied.	5 / 5
Trigger Term Quality	Strong natural terms users would say: "judgment list", "ground truth", "NDCG / MRR / Recall@k", "search-relevance", "grading", "raters". Not 5: misses common synonyms like "qrels", "relevance judgments", or "relevance ratings" that practitioners use interchangeably with judgment lists.	4 / 5
Distinctiveness Conflict Risk	Clear niche (authoring the judgment corpus rather than consuming it) with specific tooling triggers like Quepid and TREC pooling. Not 5: the "before running NDCG / MRR / Recall@k evaluations" trigger overlaps with closely related consumer skills (elasticsearch-relevance-tests, vector-search-recall-tests) a user might invoke for evaluation help.	4 / 5
	Total	18 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

3 days ago

Table of Contents

Discovery Implementation Validation