evaluate-rag

Guides evaluation of RAG pipeline retrieval and generation quality. Use when evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs for retrieval testing, or optimizing chunking strategies.

Quality

94%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

88%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A tightly written, highly actionable skill body with concrete templates, tables, and a clear sequenced workflow plus diagnostic feedback loops. The main slack is mild redundancy in restating standard metric formulas inline.

Suggestions

Trim or collapse the Recall@k/Precision@k/MRR/NDCG formula blocks to one-line definitions, since Claude already knows these metrics; keep only the RAG-specific guidance (e.g., the NDCG weak-relevance caveat).

Consider moving the chunking grid-search table and full metric reference into a references/ file, keeping SKILL.md as an overview with a one-level-deep pointer.

Dimension	Reasoning	Score
Conciseness	Mostly lean and assumes competence; the metric formula blocks (Recall@k, Precision@k, MRR, NDCG/DCG) restate definitions Claude already knows, though they serve as quick reference. Not a 5 because those formula expansions could be trimmed.	4 / 5
Actionability	Fully executable guidance: a copy-paste synthetic QA prompt template, concrete adversarial-generation example with real chunk text, a populated grid-search table, and decision tables for metric selection and failure diagnosis.	5 / 5
Workflow Clarity	Clear 5-step overview sequence with an explicit prerequisite (error analysis first) and a 'Diagnosing Failures by Metric Pattern' table that functions as a feedback loop for error recovery.	5 / 5
Progressive Disclosure	Well-organized single file with clear section headers and no nested references, but it is monolithic (~170 lines); the metric-formula reference or grid-search detail could optionally live in a separate file. Not a 5 because there is no file-splitting to evaluate, and the bulk is inline.	4 / 5
	Total	18 / 20 Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that explicitly states both capability and trigger conditions with concrete, natural-language terms. It is concise yet comprehensive with no padding or over-claims.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions — 'evaluating a retrieval-augmented generation system, measuring retrieval quality, assessing generation faithfulness or relevance, generating synthetic QA pairs... optimizing chunking strategies' — covering the domain comprehensively.	5 / 5
Completeness	Explicitly answers both what ('Guides evaluation of RAG pipeline retrieval and generation quality') and when ('Use when evaluating a retrieval-augmented generation system...') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Natural terms a user would say are well covered: 'RAG pipeline', 'retrieval-augmented generation', 'retrieval quality', 'generation faithfulness', 'chunking strategies', 'synthetic QA pairs'.	5 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (RAG evaluation) with distinct triggers; minimal overlap risk with other skills.	5 / 5
	Total	20 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: hamelsmu/evals-skills
Path: skills/evaluate-rag/SKILL.md
Commit: 22418da

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.