CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

testland/hybrid-search-eval-author

Use when a search system combines keyword (BM25) and vector retrieval, with or without a reranker, and needs numbers: judgment sets, nDCG@k and MRR on fused results, Reciprocal Rank Fusion vs weighted fusion vs single-stage, and whether the reranker earns its latency.

Quality

91%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

Quality

Content

85%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, highly actionable evaluation workflow: a clear seven-step sequence with built-in validation (CI regression gate), a decision rule for reranker ROI, an anti-patterns checklist, and genuine one-level-deep reference files holding the labeling and engine-specific detail. The main shortcomings are small wiring gaps that stop the code from being fully copy-paste runnable end-to-end, and a few duplicated statements and a redundant references block that cost token efficiency.

Suggestions

Reconcile the retriever naming and signatures: the usage lines call bm25_retrieve_fn/knn_retrieve_fn/rrf_retrieve_fn but the defined functions take (query_text, es_client, index, ...) — show the wrapper or functools.partial wiring (and how query_vector is obtained via the embedding call) so the evaluate() harness runs end-to-end, and define or reference measure_latency_p95 used in the Step 7 test.

Delete the redundant '## References' bullet list — it duplicates the bare link definitions directly beneath it, which already serve the inline citations.

Tighten Step 4 by stating the no-normalization property once ('The formula requires no score normalization because it operates on rank positions') and dropping the closing paragraph that repeats it, and mention the BEIR +12.08% figure in one place only.

DimensionReasoningScore

Conciseness

The body is dense and purposeful — no concept-teaching prose and every section maps to a workflow step — but there are trims to make: the '## References' bullet list duplicates the link-definition block immediately below it, 'requires no normalization' is stated twice within Step 4, and the BEIR +12.08% statistic is repeated in the intro and Limitations. This fits anchor 4 (efficient with minor instances that could be trimmed) rather than anchor 3, since the inline nDCG/MRR code is a canonical harness fixing evaluation conventions, not an explanation of concepts Claude already knows.

4 / 5

Actionability

Provides a copy-paste-ready evaluate() harness, three concrete Elasticsearch retriever bodies, and a runnable pytest regression gate. However, the usage lines invoke bm25_retrieve_fn/knn_retrieve_fn/rrf_retrieve_fn which are never defined (and the defined retrievers have signatures incompatible with retrieve_fn(query_text)), and query_vector and measure_latency_p95 are never shown — so the code does not run end-to-end as written. This places it at anchor 4 (mostly executable with minor gaps) rather than anchor 5 (fully executable).

4 / 5

Workflow Clarity

Seven steps are clearly sequenced (judgment set → metrics → baselines → fusion variants → reranker → regression gate), with explicit validation (the Step 7 CI threshold test), feedback loops (the weight sweep and the 'lift < 0.01 nDCG → reranker is not earning its cost' decision rule), and a checklist (the anti-patterns table). This matches anchor 5; the destructive/batch validation cap does not apply to a measurement skill.

5 / 5

Progressive Disclosure

Both referenced files (references/judgment-sets.md, references/weighted-fusion-and-reranking.md) exist, are one level deep, and are clearly signaled with descriptive lead-ins naming exactly what each holds; labeling recipes and engine-specific fusion/reranker code are appropriately moved out while the core workflow and metric harness stay inline. Matches anchor 5 (clear overview with well-signaled one-level-deep references).

5 / 5

Total

18

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it states an explicit, concrete 'Use when' trigger condition and enumerates four specific, comprehensive capabilities that map directly onto the skill's workflow. Its only weakness is that the canonical natural-language trigger phrase 'hybrid search' (along with 'semantic search'/'RAG') is absent from the description text itself, costing it full marks on trigger-term coverage.

DimensionReasoningScore

Specificity

Lists four concrete capabilities — "judgment sets, nDCG@k and MRR on fused results, Reciprocal Rank Fusion vs weighted fusion vs single-stage, and whether the reranker earns its latency" — comprehensively covering the skill's scope with no evident gaps. Anchor 4 would require minor coverage gaps, which are not present.

5 / 5

Completeness

Explicitly answers both parts: the "Use when a search system combines keyword (BM25) and vector retrieval, with or without a reranker, and needs numbers" clause gives a concrete when, and the colon-delimited list gives a clear what. Anchor 5 matches; anchor 4 would require the 'when' to be less explicit or specific.

5 / 5

Trigger Term Quality

Strong technical keyword coverage ("BM25", "vector retrieval", "reranker", "nDCG@k", "MRR", "Reciprocal Rank Fusion"), but the most natural user phrases — "hybrid search", "semantic search", "RAG" — never appear in the description itself. This fits anchor 4 (good coverage, a few natural terms missing) rather than 5, and is well above anchor 3's 'some relevant keywords'.

4 / 5

Distinctiveness Conflict Risk

The trigger is scoped to fused BM25+vector systems needing numbers, which cleanly excludes single-stage vector-recall and BM25-only relevance skills. Clear niche with distinct triggers and minimal conflict risk, matching anchor 5.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

On this page