CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

testland/hybrid-search-eval-author

Evaluates hybrid retrieval pipelines (BM25 + vector + reranker) end-to-end: authors ground-truth judgment sets, computes nDCG@k and MRR over fused results, measures the lift from Reciprocal Rank Fusion vs weighted fusion vs single-stage retrieval, and quantifies reranker (cross-encoder/Cohere/bge) impact. Use when a production system combines lexical and semantic retrieval and you need a numeric relevance baseline, fusion-strategy comparison, or evidence that a reranker is earning its latency cost.

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A strong, professionally structured evaluation skill: clearly sequenced steps, executable core code, a CI regression gate, an anti-patterns checklist, and exemplary progressive disclosure with two real one-level-deep reference files. The weaknesses are minor: duplicated facts and link definitions, a retrieve_fn signature mismatch in Step 3, and a couple of called-but-undefined helper functions.

Suggestions

Remove the duplicated content: the four reference link definitions repeated verbatim at the end of the file, the +12.08% BEIR stat stated in both the intro and Limitations, and the Elasticsearch/Cohere reranker facts repeated between Step 6 and references/weighted-fusion-and-reranking.md.

Fix the Step 3 contract mismatch: either show the functools.partial/es_client-binding wrapper that turns bm25_retrieve and knn_retrieve into retrieve_fn(query_text) callables, or redefine them to match the evaluate() signature; also provide the measure_latency_p95 implementation used by the Step 7 gate.

Add a brief error-recovery loop to Step 7 for when the regression gate fails (e.g., re-run per-stage baselines to isolate whether the regression came from the embedding swap, fusion weights, or reranker), and make the validation/test split an explicit sub-step of Step 1 or Step 5 rather than only an anti-pattern warning.

DimensionReasoningScore

Conciseness

The body is dense and mostly earns its tokens (runnable metric code, ES queries, anti-patterns table), but there is avoidable duplication: the "+12.08%" OpenSearch stat appears in both the intro and Limitations, the Elasticsearch/Cohere reranker facts in Step 6 are repeated nearly verbatim in references/weighted-fusion-and-reranking.md, and all four reference link definitions are stated twice (References section lines and again verbatim below it). This fits 'efficient; minor instances that could be trimmed' rather than the every-token-earns-its-place level 5, and is well above the noticeably-verbose level 3.

4 / 5

Actionability

Mostly executable guidance: the nDCG/MRR implementations, RRF retriever query, and pytest regression gate are copy-paste ready, but there are minor gaps — Step 3 calls evaluate(queries, bm25_retrieve_fn, ...) while the defined bm25_retrieve takes (query_text, es_client, index, k), so an adapter is implied but never shown; measure_latency_p95 and update_pipeline_weights (in the reference) are called but undefined. Matches 'concrete code or commands with minor gaps'; not score 5 because the code is not fully copy-paste runnable without user-supplied glue.

4 / 5

Workflow Clarity

A clear 7-step sequence (qrels → metrics → baselines → RRF → weighted fusion → reranker → CI gate) with explicit checkpoints: the regression gate in Step 7, the reranker ROI decision rule ("If the lift is < 0.01 nDCG, the reranker is not earning its cost"), and an anti-patterns table covering overfitting and pool-size mistakes. It falls short of score 5 because there is no error-recovery/feedback loop (what to do when the gate fails) and the validation/test split is only mentioned as an anti-pattern rather than as a step.

4 / 5

Progressive Disclosure

Excellent structure: the runnable spine (metrics, baselines, RRF, regression gate) stays inline, while engine-specific detail is split into two real, clearly signaled, one-level-deep references — [references/judgment-sets.md](references/judgment-sets.md) linked exactly at Step 1 and [references/weighted-fusion-and-reranking.md](references/weighted-fusion-and-reranking.md) linked at Steps 5-6. Both files exist, contain the promised content, and reference nothing further; navigation is trivial. This matches the score-5 anchor (clear overview, well-signaled one-level-deep references, appropriate split).

5 / 5

Total

17

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: third-person voice, dense with concrete actions, explicit 'Use when' triggers, and a well-scoped niche with named metrics, fusion strategies, and reranker vendors. The only gap is a handful of natural trigger phrases ('hybrid search', 'RAG', engine names) that live in the body and metadata but not the description itself.

DimensionReasoningScore

Specificity

The description lists multiple concrete, comprehensive actions: "authors ground-truth judgment sets, computes nDCG@k and MRR over fused results, measures the lift from Reciprocal Rank Fusion vs weighted fusion vs single-stage retrieval, and quantifies reranker (cross-encoder/Cohere/bge) impact" — this matches the anchor for multiple specific concrete actions with comprehensive coverage, and exceeds the score-4 anchor only by closing the remaining coverage gaps.

5 / 5

Completeness

Both halves are explicit: the 'what' is the enumerated action list, and the 'when' is a concrete trigger clause — "Use when a production system combines lexical and semantic retrieval and you need a numeric relevance baseline, fusion-strategy comparison, or evidence that a reranker is earning its latency cost." This clearly matches the score-5 anchor with concrete trigger phrases; the score-4 anchor's caveat ('when' could be more specific) does not apply.

5 / 5

Trigger Term Quality

Keyword coverage is strong and includes synonyms ("BM25", "vector", "reranker", "cross-encoder/Cohere/bge", "nDCG@k", "MRR", "lexical and semantic retrieval"), but a few natural phrases users would say are absent — notably "hybrid search" (it says "hybrid retrieval pipelines"), "RAG", and the engine names Elasticsearch/OpenSearch. This fits the anchor 'Good keyword coverage; a few natural terms missing' rather than the comprehensive synonym coverage of score 5.

4 / 5

Distinctiveness Conflict Risk

The description carves out a clear niche — end-to-end evaluation of fused hybrid retrieval with named metrics (nDCG, MRR), fusion strategies (RRF, weighted), and reranker ROI — so it is unlikely to trigger for the sibling vector-only or BM25-only relevance skills. Minimal conflict risk matches the score-5 anchor; score 4 ('minor overlap risk') would understate how tightly the triggers are scoped to hybrid/fused systems.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

On this page