CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/vector-search-recall-tests

Vector search benchmarking - recall@k vs latency tradeoffs, ground-truth construction via brute-force, HNSW tuning (M / ef_construct / ef per Qdrant docs), embedding-model-upgrade drift detection. Use ANN-Benchmarks framework for cross-engine comparison; per-engine clients (Qdrant, Weaviate, pgvector, Pinecone, Elasticsearch k-NN, Milvus) for in-product tests. Use when HNSW / IVF parameters are being tuned or an embedding model is swapped, and recall@k on the existing corpus has never been measured against brute-force ground truth.

70

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, highly actionable skill whose core workflow (ground truth, recall, ef sweep, latency budget) is fully executable and correctly delegates engine-specific and deprecated-API detail to real reference files. The main weaknesses are placeholder helpers in the later test steps, absent failure-recovery guidance, and modest redundancy in the HNSW parameter exposition.

Suggestions

Replace or define the placeholder helpers in Steps 5, 7, and 8 (embed_with, rebuild_index, retrieve_topk, engine.insert/engine.search) so those tests are runnable like the Qdrant sweep, or explicitly flag them as engine-agnostic skeletons to be adapted via references/per-engine-clients.md.

Add brief error-recovery guidance after each assert (e.g., if recall < 0.95 at max ef, check normalization/metric mismatch or index corruption before tuning further) to close the feedback-loop gap.

Trim the duplicated Qdrant doc quote and the HNSW parameter-effect table to just the sweep-relevant facts, since Claude already knows HNSW's M/ef_construct/ef tradeoffs.

DimensionReasoningScore

Conciseness

The body is code-forward with no concept tutorials and terse connective prose ("Recall-only is a trap"), but the HNSW parameter-effect table and the repeated Qdrant doc quote ("Increasing ef and ef_construct improves recall but increases computational latency") restate knowledge Claude already has, matching 'efficient; minor instances of over-explanation that could be trimmed'.

4 / 5

Actionability

Steps 1-4 are copy-paste executable with the real Qdrant API (query_points + SearchParams(hnsw_ef=ef)), but Steps 5, 7, and 8 depend on undefined placeholders (embed_with, rebuild_index, retrieve_topk, engine.insert, engine.search) without setup guidance — 'mostly executable guidance; concrete code or commands with minor gaps'.

4 / 5

Workflow Clarity

Steps 1-8 are clearly sequenced (ground truth → recall → sweep → latency → drift → baseline → filters → rebuild), with assert-based checkpoints and decision logic ('find smallest ef that hits the recall target'), but there are no error-recovery feedback loops telling Claude what to do when a test fails — fitting 'clear sequence with most checkpoints present; minor validation gaps' rather than the full feedback-loop anchor.

4 / 5

Progressive Disclosure

The overview keeps the core runnable Qdrant sweep inline and splits per-engine client code and version-sensitive/deprecated API notes into two real, one-level-deep reference files (references/per-engine-clients.md, references/ann-benchmarks.md), each clearly signaled with a markdown link at the exact point of need — matching 'clear overview with well-signaled one-level-deep references; easy navigation'.

5 / 5

Total

17

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that pairs a comprehensive, parameter-level account of capabilities with an explicit and genuinely discriminating 'Use when' trigger clause. The only gap is a handful of natural synonyms (ANN, similarity/semantic search) that users might say instead of 'vector search'.

DimensionReasoningScore

Specificity

The description enumerates multiple concrete actions — "recall@k vs latency tradeoffs, ground-truth construction via brute-force, HNSW tuning (M / ef_construct / ef per Qdrant docs), embedding-model-upgrade drift detection" — down to named HNSW parameters and six specific engines (Qdrant, Weaviate, pgvector, Pinecone, Elasticsearch k-NN, Milvus), which exceeds the 'several specific actions; minor gaps' anchor.

5 / 5

Completeness

It explicitly answers both questions: what ("Vector search benchmarking - recall@k vs latency tradeoffs, ground-truth construction via brute-force, HNSW tuning...") and when ("Use when HNSW / IVF parameters are being tuned or an embedding model is swapped, and recall@k on the existing corpus has never been measured against brute-force ground truth") with concrete trigger conditions.

5 / 5

Trigger Term Quality

Good natural keyword coverage ("vector search", "recall@k", "HNSW tuning", "embedding model", plus all major engine names), but common user phrasings like "ANN", "similarity search", or "semantic search" are absent, matching the 'good keyword coverage; a few natural terms missing' anchor rather than comprehensive synonym coverage.

4 / 5

Distinctiveness Conflict Risk

The niche is sharply defined — brute-force-ground-truth recall benchmarking for vector search engines with named parameters and engines — so it is clearly distinguishable from adjacent skills (e.g., term-based IR relevance testing, general benchmarking) with minimal wrong-trigger risk.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Reviewed

Table of Contents