CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/elasticsearch-relevance-tests

Author Elasticsearch relevance regression tests using the Ranking Evaluation API (`POST {index}/_rank_eval`) - judgment lists (query + expected docs at ranks), per-query metrics (Precision@K, Recall@K, MRR, DCG, ERR), reproducible test corpora; pair with Quepid + Splainer for interactive judgment authoring. Use before changing analyzers, synonyms, boosts, or query templates on an Elasticsearch index that serves user-facing search, so the NDCG / MRR baseline is captured first.

79

Quality

99%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

Quality

Content

100%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A concise, actionable SKILL.md body: a sequenced 5-step workflow with executable code and explicit assertion-based validation, supported by one real one-level-deep reference that keeps the core file lean. The only minor improvement would be spelling out an explicit error-recovery feedback loop after a failed assertion.

Suggestions

Add an explicit feedback loop in Step 4/5: when an assert fails, diagnose with Splainer, adjust the query template/analyzer, and re-run the rank_eval test until the baseline holds.

Vary the repeated "Per the [Elasticsearch Rank Eval API]" lead-ins across Steps 2–3 to reduce redundancy and tighten token use.

DimensionReasoningScore

Conciseness

Lean body that assumes Claude's competence: tight tables for judgment sources and metric selection, runnable code blocks, and no padding explaining what Elasticsearch or NDCG is. The only mild inefficiency is the repeated "Per the [Elasticsearch Rank Eval API]" lead-in before several sections, but every section earns its place.

3 / 3

Actionability

Provides fully executable guidance throughout — a complete `_rank_eval` POST body, a runnable Python test (`load_judgments` + `test_search_relevance_baseline`), a per-query regression test, and a concrete CSV judgment format — matching the copy-paste-ready score-3 anchor rather than pseudocode.

3 / 3

Workflow Clarity

A clearly sequenced 5-step workflow (build judgments → define metrics → submit rank_eval → wrap as test → per-query regression) with explicit validation checkpoints in the `assert result["metric_score"] >= 0.80` and `assert delta >= -0.10` gates. The one gap is an explicit "fail → diagnose → fix → re-run" feedback loop, which is only implicit, but the explicit assertions keep it closest to the score-3 anchor.

3 / 3

Progressive Disclosure

Well-organized overview sections with a single, clearly signaled one-level-deep reference to references/rank-eval-guide.md (verified to exist), which appropriately offloads binary thresholds, snapshot corpora, and Quepid/Splainer tooling — matching the clear-overview-with-well-signaled-references anchor.

3 / 3

Total

12

/

12

Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A high-quality description: third-person voice, multiple concrete capabilities, and an explicit "Use before..." trigger clause tying the skill to a recognizable deploy-gate scenario. It clearly answers what the skill does and when to invoke it with low conflict risk.

DimensionReasoningScore

Specificity

Lists multiple concrete actions — "Author Elasticsearch relevance regression tests", "judgment lists (query + expected docs at ranks)", per-query metrics enumerated by name, and "reproducible test corpora" — matching the score-3 anchor for specific concrete actions; far richer than naming only a domain and some actions.

3 / 3

Completeness

Explicitly answers both what (authoring judgment lists, per-query metrics, reproducible corpora) and when ("Use before changing analyzers, synonyms, boosts, or query templates on an Elasticsearch index that serves user-facing search"), satisfying the score-3 anchor for explicit what-and-when triggers.

3 / 3

Trigger Term Quality

Natural terms a search engineer would say are well covered — "relevance regression tests", "analyzers, synonyms, boosts, or query templates", "NDCG / MRR baseline", and "Quepid + Splainer"; the explicit "Use before changing..." clause is exactly how the need arises in practice.

3 / 3

Distinctiveness Conflict Risk

Pinned to Elasticsearch's `_rank_eval` endpoint with niche-specific triggers (judgment lists, NDCG/MRR, Quepid, Splainer), giving a clear niche unlikely to trigger for unrelated skills.

3 / 3

Total

12

/

12

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

metadata_version

'metadata.version' is missing

Warning

Total

15

/

16

Passed

Reviewed

Table of Contents